A supervised fine-tuning method for Text2SQL tasks in low-resource scenarios

Abstract

The present invention discloses a supervised fine-tuning method for a Text2SQL task in a low-resource scenario. The method is decomposed into two parts: a pattern connection task and an SQL generation task. The pattern connection task part is taken as the focus to accurately adjust the overall process. The core of the method is to group all tables in a telecommunications database into several slices according to specific relationships, so that the model can flexibly cover different numbers of tables according to GPU memory limitations. At the same time, the Chain of Thoughts (CoT) method is combined with the reasoning process of the pattern connection task, so that the model can gradually guide itself to generate the final result. Subsequently, a small number of problem-related tables predicted by the pattern connection model are input into the SQL generation model to perform the statement generation task. Through the present invention, the GPU memory consumption in the fine-tuning process can be effectively reduced, and the model performance comparable to that of the existing method can be maintained.

Description

Technical Field

[0001] The present invention relates to two technical fields: telecommunication field and natural language processing, and in particular to a supervised fine-tuning method for a Text2SQL task in a low-resource scenario in the telecommunication field. Background Art

[0002] With the rapid growth of data in the telecommunications field, telecommunications databases are being used more and more widely, and the number and complexity of tables in the database are also increasing. These table structures and massive records make the amount of text information (tokens) input into the model very large when fine-tuning the model for the Text2SQL (natural language query to SQL statement) task, resulting in a continuous increase in GPU memory requirements during supervised fine-tuning.

[0003] Although existing supervised fine-tuning methods have been successfully used for pattern connection and SQL generation tasks within the scope of small databases, they can skillfully perform instruction supervised fine-tuning on independent GPUs. However, telecom databases often contain telecom data in hundreds of tables, and the table structure is complex. Existing fine-tuning methods generally adopt parameter efficient fine-tuning (PEFT) technology. This technology introduces partially trainable modules into each layer of the Transformer structure of the large language model, and only trains these parameters during fine-tuning. Although the memory footprint of Text2SQL supervised fine-tuning is reduced at the parameter level, the context length of the input model is also one of the important factors affecting memory usage during fine-tuning. When facing medium to large databases containing a large amount of telecom data, the context content contained in each query instance during training requires considerable GPU memory resources to be allocated. In the supervised fine-tuning stage, if the relationship between query requirements, query predictions and overall database information is directly established, a large number of tokens are required to encode each instance. Therefore, the problem of limited GPU storage resources has become the main difficulty for large language models to provide professional services for data applications in the telecommunications field.

[0004] The current supervised fine-tuning method based on the improvement of PEFT aims to further reduce memory usage during fine-tuning, but it cannot be adapted and applied to large language models of all architectures. For example, QLoRA can reduce the memory requirements for fine-tuning by reducing the precision of model parameters. In addition, LongLoRA can both expand the context window of the LLAMA2 model and further reduce the memory footprint of supervised fine-tuning at the token level. However, both methods require the design of specific implementation codes for large language models with different architectures, and require an in-depth understanding of the underlying mechanisms of the model. Therefore, in practical applications, these techniques cannot be quickly extended to large models with different architectures.

[0005] In summary, as the amount of data contained in the telecommunications field continues to increase, how to efficiently and accurately query the required information from a large amount of data has become an important issue. In the complex scenario of natural language processing and database interaction, Text2SQL technology can convert users' natural language queries into precise SQL statements, thereby realizing accurate queries on data. However, the number of tables and columns in the telecommunications database is relatively large, and the table structure is complex. If the context length of the input model is too long, a large number of tokens are required to encode each feature. Therefore, in practical applications, the scarcity of GPU resources has become an important factor restricting the performance improvement of the Text2SQL model. Current supervised fine-tuning methods, such as PEFT, have effectively alleviated the memory usage of Text2SQL supervised fine-tuning by reducing the number of parameters involved in training, but it is difficult to solve the GPU resource pressure caused by the large number of tokens, and it cannot be implemented on large models of different architectures. Summary of the invention

[0006] The present invention aims to solve one of the technical problems existing in the related art at least to a certain extent.

[0007] An object of the present invention is to provide a supervised fine-tuning method for a Text2SQL task in a low-resource scenario, to perform supervised fine-tuning of the Text2SQL task under limited GPU resource conditions in the telecommunications field, and to solve the problem of large GPU memory consumption when fine-tuning a large model due to the large number of tables and columns in the telecommunications database, the complex table structure, and the excessive length of the input model context.

[0008] In order to achieve the above-mentioned object, the present invention provides a supervised fine-tuning method for Text2SQL task in low-resource scenario, which is decomposed into two parts: pattern connection task and SQL generation task; wherein the pattern connection task part includes:

[0009] S100, decomposing the entire telecommunications database into multiple slices, each slice containing an adjustable number of table information; and dynamically adjusting the number of tables covered by each slice according to the GPU memory limit;

[0010] S200, training a pattern connection model using a supervised fine-tuning method, and performing model inference to predict a table related to the user query question;

[0011] The SQL generation task part includes:

[0012] S300: Based on the tables and corresponding questions predicted by the pattern connection model, a SQL generation model is trained, and the SQL generation model is used to generate SQL query statements related to the user's query questions.

[0013] A further preferred technical solution of the present invention is that the specific method of step S100 is:

[0014] S110, dividing all tables in the telecommunications database into two sets, namely, a set of tables interconnected by foreign keys, and a set of tables without foreign key correlation;

[0015] S120. Decompose the two sets of collections into multiple slices according to GPU memory to obtain a slice set.

[0016] Preferably, in step S110, all tables in the telecommunication database are divided into two sets, namely, a set of tables interconnected by foreign keys, and a set of tables without foreign key correlation. The specific method is:

[0017] Define the set of all tables in the telecommunications database as , which is divided into and Two groups, represented by:

[0018] ;

[0019] in, represents a collection of tables interconnected by foreign keys, ;in, Represents a related group in which each table is subject to The constraints imposed by a foreign key or its related foreign keys; define ,in Indicated in The first The name of a table and all its columns; ,in Indicates Related groups In the table List, Indicates of the related groups Table related to foreign keys, and ;

[0020] represents a collection of tables with no foreign key dependencies, ;in, Represents tables without foreign key dependencies and all their respective columns, defining ,in Indicates In the table List.

[0021] Preferably, in step S120, the two sets of collections are decomposed into multiple slices according to the GPU memory to obtain a slice set, and the specific method is:

[0022] right and Perform a more detailed decomposition and define the smallest slicing unit of table information in supervised fine-tuning , the slice set is ,definition For when the model is The maximum token capacity of a single data instance when fine-tuning with a batch size under the GPU memory capacity is defined as Token counts for encoding special characters and supervised fine-tuning templates for the model; After encoding, the total token count satisfy: ;

[0023] Initialize two empty collections and ;

[0024] First process the collection of tables interconnected by foreign keys , for each relevant group , for each of its tables Make a judgment, if and The combined length is less than , the table Add to If the length is greater than the given ,Will Add to and clear , and then the table Add to clear middle;

[0025] Then process the table collection without foreign key correlation , for each of the tables ,judge and Is the combined length less than If so, then the table Add to If not, then Add to and clear , and then the table Add to clear middle;

[0026] Get the final slice collection .

[0027] Preferably, the pattern connection model in step S200 is iterated multiple times from each slice The target is extracted from the target, and its target is expressed as:

[0028] ;

[0029] ;

[0030] ;

[0031] in, Represents a database, Large language models for representing pattern connection tasks, represents the total number of questions contained in the dataset, Indicates the total number of slices, Indicates the requirements of SQL query, Represents the total slice collection The slices, and , represents all parameters involved in training, is the loss function used in the supervised fine-tuning process; Indicates in slice The name of the target table related to the problem is found in the slice. If no table related to the problem is found in the slice, a special token is assigned to it, defined as #None#; The prediction of all relevant slices is completed, and a definite table prediction result is derived .

[0032] As a preferred method, a supervised fine-tuning method is used to train a pattern connection model, and the reasoning method of thought chain is introduced; as the number of samples increases during the fine-tuning process, the pattern connection model remembers the reasoning sequence and incorporates the previous reasoning information into the reasoning process, and the task goal is transformed into:

[0033] ;

[0034] ;

[0035] ;

[0036] in, Indicates The target table information covered by the slices, that is, The target table information covered by the slices Integrate into The training target of each slice.

[0037] Preferably, the supervised fine-tuning method is used to train a mode connection model, and the specific method is:

[0038] Join task datasets in a pattern containing paired questions and answers , and slice collections , as model input, set the number of training steps ;

[0039] Initialization mode connects the model and optimizer, and then enters the training loop; in each training, the loss is initialized first, and then Each pair of questions in and answers Processing; for slice collection Each slice in If the answer is A related table in In the slice In the case of , the mode connection model is used to make predictions and calculate the loss, and the number of balanced samples is used to make predictions and calculate the loss again, and then Add to In the collection; if Not Available , only the mode connection model is used to predict and calculate the loss;

[0040] After processing all slices and answers, perform back propagation of loss and optimizer steps and parameter updates; repeat the above process until the number of training steps reaches .

[0041] Preferably, model reasoning is performed in step S200, and the specific method is as follows:

[0042] The query question , and all tables under the target database D inputting the mode connection model;

[0043] By calling the function For all tables Perform a slice operation to obtain a slice set ;

[0044] Slice a collection Each slice in Processing, that is, using the language model function Combined query questions , Current Slice and a collection of related tables Get a related table , and then get the relevant table Add to table collection middle;

[0045] Finally, we get a collection of related tables .

[0046] Another aspect of the present invention provides a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions enabling a computer to execute the above-mentioned supervised fine-tuning method for the Text2SQL task in a low-resource scenario.

[0047] Yet another aspect of the present invention provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus, and the processor calls the logic instructions in the memory to execute the above-mentioned supervised fine-tuning method of the Text2SQL task in the low-resource scenario.

[0048] On the other hand, the present invention provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer executes the above-mentioned supervised fine-tuning method for the Text2SQL task in the low-resource scenario.

[0049] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0050] (1) Reduced GPU memory usage: In supervised fine-tuning, the entire database is decomposed into slices, and the slices flexibly cover different numbers of tables according to computing resource constraints. The goal of supervised fine-tuning for the pattern connection task is redesigned. Since the number of tokens in the input model is reduced, memory resource consumption is reduced during the fine-tuning process and inference process.

[0051] (2) Combined with the thinking chain method: The thinking chain method is added in the training and reasoning process to enhance the model's ability to perceive the relationship between various discrete slices and effectively improve the generation effect of subsequent results.

[0052] (3) Large models applicable to different architectures: The present invention is designed for low-resource scenarios and telecommunication databases, taking into account the underlying mechanisms of the model and having wider applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is an overall flow chart of the supervised fine-tuning method for the Text2SQL task in a low-resource scenario of the present invention.

[0054] Figure 2 This is a flowchart of model reasoning in step S200 of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0056] Combine the following Figure 1-Figure 2 The present invention describes a supervised fine-tuning method, medium, electronic device, and computer program product for a Text2SQL task in a low-resource scenario.

[0057] First, some technical terms involved in the present invention are explained and illustrated.

[0058] Large Language Model (LLM): refers to a deep learning model trained with large amounts of text data, designed to understand and generate human language. These models typically contain tens of billions (or more) of parameters and gain a deep understanding of language by training on massive amounts of text data. Large language models can handle a variety of natural language tasks, such as text classification, question-answering, and conversation, and are an important path to artificial intelligence. Currently, large language models use a Transformer architecture and pre-training objectives (such as Language Modeling) similar to small models. The difference from small models is the increase in model size, training data, and computing resources.

[0059] Database: A database is a computer software system that stores and manages data according to data structures. It is an entity that can reasonably store data in a "warehouse". Users can store transaction data to be managed in the "warehouse" according to certain rules. The two concepts of "data" and "library" are combined into a database. The storage space of a database is very large and can store millions, tens of millions, or even hundreds of millions of data. Database is a new method and technology for data management. It can organize data more appropriately, maintain data more conveniently, control data more strictly, and use data more effectively.

[0060] Text2SQL: It is a subtask in the field of semantic analysis in natural language processing (NLP). It allows users to make query requests to the database in natural language, and the system will automatically convert these requests into SQL statements to obtain the required information from the database. This technology simplifies the query creation process, allowing people who are not familiar with SQL to easily query the database. Text2SQL technology is widely used in data analysis, data mining, voice assistants, chatbots and other fields to help users efficiently obtain database information. Implementing Text2SQL requires steps such as semantic understanding, information extraction, SQL generation and execution return, involving complex technologies such as natural language processing and database structure understanding.

[0061] Token: In the context of a large model, it refers to the smallest unit into which text data is segmented before or during model processing. These units can be words, punctuation marks, subwords, etc., depending on the model's vocabulary and word segmentation strategy.

[0062] Supervised fine-tuning (SFT): is a commonly used technical method in the field of machine learning and natural language processing. It refers to further training on a pre-trained model (such as a large language model) using a well-labeled dataset to make the model perform better on a specific task or field. Supervised fine-tuning can utilize the parameters and structure of the pre-trained model to avoid training the model from scratch, thereby accelerating the model training process and improving the model's performance on the target task. However, it also requires a large amount of labeled data for fine-tuning the target task. If the labeled data is insufficient, the fine-tuned model may perform poorly.

[0063] Parameter Efficient Fine-tuning (PEFT): is a technique used in Natural Language Processing (NLP) to improve the performance of pre-trained language models on specific downstream tasks. It involves reusing the parameters of a pre-trained model and fine-tuning it on a smaller dataset, which can save computing resources and time compared to training the entire model from scratch. PEFT is a technique that adapts to new tasks by adjusting only a small number of additional parameters. These additional parameters can be newly added embedding layers, low-rank matrices, or other types of parameters, which are used to "guide" or "adjust" the output of the pre-trained model to make it more suitable for the new task.

[0064] Chain of Thought (CoT): refers to the process of breaking down logically complex problems into complete thinking processes through a series of logically related thoughts. It aims to show a small number of examples to the large language model and explain the reasoning process through these examples, so that the large language model can learn to show the reasoning process when generating answers and guide it to obtain more accurate answers.

[0065] GPU resource efficiency: refers to the extent to which the computing resources of the GPU are effectively utilized when performing computing tasks. Specifically, it measures the actual utilization efficiency of the GPU's computing power, memory interface and other resources when processing tasks such as graphics rendering and data parallel computing. Efficient GPU resource utilization means that more data can be processed and more complex computing tasks can be completed in the same amount of time. This usually depends on factors such as the GPU's architectural design, number of cores, core frequency, video memory capacity and bandwidth, as well as the degree of optimization of the software and drivers. Improving GPU resource efficiency helps reduce computing costs and speed up task processing, and is widely used in deep learning, scientific computing, graphics rendering and other fields.

[0066] Example 1: This example provides a supervised fine-tuning method for the Text2SQL task in a low-resource scenario. The method of this example decomposes the entire Text2SQL task into two different subtasks: one is defined as a pattern connection task, whose purpose is to predict tables related to the problem, and the other is a SQL generation task, which is responsible for generating query statements related to the problem. In the fine-tuning process of the pattern connection model, this example decomposes the entire database into a flexible combination with an adjustable number of tables, so that the model can learn the relationships in the entire database from these scattered slices, thereby effectively reducing the number of tokens input to the model to reduce GPU memory usage. In addition, this example trains the model's thinking chain ability for this task, enhancing the model's ability to perceive the connections between various discrete slices during the reasoning process.

[0067] This method aims to solve the problem of large GPU memory consumption due to the large input model context length when fine-tuning large models. By redesigning the goal of supervised fine-tuning of the pattern connection task, all tables in the database are flexibly decomposed into several slices, and the problem, database, information from the target table of the slice and the connection between them are learned under limited GPU memory. At the same time, the function of the chain of thought (CoT) technology is added to enhance the model's ability to perceive the entire database from some slices during the reasoning stage. This method overcomes the limitations of existing methods, effectively alleviates the resource bottleneck problem in low-resource environments, and is applicable to large models of different architectures, providing a practical technical path for the application of large-scale telecommunications data.

[0068] Specifically, if Figure 1 As shown, the method is decomposed into two parts: the schema connection task and the SQL generation task.

[0069] The mode connection task part includes:

[0070] S100, decomposing the entire telecommunications database into multiple slices, each slice containing an adjustable number of table information; and dynamically adjusting the number of tables covered by each slice according to the GPU memory limit;

[0071] S200, training a pattern connection model using a supervised fine-tuning method, and performing model inference to predict a table related to the user query question;

[0072] The SQL generation task part includes:

[0073] S300: Based on the tables and corresponding questions predicted by the pattern connection model, a SQL generation model is trained, and the SQL generation model is used to generate SQL query statements related to the user's query questions.

[0074] In this embodiment, the pattern connection model is used as the focus to precisely adjust the overall process. The core is to group all tables in the telecommunications database into several slices according to specific relationships. The model can flexibly cover different numbers of tables according to the GPU memory limit. The method is as follows:

[0075] S110, dividing all tables in the telecommunications database into two sets, namely, a set of tables interconnected by foreign keys, and a set of tables without foreign key correlation; the specific method is:

[0076] Define the set of all tables in the telecommunications database as , which is divided into and Two groups, represented by:

[0077] ;

[0078] in, represents a collection of tables interconnected by foreign keys, ;in, Represents a related group in which each table is subject to The constraints imposed by a foreign key or its related foreign keys; define ,in Indicated in The first The name of a table and all its columns; ,in Indicates Related groups In the table List, Indicates of the related groups Table related to foreign keys, and ;

[0079] represents a collection of tables with no foreign key dependencies, ;in, Represents tables without foreign key dependencies and all their respective columns, defining ,in Indicates In the table List.

[0080] S120, decomposing the two sets of collections into multiple slices according to the GPU memory to obtain a slice set, the specific method is as follows:

[0081] right and Perform a more detailed decomposition and define the smallest slicing unit of table information in supervised fine-tuning , the slice set is ,definition For when the model is The maximum token capacity of a single data instance when fine-tuning with a batch size under the GPU memory capacity is defined as Token counts for encoding special characters and supervised fine-tuning templates for the model; After encoding, the total token count satisfy: ;

[0082] Initialize two empty collections and ;

[0083] First process the collection of tables interconnected by foreign keys , for each relevant group , for each of its tables Make a judgment, if and The combined length is less than , the table Add to If the length is greater than the given ,Will Add to and clear , and then the table Add to clear middle;

[0084] Then process the table collection without foreign key correlation , for each of the tables ,judge and Is the combined length less than If so, then the table Add to If not, then Add to and clear , and then the table Add to clear middle;

[0085] Get the final slice collection .

[0086] The following computer program describes Transformed into a collection of total slices The detailed process:

[0087]

[0088] Converted into a collection ,gather Including expressed as That is, the table in the database is decomposed into multiple fragments, each fragment contains a certain number of tables and their column descriptions. Each slice As the smallest unit of table information, the table information of a unit is distinguished during the supervised fine-tuning process.

[0089] The pattern connection task requires multiple iterations from each slice The target can be expressed as:

[0090] ;

[0091] ;

[0092] ;

[0093] in, Represents a database, Large language models for representing pattern connection tasks, represents the total number of questions contained in the dataset, Indicates the total number of slices, Indicates the requirements of SQL query, Represents the total slice collection The slices, and , represents all parameters involved in training, is the loss function used in the supervised fine-tuning process; Indicates in slice The name of the target table related to the problem is found in the slice. If no table related to the problem is found in the slice, a special token is assigned to it, defined as #None#; The prediction of all relevant slices is completed, and a definite table prediction result is derived .

[0094] In supervised fine-tuning, database knowledge is added to the model together with the reasoning ability of the thinking chain, e.g. Figure 2 As shown in Figure 2, as the number of samples increases during fine-tuning, the model will remember this reasoning sequence and incorporate previous reasoning information during reasoning. Therefore, the task objective is transformed into:

[0095] ;

[0096] ;

[0097] ;

[0098] in, Indicates The target table information covered by the slices, that is, The target table information covered by the slices Integrate into The training target of each slice.

[0099] Train a mode connection model using supervised fine-tuning as follows:

[0100] Join task datasets in a pattern containing paired questions and answers , and slice collections , as model input, set the number of training steps ;

[0101] Initialization mode connects the model and optimizer, and then enters the training loop; in each training, the loss is initialized first, and then Each pair of questions in and answers Processing; for slice collection Each slice in If the answer is A related table in In the slice In the case of , the mode connection model is used to make predictions and calculate the loss, and the number of balanced samples is used to make predictions and calculate the loss again, and then Add to In the collection; if Not Available In , only the mode connection model is used to predict and calculate the loss;

[0102] After processing all slices and answers, perform back propagation of loss and optimizer steps and parameter updates; repeat the above process until the number of training steps reaches .

[0103] The process of training the pattern connection model is represented by a computer program as follows:

[0104]

[0105] The main purpose of the algorithm is to train the model using pairs of questions and answers in a given dataset, make model predictions using slices in the slice set W, calculate the loss, and finally update the model parameters through backpropagation and the optimizer.

[0106] The process of solving the pattern connection task is transformed into a multi-step prediction process using a thought chaining approach to infer the potential table names from multiple slices. When performing model inference, the size of each slice is Set equal to . Model reasoning, the specific method is:

[0107] The query question , and all tables under the target database D inputting the mode connection model;

[0108] By calling the function For all tables Perform a slice operation to obtain a slice set ;

[0109] Slice a collection Each slice in Processing, that is, using the language model function Combined query questions , Current Slice and a collection of related tables Get a related table , and then get the relevant table Add to table collection middle;

[0110] Finally, we get a collection of related tables .

[0111] The computer program for the reasoning phase of the pattern connection task is as follows:

[0112]

[0113] Summarizing the above technical solutions, the key points of this embodiment are summarized as follows:

[0114] 1. Pattern connection model: The main function of the pattern connection model is to simplify the entire process by decomposing the complete database into flexible table combinations, so that the model can learn the relationships within the database from these scattered slices and predict the tables related to the problem.

[0115] 2. Data decomposition: In the fine-tuning process of the schema connection model, this embodiment decomposes the database into multiple slices, and the model allows each slice to flexibly cover different numbers of tables according to the GPU memory limit. The model is trained in fine-tuning to establish the relationship between the problem, the entire database, and the target table from the distributed slices.

[0116] 3. Chain of Thought: In the training and reasoning process of the pattern connection model, in order to enhance the ability to perceive the relationship between each discrete slice, the chain thinking ability of the model is trained. The Chain of Thought (CoT) capability effectively merges the table information inferred from the previous slice into the known information of the current reasoning, enabling the model to gradually guide the model to generate all table names related to the problem.

[0117] 4. Target table prediction: During inference, the model infers the table relevant to the question from each slice in turn.

[0118] 5. SQL Generation Model: After the fine-tuning process and inference process of the pattern connection model are completed, the predicted target table and question are sent to the SQL generation model to generate the final SQL query statement. Since the pattern connection model filters out a large number of irrelevant tables, the model is fine-tuned using only a small number of tables including the target table, which significantly reduces the required memory.

[0119] The present invention can significantly reduce the consumption of GPU memory resources during the supervised fine-tuning of the Text2SQL task in a large telecommunications database, and ensure that the task has good SQL generation capabilities. It is applicable to multiple fields such as telecommunications data, data analysis, and intelligent question answering. It is expected that methods for reducing the total time required for the reasoning process of the present invention will be studied in the future while maintaining its strong performance.

[0120] Tables 1 to 8 are the results of verifying the method of this embodiment.

[0121] The present invention is compared with three different methods for predicting database tables (zero-shot, compromise, and DTS-SQL) on two data sets. The results are shown in Table 1. In the table, “-” is used to indicate that no experimental results can be obtained.

[0122] Table 1 Performance of different models in the pattern connection model table prediction task

[0123]

[0124] Table 1 shows that the zero-shot method exhibits the worst performance, indicating that the method that only relies on the inherent pre-training knowledge of the calling model to solve complex Text2SQL tasks still has a gap compared with the supervised fine-tuning method. The compromise method only establishes the relationship between the question and the table name, but ignores the column name, which can complete the table prediction task with lower computing resources. However, due to the lack of information, it is not enough to handle large-scale Text2SQL tasks in telecommunications databases. DTS-SQL demonstrates the best performance on both datasets by seamlessly establishing direct associations between answers, databases, and related tables. However, during the supervised fine-tuning process, there is often a risk of exceeding the current GPU resource limit, which ultimately leads to an out-of-memory (OOM) error. By comparing with these three existing methods, the method proposed in the present invention achieves very impressive performance on all three open source models while significantly reducing GPU memory requirements.

[0125] Table 2 Prediction evaluation results of the thinking chain CoT method

[0126]

[0127] Table 2 shows that in the supervised fine-tuning process, adding the previously selected table to the current training information can improve the model's table accuracy to a certain extent, but may cause a slight decrease in the model's table coverage prediction ability.

[0128] Table 3 The impact of slices of different granularities on GPU memory and table prediction accuracy during training

[0129]

[0130] Table 4 The impact of different granularity slices on table prediction accuracy and inference time during inference

[0131]

[0132] Table 3 shows that during the training process, the more slices there are, the fewer tokens slice_token each slice contains, which can reduce the use of GPU memory resources, but will lose some training time. Table 4 shows that during the inference process, the more slices are selected, the higher the accuracy of the model prediction, but it will increase some inference time.

[0133] In the supervised fine-tuning method proposed in the present invention, the GPU memory size required for training is adjusted by adjusting slice_token. Slice_tokens of different sizes mean that different amounts of table information are encapsulated in the slice. Therefore, using slice_tokens of different sizes also affects the inference performance of the model. The results show that when the slice size is moderate, the model performs best.

[0134] Total_GPU_usage is used to illustrate the maximum GPU memory required for the entire workflow, Inference_time represents the average inference time of a single data instance, and Execution Accuracy (EX) and Logical Form Accuracy (EM) are used to evaluate the quality of the generated SQL statements. In the result evaluation of the SQL generation model, the present invention is benchmarked against three supervised fine-tuning methods: Zero-shot, DTS-SQL, and DB-GPT. The results are shown in Table 5.

[0135] Table 5 Evaluation results of TEXT2SQL task using different supervised fine-tuning methods

[0136]

[0137] Table 5 shows that the proposed method is significantly better than the other three methods in SQL generation performance and performs well in reducing total memory consumption. Comparing the experimental results of the proposed method and DTS-SQL, it can be observed that in the complex Text2SQL task of a large-scale telecommunications database, designing a table prediction model and using a method that combines the table prediction results from the pattern connection model with the table involved in the target SQL in the supervised fine-tuning stage to train the SQL generation model can reduce the information processing complexity of the SQL generation model, thereby improving the execution accuracy to a certain extent.

[0138] Embodiment 2: This embodiment provides a non-transitory computer-readable storage medium on which computer instructions are stored. The computer instructions enable a computer to perform a supervised fine-tuning method for a Text2SQL task in a low-resource scenario. The method includes the following steps:

[0139] S100, decomposing the entire telecommunications database into multiple slices, each slice containing an adjustable number of table information; and dynamically adjusting the number of tables covered by each slice according to the GPU memory limit;

[0140] S200, training a pattern connection model using a supervised fine-tuning method, and performing model inference to predict a table related to the user query question;

[0141] S300: Based on the tables and corresponding questions predicted by the pattern connection model, a SQL generation model is trained, and the SQL generation model is used to generate SQL query statements related to the user's query questions.

[0142] Embodiment 3: This embodiment provides an electronic device, which may include: a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other via the communication bus. The processor may call the logic instructions in the memory to execute a supervised fine-tuning method for a Text2SQL task in a low-resource scenario, and the method includes the following steps:

[0143] S100, decomposing the entire telecommunications database into multiple slices, each slice containing an adjustable number of table information; and dynamically adjusting the number of tables covered by each slice according to the GPU memory limit;

[0144] S200, training a pattern connection model using a supervised fine-tuning method, and performing model inference to predict a table related to the user query question;

[0145] S300: Based on the tables and corresponding questions predicted by the pattern connection model, a SQL generation model is trained, and the SQL generation model is used to generate SQL query statements related to the user's query questions.

[0146] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0147] Embodiment 4: This embodiment provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform a supervised fine-tuning method for a Text2SQL task in a low-resource scenario. The method includes the following steps:

[0148] S100, decomposing the entire telecommunications database into multiple slices, each slice containing an adjustable number of table information; and dynamically adjusting the number of tables covered by each slice according to the GPU memory limit;

[0149] S200, training a pattern connection model using a supervised fine-tuning method, and performing model inference to predict a table related to the user query question;

[0150] S300: Based on the tables and corresponding questions predicted by the pattern connection model, a SQL generation model is trained, and the SQL generation model is used to generate SQL query statements related to the user's query questions.

[0151] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A supervised fine-tuning method for Text2SQL tasks in low-resource scenarios, characterized in that: This method is decomposed into two parts: schema connection task and SQL generation task; The mode connection task part includes: S100, decomposing the entire telecommunications database into multiple slices, each slice containing an adjustable number of table information; and dynamically adjusting the number of tables covered by each slice according to the GPU memory limit; the specific method is: S110, dividing all tables in the telecommunications database into two sets, namely, a set of tables interconnected by foreign keys, and a set of tables without foreign key correlation; the specific method is: Define the set of all tables in the telecommunications database as , which is divided into and Two groups, represented by: ； in, represents a collection of tables interconnected by foreign keys, ;in, Represents a related group in which each table is subject to The constraints imposed by a foreign key or its related foreign keys; define ,in Indicated in The first The name of a table and all its columns; ,in Indicates Related groups In the table List, Indicates of the related groups Table related to foreign keys, and ; represents a collection of tables with no foreign key dependencies, ;in, Represents tables without foreign key dependencies and all their respective columns, defining ,in Indicates In the table List; S120, decomposing the two sets of collections into multiple slices according to the GPU memory to obtain a slice set; the specific method is: right and Perform a more detailed decomposition and define the smallest slicing unit of table information in supervised fine-tuning , the slice set is ,definition For when the model is The maximum token capacity of a single data instance when fine-tuning with a batch size under the GPU memory capacity is defined as Token counts for encoding special characters and supervised fine-tuning templates for the model; After encoding, the total token count satisfy: ; Initialize two empty collections and ; First process the collection of tables interconnected by foreign keys , for each relevant group , for each of its tables Make a judgment, if and The combined length is less than , the table Add to If the length is greater than the given ,Will Add to and clear , and then the table Add to clear middle; Then process the table collection without foreign key correlation , for each of the tables ,judge and Is the combined length less than If so, then the table Add to If not, then Add to and clear , and then the table Add to clear middle; Get the final slice set ; S200, training a pattern connection model using a supervised fine-tuning method, and performing model inference to predict a table related to the user query question; Train a mode connection model using supervised fine-tuning as follows: Join task datasets in a pattern containing paired questions and answers , and slice collections , as model input, set the number of training steps ; Initialization mode connects the model and optimizer, and then enters the training loop; in each training, the loss is initialized first, and then Each pair of questions in and answers Processing; for slice collection Each slice in If the answer is A related table in In the slice In the case of , the mode connection model is used to make predictions and calculate the loss, and the number of balanced samples is used to make predictions and calculate the loss again, and then Add to In the collection; if Not Available In , only the mode connection model is used to predict and calculate the loss; After processing all slices and answers, perform back propagation of loss and optimizer steps and parameter updates; repeat the above process until the number of training steps reaches ; In step S200, model inference is performed, and the specific method is as follows: The query question , and all tables under the target database D inputting the mode connection model; By calling the function For all tables Perform a slice operation to obtain a slice set ; Slice a collection Each slice in Processing, that is, using the language model function Combined query questions , Current Slice and a collection of related tables Get a related table , and then get the relevant table Add to table collection middle; Finally, we get a collection of related tables ; The SQL generation task part includes: S300: Based on the tables and corresponding questions predicted by the pattern connection model, a SQL generation model is trained, and the SQL generation model is used to generate SQL query statements related to the user's query questions.

2. The supervised fine-tuning method for Text2SQL tasks in low-resource scenarios according to claim 1 is characterized in that: In step S200, the pattern connection model is iterated from each slice The target is extracted from the target, and its target is expressed as: ； ； ； in, Represents a database, Large language models for representing pattern connection tasks, represents the total number of questions contained in the dataset, Indicates the total number of slices, Indicates the requirements of SQL query, Represents the total slice collection The slices, and , represents all parameters involved in training, is the loss function used in the supervised fine-tuning process; Indicates in slice The name of the target table related to the problem is found in the slice. If no table related to the problem is found in the slice, a special token is assigned to it, defined as #None#; The prediction of all relevant slices is completed, and a definite table prediction result is derived .

3. The supervised fine-tuning method for Text2SQL tasks in low-resource scenarios according to claim 2 is characterized in that: The supervised fine-tuning method is used to train a pattern connection model, and the reasoning method of thought chain is introduced; as the number of samples increases during the fine-tuning process, the pattern connection model memorizes the reasoning process and incorporates previous reasoning information into the reasoning process. The task goal is transformed into: ； ； ； in, Indicates The target table information covered by the slices, that is, The target table information covered by the slices Integrate into The training target of each slice.

4. A non-transitory computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and the computer instructions enable the computer to execute the supervised fine-tuning method for the Text2SQL task in a low-resource scenario as described in any one of claims 1-3.

5. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus, and the processor calls the logic instructions in the memory to execute the supervised fine-tuning method of the Text2SQL task in the low-resource scenario described in any one of claims 1-3.

Images