Large model knowledge distillation method and device for text2sql task

Abstract

The application relates to a large model knowledge distillation method and device for a Text2SQL task, wherein the method comprises the following steps: based on original Text2SQL training samples, a target large language model is supervised and trained to generate a teacher model corresponding to a Text2SQL task; a random mask operation is performed on SQL sentences in the original Text2SQL training samples to obtain a mask sequence, the mask sequence is input into an initial student model to predict mask tokens at mask positions in the mask sequence, mask prediction tokens are obtained, and the mask tokens in the mask sequence are replaced with the corresponding mask prediction tokens to construct a noisy sample; target training samples are determined according to the noisy sample, and the initial student model is supervised and distilled based on the target training samples to generate a final target student model. Thus, the problems that the model reasoning delay and model deployment difficulty are large in the prior art, the model distillation training and reasoning are obviously different, and the performance and efficiency of knowledge distillation are difficult to be considered are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to a large model knowledge distillation method and device for a Text2SQL task. BACKGROUND

[0002] In today's information age, database query and information retrieval have become increasingly important. The Text2SQL task aims to convert natural language questions into SQL statements for query, and is one of the key technologies to achieve this goal. However, traditional Text2SQL models often struggle to handle complex query logic and multi-hop reasoning, especially when faced with large-scale databases, and their performance and accuracy are difficult to meet the requirements of practical applications. In order to improve the processing capacity of the Text2SQL task, existing technologies can use large language models, which can better understand user query intent and generate more accurate SQL statements, thereby achieving better Text2SQL performance. However, large language models often have super-large-scale parameters, low model inference efficiency, and high deployment cost, which greatly limits the development of Text2SQL systems based on large language models.

[0003] In addition, existing technologies can solve the problems of low inference efficiency and high deployment cost of Text2SQL systems based on large language models through knowledge distillation, quantization, and model pruning, etc. Among them, knowledge distillation has attracted widespread attention in the industry due to its ease of use and efficient model compression capabilities, and more and more research attempts to use knowledge distillation technology to achieve lightweight Text2SQL.

[0004] Specifically, knowledge distillation technology extracts knowledge from large models (teacher models) and transfers it to small models (student models), aiming to reduce the complexity and resource consumption of the model as much as possible without significantly reducing the performance of the model. In order to further improve the effect of distillation, the current leading knowledge distillation method uses the samples generated by the student model being optimized to train (on-policy distillation), that is, it needs to generate corresponding training samples in a self-recurrent manner online while training the student model. This on-policy distillation method reduces the difference between training and inference by using the same target strategy in the training phase as in the inference phase, thereby achieving better distillation effect. However, this distillation method requires the student model to generate samples in real time, resulting in serious training delay and low training efficiency.

[0005] In summary, the existing technology has large model inference delay and model deployment difficulty, and the difference between model distillation training and inference is obvious, making it difficult to balance the performance and efficiency of knowledge distillation, which needs to be solved urgently. SUMMARY

[0006] The application provides a large model knowledge distillation method and device for a Text2SQL task to solve the problems of model reasoning delay and large model deployment difficulty in the prior art, obvious differences between model distillation training and reasoning, and difficulty in balancing the performance and efficiency of knowledge distillation.

[0007] The first aspect of the application provides a large model knowledge distillation method for a Text2SQL task, including the following steps: based on a preset original Text2SQL training sample corresponding to a Text2SQL task, performing supervised training on a target large language model to generate a teacher model corresponding to the Text2SQL task; performing random mask operation on an SQL statement in the original Text2SQL training sample to obtain a mask sequence, inputting the mask sequence into an initial student model to predict mask tokens at mask positions in the mask sequence and obtain mask prediction tokens, and replacing the mask tokens in the mask sequence with the corresponding mask prediction tokens to construct a noisy sample corresponding to the original Text2SQL training sample; determining a target training sample according to the noisy sample, and based on the target training sample, performing distillation training on the initial student model in combination with the teacher model to generate a final target student model. Optionally, in an embodiment of the application,

[0008] Optionally, in an embodiment of the application, the supervised training on the target large language model based on the preset original Text2SQL training sample corresponding to the Text2SQL task includes: performing word segmentation on a natural language question and an SQL statement in the original Text2SQL training sample to obtain a natural language question token sequence and an SQL query token sequence respectively; constructing an original Text2SQL training data set based on the natural language question token sequence and the SQL query token sequence, and performing supervised training on the target large language model through the original Text2SQL training data set to generate the teacher model.

[0009] Optionally, in an embodiment of the application, the inputting of the mask sequence into the initial student model to predict the mask tokens at the mask positions in the mask sequence and obtain the mask prediction tokens, and the replacement of the mask tokens in the mask sequence with the corresponding mask prediction tokens to construct the noisy sample corresponding to the original Text2SQL training sample include: splicing the natural language question token sequence and the mask sequence to obtain a spliced sequence, and predicting the mask tokens at the mask positions in the mask sequence based on the initial student model and the spliced sequence to obtain the mask prediction tokens.

[0010] replace the masked tokens in the masked sequence with corresponding predicted masked tokens to obtain a rewritten SQL query statement token sequence corresponding to the SQL query token sequence; and concatenate the natural language question token sequence and the rewritten SQL query statement token sequence to establish the noisy sample.

[0011] Optionally, in an embodiment of the present application, the determining, based on the noisy sample, a target training sample, supervising, based on the target training sample, the initial student model in combination with the teacher model to perform distillation training to generate a final target student model, comprises: inputting the target training sample into the teacher model and the initial student model respectively to generate a teacher model output probability and a student model output probability respectively; determining a distillation training loss function based on the teacher model output probability, the student model output probability, and a preset probability distribution distance measurement function; and performing back propagation training on the initial student model according to the distillation training loss function and a preset NSP training objective function to generate the final target student model.

[0012] The second aspect embodiment of the present application provides a large model knowledge distillation device for a Text2SQL task, comprising: a supervised training module configured to supervise training of a target large language model based on a preset original Text2SQL training sample corresponding to a Text2SQL task to generate a teacher model corresponding to the Text2SQL task; a noisy module configured to perform a random masking operation on an SQL statement in the original Text2SQL training sample to obtain a masked sequence, and input the masked sequence into an initial student model to predict masked tokens at masked positions in the masked sequence to obtain predicted masked tokens, and replace the masked tokens in the masked sequence with the corresponding predicted masked tokens to construct a noisy sample corresponding to the original Text2SQL training sample; and a distillation training module configured to determine a target training sample according to the noisy sample, supervise distillation training of the initial student model in combination with the teacher model based on the target training sample to generate a final target student model.

[0013] Optionally, in an embodiment of the present application, the supervised training module comprises: a tokenization unit configured to tokenize a natural language question and an SQL statement in the original Text2SQL training sample to obtain a natural language question token sequence and an SQL query token sequence respectively; and a construction unit configured to construct an original Text2SQL training dataset based on the natural language question token sequence and the SQL query token sequence, and supervise training of the target large language model through the original Text2SQL training dataset to generate the teacher model.

[0014] Optionally, in an embodiment of the present application, the noise adding module comprises: a first splicing unit configured to splice the natural language question token sequence and the mask sequence to obtain a spliced sequence, and predict a mask token at a mask position in the mask sequence based on the initial student model and the spliced sequence to obtain a mask predicted token; a replacement unit configured to replace the mask token in the mask sequence with the corresponding mask predicted token to obtain a rewritten SQL query statement token sequence corresponding to the SQL query token sequence; and a second splicing unit configured to splice the natural language question token sequence and the rewritten SQL query statement token sequence to establish the noise-added sample.

[0015] Optionally, in an embodiment of the present application, the distillation training module comprises: a generation unit configured to input the target training sample into the teacher model and the initial student model respectively to generate a teacher model output probability and a student model output probability respectively; a determination unit configured to determine a distillation training loss function based on the teacher model output probability, the student model output probability, and a preset probability distribution distance measurement function; and a back propagation training unit configured to perform back propagation training on the initial student model according to the distillation training loss function and a preset NSP training objective function to generate the final target student model.

[0016] An embodiment of the third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large model knowledge distillation method for a Text2SQL task as described in the above embodiments.

[0017] An embodiment of the fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the large model knowledge distillation method for a Text2SQL task as described above.

[0018] An embodiment of the fifth aspect of the present application provides a computer program product comprising a computer program, which is executed to implement the large model knowledge distillation method for a Text2SQL task as described above.

[0019] Therefore, the embodiments of the present application have the following beneficial effects:

[0020] Embodiments of the present application can supervise training of a target large language model based on a preset original Text2SQL training sample corresponding to a Text2SQL task, to generate a teacher model corresponding to the Text2SQL task; perform random mask operation on an SQL statement in the original Text2SQL training sample to obtain a masked sequence, and input the masked sequence into an initial student model to predict masked tokens at masked positions in the masked sequence, to obtain masked prediction tokens, and replace the masked tokens in the masked sequence with the corresponding masked prediction tokens, to construct a noisy sample corresponding to the original Text2SQL training sample; determine a target training sample according to the noisy sample, and perform distillation training of the initial student model based on the target training sample and in combination with the teacher model, to generate a final target student model. The present application actively adds noise to the training sample in the training stage, simulates error output in the self-recurrent generation in the model inference stage, to alleviate the difference between model training and inference, and the generation of the noisy sample only needs one forward propagation, so that the training performance and efficiency can be well balanced. Thus, the problems of the prior art, such as model inference delay and large model deployment difficulty, obvious difference between model distillation training and inference, and difficulty in balancing the performance and efficiency of knowledge distillation, are solved.

[0021] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the accompanying drawings, wherein:

[0023] Figure 1 A flowchart of a large model knowledge distillation method for a Text2SQL task according to an embodiment of the present application;

[0024] Figure 2 A logical architecture schematic diagram of a large model knowledge distillation method for a Text2SQL task according to an embodiment of the present application;

[0025] Figure 3 An effect comparison schematic diagram of a student model on a Text2SQL task according to an embodiment of the present application;

[0026] Figure 4 An example diagram of a large model knowledge distillation apparatus for a Text2SQL task according to an embodiment of the present application;

[0027] Figure 5 A structural schematic diagram of an electronic device according to an embodiment of the present application.

[0028] The method comprises the following steps: 10, a large model knowledge distillation device for a Text2SQL task; 100, a supervised training module; 200, a noise adding module; 300, a distillation training module; 501, a memory; 502, a processor; and 503, a communication interface. DETAILED DESCRIPTION

[0029] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, in which the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0030] A large model knowledge distillation method and device for a Text2SQL task are described below with reference to the accompanying drawings. In view of the problems mentioned in the above background art, the present application provides a large model knowledge distillation method for a Text2SQL task, in which a teacher model corresponding to a Text2SQL task is generated by supervising training of a target large language model based on a pre-set original Text2SQL training sample corresponding to the Text2SQL task; a random mask operation is performed on an SQL statement in the original Text2SQL training sample to obtain a masked sequence, and the masked sequence is input into an initial student model to predict masked tokens at masked positions in the masked sequence, obtaining masked prediction tokens, and replacing the masked tokens in the masked sequence with the corresponding masked prediction tokens to construct a noisy sample corresponding to the original Text2SQL training sample; a target training sample is determined according to the noisy sample, and the initial student model is supervised and distilled based on the target training sample and the teacher model to generate a final target student model. The present application actively adds noise to the training sample during the training phase to simulate the error output in the self-recurrent generation during the model inference phase, so as to alleviate the difference between the model training and inference, and the generation of the noisy sample only needs one forward propagation, thereby well balancing the training performance and efficiency. Thus, the problems of the prior art, such as model inference delay and difficulty in model deployment, obvious difference between model distillation training and inference, and difficulty in balancing the performance and efficiency of knowledge distillation, are solved.

[0031] Specifically, Figure 1 A flowchart of a large model knowledge distillation method for a Text2SQL task provided by an embodiment of the present application is shown in FIG. 10.

[0032] As shown in FIG. 10, the large model knowledge distillation method for a Text2SQL task comprises the following steps: Figure 1

[0033] ​In step S101, the target large language model is trained under supervision based on the original Text2SQL training samples corresponding to the preset Text2SQL task to generate the teacher model corresponding to the Text2SQL task.

[0034] In this embodiment, a large language model with a larger parameter scale can be trained in a supervised manner on the original Text2SQL training samples to obtain a teacher model, which can show excellent performance on the Text2SQL task.

[0035] Optionally, in one embodiment of this application, a teacher model is trained under supervision based on the original Text2SQL training samples corresponding to a preset Text2SQL task to generate a teacher model corresponding to the Text2SQL task. This includes: segmenting the natural language questions and SQL statements in the original Text2SQL training samples to obtain natural language question word sequences and SQL query word sequences, respectively; constructing an original Text2SQL training dataset based on the natural language question word sequences and SQL query word sequences; and training the target large language model under supervision using the original Text2SQL training dataset to generate a teacher model corresponding to the Text2SQL task.

[0036] It should be noted that the embodiments of this application can use the original Text2SQL samples as the training set, and use a word segmenter to segment the natural language questions and SQL statements in the Text2SQL, respectively obtaining the natural language question word sequence x = {x1, x2, ..., x...} m} and the SQL query term sequence y={y1,y2,...,y n}, where m and n represent the lengths of the question word sequence and the SQL statement word sequence, respectively.

[0037] Considering that existing large-scale language models are generally autoregressive model architectures, this embodiment of the application can input the word sequence (x,y) into the model for forward propagation, and M p The training objective is to predict the next word (Next TokenPrediction, NSP). Therefore, the specific expression for the prediction loss corresponding to the position of the SQL statement lexical sequence in this embodiment is shown in Equation 1:

[0038]

[0039] Where p(y t |x,y ＜t Let be the probability distribution of the teacher model's prediction for the t-th SQL statement term; This is the training loss function for NSP.

[0040] Subsequently, in this embodiment of the application, the Text2SQL training samples (x, y) can be used to perform supervised training on a larger language model with a larger parameter scale, thereby obtaining the trained teacher model M. p ;

[0041] Therefore, embodiments of this application can update the parameters of the teacher model through gradient backpropagation, thereby obtaining a high-performance teacher model.

[0042] In step S102, a random masking operation is performed on the SQL statements in the original Text2SQL training samples to obtain a mask sequence. The mask sequence is then input into the initial student model to predict the mask words at the mask positions in the mask sequence, thereby obtaining mask prediction words. The mask words in the mask sequence are then replaced with the corresponding mask prediction words to construct the noisy sample corresponding to the original Text2SQL training samples.

[0043] Furthermore, this embodiment of the application also requires random masking of the SQL statements in the original Text2SQL data, such as... Figure 2 As shown, the noisy sample is input into a student model with a smaller parameter size to predict the word corresponding to the mask position, and the mask at the corresponding position is replaced with the predicted word to construct the noisy sample.

[0044] Optionally, in one embodiment of this application, a mask sequence is input into an initial student model to predict mask words at mask positions in the mask sequence, obtaining mask prediction words. The mask words in the mask sequence are then replaced with the corresponding mask prediction words to construct a noisy sample corresponding to the original Text2SQL training sample. This includes: concatenating a natural language question word sequence and a mask sequence to obtain a concatenated sequence; predicting mask words at mask positions in the mask sequence based on the initial student model and the concatenated sequence to obtain mask prediction words; replacing the mask words in the mask sequence with the corresponding mask prediction words to obtain a rewritten SQL query word sequence corresponding to the SQL query word sequence; and concatenating the natural language question word sequence and the rewritten SQL query word sequence to establish a noisy sample.

[0045] It should be noted that the specific steps for adding noise to the input samples (i.e., the original Text2SQL training samples) in each training batch in this embodiment are as follows:

[0046] Step 1: Randomly mask the SQL statement word sequence y in the original Text2SQL (i.e., the data (x, y) of the current training batch), that is, replace some words in y with "|mask|", to obtain the mask sequence y′={y1,|mask|,...,y i ,|mask|,yi+2 ,...,y n};

[0047] Step 2: Concatenate the original natural language problem word sequence x and the mask sequence y′ from Step 1 to obtain (x, y′), and then input it into the student model M with a smaller parameter size obtained from the previous training round. q In the first round of training (if it's the first round, use an untrained student model as initialization), predict the words corresponding to the mask positions to obtain the predicted words at the mask positions. The subscript indicates the position of the mask word in the sequence;

[0048] It is important to note that the mask prediction in this step only requires inputting (x,y′) into the student model once, without performing autoregressive word prediction, which can significantly reduce the additional training latency.

[0049] Step 3: Replace the mask tokens in the mask sequence y′ with the corresponding mask position predicted tokens from Step 2 to obtain the rewritten SQL query statement token sequence. The original natural language problem word sequence x and Noisy samples are obtained by splicing.

[0050] Therefore, the embodiments of this application can simulate the problem of early positional lexical errors affecting subsequent lexical predictions in the autoregressive inference process of the model by using the above-mentioned mask prediction method. That is, by efficiently adding noise to the training samples, the errors in the model inference stage are simulated, and the generation of the noise samples only requires one forward propagation, so as to effectively alleviate the difference between the training stage and the inference stage, thereby improving and taking into account the performance and efficiency of subsequent distillation training.

[0051] In step S103, target training samples are determined based on the noisy samples. Based on the target training samples, the initial student model is distilled and trained in conjunction with the teacher model to generate the final target student model.

[0052] Furthermore, embodiments of this application can use noisy samples as training samples, and use the teacher model as the teacher to supervise the distillation training of the student model, ultimately obtaining a student model with good Text2SQL performance.

[0053] It is understood that the knowledge distillation framework of this application has the advantages of simple implementation and wide applicability. This framework is applicable to any Text2SQL large language model and can significantly improve the distillation training effect without introducing additional models and excessive training overhead.

[0054] Optionally, in one embodiment of this application, determining target training samples based on noisy samples, and performing distillation training on an initial student model supervised by a teacher model based on the target training samples to generate a final target student model, includes: inputting the target training samples into the teacher model and the initial student model respectively to generate the output probability of the teacher model and the output probability of the student model respectively; determining a distillation training loss function based on the output probability of the teacher model, the output probability of the student model, and a preset probability distribution distance metric function; and performing backpropagation training on the initial student model according to the distillation training loss function and a preset NSP training objective function to generate the final target student model.

[0055] In actual implementation, embodiments of this application can add noise to samples. As target training samples, they are respectively input into the teacher model M. p And the current round student model M with a smaller parameter size q In the initial student model, the output probability of the teacher model at the SQL statement location is obtained. And student model output probability

[0056] Subsequently, embodiments of this application can use any probability distribution distance metric function (such as inverse KL divergence) to calculate the difference between the output probability of the student model and the output probability of the teacher model, and use this as the distillation training loss function, and combine it with the NSP training objective to perform backpropagation training on the student model, as shown in equations (2) and (3):

[0057]

[0058] in, This represents the distance between probability distributions, specifically the classic forward KL divergence or backward KL divergence. This is the training loss function for distillation.

[0059] It should be noted that, in the embodiments of this application, only the distillation training is performed using... The parameters of the student model are updated, but the parameters of the teacher model are not. In addition, the above-mentioned noisy sample construction and distillation training operations need to be performed iteratively. That is, at the beginning of each training step, the training samples are noisy to obtain noisy samples, and then the student model is distilled and trained. The trained student model participates in the sample noisy processing of the next training step until all training steps are completed, so as to obtain the final student model with good Text2SQL performance (i.e., the final target student model).

[0060] Therefore, the embodiments of this application can learn Text2SQL related knowledge from the teacher model and achieve good performance on the publicly available Text2SQL dataset. Furthermore, considering that the student model parameter size is much smaller than that of the teacher model, it can significantly reduce model inference latency and lower the difficulty of model deployment.

[0061] Figure 3 This is a diagram illustrating the performance comparison of the student model on the Text2SQL task. (Example) Figure 3 As shown, the horizontal axis represents the distillation training latency, and the vertical axis represents the average execution accuracy of the student model on the publicly available Text2SQL dataset. The traditional distillation method refers to the conventional distillation training using only the original Text2SQL training data, while the on-policy distillation method refers to using the inference data generated online by the student model as the distillation training data. This application embodiment effectively alleviates the difference between model training and inference by simulating the erroneous output during inference in the training phase, and induces the student model to remove noisy samples to help it better learn the Text2SQL knowledge in the teacher model, ultimately improving the Text2SQL performance of the student model efficiently.

[0062] Therefore, the distillation training framework of this application can effectively improve the performance and training efficiency of the model. Without increasing the training latency too much, the trained student model can achieve significantly better results than other knowledge distillation methods on multiple publicly available Text2SQL datasets.

[0063] According to the knowledge distillation method for large-scale models targeting Text2SQL tasks proposed in this application, a teacher model is generated by supervised training of a target large-scale language model based on original Text2SQL training samples corresponding to a preset Text2SQL task. The SQL statements in the original Text2SQL training samples are randomly masked to obtain a mask sequence, which is then input into an initial student model to predict the masked terms at the masked positions in the mask sequence. The masked terms in the mask sequence are then replaced with the corresponding predicted masked terms to construct a noisy sample corresponding to the original Text2SQL training samples. A target training sample is determined based on the noisy sample. Based on the target training sample, the initial student model is distilled and trained under the supervision of the teacher model to generate the final target student model. This application mitigates the difference between model training and inference by actively adding noise to the training samples during the training phase, simulating the erroneous output generated in the autoregressive generation phase of the model inference phase. Furthermore, the generation of noisy samples requires only one forward propagation, thus effectively balancing training performance and efficiency.

[0064] Secondly, the large model knowledge distillation apparatus for Text2SQL tasks proposed according to embodiments of this application is described with reference to the accompanying drawings.

[0065] Figure 4 This is a block diagram of a large model knowledge distillation apparatus for Text2SQL tasks according to an embodiment of this application.

[0066] like Figure 4 As shown, the large model knowledge distillation device 10 for Text2SQL tasks includes: a supervised training module 100, a noise-adding module 200, and a distillation training module 300.

[0067] The supervised training module 100 is used to supervise the training of the target large language model based on the original Text2SQL training samples corresponding to the preset Text2SQL task, so as to generate the teacher model corresponding to the Text2SQL task.

[0068] The noise-adding module 200 is used to perform random masking operations on the SQL statements in the original Text2SQL training samples to obtain a mask sequence. The mask sequence is then input into the initial student model to predict the mask words at the mask positions in the mask sequence to obtain the mask prediction words. The mask words in the mask sequence are then replaced with the corresponding mask prediction words to construct the noise-adding samples corresponding to the original Text2SQL training samples.

[0069] The distillation training module 300 is used to determine the target training sample based on the noisy sample, and to perform distillation training on the initial student model in conjunction with the teacher model to generate the final target student model.

[0070] Optionally, in one embodiment of this application, the supervised training module 100 includes: a word segmentation unit and a construction unit.

[0071] The word segmentation unit is used to segment the natural language questions and SQL statements in the original Text2SQL training samples, and obtain the natural language question word sequence and the SQL query word sequence, respectively.

[0072] The building unit is used to construct the original Text2SQL training dataset based on the natural language question word sequence and the SQL query word sequence, and to supervise the training of the target large language model using the original Text2SQL training dataset to generate the teacher model corresponding to the Text2SQL task.

[0073] Optionally, in one embodiment of this application, the noise-adding module 200 includes: a first splicing unit, a replacement unit, and a second splicing unit.

[0074] The first concatenation unit is used to concatenate the natural language question word sequence and the mask sequence to obtain the concatenated sequence, and based on the initial student model and the concatenated sequence, to predict the mask words at the mask positions in the mask sequence to obtain the mask prediction words.

[0075] The replacement unit is used to replace the mask tokens in the mask sequence with the corresponding mask prediction tokens to obtain the rewritten SQL query statement token sequence corresponding to the SQL query token sequence.

[0076] The second concatenation unit is used to concatenate the natural language problem word sequence and the rewritten SQL query word sequence to create a noisy sample.

[0077] Optionally, in one embodiment of this application, the distillation training module 300 includes: a generation unit, a determination unit, and a backpropagation training unit.

[0078] The generation unit is used to input the target training samples into the teacher model and the initial student model respectively, so as to generate the output probabilities of the teacher model and the student model respectively.

[0079] The unit is defined to determine the distillation training loss function based on the output probabilities of the teacher model, the output probabilities of the student model, and a preset probability distribution distance metric function.

[0080] The backpropagation training unit is used to backpropagate the initial student model based on the distillation training loss function and the preset NSP training objective function to generate the final target student model.

[0081] It should be noted that the foregoing explanation of the embodiment of the large model knowledge distillation method for Text2SQL tasks also applies to the large model knowledge distillation apparatus for Text2SQL tasks in this embodiment, and will not be repeated here.

[0082] The knowledge distillation apparatus for large-scale models targeting Text2SQL tasks proposed in this application includes a supervised training module for supervising the training of a target large language model based on original Text2SQL training samples corresponding to a preset Text2SQL task, thereby generating a teacher model corresponding to the Text2SQL task; a noise-adding module for performing random masking operations on the SQL statements in the original Text2SQL training samples to obtain a mask sequence, inputting the mask sequence into an initial student model to predict the masked terms at the masked positions in the mask sequence, obtaining masked predicted terms, and replacing the masked terms in the mask sequence with the corresponding masked predicted terms to construct a noisy sample corresponding to the original Text2SQL training samples; and a distillation training module for determining a target training sample based on the noisy sample, and performing distillation training on the initial student model in conjunction with the teacher model to generate the final target student model. This application mitigates the difference between model training and inference by actively adding noise to the training samples during the training phase, simulating the erroneous output in the autoregressive generation during the model inference phase. Furthermore, the generation of the noisy sample only requires one forward propagation, thus effectively balancing training performance and efficiency.

[0083] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0084] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0085] When the processor 502 executes the program, it implements the large model knowledge distillation method for Text2SQL tasks provided in the above embodiments.

[0086] Furthermore, electronic devices also include:

[0087] Communication interface 503 is used for communication between memory 501 and processor 502.

[0088] The memory 501 is used to store computer programs that can run on the processor 502.

[0089] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0090] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0091] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0092] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0093] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described large model knowledge distillation method for Text2SQL tasks.

[0094] This application also provides a computer program product, including a computer program that, when executed, is used to implement the above-described large model knowledge distillation method for Text2SQL tasks.

[0095] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0097] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0098] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0099] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0100] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0102] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A large model knowledge distillation method for a Text2SQL task, characterized in that, The method comprises the following steps: Based on the preset original Text2SQL training sample corresponding to the Text2SQL task, the target large language model is supervised and trained to generate a teacher model corresponding to the Text2SQL task; Random mask operation is performed on the SQL statement in the original Text2SQL training sample to obtain a mask sequence, and the mask sequence is input into an initial student model to predict the mask token at the mask position in the mask sequence and obtain a mask prediction token, and the mask token in the mask sequence is replaced with the corresponding mask prediction token to construct a noisy sample corresponding to the original Text2SQL training sample; According to the noisy sample, a target training sample is determined, and based on the target training sample, the initial student model is supervised and distilled trained in combination with the teacher model to generate a final target student model.

2. The method of claim 1, wherein, The method comprises the following steps: The natural language question and the SQL statement in the original Text2SQL training sample are segmented to obtain a natural language question token sequence and a SQL query token sequence, respectively; Based on the natural language question token sequence and the SQL query token sequence, an original Text2SQL training dataset is constructed, and the target large language model is supervised and trained based on the original Text2SQL training dataset to generate the teacher model.

3. The method of claim 2, wherein, The method comprises the following steps: The natural language question token sequence and the SQL query token sequence are concatenated to establish the noisy sample. The method comprises the following steps: The target training sample is input into the teacher model and the initial student model respectively to generate a teacher model output probability and a student model output probability respectively; 4. The method of claim 3, wherein, Based on the teacher model output probability, the student model output probability, and a preset probability distribution distance measurement function, a distillation training loss function is determined. ​ ​ The initial student model is trained in a back propagation manner according to the distillation training loss function and a preset NSP training objective function, so as to generate the final target student model.

5. A large model knowledge distillation device for a Text2SQL task, characterized in that, Comprise: The supervision training module is used for supervising and training the target large language model based on the original Text2SQL training sample corresponding to the preset Text2SQL task, so as to generate the teacher model corresponding to the Text2SQL task; The noise adding module is used for performing random mask operation on the SQL statement in the original Text2SQL training sample, so as to obtain a mask sequence, inputting the mask sequence into an initial student model, predicting the mask token at the mask position in the mask sequence to obtain a mask prediction token, and replacing the mask token in the mask sequence with the corresponding mask prediction token to construct a noise-added sample corresponding to the original Text2SQL training sample; The distillation training module is used for determining a target training sample according to the noise-added sample, and supervising and distilling the initial student model based on the target training sample and the teacher model, so as to generate a final target student model.

6. The apparatus of claim 5, wherein, The supervision training module comprises: The word segmentation unit is used for segmenting the natural language question and the SQL statement in the original Text2SQL training sample to obtain a natural language question token sequence and a SQL query token sequence, respectively; The construction unit is used for constructing an original Text2SQL training data set based on the natural language question token sequence and the SQL query token sequence, and supervising and training the target large language model through the original Text2SQL training data set to generate the teacher model.

7. The apparatus of claim 6, wherein, The noise adding module comprises: The first splicing unit is used for splicing the natural language question token sequence and the mask sequence to obtain a spliced sequence, and predicting the mask token at the mask position in the mask sequence based on the initial student model and the spliced sequence to obtain the mask prediction token; The replacement unit is used for replacing the mask token in the mask sequence with the corresponding mask prediction token to obtain a rewritten SQL query statement token sequence corresponding to the SQL query token sequence; The second splicing unit is used for splicing the natural language question token sequence and the rewritten SQL query statement token sequence to establish the noise-added sample.

8. An electronic device, comprising: Comprise: The memory, the processor and the computer program stored on the memory and executable on the processor, the processor executes the program to realize the large model knowledge distillation method for the Text2SQL task as claimed in any one of claims 1-4.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor for realizing the large model knowledge distillation method for the Text2SQL task as claimed in any one of claims 1-4.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed for realizing the large model knowledge distillation method for the Text2SQL task as claimed in any one of claims 1-4.

Images