Large model knowledge distillation method and device oriented to Text2SQL (Structured Query Language) task

Abstract

The invention relates to a large model knowledge distillation method and device for a Text2SQL task, and the method comprises the steps: carrying out the supervision training of a target large language model based on an original Text2SQL training sample, so as to generate a teacher model corresponding to the Text2SQL task; random mask operation is carried out on SQL statements in the original Text2SQL training sample to obtain a mask sequence, the mask sequence is input into the initial student model to predict mask lexical elements at mask positions in the mask sequence to obtain mask prediction lexical elements, the mask lexical elements in the mask sequence are replaced with the corresponding mask prediction lexical elements, and the mask prediction lexical elements in the original Text2SQL training sample are obtained. A noise adding sample is constructed; and determining a target training sample according to the noise adding sample, and on the basis of the target training sample, combining with the teacher model to supervise the initial student model to perform distillation training to generate a final target student model. Therefore, the problems that in the prior art, model reasoning is delayed, the model deployment difficulty is large, the difference between model distillation training and reasoning is obvious, and the performance and efficiency of knowledge distillation are difficult to consider at the same time are solved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a large-model knowledge distillation method and device for Text2SQL tasks. Background Art

[0002] In today's information age, database query and information retrieval are becoming increasingly important. The Text2SQL task aims to convert natural language questions into SQL statement queries, which is one of the key technologies to achieve this goal. However, traditional Text2SQL models are often unable to handle complex query logic and multi-hop reasoning, especially when facing large-scale databases, and their performance and accuracy are difficult to meet the requirements of practical applications. In order to improve the processing capability of Text2SQL tasks, existing technologies can use large language models, which can better understand user query intent and generate more accurate SQL statements, thereby achieving better Text2SQL performance. However, large language models often have super-large-scale parameters, low model reasoning efficiency, and high deployment costs, which greatly restricts the development of Text2SQL systems based on large language models.

[0003] In addition, existing technologies can solve the problems of low reasoning efficiency and high deployment cost of Text2SQL systems based on large language models through various model compression methods such as knowledge distillation, quantization and model pruning. Among them, knowledge distillation has received widespread attention in the industry due to its ease of use and efficient model compression capabilities. More and more studies are trying to use knowledge distillation technology to achieve lightweight Text2SQL.

[0004] Specifically, knowledge distillation technology extracts knowledge from a large model (teacher model) and passes it to a small model (student model), aiming to minimize the complexity and resource consumption of the model without excessively reducing the performance of the model. In order to further improve the effect of distillation, cutting-edge knowledge distillation methods use samples generated by the student model currently being optimized for training (on-policy distillation), that is, it is necessary to generate corresponding training samples online in the form of autoregression while the student model is being trained. This on-policy distillation method uses the same target strategy in the training phase as in the inference phase to reduce the difference between training and inference, thereby achieving better distillation effects. However, this distillation method requires the student model to generate samples in real time by autoregression, and the training delay is relatively serious, resulting in low training efficiency.

[0005] In summary, the existing technologies have high model inference delays and high difficulty in model deployment. The difference between model distillation training and inference is obvious, and it is difficult to balance the performance and efficiency of knowledge distillation, which needs to be solved urgently. Summary of the invention

[0006] The present application provides a large-model knowledge distillation method and device for Text2SQL tasks to solve the problems of large model reasoning delay and model deployment difficulty in the prior art, obvious differences between model distillation training and reasoning, and difficulty in balancing the performance and efficiency of knowledge distillation.

[0007] The first aspect of the present application provides a large model knowledge distillation method for Text2SQL tasks, including the following steps: based on the original Text2SQL training samples corresponding to the preset Text2SQL tasks, supervised training is performed on the target large language model to generate a teacher model corresponding to the Text2SQL task; random masking operations are performed on the SQL statements in the original Text2SQL training samples to obtain a mask sequence, and the mask sequence is input into the initial student model to predict the masked word units at the masked positions in the mask sequence to obtain masked predicted word units, and the masked word units in the mask sequence are replaced with corresponding masked predicted word units to construct a noisy sample corresponding to the original Text2SQL training sample; a target training sample is determined based on the noisy sample, and based on the target training sample, the initial student model is supervised for distillation training in combination with the teacher model to generate a final target student model. Optionally, in one embodiment of the present application,

[0008] Optionally, in one embodiment of the present application, the original Text2SQL training samples corresponding to the preset Text2SQL task are used to perform supervised training on the target large language model to generate a teacher model corresponding to the Text2SQL task, including: performing word segmentation on the natural language questions and SQL statements in the original Text2SQL training samples to obtain a natural language question word unit sequence and a SQL query word unit sequence, respectively; constructing an original Text2SQL training data set based on the natural language question word unit sequence and the SQL query word unit sequence, and performing supervised training on the target large language model through the original Text2SQL training data set to generate the teacher model.

[0009] Optionally, in one embodiment of the present application, the step of inputting the mask sequence into the initial student model to predict the masked word-gram at the masked position in the masked sequence to obtain the masked predicted word-gram, and replacing the masked word-gram in the masked sequence with the corresponding masked predicted word-gram to construct the noisy sample corresponding to the original Text2SQL training sample, includes: concatenating the natural language question word-gram sequence and the masked sequence to obtain a concatenated sequence, and predicting the masked word-gram at the masked position in the masked sequence based on the initial student model and the concatenated sequence to obtain the masked predicted word-gram;

[0010] The masked words in the masked sequence are replaced with corresponding masked predicted words to obtain a rewritten SQL query statement word sequence corresponding to the SQL query word sequence; and the natural language question word sequence and the rewritten SQL query statement word sequence are concatenated to establish the noisy sample.

[0011] Optionally, in one embodiment of the present application, the target training sample is determined according to the noisy sample, and based on the target training sample, the initial student model is supervised by the teacher model to perform distillation training to generate a final target student model, including: inputting the target training sample into the teacher model and the initial student model, respectively, to generate the teacher model output probability and the student model output probability, respectively; determining the distillation training loss function based on the teacher model output probability, the student model output probability and a preset probability distribution distance metric function; and performing back propagation training on the initial student model according to the distillation training loss function and a preset NSP training objective function to generate the final target student model.

[0012] The second aspect of the present application provides a large model knowledge distillation device for Text2SQL tasks, including: a supervised training module, which is used to perform supervised training on a target large language model based on an original Text2SQL training sample corresponding to a preset Text2SQL task, so as to generate a teacher model corresponding to the Text2SQL task; a noise adding module, which is used to perform a random masking operation on the SQL statements in the original Text2SQL training sample to obtain a mask sequence, and input the mask sequence into an initial student model to predict the masked word units at the masked positions in the mask sequence to obtain masked predicted word units, and replace the masked word units in the mask sequence with corresponding masked predicted word units to construct a noisy sample corresponding to the original Text2SQL training sample; a distillation training module, which is used to determine a target training sample based on the noisy sample, and based on the target training sample, supervise the initial student model to perform distillation training in combination with the teacher model to generate a final target student model.

[0013] Optionally, in one embodiment of the present application, the supervised training module includes: a word segmentation unit, used to segment the natural language questions and SQL statements in the original Text2SQL training samples to obtain natural language question word unit sequences and SQL query word unit sequences, respectively; a construction unit, used to construct an original Text2SQL training data set based on the natural language question word unit sequence and the SQL query word unit sequence, and supervise the target large language model through the original Text2SQL training data set to generate the teacher model.

[0014] Optionally, in one embodiment of the present application, the noise addition module includes: a first splicing unit, used to splice the natural language question word unit sequence and the mask sequence to obtain a spliced ​​sequence, and based on the initial student model and the spliced ​​sequence, predict the mask word unit at the mask position in the mask sequence to obtain the masked predicted word unit; a replacement unit, used to replace the masked word unit in the mask sequence with the corresponding masked predicted word unit to obtain a rewritten SQL query statement word unit sequence corresponding to the SQL query word unit sequence; a second splicing unit, used to splice the natural language question word unit sequence and the rewritten SQL query statement word unit sequence to establish the noise addition sample.

[0015] Optionally, in one embodiment of the present application, the distillation training module includes: a generation unit, used to input the target training samples into the teacher model and the initial student model, respectively, to generate the teacher model output probability and the student model output probability, respectively; a determination unit, used to determine the distillation training loss function based on the teacher model output probability, the student model output probability and a preset probability distribution distance metric function; a back propagation training unit, used to perform back propagation training on the initial student model according to the distillation training loss function and a preset NSP training objective function to generate the final target student model.

[0016] The third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large model knowledge distillation method for Text2SQL tasks as described in the above embodiment.

[0017] The fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program that, when executed by a processor, implements the above-mentioned large model knowledge distillation method for Text2SQL tasks.

[0018] The fifth aspect of the present application provides a computer program product, including a computer program, which is executed to implement the above-mentioned large model knowledge distillation method for Text2SQL tasks.

[0019] Therefore, the embodiments of the present application have the following beneficial effects:

[0020] The embodiment of the present application can supervise the training of the target large language model based on the original Text2SQL training samples corresponding to the preset Text2SQL task to generate a teacher model corresponding to the Text2SQL task; perform random masking operations on the SQL statements in the original Text2SQL training samples to obtain a mask sequence, and input the mask sequence into the initial student model to predict the masked word units at the masked positions in the mask sequence to obtain masked predicted word units, and replace the masked word units in the mask sequence with corresponding masked predicted word units to construct the noised samples corresponding to the original Text2SQL training samples; determine the target training samples according to the noised samples, and supervise the initial student model for distillation training based on the target training samples in combination with the teacher model to generate the final target student model. The present application actively adds noise to the training samples in the training stage to simulate the error output in the autoregressive generation in the model inference stage to alleviate the difference between model training and inference, and the generation of noise samples only requires one forward propagation, so that it can well balance training performance and efficiency. This solves the problems of existing technologies such as model inference delay and model deployment difficulty, obvious differences between model distillation training and reasoning, and difficulty in balancing the performance and efficiency of knowledge distillation.

[0021] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A flowchart of a large model knowledge distillation method for Text2SQL tasks provided according to an embodiment of the present application;

[0024] Figure 2 A schematic diagram of the logical architecture of a large model knowledge distillation method for Text2SQL tasks provided according to an embodiment of the present application;

[0025] Figure 3 A schematic diagram showing a comparison of the effects of a student model on a Text2SQL task provided according to an embodiment of the present application;

[0026] Figure 4 This is an example diagram of a large model knowledge distillation device for Text2SQL tasks according to an embodiment of the present application;

[0027] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0028] Among them, 10-large model knowledge distillation device for Text2SQL tasks; 100-supervised training module, 200-noise module, 300-distillation training module; 501-memory, 502-processor, 503-communication interface. DETAILED DESCRIPTION

[0029] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0030] The following describes a large model knowledge distillation method and device for Text2SQL tasks according to an embodiment of the present application with reference to the accompanying drawings. In view of the problems mentioned in the above background technology, the present application provides a large model knowledge distillation method for Text2SQL tasks, in which a target large language model is supervised and trained based on the original Text2SQL training samples corresponding to the preset Text2SQL task to generate a teacher model corresponding to the Text2SQL task; a random masking operation is performed on the SQL statements in the original Text2SQL training samples to obtain a mask sequence, and the mask sequence is input into the initial student model to predict the masked word units at the mask positions in the mask sequence to obtain masked predicted word units, and the masked word units in the mask sequence are replaced with corresponding masked predicted word units to construct a noised sample corresponding to the original Text2SQL training sample; a target training sample is determined according to the noised sample, and based on the target training sample, the initial student model is supervised for distillation training in combination with the teacher model to generate the final target student model. This application actively adds noise to the training samples during the training phase to simulate the erroneous output in the autoregressive generation during the model inference phase, so as to alleviate the difference between model training and inference. The generation of noise samples only requires one forward propagation, so that the training performance and efficiency can be well balanced. This solves the problems of the prior art, such as the large delay in model inference and the difficulty in model deployment, the obvious difference between model distillation training and inference, and the difficulty in balancing the performance and efficiency of knowledge distillation.

[0031] Specifically, Figure 1 A flowchart of a large model knowledge distillation method for Text2SQL tasks provided in an embodiment of the present application.

[0032] like Figure 1 As shown in FIG. 1 , the large model knowledge distillation method for the Text2SQL task includes the following steps:

[0033] In step S101, supervised training is performed on the target large language model based on the original Text2SQL training samples corresponding to the preset Text2SQL task to generate a teacher model corresponding to the Text2SQL task.

[0034] The embodiment of the present application can first perform supervised training on a large language model with a larger parameter scale on the original Text2SQL training samples to obtain a teacher model, which can demonstrate excellent performance on the Text2SQL task.

[0035] Optionally, in one embodiment of the present application, based on the original Text2SQL training samples corresponding to the preset Text2SQL task, the target large language model is supervised trained to generate a teacher model corresponding to the Text2SQL task, including: segmenting the natural language questions and SQL statements in the original Text2SQL training samples to obtain natural language question word sequences and SQL query word sequences, respectively; based on the natural language question word sequences and SQL query word sequences, constructing an original Text2SQL training data set, and supervising the target large language model through the original Text2SQL training data set to generate a teacher model corresponding to the Text2SQL task.

[0036] It should be noted that the embodiments of the present application can use the original Text2SQL sample as a training set, use a word segmenter to segment the natural language questions and SQL statements in Text2SQL, and obtain the natural language question word sequence x = {x1, x2, ..., x m} and SQL query word sequence y = {y1,y2,...,y n}, where m and n represent the length of the question word sequence and the SQL statement word sequence respectively.

[0037] Considering that the existing language model is generally an autoregressive model architecture, the embodiment of the present application can input the word sequence (x, y) into the model for forward propagation, and M p The training goal is to predict the next word (Next Token Prediction, NSP), so the embodiment of the present application calculates the prediction loss corresponding to the position of the SQL statement word sequence. The specific loss expression is shown in Formula 1:

[0038]

[0039] Among them, p(y t |x,y ＜t ) is the predicted probability distribution of the teacher model for the t-th SQL statement word; Training loss function for NSP.

[0040] After that, the embodiment of the present application can use the Text2SQL training sample (x, y) to perform supervised training on a large language model with a larger parameter scale to obtain a trained teacher model M p ;

[0041] Therefore, the embodiments of the present application can update the parameters of the teacher model through gradient back propagation, thereby obtaining a teacher model with powerful performance.

[0042] In step S102, a random masking operation is performed on the SQL statements in the original Text2SQL training sample to obtain a masked sequence, and the masked sequence is input into the initial student model to predict the masked word units at the masked positions in the masked sequence to obtain masked predicted word units, and the masked word units in the masked sequence are replaced with corresponding masked predicted word units to construct a noisy sample corresponding to the original Text2SQL training sample.

[0043] Furthermore, the embodiment of the present application also needs to randomly mask the SQL statements in the original Text2SQL data, such as Figure 2 As shown, the noisy sample is input into a student model with a smaller parameter scale to predict the word unit corresponding to the mask position, and the mask at the corresponding position is replaced with the predicted word unit to construct the noisy sample.

[0044] Optionally, in one embodiment of the present application, the mask sequence is input into the initial student model to predict the masked words at the masked positions in the masked sequence to obtain masked predicted words, and the masked words in the masked sequence are replaced with corresponding masked predicted words to construct a noisy sample corresponding to the original Text2SQL training sample, including: concatenating the natural language question word sequence and the masked sequence to obtain a concatenated sequence, and based on the initial student model and the concatenated sequence, predicting the masked words at the masked positions in the masked sequence to obtain masked predicted words; replacing the masked words in the masked sequence with corresponding masked predicted words to obtain a rewritten SQL query statement word sequence corresponding to the SQL query word sequence; and concatenating the natural language question word sequence and the rewritten SQL query statement word sequence to establish a noisy sample.

[0045] It should be noted that the specific steps of performing noise processing on the input samples (i.e., the original Text2SQL training samples) in each round of training batches in the embodiment of the present application are as follows:

[0046] Step 1: Randomly mask the SQL statement word sequence y in the original Text2SQL (i.e., the data (x, y) of the current training batch), that is, use “|mask|” to replace some words in y, and obtain the mask sequence y′={y1,|mask|,...,y i ,|mask|,yi+2 ,...,y n};

[0047] Step 2: Concatenate the original natural language question word sequence x and the mask sequence y′ in step 1 to obtain (x, y′), and then input it into the student model M with a smaller parameter scale obtained in the previous round of training. q In the first round of training (if it is the first round of training, the untrained student model is used as initialization), the word unit corresponding to the mask position is predicted to obtain the mask position predicted word unit The subscripts indicate the positions of the masked tokens in the sequence;

[0048] It should be noted that the mask prediction in this step only requires (x, y′) to be input into the student model once, without the need for autoregressive word prediction, which can significantly reduce the additional training delay.

[0049] Step 3: Replace the masked tokens in the masked sequence y′ with the corresponding predicted tokens at the masked positions in step 2 to obtain the rewritten SQL query token sequence. The original natural language question word sequence x and Splicing to get the noisy sample

[0050] Therefore, the embodiments of the present application can simulate the problem of early-position word-unit errors affecting subsequent word-unit predictions in the model's autoregressive reasoning process through the above-mentioned mask prediction method, that is, by efficiently adding noise to the training samples, simulating the errors in the model reasoning stage, and the generation of the noise sample only requires one forward propagation to effectively alleviate the difference between the training stage and the reasoning stage, thereby improving and taking into account the performance and efficiency of subsequent distillation training.

[0051] In step S103, a target training sample is determined according to the noisy sample, and based on the target training sample, the initial student model is supervised by the teacher model to perform distillation training to generate a final target student model.

[0052] Furthermore, the embodiments of the present application may use the noisy samples as training samples, and utilize the teacher model as a teacher to supervise the distillation training of the student model, and finally obtain a student model with good Text2SQL performance.

[0053] It can be understood that the knowledge distillation framework of the embodiment of the present application has the advantages of simple implementation and wide applicability. The framework is applicable to any Text2SQL large language model and can significantly improve the distillation training effect without introducing additional models and excessive training overhead.

[0054] Optionally, in one embodiment of the present application, a target training sample is determined based on the noisy sample, and based on the target training sample, distillation training is performed on the initial student model in combination with the teacher model to supervise the initial student model to generate a final target student model, including: inputting the target training sample into the teacher model and the initial student model respectively to generate the teacher model output probability and the student model output probability respectively; determining the distillation training loss function based on the teacher model output probability, the student model output probability and a preset probability distribution distance metric function; and performing back-propagation training on the initial student model according to the distillation training loss function and the preset NSP training objective function to generate the final target student model.

[0055] In the actual implementation process, the embodiment of the present application can add noise samples as target training samples and input them into the teacher model M p and the current round student model M with smaller parameters q (i.e. the initial student model), the output probability of the teacher model at the SQL statement position is obtained And the student model output probability

[0056] Afterwards, the embodiment of the present application can use any probability distribution distance metric function (such as reverse KL divergence, etc.) to calculate the difference between the output probability of the student model and the output probability of the teacher model, and use this as the distillation training loss function, and perform back propagation training on the student model in conjunction with the NSP training target, as shown in equations (2) and (3):

[0057]

[0058] in, Represents the probability distribution distance, which can be the classic forward KL divergence or reverse KL divergence; Training loss function for distillation.

[0059] It should be noted that in the distillation training of the present application embodiment, only The student model parameters are updated, but the teacher model parameters are not updated. In addition, the above-mentioned noise sample construction and distillation training operations need to be performed iteratively, that is, at the beginning of each training step, the training samples are denoised to obtain noisy samples, and then the student model is distilled and trained. The trained student model participates in the sample denoising process of the next training step until all training steps are completed, so as to obtain the final student model with good Text2SQL performance (i.e., the final target student model).

[0060] Therefore, the embodiments of the present application can learn Text2SQL related knowledge from the teacher model and can achieve good performance on the Text2SQL public dataset. Considering that the parameter scale of the student model is much smaller than that of the teacher model, it can greatly reduce the model reasoning delay and reduce the difficulty of model deployment.

[0061] Figure 3 The following is a schematic diagram showing the comparison of the student model’s performance on the Text2SQL task. Figure 3 As shown, the horizontal axis represents the distillation training delay, and the vertical axis represents the average execution accuracy of the student model in the Text2SQL public data set; the traditional distillation method represents the conventional use of only the original Text2SQL training data for distillation training, and the on-policy distillation method represents the use of the inference data generated online by the student model as the distillation training data; the embodiment of the present application effectively alleviates the difference between model training and inference by simulating the erroneous output during inference in the training phase, and induces the student model to remove noise samples to help it better learn the Text2SQL knowledge in the teacher model, and finally efficiently improve the Text2SQL performance of the student model.

[0062] Therefore, the distillation training framework of the embodiment of the present application can effectively improve the performance and training efficiency of the model. Without increasing too much training delay, the trained student model can achieve significantly better results than other knowledge distillation methods on multiple Text2SQL public datasets.

[0063] According to the large model knowledge distillation method for Text2SQL tasks proposed in the embodiment of the present application, supervised training is performed on the target large language model based on the original Text2SQL training samples corresponding to the preset Text2SQL task to generate a teacher model corresponding to the Text2SQL task; random masking operations are performed on the SQL statements in the original Text2SQL training samples to obtain a mask sequence, and the mask sequence is input into the initial student model to predict the masked word units at the masked positions in the mask sequence to obtain masked predicted word units, and the masked word units in the mask sequence are replaced with corresponding masked predicted word units to construct the noised samples corresponding to the original Text2SQL training samples; the target training samples are determined according to the noised samples, and the initial student model is supervised for distillation training based on the target training samples in combination with the teacher model to generate the final target student model. The present application actively adds noise to the training samples in the training stage to simulate the error output in the autoregressive generation in the model inference stage to alleviate the difference between model training and inference, and the generation of noise samples only requires one forward propagation, so that training performance and efficiency can be well balanced.

[0064] Secondly, the large model knowledge distillation device for Text2SQL tasks proposed according to the embodiment of the present application is described with reference to the accompanying drawings.

[0065] Figure 4 It is a block diagram of a large model knowledge distillation device for Text2SQL tasks in an embodiment of the present application.

[0066] like Figure 4 As shown, the large model knowledge distillation device 10 for the Text2SQL task includes: a supervised training module 100, a noise addition module 200 and a distillation training module 300.

[0067] Among them, the supervised training module 100 is used to perform supervised training on the target large language model based on the original Text2SQL training samples corresponding to the preset Text2SQL task to generate a teacher model corresponding to the Text2SQL task.

[0068] The noise adding module 200 is used to perform random masking operation on the SQL statements in the original Text2SQL training sample to obtain a masked sequence, and input the masked sequence into the initial student model to predict the masked word units at the masked positions in the masked sequence to obtain masked predicted word units, and replace the masked word units in the masked sequence with corresponding masked predicted word units to construct a noise adding sample corresponding to the original Text2SQL training sample.

[0069] The distillation training module 300 is used to determine the target training sample according to the noisy sample, and perform distillation training on the initial student model in combination with the teacher model to supervise the target training sample, so as to generate the final target student model.

[0070] Optionally, in one embodiment of the present application, the supervised training module 100 includes: a word segmentation unit and a construction unit.

[0071] The word segmentation unit is used to segment the natural language questions and SQL statements in the original Text2SQL training samples to obtain natural language question word sequences and SQL query word sequences respectively.

[0072] The construction unit is used to construct an original Text2SQL training data set based on the natural language question word sequence and the SQL query word sequence, and supervise the target large language model through the original Text2SQL training data set to generate a teacher model corresponding to the Text2SQL task.

[0073] Optionally, in one embodiment of the present application, the noise adding module 200 includes: a first splicing unit, a replacement unit and a second splicing unit.

[0074] Among them, the first splicing unit is used to splice the natural language question word unit sequence and the mask sequence to obtain a spliced ​​sequence, and based on the initial student model and the spliced ​​sequence, predict the masked word unit at the mask position in the mask sequence to obtain the masked predicted word unit.

[0075] The replacement unit is used to replace the masked tokens in the masked sequence with corresponding masked predicted tokens to obtain a rewritten SQL query statement token sequence corresponding to the SQL query token sequence.

[0076] The second concatenation unit is used to concatenate the natural language question word unit sequence and the rewritten SQL query statement word unit sequence to establish a noise-added sample.

[0077] Optionally, in one embodiment of the present application, the distillation training module 300 includes: a generation unit, a determination unit and a back-propagation training unit.

[0078] Among them, the generation unit is used to input the target training samples into the teacher model and the initial student model respectively to generate the teacher model output probability and the student model output probability respectively.

[0079] The determination unit is used to determine the distillation training loss function based on the teacher model output probability, the student model output probability and a preset probability distribution distance metric function.

[0080] The back-propagation training unit is used to perform back-propagation training on the initial student model according to the distillation training loss function and the preset NSP training objective function to generate the final target student model.

[0081] It should be noted that the aforementioned explanation of the embodiment of the large model knowledge distillation method for Text2SQL tasks is also applicable to the large model knowledge distillation device for Text2SQL tasks in this embodiment, and will not be repeated here.

[0082] According to the large model knowledge distillation device for Text2SQL tasks proposed in the embodiment of the present application, it includes a supervised training module, which is used to supervise the target large language model based on the original Text2SQL training samples corresponding to the preset Text2SQL task to generate a teacher model corresponding to the Text2SQL task; a noise adding module, which is used to perform random masking operations on the SQL statements in the original Text2SQL training samples to obtain a mask sequence, and input the mask sequence into the initial student model to predict the masked word units at the masked positions in the mask sequence to obtain masked predicted word units, and replace the masked word units in the mask sequence with corresponding masked predicted word units to construct the noised samples corresponding to the original Text2SQL training samples; a distillation training module, which is used to determine the target training samples according to the noised samples, and based on the target training samples, the initial student model is supervised for distillation training in combination with the teacher model to generate the final target student model. The present application actively adds noise to the training samples in the training stage to simulate the error output in the autoregressive generation in the model reasoning stage to alleviate the difference between model training and reasoning, and the generation of noise samples only requires one forward propagation, so that training performance and efficiency can be well balanced.

[0083] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0084] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0085] When the processor 502 executes the program, the large model knowledge distillation method for the Text2SQL task provided in the above embodiment is implemented.

[0086] Furthermore, the electronic device further comprises:

[0087] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0088] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0089] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0090] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0091] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0092] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0093] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the large model knowledge distillation method for the Text2SQL task as described above is implemented.

[0094] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned large model knowledge distillation method for Text2SQL tasks.

[0095] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0096] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0097] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0098] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0099] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0100] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0101] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0102] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A large model knowledge distillation method for Text2SQL tasks, characterized by: The following steps are involved: Based on the original Text2SQL training samples corresponding to the preset Text2SQL task, supervised training is performed on the target large language model to generate a teacher model corresponding to the Text2SQL task; Performing random masking operations on the SQL statements in the original Text2SQL training sample to obtain a masked sequence, and inputting the masked sequence into the initial student model to predict the masked word units at the masked positions in the masked sequence to obtain masked predicted word units, and replacing the masked word units in the masked sequence with corresponding masked predicted word units to construct a noisy sample corresponding to the original Text2SQL training sample; A target training sample is determined according to the noisy sample, and based on the target training sample, the initial student model is supervised to perform distillation training in combination with the teacher model to generate a final target student model.

2. The method according to claim 1, characterized in that The method of performing supervised training on the target large language model based on the original Text2SQL training sample corresponding to the preset Text2SQL task to generate a teacher model corresponding to the Text2SQL task includes: Segmenting the natural language questions and SQL statements in the original Text2SQL training sample to obtain natural language question word unit sequences and SQL query word unit sequences respectively; Based on the natural language question word-unit sequence and the SQL query word-unit sequence, an original Text2SQL training data set is constructed, and supervised training is performed on the target large language model through the original Text2SQL training data set to generate the teacher model.

3. The method according to claim 2, characterized in that The step of inputting the mask sequence into the initial student model to predict the masked word-grams at the masked positions in the mask sequence to obtain masked predicted word-grams, and replacing the masked word-grams in the mask sequence with corresponding masked predicted word-grams to construct a noisy sample corresponding to the original Text2SQL training sample, includes: splicing the natural language question word-unit sequence and the mask sequence to obtain a spliced ​​sequence, and predicting the masked word-unit at the masked position in the masked sequence based on the initial student model and the spliced ​​sequence to obtain the masked predicted word-unit; Replacing the masked tokens in the masked sequence with corresponding masked predicted tokens to obtain a rewritten SQL query statement token sequence corresponding to the SQL query token sequence; The natural language question word-gram sequence and the rewritten SQL query statement word-gram sequence are concatenated to create the noisy sample.

4. The method according to claim 3, characterized in that The step of determining a target training sample according to the noise-added sample, and performing distillation training on the initial student model in combination with the teacher model to supervise the target training sample to generate a final target student model includes: Inputting the target training sample into the teacher model and the initial student model respectively to generate the teacher model output probability and the student model output probability respectively; Determine a distillation training loss function based on the teacher model output probability, the student model output probability and a preset probability distribution distance metric function; The initial student model is back-propagated and trained according to the distillation training loss function and the preset NSP training objective function to generate the final target student model.

5. A large model knowledge distillation device for Text2SQL tasks, characterized by: include: A supervised training module, used to perform supervised training on a target large language model based on an original Text2SQL training sample corresponding to a preset Text2SQL task, so as to generate a teacher model corresponding to the Text2SQL task; A noise adding module, used for performing a random masking operation on the SQL statements in the original Text2SQL training sample to obtain a masked sequence, and inputting the masked sequence into the initial student model to predict the masked word units at the masked positions in the masked sequence to obtain masked predicted word units, and replacing the masked word units in the masked sequence with corresponding masked predicted word units to construct a noise adding sample corresponding to the original Text2SQL training sample; A distillation training module is used to determine a target training sample based on the noisy sample, and to perform distillation training on the initial student model in combination with the teacher model based on the target training sample to generate a final target student model.

6. The device according to claim 5, characterized in that The supervised training module includes: A word segmentation unit is used to segment the natural language questions and SQL statements in the original Text2SQL training sample to obtain a natural language question word unit sequence and an SQL query word unit sequence respectively; A construction unit is used to construct an original Text2SQL training data set based on the natural language question word sequence and the SQL query word sequence, and to perform supervised training on the target large language model through the original Text2SQL training data set to generate the teacher model.

7. The device according to claim 6, characterized in that The noise adding module comprises: A first concatenation unit is used to concatenate the natural language question word-unit sequence and the mask sequence to obtain a concatenated sequence, and predict the mask word-unit at the mask position in the mask sequence based on the initial student model and the concatenated sequence to obtain the masked predicted word-unit; A replacement unit, configured to replace the masked tokens in the masked sequence with corresponding masked predicted tokens to obtain a rewritten SQL query statement token sequence corresponding to the SQL query token sequence; The second concatenation unit is used to concatenate the natural language question word-gram sequence and the rewritten SQL query statement word-gram sequence to establish the noisy sample.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the large model knowledge distillation method for Text2SQL tasks as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the large model knowledge distillation method for Text2SQL tasks as described in any one of claims 1-4.

10. A computer program product, comprising a computer program, characterized in that The computer program is executed to implement the large model knowledge distillation method for Text2SQL tasks as described in any one of claims 1-4.

Images