Large language model knowledge distillation method, device, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, the technical scheme can be applied to the field of financial science and technology / medical health, and discloses a big language model knowledge distillation method, device, equipment and medium, the method comprises the following steps: obtaining business data, the business data comprising an input text sequence, an expected output text sequence and a corresponding attention mask sequence; inputting the input text sequence and the attention mask sequence into a student model and a teacher model, and respectively obtaining first prediction probability distribution output by the student model and second prediction probability distribution output by the teacher model; calculating a reverse KL divergence loss value based on the first prediction probability distribution and the second prediction probability distribution; and optimizing parameters of the student model by using the reverse KL divergence loss value. According to the method, the student model is guided to accurately fit the core probability distribution of the teacher model through the reverse KL divergence loss function, so that the generation accuracy of the lightweight model is obviously improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, fintech, and healthcare, and particularly to a method, apparatus, device, and medium for knowledge distillation of large language models. Background Technology

[0002] With the acceleration of digital transformation, large language models are being used more and more widely in the fields of fintech and healthcare. Therefore, it is crucial to leverage knowledge distillation to transfer knowledge from complex teacher models in fintech and healthcare to lightweight student models, ensuring that the student models maintain good performance.

[0003] Existing knowledge distillation (KD) methods (such as sequence-level KD and word-level KD) typically use forward KL divergence (KL[p||q]) as the optimization objective, forcing the student model to cover all distribution patterns of the teacher model. However, when the output space of the teacher model is complex (such as in open-domain text generation tasks), the student model, due to its limited capacity, cannot accurately fit all patterns, resulting in low-quality text generation (such as overestimating low-probability regions of the teacher distribution). Furthermore, traditional methods suffer from exposure bias, i.e., using the true context (TeacherForcing) during training but relying on the context generated by the model itself during inference, leading to inconsistency between training and inference and a decline in generation performance. Existing black-box distillation (such as mimicking ChatGPTAPI) can only utilize the output text of the teacher model and cannot utilize its internal probability distribution or hidden states, limiting the performance ceiling of the student model. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and medium for knowledge distillation of large language models, aiming to solve the problem of poor reasoning performance after distillation of large language models.

[0005] Firstly, a knowledge distillation method for large language models is provided, including the following steps:

[0006] S10. Obtain business data, which originates from a business analysis task. The business data includes the input text sequence, the expected output text sequence, and the corresponding attention mask sequence of the business analysis task.

[0007] S20. Input the input text sequence of the business analysis task and the attention mask sequence into the student model and the teacher model respectively to obtain the first prediction probability distribution output by the student model and the second prediction probability distribution output by the teacher model, wherein the second prediction probability distribution is generated based on the supervision signal of the expected output text sequence;

[0008] S30. Based on the first predicted probability distribution and the second predicted probability distribution, calculate the reverse KL divergence loss value, wherein the reverse KL divergence loss value is used to measure the difference between the second predicted probability distribution and the first predicted probability distribution.

[0009] S40. Optimize the parameters of the student model using the inverse KL divergence loss value to generate an optimized student model for business analysis. In the optimization process, the student model generates contextual information for subsequent predictions based on the business analysis prediction results of its previous steps.

[0010] Secondly, a knowledge distillation device for a large language model is provided, comprising:

[0011] The data acquisition module is used to acquire business data, which originates from a business analysis task. The business data includes the input text sequence, the expected output text sequence, and the corresponding attention mask sequence of the business analysis task.

[0012] The model prediction module is used to input the input text sequence of the business analysis task and the attention mask sequence into the student model and the teacher model, respectively, to obtain the first prediction probability distribution output by the student model and the second prediction probability distribution output by the teacher model, wherein the second prediction probability distribution is generated based on the supervision signal of the expected output text sequence;

[0013] The loss calculation module is used to calculate the reverse KL divergence loss value based on the first predicted probability distribution and the second predicted probability distribution, wherein the reverse KL divergence loss value is used to measure the difference between the second predicted probability distribution and the first predicted probability distribution.

[0014] The model optimization module is used to optimize the parameters of the student model using the inverse KL divergence loss value, and generate an optimized student model for business analysis. During the optimization process, the student model generates contextual information for subsequent predictions based on the business analysis prediction results of its previous steps.

[0015] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described large language model knowledge distillation method.

[0016] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described large language model knowledge distillation method.

[0017] In the above-mentioned solution implemented by the knowledge distillation method, apparatus, device, and medium for large language models, business data is acquired. This business data originates from a business analysis task and includes an input text sequence, a desired output text sequence, and a corresponding attention mask sequence. The input text sequence and the attention mask sequence are input to a student model and a teacher model, respectively, to obtain a first predicted probability distribution output by the student model and a second predicted probability distribution output by the teacher model. The second predicted probability distribution is generated based on the supervision signal of the desired output text sequence. Based on the first and second predicted probability distributions, a reverse KL divergence loss value is calculated, whereby the reverse KL divergence loss value measures the difference between the second predicted probability distribution and the first predicted probability distribution. The parameters of the student model are optimized using the reverse KL divergence loss value to generate an optimized student model for business analysis. During the optimization process, the contextual information used by the student model to generate subsequent prediction content is generated based on the business analysis prediction results of its previous steps. By using the inverse KL divergence loss function to guide the student model to accurately fit the core probability distribution of the teacher model, overfitting to low confidence regions is avoided, resulting in a significant improvement in the generation accuracy of the lightweight model. At the same time, an autoregressive training mechanism is used to eliminate exposure bias and significantly improve the coherence of long texts. After the model parameter size is compressed, the student model can still maintain performance close to that of the teacher model. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an application environment for the knowledge distillation method of a large language model in one embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of the knowledge distillation method for large language models in one embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of a business data preprocessing process in one embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of a process for calculating the inverse KL divergence loss value in one embodiment of the present invention;

[0023] Figure 5 This is a flowchart illustrating the optimization of student model parameters in one embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of a large language model knowledge distillation device in one embodiment of the present invention;

[0025] Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0026] Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] The knowledge distillation method for large language models provided in this invention is applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The client sets distillation parameters and sends them to the server, then views the distillation results returned by the server. The server acquires business data, which originates from a business analysis task. This business data includes the input text sequence, the expected output text sequence, and the corresponding attention mask sequence of the business analysis task. The input text sequence and the attention mask sequence of the business analysis task are input into a student model and a teacher model, respectively, to obtain a first prediction probability distribution output by the student model and a second prediction probability distribution output by the teacher model. The second prediction probability distribution is generated based on the supervision signal of the expected output text sequence. Based on the first and second prediction probability distributions, a reverse KL divergence loss value is calculated, which measures the difference between the second prediction probability distribution and the first prediction probability distribution. The parameters of the student model are optimized using the reverse KL divergence loss value to generate an optimized student model for business analysis. During the optimization process, the context information used by the student model to generate subsequent prediction content is generated based on the business analysis prediction results of its previous steps.

[0029] The solution provided in this invention guides the student model to accurately fit the core probability distribution of the teacher model through the inverse KL divergence loss function, avoiding overfitting to low-confidence regions and significantly improving the generation accuracy of the lightweight model. Simultaneously, an autoregressive training mechanism is employed to eliminate exposure bias, significantly improving the coherence of long texts. Even after parameter compression, the student model maintains performance close to that of the teacher model. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0030] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the knowledge distillation method for large language models provided in this embodiment of the invention includes the following steps:

[0031] S10. Obtain business data, which originates from a business analysis task. The business data includes the input text sequence, the expected output text sequence, and the corresponding attention mask sequence of the business analysis task.

[0032] In this embodiment, the business data includes input text A, output text B, and attention mask M for the business analysis task. The model input consists of input text A and attention mask M, and the output text is the text that B is expected to output. The model output is the predicted text C. For financial business, the acquired business data is financial business data, and a student model for financial business analysis is distilled out; for healthcare business, the acquired data is healthcare data, and a student model for healthcare business analysis is distilled out.

[0033] like Figure 3 As shown, in one specific embodiment, before step S20, the method further includes: preprocessing the input text sequence and the expected output text sequence of the business analysis task, wherein the preprocessing includes:

[0034] S111. Use a word segmenter to convert the input text sequence and the expected output text sequence into corresponding word identifier sequences;

[0035] S112. Perform a padding operation on the word identifier sequence to make the sequence length uniform to the target length, and use padding markers to indicate the padding positions.

[0036] S113. Generate the attention mask sequence based on the padded sequence. The attention mask sequence is used to distinguish between valid word positions and padded positions.

[0037] In this embodiment, before inputting the data into the model, the text first needs to be converted into a TokenID sequence acceptable to the model. Inputting the input text A into the model's built-in tokenizer will yield the Token ID sequence. Then, the Padding function is used to fill the Token ID sequence, unifying the length of each segment to the longest sequence length in the model, and using <\PAD> for padding. For example, if a Token ID is [1211,1130,988] and the longest sequence length is 5, then the padded sequence will be [1211,1130,988,<\PAD>,<\PAD>].

[0038] S20. Input the input text sequence of the business analysis task and the attention mask sequence into the student model and the teacher model respectively to obtain the first prediction probability distribution output by the student model and the second prediction probability distribution output by the teacher model, wherein the second prediction probability distribution is generated based on the supervision signal of the expected output text sequence.

[0039] In one specific embodiment, the teacher model is a neural network model with a parameter size greater than 3B that has text generation capabilities; the student model is a neural network model with a parameter size less than or equal to 3B that has text generation capabilities.

[0040] In this embodiment, large models with strong generative capabilities are selected as teacher models, such as GPT-2, GPT-J, and LLaMA. Models with compact structures and small parameter counts are selected as student models, such as GPT-2 Small and OPT-350M.

[0041] In one specific embodiment, in step S20, the first predicted probability distribution and the second predicted probability distribution are obtained in the following way:

[0042] The original prediction scores output by the student model and the teacher model are normalized to obtain the probability distribution of each word position on the entire vocabulary.

[0043] In this embodiment, specifically, the teacher model outputs a logits matrix with a shape of [512, 32000] (LLaMA vocabulary), and calculates the vocabulary probability distribution at each position using torch.softmax(dim = -1).

[0044] This invention ensures compatibility of heterogeneous data through standardized preprocessing, and soft normalization preserves the details of probability distribution (e.g., the probability of "migraine" is 0.7 vs. "tension headache" is 0.3).

[0045] S30. Based on the first predicted probability distribution and the second predicted probability distribution, calculate the reverse KL divergence loss value, wherein the reverse KL divergence loss value is used to measure the difference between the second predicted probability distribution and the first predicted probability distribution.

[0046] like Figure 4 As shown, in a specific embodiment, step S30, calculating the reverse KL divergence loss value specifically includes:

[0047] S31. Take the first predicted probability distribution as the target distribution;

[0048] S32. Use the second predicted probability distribution as a reference distribution;

[0049] S33. Calculate the inverse KL divergence value of the target distribution relative to the reference distribution as the loss value.

[0050] In this embodiment, the loss calculation is specifically implemented in three stages:

[0051] Target distribution setting: Log probability distribution of the student model (dimension [batch, 1024, 50257]);

[0052] Reference distribution setting: Normalized probability distribution of the teacher model;

[0053] Divergence calculation: Batch-averaged KL divergence is used.

[0054] Inverse KLD Loss Calculation: This stage is the loss function calculation part of the model training process. The padded Token ID and attention mask are input into the model, and the model outputs a logits matrix of size [Sequence, vocab_size], where Sequence is the longest sequence length and vocab_size is the model vocabulary length. The vocab_size represents the score of the word predicted by the model at each position. For example, in a logits matrix of size [2,3], a0 = logits[0][0], a1 = logits[0][1], a2 = logits[0][2], the value of a0 represents the score of the model's assumption that the ID of the word output at position 0 is 0, and the value of a1 represents the score of the model's assumption that the ID of the word output at position 0 is 1. The logits of the teacher model and the student model are transformed into a probability distribution using the log_softmax function (code: F.log_softmax(student_logits, dim = -1)), and then the inverse KLD is calculated to obtain the loss function.

[0055] loss = F.kl_div(

[0056] F.log_softmax(student_logits,dim=-1),

[0057] F.softmax(teacher_logits,dim=-1),

[0058] reduction = 'batchmean' ))

[0060] The `kl_div` function is used to calculate the KL loss, and the `log_softmax` function is used to calculate the softmax first and then the log loss. The `softmax` function is used to calculate the softmax. `student_logits` and `teacher_logits` are the outputs of the student model and the teacher model, respectively, and can be obtained directly through the model `student["logits"]`.

[0061] S40. Optimize the parameters of the student model using the inverse KL divergence loss value to generate an optimized student model for business analysis. In the optimization process, the student model generates contextual information for subsequent predictions based on the business analysis prediction results of its previous steps.

[0062] like Figure 5 As shown, in a specific embodiment, step S40, which involves optimizing the parameters of the student model using the inverse KL divergence loss value to generate an optimized student model for business analysis, specifically includes:

[0063] S41. Calculate the gradient of the backpropagation loss value with respect to the student model parameters using the backpropagation function of the deep learning framework;

[0064] S42. Using the gradient descent algorithm, update the parameters of the student model based on the gradient to generate an optimized student model for business analysis.

[0065] In this embodiment, gradient updates use the AdamW optimizer (learning rate = 3e-5, weight decay = 0.01). After calculating the inverse KLD loss for each batch of data, loss.backward() is called to automatically calculate the derivative. Gradient clipping (threshold = 1.0) is used to prevent gradient explosion.

[0066] In one specific embodiment, in step S40, during the optimization process, the context information used by the student model to generate subsequent prediction content is generated based on the business analysis prediction results of its previous steps, specifically including:

[0067] At each time step of the training iteration, for the student model, the lexical units it generated before the current time step are used as contextual input to predict the lexical units for the next time step.

[0068] In this embodiment, replacing the traditional forward KLD with inverse KLD as the optimization objective can more effectively avoid overestimation of low-probability regions. This improvement is more suitable for generation tasks, as it better matches the distribution of the student model with the generation strategy of the teacher model, thereby improving generation quality and accuracy. By allowing the student model to see more of its own output during the generation process, rather than relying entirely on the real labels, the exposure bias caused by the inconsistency between training and inference distributions is significantly reduced. This strategy significantly reduces the cumulative error in long-term text generation, improving the quality and coherence of long text generation. This method is not only applicable to small models, but can also effectively compress different model families with parameters ranging from 120M to 13B, and shows consistent performance improvement in large-scale model compression, demonstrating its broad applicability in large model compression.

[0069] This invention also provides an application example in a fintech scenario (intelligent generation of insurance claim reports). In the remote auto insurance claim scenario, surveyors need to generate structured reports in real time based on on-site photos / dialogue records. Traditional models based on GPT-3.5 (17.5 billion parameters) cannot be deployed on mobile devices due to high computational latency (>8 seconds). The compressed student model (using forward KL distillation) has an error rate as high as 32% when describing key responsibilities, overfits non-critical details (such as "minor scratches on windshield wipers" with a probability of 0.03%), and ignores core risk points (such as "brake pad wear" with a probability of 91%).

[0070] This invention employs reverse KL distillation optimization: the teacher model (Llama2-13B) analyzes on-site images and text to generate report templates; KL(q||p) enables the student model (DistilBERT-110M) to focus on high-probability liability features. The accuracy of generating key liability items (such as "brake system failure") is significantly improved, while the generation rate of low-frequency noise features is significantly reduced. Key-value caching technology is used to dynamically adjust the length of the autoregressive generation history window in real time on the claims adjuster's tablet (512 words in the first 3 sentences → 128 words in the later section). When personal injury cases are involved, the injury ICD code is automatically embedded (e.g., "S42.0 humeral fracture"). The system's memory pool mechanism verifies logical consistency (e.g., excluding contradictory combinations like "no fracture + severe pain"). Industry breakthrough: After an insurance company adopted this system, its disputed claims rate decreased by 27%, and mobile coverage reached 100%.

[0071] This invention also provides an application example in a healthcare scenario (intelligent assisted generation of image reports). A tertiary hospital processes over 2000 CT images daily. Radiologists manually write reports, taking approximately 15 minutes per report, and 9.7% of these reports omit key indicators (such as changes in tumor size). Traditional distillation models (BERT→TinyBERT) suffer from exposure bias, leading to a break in the logical flow between "image description" and "clinical recommendations" (e.g., finding a 3cm lesion but recommending "regular follow-up").

[0072] This invention employs a teacher model (Med-PaLM 2) to simultaneously process CT images (Vision Transformer) and patient medical history (Text Transformer); the student model retains only the text branch but learns semantic associations of images through cross-modal loss (Lcross). The completeness of key indicator descriptions is significantly improved (e.g., the detection rate of "spiculated sign" and "vascular cluster sign" is over 95%). Five descriptive statements are generated, and entity logic verification is performed once; revisions are automatically triggered when contradictory combinations are detected (e.g., "pancreatic necrosis + normal amylase"). A gradient confusion layer is used to separate sensitive data (e.g., patient ID) from medical features; under a federated learning framework, joint distillation by 10 hospitals increased the rare disease identification rate (e.g., Castleman's disease) to 89%. Report generation time was reduced from 15 minutes to 1.2 minutes; the misdiagnosis rate decreased to 0.9% (2000 blind tests). Prescription recommendations that do not conform to DDI (drug interaction rules) are highlighted (e.g., the contraindicated combination of "lung cancer targeted drug + St. John's wort extract").

[0073] In summary, by employing inverse KLD, the student model in this invention can more accurately simulate the output distribution of the teacher model on complex generation tasks, resulting in more predictable generation outcomes. This method demonstrates consistent performance improvement across model families of different sizes and structures, exhibiting strong generalization ability.

[0074] like Figure 6 As shown, this embodiment of the invention also provides a large language model knowledge distillation apparatus, characterized in that it includes:

[0075] Data acquisition module 10 is used to acquire business data, which originates from a business analysis task. The business data includes the input text sequence, the expected output text sequence, and the corresponding attention mask sequence of the business analysis task.

[0076] The model prediction module 20 is used to input the input text sequence of the business analysis task and the attention mask sequence into the student model and the teacher model, respectively, to obtain the first prediction probability distribution output by the student model and the second prediction probability distribution output by the teacher model, wherein the second prediction probability distribution is generated based on the supervision signal of the expected output text sequence;

[0077] The loss calculation module 30 is used to calculate the reverse KL divergence loss value based on the first predicted probability distribution and the second predicted probability distribution, wherein the reverse KL divergence loss value is used to measure the difference between the second predicted probability distribution and the first predicted probability distribution.

[0078] The model optimization module 40 is used to optimize the parameters of the student model using the inverse KL divergence loss value, and generate an optimized student model for business analysis. In the optimization process, the student model generates contextual information for subsequent predictions based on the business analysis prediction results of its previous steps.

[0079] In one specific embodiment, a preprocessing module is further included before the model prediction module 20, for:

[0080] The input text sequence and the expected output text sequence are preprocessed, the preprocessing including:

[0081] Use a word segmenter to convert the input text sequence and the expected output text sequence into the corresponding word identifier sequence;

[0082] The word identifier sequence is padded to make the sequence length uniform to the target length, and the padded position is marked with a padded marker.

[0083] The attention mask sequence is generated based on the padded sequence, and the attention mask sequence is used to distinguish between valid word positions and padded positions.

[0084] In one specific embodiment, in the model prediction module 20, the first prediction probability distribution and the second prediction probability distribution are obtained in the following way:

[0085] The original prediction scores output by the student model and the teacher model are normalized to obtain the probability distribution of each word position on the entire vocabulary.

[0086] In one specific embodiment, the calculation of the reverse KL divergence loss value in the loss calculation module 30 specifically includes:

[0087] Use the first predicted probability distribution as the target distribution;

[0088] The second predicted probability distribution is used as the reference distribution;

[0089] The inverse KL divergence value of the target distribution relative to the reference distribution is calculated as the loss value.

[0090] In one specific embodiment, in the model optimization module 40, optimizing the parameters of the student model using the inverse KL divergence loss value to generate an optimized student model for business analysis specifically includes:

[0091] The gradient of the backpropagation loss value with respect to the student model parameters is calculated using the backpropagation function of the deep learning framework.

[0092] Using the gradient descent algorithm, the parameters of the student model are updated based on the gradient to generate an optimized student model for business analysis.

[0093] In one specific embodiment, in the model optimization module 40, during the optimization process, the context information used by the student model to generate subsequent prediction content is generated based on the prediction results of its previous steps, specifically including:

[0094] At each time step of the training iteration, for the student model, the lexical units generated before the current time step are used as contextual input to predict the lexical units for the next time step.

[0095] In one specific embodiment, the teacher model is a neural network model with a parameter size greater than 3B that has text generation capabilities; the student model is a neural network model with a parameter size less than or equal to 3B that has text generation capabilities.

[0096] Specific limitations regarding the large language model knowledge distillation apparatus can be found in the limitations of the large language model knowledge distillation method described above, and will not be repeated here. Each module in the aforementioned large language model knowledge distillation apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0097] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a large language model knowledge distillation method on the server side.

[0098] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the client-side functions or steps of a large language model knowledge distillation method.

[0099] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0100] S10. Obtain business data, which originates from a business analysis task. The business data includes the input text sequence, the expected output text sequence, and the corresponding attention mask sequence of the business analysis task.

[0101] S20. Input the input text sequence of the business analysis task and the attention mask sequence into the student model and the teacher model respectively to obtain the first prediction probability distribution output by the student model and the second prediction probability distribution output by the teacher model, wherein the second prediction probability distribution is generated based on the supervision signal of the expected output text sequence;

[0102] S30. Based on the first predicted probability distribution and the second predicted probability distribution, calculate the reverse KL divergence loss value, wherein the reverse KL divergence loss value is used to measure the difference between the second predicted probability distribution and the first predicted probability distribution.

[0103] S40. Optimize the parameters of the student model using the inverse KL divergence loss value to generate an optimized student model for business analysis. In the optimization process, the student model generates contextual information for subsequent predictions based on the business analysis prediction results of its previous steps.

[0104] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0105] S10. Obtain business data, which originates from a business analysis task. The business data includes the input text sequence, the expected output text sequence, and the corresponding attention mask sequence of the business analysis task.

[0106] S20. Input the input text sequence and the attention mask sequence into the student model and the teacher model respectively to obtain the first prediction probability distribution output by the student model and the second prediction probability distribution output by the teacher model, wherein the second prediction probability distribution is generated based on the supervision signal of the expected output text sequence;

[0107] S30. Based on the first predicted probability distribution and the second predicted probability distribution, calculate the reverse KL divergence loss value, wherein the reverse KL divergence loss value is used to measure the difference between the second predicted probability distribution and the first predicted probability distribution.

[0108] S40. Optimize the parameters of the student model using the inverse KL divergence loss value to generate an optimized student model for business analysis. In the optimization process, the student model generates contextual information for subsequent predictions based on the business analysis prediction results of its previous steps.

[0109] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0112] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A knowledge distillation method for large language models, characterized in that, Includes the following steps: S10. Obtain business data, which originates from a business analysis task. The business data includes the input text sequence, the expected output text sequence, and the corresponding attention mask sequence of the business analysis task. S20. Input the input text sequence of the business analysis task and the attention mask sequence into the student model and the teacher model respectively to obtain the first prediction probability distribution output by the student model and the second prediction probability distribution output by the teacher model, wherein the second prediction probability distribution is generated based on the supervision signal of the expected output text sequence; S30. Based on the first predicted probability distribution and the second predicted probability distribution, calculate the reverse KL divergence loss value, wherein the reverse KL divergence loss value is used to measure the difference between the second predicted probability distribution and the first predicted probability distribution. S40. Optimize the parameters of the student model using the inverse KL divergence loss value to generate an optimized student model for business analysis. In the optimization process, the student model generates contextual information for subsequent predictions based on the business analysis prediction results of its previous steps.

2. The knowledge distillation method for large language models according to claim 1, characterized in that, Before step S20, the method further includes: The input text sequence and the expected output text sequence of the business analysis task are preprocessed, and the preprocessing includes: Use a word segmenter to convert the input text sequence and the expected output text sequence into the corresponding word identifier sequence; The word identifier sequence is padded to make the sequence length uniform to the target length, and the padded position is marked with a padded marker. The attention mask sequence is generated based on the padded sequence, and the attention mask sequence is used to distinguish between valid word positions and padded positions.

3. The knowledge distillation method for large language models according to claim 1, characterized in that, In step S20, the first predicted probability distribution and the second predicted probability distribution are obtained in the following way: The original prediction scores output by the student model and the teacher model are normalized to obtain the probability distribution of each word position on the entire vocabulary.

4. The knowledge distillation method for large language models according to claim 1, characterized in that, In step S30, calculating the reverse KL divergence loss specifically includes: Use the first predicted probability distribution as the target distribution; The second predicted probability distribution is used as the reference distribution; The inverse KL divergence value of the target distribution relative to the reference distribution is calculated as the loss value.

5. The knowledge distillation method for large language models according to claim 1, characterized in that, In step S40, optimizing the parameters of the student model using the inverse KL divergence loss value to generate an optimized student model for business analysis specifically includes: The gradient of the backpropagation loss value with respect to the student model parameters is calculated using the backpropagation function of the deep learning framework. Using the gradient descent algorithm, the parameters of the student model are updated based on the gradient to generate an optimized student model for business analysis.

6. The knowledge distillation method for large language models according to any one of claims 1 to 5, characterized in that, In step S40, during the optimization process, the contextual information used by the student model to generate subsequent prediction content is generated based on the business analysis prediction results of its previous steps, specifically including: At each time step of the training iteration, for the student model, the lexical units it generated before the current time step are used as contextual input to predict the lexical units for the next time step.

7. The knowledge distillation method for large language models according to claim 1, characterized in that: The teacher model is a neural network model with a parameter size greater than 3B that has text generation capabilities; the student model is a neural network model with a parameter size less than or equal to 3B that has text generation capabilities.

8. A knowledge distillation device for a large language model, characterized in that, include: The data acquisition module is used to acquire business data, which originates from a business analysis task. The business data includes the input text sequence, the expected output text sequence, and the corresponding attention mask sequence of the business analysis task. The model prediction module is used to input the input text sequence of the business analysis task and the attention mask sequence into the student model and the teacher model, respectively, to obtain the first prediction probability distribution output by the student model and the second prediction probability distribution output by the teacher model, wherein the second prediction probability distribution is generated based on the supervision signal of the expected output text sequence; The loss calculation module is used to calculate the reverse KL divergence loss value based on the first predicted probability distribution and the second predicted probability distribution, wherein the reverse KL divergence loss value is used to measure the difference between the second predicted probability distribution and the first predicted probability distribution. The model optimization module is used to optimize the parameters of the student model using the inverse KL divergence loss value, and generate an optimized student model for business analysis. During the optimization process, the student model generates contextual information for subsequent predictions based on the business analysis prediction results of its previous steps.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the large language model knowledge distillation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the large language model knowledge distillation method as described in any one of claims 1 to 7.

Images