Cross-modal large language model distillation method based on structural difference perception

Abstract

The invention discloses a cross-modal large language model distillation method based on structural difference perception. The method comprises the following steps: 1) finely tuning a model, and learning an output mode; 2) constructing a structural label; and 3) performing multi-task distillation training. According to the method, the structure perception ability can be added to the distillation process of the large language model, the code structured difference and loss are transmitted to the student model, the defect that the large language model can only pass text probability training in a structure related task is improved, the structure expression ability represented by a code generation task is improved, and the structure perception ability is improved. And thus, the small parameter model performance on the code generation task is improved.

Description

Technical Field

[0001] This invention relates to large model distillation technology and code structured difference assessment technology, particularly to the optimization and adjustment technology of loss function in large model white-box distillation, and to code structured assessment algorithm technology, specifically a cross-modal large language model distillation method based on structured difference awareness. Background Technology

[0002] Large Language Models (LLMs) are question-answering language models trained on massive amounts of text training data and using Natural Language Processing (NLP) techniques, such as Transformers and BERT, with hundreds of millions of parameters. LLMs can answer user questions, providing specialized knowledge across various domains and generating specified content. They possess extensive knowledge graphs and efficient inductive abilities. LLMs are widely used in knowledge retrieval, education, content creation, and text verification and polishing. Furthermore, their multimodal (text, image, video, audio) expansion training extends their application potential to even more fields. LLM is now widely used in code generation and completion tasks. However, limited by its initial training method—guided learning based on text masking—its code generation relies heavily on text features, easily ignoring the inherent structural information of the code, thus limiting its accuracy. Furthermore, LLM is influenced by the provided text information and its unsupervised learning method. When incorrect knowledge or information is mentioned and input excessively, LLM may mistakenly identify it as correct information, resulting in incorrect answers. This is known as the LLM illusion phenomenon, which occasionally occurs in code generation tasks. These phenomena and characteristics can potentially affect the reliability and stability of the code generated by LLM.

[0003] Abstract Syntax Trees (ASTs) are tree-structured information describing code, widely used in code completion, compilation, translation, and other technical fields. The core of ASTs lies in their focus on describing the structure and hierarchy of code, rather than specific syntactic details and keywords. In the compilation field, they are an important tool for measuring whether code conforms to syntactic structure specifications and is correct. In early NLP tasks, many researchers proposed using ASTs and their description language, Abstract Syntax Description Language (ASDL), as the starting or intermediate step in code generation to constrain code generation standards. This approach achieved good results before the advent of LLMs. Although current LLMs improve the accuracy of code generation tasks through training with large amounts of correct text, due to considerations of model generalization and multi-domain problem-solving capabilities, the widely used LLMs do not currently use code structure constraints as an intermediate step in training. Therefore, the correctness of code generated by LLMs is essentially the correctness of the text, lacking rigorous structural constraints on the code. When new languages ​​emerge and there is a lack of large amounts of text as training data, the code generation capabilities of LLMs will be challenged. Building efficient and rigorous LLM constraint training for code tasks is crucial for the development of LLM and for ensuring code stability and reliability. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a cross-modal large language model distillation method based on structure difference awareness. This method adds structure awareness to the large language model distillation process, passing code structure differences and losses to the student model. This overcomes the limitation of large language models being trained solely on text probabilities in structure-related tasks, improving the structural representation ability in tasks such as code generation, and thus enhancing the performance of small-parameter models in code generation tasks.

[0005] The technical solution to achieve the objective of this invention is:

[0006] A method for distilling cross-modal large language models based on structural difference perception includes the following steps: 1) Model fine-tuning and learning output modes: including: 1-1) Preprocessing the dataset: The original training dataset is a collection of prompt-response pairs from HumanEval and publicly available data repositories on GitHub. The original training dataset is preprocessed by cleaning the prompt-response pairs into prompt-pure code pairs, removing the natural language portion and Markdown output format from the original training dataset's response field, and constructing the preprocessed training dataset. ,in Indicates the first One prompt word, This represents the corresponding pure code snippet, which performs geometric random sampling within the same language on a training dataset with a size greater than or equal to 10,000 data points, controlling the training dataset. The scale of Within this range, in order to maintain efficient and accurate training; 1-2) Role Definition and Fine-tuning of Teacher and Student Models: Defining the Teacher Model and student model ,in and All are large language models. The parameter size is greater than And possess stronger thinking abilities, based on the cleaned training set in step 1-1). Low-rank adaptive LoRA technology was used to respectively... and Perform supervised fine-tuning, with the fine-tuning objective being a given... Time prediction This enables the model to learn instructions and generate pure code output patterns. 1-3) Convergence Judgment: Monitor the loss changes during the fine-tuning process in step 1-2). When the loss changes less than 1 / 3 of the total loss over three consecutive training epochs, the convergence judgment is made. When convergence is reached, fine-tuning is stopped, and the fine-tuned teacher model is obtained. and student model ; 2) Constructing structural tags and Create distillation labels containing structural information, including: 2-1) Structured Transformation: A cross-modal structure-addition algorithm is used to construct structure labels. In each batch of training, the fine-tuned teacher model obtained in steps 1-3) is transformed. and student model For the same batch of prompt words Generated plain text code and Take it out, and at the same time... corresponding Extract the code as a comparison of the training dataset. , respectively , and The structure transformation is performed using Abstract Syntax Tree (AST) mode transformation, and the transformation operator is shown in formula (1): (1), in Source code, Indicates the first part of the source program Each grammatical unit. For context analysis functions, The recursive replacement rule is shown in formula (2): (2), Will , and Substituting these values ​​into formula (1), we obtain the structured representations of the output codes for the same set of prompt words from the teacher model, student model, and training data. , and ; 2-2) Quantification of structural differences: The structural differences obtained in step 2-1) Using the Levinstein edit distance algorithm and respectively with and A quantitative comparison is performed, as shown in formula (3): (3), Formula (3) is defined as the graph space of the two trees. and The minimum editing cost between them is solved by dynamic programming as shown in formulas (4), (5), and (6): (4), (5), (6), in Indicates an empty forest. Indicates deleting a node The remaining forest after that, For node label functions, For indicator functions, Representing tree space The root node, substitute , and Calculated using formulas (3), (4), and (5) and and Structural difference numerical values and ; 2-3) Structural label construction: using the structural difference numerical values ​​from step 2-2). and Calculate quantifiable scores as direct structure hard labels and direct structure soft tags The quantifiable fraction can be calculated as shown in formula (7): (7), in To standardize the function, considering the discreteness and scale uncertainty of the graph edit distance, an exponential standardization function is adopted as shown in formula (8): (8); This function is different from the traditional It has better gradient properties, is sensitive to small distances, moderately saturates over large distances, and exhibits smooth decay. 3) Multi-task distillation training: using the training set cleaned in step 1). For the teacher model that was fine-tuned in step 2) and student model Distillation training for structural reinforcement includes: 3-1) Multi-task loss construction: Construct a composite loss model that includes knowledge distillation and structural loss, defining the loss for knowledge distillation as... The loss of structural information is The composite loss model is then constructed as shown in equation (9): (9); 3-2) Dynamic adjustment of structural loss constraints: combined with the direct structural hard labeling in step 2). and direct structure soft tags An additional cross-modal loss is constructed using an augmented Lagrangian framework. First, the constraints are defined as shown in equations (10) and (11): Structural hard label constraint violation: (10) Structural soft label constraint violation: (11), in and They are respectively and The upper limit of structural difference tolerance is a hyperparameter. Each batch of training is penalized according to the degree of violation. The augmented Lagrangian framework contains linear and quadratic penalty terms, forming a dynamic adjustment constraint on training and an aggravated penalty effect when the constraint is violated. The structural loss information loss function is constructed as shown in formula (12) when using the augmented Lagrangian framework: (12), in and The label multiplier and the label constraint violation degree form a linear penalty term to determine the penalty strength. This is a learnable parameter. In the Lagrange method, the label multiplier is iterated based on the dual ascent method as shown in formula (13): (13) in The learning rate of the multipliers controls the speed of multiplier updates. In this iteration, when constraints are violated more frequently, the label multiplier will increase, increasing the penalty for training. In this framework, the label multiplier and the violation degree constitute a first-order linear penalty, maintaining dynamic constraints and flexibility, and preventing getting trapped in local optima. Here, is the quadratic penalty coefficient, and is a hyperparameter. This makes the optimization problem more "prominent," and the penalty grows faster. Convergence can be guaranteed even under unstable and precise conditions. The quadratic multiplication coefficients and the square of the violation degree constitute an augmentation term, which aggravates the penalty effect.

[0007] This technical solution introduces structured code evaluation into the distillation loss function, enabling lightweight student models to perform structure-aware and constraint-controlled training in code generation tasks. First, the training data is cleaned to remove interference from natural language and Markdown formatting, obtaining clean code output samples. Then, the teacher and student models are professionally fine-tuned based on LoRA technology to adapt to the pure code output mode. Next, the abstract syntax tree transformation operator is used to convert the tensor of the model output into a structured graph representation. Levinstein edit distance is used to quantify the structural differences between student output and training data, as well as teacher output, constructing hard and soft structural labels. An exponential normalization function is used to enhance the gradient sensitivity of difference perception. During the distillation training phase, this solution combines traditional knowledge distillation loss and structured difference loss to construct a multi-task optimization objective based on an augmented Lagrange framework. It introduces structural constraint violation degree and dynamically updated Lagrange multipliers to achieve adaptive adjustment of the structural constraints of the student model: increasing the penalty when structural differences exceed the tolerance threshold, and maintaining training degrees of freedom when constraints are met, avoiding getting trapped in local optima.

[0008] This technical solution not only preserves semantic alignment at the text level, but also uses code syntax and hierarchical structure information as optimizable supervision signals, enabling the student model to generate grammatically correct and structurally reliable code even with a small parameter size.

[0009] This technical solution adopts a structured difference evaluation and white-box distillation mechanism that integrates code modalities, which improves the structural accuracy and generalization ability of lightweight models in code generation tasks. It is especially suitable for code generation scenarios of low-resource languages ​​or emerging programming languages, and can be widely used in fields such as intelligent programming assistants, code completion, educational tools and software testing. It provides a new training paradigm for improving the reliability and stability of large model code generation.

[0010] This method adds structure awareness to the distillation process of large language models, passing code structure differences and losses to the student model. It improves the limitation of large language models in structure-related tasks that can only be trained by text probabilities, enhances the structural representation ability in tasks such as code generation, and thus improves the performance of small parameter models in code generation tasks. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the data cleaning and fine-tuning workflow in the embodiment; Figure 2 This is a schematic diagram illustrating the structural distance loss calculation workflow in the embodiment; Figure 3 This is a schematic diagram of the cross-modal loss calculation workflow based on the augmented Lagrange framework in the embodiment. Detailed Implementation

[0012] The following description, in conjunction with the accompanying drawings and embodiments, further elaborates on the content of the present invention, but is not intended to limit the scope of the invention.

[0013] Example:

[0014] A method for distilling cross-modal large language models based on structural difference perception includes the following steps: 1) Model fine-tuning and learning output modes: such as Figure 1 As shown, it includes: 1-1) Preprocessing the dataset: The original training dataset is a collection of prompt-response pairs from HumanEval and publicly available data repositories on GitHub. The original training dataset is preprocessed by cleaning the prompt-response pairs into prompt-pure code pairs, removing the natural language portion and Markdown output format from the original training dataset's response field, and constructing the preprocessed training dataset. ,in Indicates the first One prompt word, This represents the corresponding pure code snippet, which performs geometric random sampling within the same language on a training dataset with a size greater than or equal to 10,000 data points, controlling the training dataset. The scale of Within this range, in order to maintain efficient and accurate training; 1-2) Role Definition and Fine-tuning of Teacher and Student Models: Defining the Teacher Model and student model ,in and All are large language models. The parameter size is greater than And possess stronger thinking abilities, based on the cleaned training set in step 1-1). Low-rank adaptive LoRA technology was used to respectively... and Perform supervised fine-tuning, with the fine-tuning objective being a given... Time prediction This enables the model to learn instructions and generate pure code output patterns. 1-3) Convergence Judgment: Monitor the loss changes during the fine-tuning process in step 1-2). When the loss changes less than 1 / 3 of the total loss over three consecutive training epochs, the convergence judgment is made. When convergence is reached, fine-tuning is stopped, and the fine-tuned teacher model is obtained. and student model ; 2) Constructing structural tags and Create distillation labels that contain structural information, such as... Figure 2 As shown, it includes: 2-1) Structured Transformation: A cross-modal structure-addition algorithm is used to construct structure labels. In each batch of training, the fine-tuned teacher model obtained in steps 1-3) is transformed. and student model For the same batch of prompt words Generated plain text code and Take it out, and at the same time... corresponding Extract the code as a comparison of the training dataset. , respectively , and The structure transformation is performed using Abstract Syntax Tree (AST) mode transformation, and the transformation operator is shown in formula (1): (1), in Source code, Indicates the first part of the source program Each grammatical unit. For context analysis functions, The recursive replacement rule is shown in formula (2): (2), Will , and Substituting these values ​​into formula (1), we obtain the structured representations of the output codes for the same set of prompt words from the teacher model, student model, and training data. , and ; 2-2) Quantification of structural differences: The structural differences obtained in step 2-1) Using the Levinstein edit distance algorithm and respectively with and A quantitative comparison is performed, as shown in formula (3): (3), Formula (3) is defined as the graph space of the two trees. and The minimum editing cost between them is solved by dynamic programming as shown in formulas (4), (5), and (6): (4), (5), (6), in Indicates an empty forest. Indicates deleting a node The remaining forest after that, For node label functions, For indicator functions, Representing tree space The root node, substitute , and Calculated using formulas (4), (5), and (6) and and Structural difference numerical values and ; 2-3) Structural label construction: using the structural difference numerical values ​​from step 2-2). and Calculate quantifiable scores as direct structure hard labels and direct structure soft tags The quantifiable fraction can be calculated as shown in formula (7): (7), in To standardize the function, considering the discreteness and scale uncertainty of the graph edit distance, an exponential standardization function is adopted as shown in formula (8): (8); This function is different from the traditional It has better gradient properties, is sensitive to small distances, moderately saturates over large distances, and exhibits smooth decay. 3) Multi-task distillation training: using the training set cleaned in step 1). For the teacher model that was fine-tuned in step 2) and student model Perform distillation training to strengthen the structure, such as Figure 3 As shown, it includes: 3-1) Multi-task loss construction: Construct a composite loss model that includes knowledge distillation and structural loss, defining the loss for knowledge distillation as... The loss of structural information is The composite loss model is then constructed as shown in equation (9): (9); 3-2) Dynamic adjustment of structural loss constraints: combined with the direct structural hard labeling in step 2). and direct structure soft tags An additional cross-modal loss is constructed using an augmented Lagrangian framework. First, the constraints are defined as shown in equations (10) and (11): Structural hard label constraint violation: (10) Structural soft label constraint violation: (11), in and They are respectively and The upper limit of structural difference tolerance is a hyperparameter. Each batch of training is penalized according to the degree of violation. The augmented Lagrangian framework contains linear and quadratic penalty terms, forming a dynamic adjustment constraint on training and an aggravated penalty effect when the constraint is violated. The structural loss information loss function is constructed as shown in formula (12) when using the augmented Lagrangian framework: (12), in and The label multiplier and the label constraint violation degree form a linear penalty term to determine the penalty strength. This is a learnable parameter. In the Lagrange method, the label multiplier is iterated based on the dual ascent method as shown in formula (13): (13) in The learning rate of the multipliers controls the speed of multiplier updates. In this iteration, when constraints are violated more frequently, the label multiplier will increase, increasing the penalty for training. In this framework, the label multiplier and the violation degree constitute a first-order linear penalty, maintaining dynamic constraints and flexibility, and preventing getting trapped in local optima. Here, is the quadratic penalty coefficient, and is a hyperparameter. This makes the optimization problem more "prominent," and the penalty grows faster. Convergence can be guaranteed even under unstable and precise conditions. The quadratic multiplication coefficients and the square of the violation degree constitute an augmentation term, which aggravates the penalty effect.

[0015] Specifically, the experimental procedure in this example is as follows: Step 1: Clean the training dataset, removing Markdown formatting and natural language; Step 2: Obtain the cleaned Prompt-Pure_Code instruction-code pairs; Step 3: Fine-tune the teacher and student models; Step 4: Obtain a fine-tuned model that has learned the output pattern; Step 5: Use the intermediate outputs of the fine-tuned teacher and student models, as well as the Response field from the training set, as input. Input to the structural difference calculation module; Step 6: Define the AST transformation operator ; Step 7: [The sentence is incomplete and likely refers to a specific step or instruction.] Perform AST transformation to obtain a graph space representation; Step 8: Calculate using the Rhinestein distance algorithm and Structural differences; Step 9: Standardize the structural differences from Step 8; Step 10: Use the standardized scores from Step 9 as direct structural hard labels. ; Step 11: Calculate using the Rhinestein distance algorithm and Structural differences; Step 12: Standardize the structural differences from Step 11; Step 13: Use the standardized scores from Step 12 as direct structural hard labels ; Step 14: In the multi-task training module, input the knowledge distillation loss. ; Step 15: In the multi-task training module, input the direct structure labels from Step 10 and Step 13; Step 16: Perform constraint checks. If a hard label is being checked, when... At that time, jump to Step 17. If so, proceed to Step 18; if determining a soft tag, when At that time, jump to Step 21. If necessary, proceed to Step 22; Step 17: Set the violation level of this round of hard labels Proceed to Step 19; Step 18: Set the hard label violation level for this round Proceed to Step 19; Step 19: Calculate the linear penalty term ; Step 20: Calculate the secondary penalty term Proceed to Step 16 to determine the soft label; Step 21: Set the violation level of soft tags for this round Proceed to Step 23; Step 22: Set the violation level of this round of soft tags Proceed to Step 23; Step 23: Calculate the linear penalty term ; Step 24: Calculate the secondary penalty term ; Step 25: Aggregate the total loss based on the calculated values ​​from Step 19, Step 20, Step 23, and Step 24; Step 26: Determine whether to update the Lagrange multipliers. If an update is needed, proceed to Step 27; otherwise, proceed to Step 28. Step 27: Using the dual ascent method Update the multipliers; Step 28: Keep the multipliers unchanged; Step 29: Determine if the current training round is over. If it is, end the round; otherwise, output the updated version. Proceed to Step 14.

[0016] In this example, the implementation uses the Hugginface public dataset for training and HeratStone, Django, and live_code_bench datasets widely used in the code generation field for evaluation. This method is implemented on an Nvidia H100-Nvlink-80G computing platform, with PyCharm as the development environment and Python 3.10 as the programming language. Eleven programming languages ​​were used in the training: Ruby, Rust, JavaScript, Python, Java, C++, Go, and C#. The original dataset had a difficulty distribution of Low-33.6%, High-33.4%, and Extreme-33.0%, with a 1:2:1 sampling ratio for each difficulty level. After processing, a total of 1978 data points were obtained. Initial LoRA fine-tuning training used a full-parameter model, training 0.217% of the parameters, for a total of 32,112,640 parameters. The batch size for each device was constrained to 4, and the rank was increased to 8. The total number of training epochs was set to 3, achieving high-precision training and optimized space utilization under these settings. In the experimental example, the loss converged to around 0.5 after three rounds, which is considered excellent. During distillation, the batch size of each device was constrained to 1, while keeping the rank and epoch unchanged. Since two models need to be loaded simultaneously—the teacher model - Qwen / Qwen3-14B and the student model - Qwen / Qwen3-1.7B—memory usage needs to be considered. Under these parameter settings, memory usage is between 46GB and 77GB. To maximize efficiency, a tolerance coefficient is set during multi-task training. All values ​​start at 0.2. Automatic iteration is performed during training. When the multiplier learning rate is controlled at 0.001, the convergence effect is good, preventing gradient explosion and loss distortion. After three rounds of training, the loss converges to about 1.2, which is a good acceptable range, demonstrating its good training effect.

Claims

1. A cross-modal large language model distillation method based on structural difference perception, characterized in that, Includes the following steps: 1) Model fine-tuning and learning output modes: including: 1-1) Preprocessing the dataset: The original training dataset is a collection of prompt-response pairs from HumanEval and publicly available data repositories on GitHub. The original training dataset is preprocessed by cleaning the prompt-response pairs into prompt-pure code pairs, removing the natural language portion and Markdown output format from the original training dataset's response field, and constructing the preprocessed training dataset. ,in Indicates the first One prompt word, This represents the corresponding pure code snippet, which performs geometric random sampling within the same language on a training dataset with a size of 10,000 or more, controlling the training dataset. The scale of Within the scope; 1-2) Role Definition and Fine-tuning of Teacher and Student Models: Defining the Teacher Model and student model ,in and All are large language models. The parameter size is greater than And possess stronger thinking abilities, based on the cleaned training set in step 1-1). Low-rank adaptive LoRA technology was used to respectively... and Perform supervised fine-tuning, with the fine-tuning objective being a given... Time prediction This enables the model to learn instructions and generate pure code output patterns. 1-3) Convergence Judgment: Monitor the loss changes during the fine-tuning process in step 1-2). When the loss changes less than 1 / 3 of the total loss over three consecutive training epochs, the convergence judgment is made. When convergence is reached, fine-tuning is stopped, and the fine-tuned teacher model is obtained. and student model ; 2) Constructing structural tags and Construct a distillation label containing structural information, including: 2-1) Structured Transformation: A cross-modal structure-addition algorithm is used to construct structure labels. In each batch of training, the fine-tuned teacher model obtained in steps 1-3) is transformed. and student model For the same batch of prompt words Generated plain text code and Take it out, and at the same time... corresponding Extract the code as a comparison of the training dataset. , respectively , and The structure transformation is performed using Abstract Syntax Tree (AST) mode transformation, and the transformation operator is shown in formula (1): （1）， in Source code, Indicates the first part of the source program Each grammatical unit. For context analysis functions, The recursive replacement rule is shown in formula (2): （2）， Will , and Substituting these values ​​into formula (1), we obtain the structured representations of the output codes for the same set of prompt words from the teacher model, student model, and training data. , and ; 2-2) Quantification of structural differences: The structural differences obtained in step 2-1) Using the Levinstein edit distance algorithm and respectively with and A quantitative comparison is performed, as shown in formula (3): （3）， Formula (3) is defined as the graph space of the two trees. and The minimum editing cost between them is solved by dynamic programming as shown in formulas (4), (5), and (6): （4）， （5）， （6）， in Indicates an empty forest. Indicates deleting a node The remaining forest after that, For node label functions, For indicator functions, Representing tree space The root node, Substitution , and Calculated using formulas (4), (5), and (6) and and Structural difference numerical values and ; 2-3) Structural label construction: using the structural difference numerical values ​​from step 2-2). and Calculate quantifiable scores as direct structure hard labels and direct structure soft tags The quantifiable fraction can be calculated as shown in formula (7): （7）， in To standardize the function, the exponential standardization function is used as shown in formula (8): （8）； 3) Multi-task distillation training: including: 3-1) Multi-task loss construction: Construct a composite loss model that includes knowledge distillation and structural loss, defining the loss for knowledge distillation as... The loss of structural information is The composite loss model is then constructed as shown in equation (9): （9）； 3-2) Dynamic adjustment of structural loss constraints: combined with the direct structural hard labeling in step 2). and direct structure soft tags An additional cross-modal loss is constructed using an augmented Lagrangian framework. First, the constraints are defined as shown in equations (10) and (11): Structural hard label constraint violation: (10) Structural soft label constraint violation: (11), in and They are respectively and The upper limit of structural difference tolerance is a hyperparameter. Each batch of training is penalized according to the degree of violation. The augmented Lagrangian framework contains linear and quadratic penalty terms, forming a dynamic adjustment constraint on training and an aggravated penalty effect when the constraint is violated. The structural loss information loss function is constructed as shown in formula (12) when using the augmented Lagrangian framework: （12）， in and The label multiplier and the label constraint violation degree form a linear penalty term to determine the penalty strength. This is a learnable parameter. In the Lagrange method, the label multiplier is iterated based on the dual ascent method as shown in formula (13): （13）， in The learning rate of the multipliers controls the speed of multiplier updates. In this iteration, the label multiplier increases when constraints are violated more frequently. In this framework, the label multiplier and the degree of violation constitute a first-order linear penalty. is the quadratic penalty coefficient, and is a hyperparameter.

Images