Code generation model fine-tuning method and device based on clustering and natural language strategy optimization algorithm

Abstract

The application discloses a code generation model fine-tuning method and device based on a clustering and natural language strategy optimization algorithm, and the method comprises the following steps: adopting a clustering algorithm to cluster preprocessed code samples to obtain core code samples, and performing preliminary supervision fine-tuning on a pre-trained large language model based on the core code samples; collecting code generation results of different large language models based on the same instruction text, performing quality sorting on the code generation results, and then constructing a fine-tuning data set, and training a reward model based on the fine-tuning data set; and using the trained reward model to perform strategy supervision fine-tuning on the large language model after preliminary supervision fine-tuning by adopting a natural language strategy optimization algorithm to obtain a target strategy model. The method can reduce the fine-tuning cost of the code generation large language model and align the intention about the code generation task, and improve the accuracy and professionalism of the large language model in the specific field code generation.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and deep learning technology, specifically relating to a method and apparatus for fine-tuning a code generation model based on clustering and natural language policy optimization algorithms. Background Technology

[0002] Large Language Models (LLMs) possess excellent semantic understanding and natural language generation capabilities, handling the complexity of natural language and capturing its semantic relationships and contextual information. With the development of deep learning technology, researchers have begun to explore the application of LLMs in code generation. Code generation techniques utilize LLMs to model and generate source code, precisely capturing the complexity of natural language descriptions and generating corresponding code.

[0003] When using large language models to generate code, it is necessary to fine-tune the pre-trained large language models for specific tasks. Fine-tuning, as an optimization strategy, makes full use of the performance and pre-trained knowledge of large language models. Through the process of fine-tuning for specific tasks, the model's specialization, accuracy, and generalization capabilities are improved, so as to better meet the needs of different natural language processing tasks and generate more accurate and high-quality code.

[0004] Chinese patent application CN116991467A discloses a code repair system and method based on a large language model. This system processes and vectorizes code defect information, inputs it into a pre-trained large-scale programming language model, and uses prompt words to control the large language model to generate intelligent explanations and repair codes for specific code defects based on static analysis results. Chinese patent application CN116909532A discloses a code generation and defect repair method and apparatus that combine the advantages of a large language model trained on massive amounts of text data with reinforcement learning methods. It uses test cases as a reward function to optimize the output of the language model.

[0005] However, existing fine-tuning strategies for large language models used in code generation still face several challenges. First, fine-tuning typically requires a large and diverse dataset to allow the large language model to better adapt to new tasks or domains. If the available dataset is small, the model may rely too heavily on the knowledge of the original model and struggle to adequately adapt to new tasks. Second, fine-tuning on large datasets requires powerful computing capabilities and expensive hardware, which can be challenging for small teams or independent researchers. Furthermore, while probabilistic large language models can generate reasonable code text after fine-tuning, they do not perfectly align with user intent. This misalignment means the model cannot understand the user's semantics and context, as well as the specific task's background and objectives. The fine-tuning process is limited by sample bias and constraints of the pre-training data, which can lead to inconsistent results in different contexts, and even inconsistent answers in the same task. This reduces the model's reliability and interpretability, causing the generated code text to deviate from user expectations.

[0006] Therefore, it is necessary to further post-process and adjust the large language model generated by the code, while reducing the cost of fine-tuning, in order to meet the requirements of accuracy and fluency. Summary of the Invention

[0007] In view of the above, the purpose of this invention is to provide a method and apparatus for fine-tuning a code generation model based on clustering and natural language strategy optimization algorithms, so as to reduce fine-tuning costs and align intent regarding code generation tasks, thereby improving the accuracy and professionalism of large language models as code generation models in specific domains.

[0008] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0009] In a first aspect, the present invention provides a method for fine-tuning a code generation model based on clustering and natural language strategy optimization algorithms, comprising the following steps:

[0010] Clustering algorithms are used to cluster the preprocessed code samples to obtain core code samples, and the pre-trained large language model is initially supervised and fine-tuned based on the core code samples;

[0011] We collect code generation results from different large language models based on the same instruction text, sort the code generation results by quality, construct a fine-tuning dataset, and train the reward model based on the fine-tuning dataset.

[0012] A code generation model is obtained by using a natural language policy optimization algorithm to perform supervised fine-tuning on a large language model after initial supervised fine-tuning, based on a trained reward model. This includes: using the initially supervised fine-tuned large language model as a reference policy model to provide initial performance and user intent benchmarks; using the final target model of the policy-supervised fine-tuning as the target policy model; generating a masked policy model based on the target policy model using masking techniques to address instability during reinforcement learning training; calculating the original reward value for the predicted code of the masked policy model using the trained reward model; calculating the KL divergence between the predicted codes of the reference policy model and the masked policy model; calculating the KL penalty reward value based on the original reward value and KL divergence; constructing an objective function based on the KL penalty reward value to update the parameters of the target policy model and simultaneously updating the parameters of the masked policy model; and obtaining the final target policy model after multiple iterations and parameter updates, which serves as the code generation model for generating accurate code aligned with user intent.

[0013] Preferably, the step of using a clustering algorithm to cluster the preprocessed code samples to obtain core code samples includes:

[0014] The acquired code is preprocessed to obtain code samples. The preprocessing includes: filtering code fragments that do not meet the length requirements and deleting code with ambiguous variable names or syntax errors; encoding the code samples using the encoder of a pre-trained large language model and generating corresponding word vector representations; mapping the word vector representations to a low-dimensional space using dimensionality reduction techniques to obtain one-dimensional vectors as the representation vectors of the code samples; and using a clustering algorithm to cluster the code samples based on their representation vectors to obtain core code samples.

[0015] Preferably, the clustering algorithm employs a K-center greedy algorithm, comprising:

[0016] Initialize the non-center point set S1 and the center point set S2. Select any point from the non-center point set S1 as the initial center point and add it to the center point set S2. Each point represents a code sample.

[0017] Calculate the minimum distance from each point in the non-center point set S1 to all points in the center point set S2, and select the point with the maximum distance in the non-center point set S1 as a candidate for the next center point;

[0018] Remove the point with the largest distance from the non-center point set S1 and add it to the center point set S2;

[0019] Repeat the above distance calculation and point movement steps until the center point set S2 contains K points as the clustering result;

[0020] Based on the center point set S2 and a given ratio, select a set of samples that are close to the center point samples from the code samples as the core code samples.

[0021] Preferably, the preliminary supervised fine-tuning of the pre-trained large language model based on core code samples includes:

[0022] The core code sample is input into the pre-trained large language model for forward propagation to generate predicted code. The predicted code is then compared with the expected generated code and the loss function is calculated. The gradient of the loss function is backpropagated from the output layer to the input layer through backpropagation. An optimizer is used to update the model parameters to minimize the loss function, resulting in a large language model with basic code generation capabilities.

[0023] Preferably, training the reward model based on the fine-tuning dataset includes:

[0024] The fine-tuning dataset includes an initial dataset constructed from code generation results obtained by different large language models based on the same instruction text, and a code ranking dataset constructed after ranking the code generation results by quality. The code ranking dataset is used as a label, and the reward model is trained under supervision using the initial dataset and the code ranking dataset to maximize consistency with the quality ranking. This enables the reward model to calculate the relative order of predicted code quality and calculate the original reward value of the predicted code based on the relative order.

[0025] Preferably, the formula for calculating the KL penalty reward value is as follows:

[0026]

[0027] in, R(s) represents the KL penalty / reward value. t ,a t ,y t ) represents the original reward value calculated by the reward model, KL(·) represents the KL divergence, and s t Let a be the state, representing the t-th instruction text input. t For the action, y represents the code text output corresponding to the large language model. t This indicates standard code text output, π ψ (a t |s t ) represents the probability of the code text generated by the masking strategy model, π0(a t |s t ) represents the probability of the code text generated by the reference policy model, and β represents the KL penalty factor;

[0028] By constraining the reward model with KL penalty reward values, the target policy model is prevented from deviating from the reference policy model.

[0029] Preferably, the process of constructing the objective function includes:

[0030] First, a batch of code samples is sampled from dataset D, which is used for policy-supervised fine-tuning of large language models. t represents D t The t-th code sample out of a total of T code samples, x t This represents the t-th instruction text input. This represents the corresponding standard code text output, and then the code text generated by the reference policy model π0 is collected. And the masking strategy model π ψ The generated code text Let it be denoted as the set of trajectories.

[0031] Then, based on the V-value function representing the state value function. and the Q-value function representing the state-action value function. Calculate the advantage function The calculation formula is:

[0032]

[0033] in, Let s represent the advantage function, s represent the state (instruction input), a represent the action (model output), and π represent the model output. θ Represents the target strategy model. Let t represent the Q-value function, and t represent the set of trajectories τ. t The t-th trajectory among a total of T trajectories;

[0034] Next, the objective function LOSS for the natural language policy optimization algorithm is constructed, and its calculation formula is as follows:

[0035]

[0036] in, Represents the policy function. Let m+1 represent the new target policy model, and m represent the current target policy model.

[0037] The strategy function Let the objective function of the policy network be the action that guides the code generation model's decision-making, providing a probability distribution for choosing an action given a state. This can be represented as:

[0038]

[0039] Among them, clip(r) t (θ), 1-ε, 1+ε) is the cutoff, indicating that r is cut off. t (θ) is restricted to the range [1-ε, 1+ε], where ε represents the truncation range to prevent the difference between the model update and the truncation range from being too large. Importance weights are used to measure the current target strategy model. The deviation from the reference policy model π0 ensures that the model does not change excessively when it is updated, thus maintaining the stability of training. Describe the dominance function, if This indicates that the value of the code text is above average, maximizing... It will increase r t (θ), but it will not exceed 1+ε; conversely, if... maximize It will reduce r t (θ), but will not allow it to exceed 1-ε;

[0040] The value function Let be the objective function of the value network. The value network evaluates the long-term value of a state or action, helping the code generation model learn and optimize its policy, and is expressed as:

[0041]

[0042] Among them, s t V represents the t-th instruction text input. φ (s t ) indicates that in state s t The value function under the following conditions This represents the corresponding KL penalty / reward value.

[0043] Preferably, the synchronous update of the mask strategy model parameters includes:

[0044] After each preset number of updates, the updated target strategy model is copied, retaining the words in the vocabulary with a probability greater than or equal to a preset threshold, and masking irrelevant words with a probability less than the preset threshold, thus obtaining a new masking strategy model.

[0045] Secondly, embodiments of the present invention also provide a code generation model fine-tuning device based on clustering and natural language policy optimization algorithms, including: a model preliminary fine-tuning module, a reward model training module, and a model policy fine-tuning module;

[0046] The model preliminary fine-tuning module is used to cluster the preprocessed code samples using a clustering algorithm to obtain core code samples, and to perform preliminary supervised fine-tuning of the pre-trained large language model based on the core code samples.

[0047] The reward model training module is used to collect code generation results of different large language models based on the same instruction text, and construct a fine-tuning dataset after ranking the code generation results by quality. The reward model is then trained based on the fine-tuning dataset.

[0048] The model policy fine-tuning module is used to perform supervised fine-tuning of the large language model after preliminary supervised fine-tuning using a natural language policy optimization algorithm on the trained reward model to obtain a code generation model. This includes: using the pre-trained, fine-tuned large language model as a reference policy model to provide initial performance and user intent benchmarks; using the final target model of the policy-supervised fine-tuning as the target policy model; generating a masked policy model based on the target policy model using masking techniques to address instability during reinforcement learning training; calculating the original reward value for the predicted code of the masked policy model using the trained reward model; calculating the KL divergence between the predicted codes of the reference policy model and the masked policy model; calculating the KL penalty reward value based on the original reward value and KL divergence; constructing an objective function based on the KL penalty reward value to update the parameters of the target policy model and simultaneously update the parameters of the masked policy model; and obtaining the final target policy model after multiple iterations and parameter updates, which serves as the code generation model for generating accurate code aligned with user intent.

[0049] Thirdly, embodiments of the present invention also provide a code generation model fine-tuning device based on clustering and natural language policy optimization algorithms, including a memory and a processor. The memory is used to store a computer program, and the processor is used to implement the above-mentioned code generation model fine-tuning method based on clustering and natural language policy optimization algorithms when the computer program is executed.

[0050] Compared with existing technologies, the beneficial effects of this invention include at least the following: 1) This invention utilizes clustering algorithms to reduce the fine-tuning cost of large language models for code generation. Clustering can maximize the code diversity of the core set while reducing the amount of training data, and approximate the distribution of the complete dataset to the greatest extent possible; 2) This invention ranks the code generated by the large model through a reward model. By comparing the different code generation results of different large language models for the same instruction text input through ranking, the reward model is trained to automatically complete code quality scoring, overcoming the subjective bias problem of manual scoring and reducing manual costs; 3) This invention utilizes a natural language policy optimization algorithm. By using the large language model for preliminary supervised fine-tuning as the reference policy model and the target policy model as the fine-tuning target, the mask policy model and the target policy model are updated synchronously. The mask policy model is used to predict code and update the parameters of the target policy model. Introducing a mask policy can shield some irrelevant tokens, thereby reducing the action space of the policy model. This allows the action space to reach tens of thousands of tokens, which is larger than the action space of common reinforcement learning policy models. This makes the fine-tuning process more stable and the model converges faster, improving the efficiency and accuracy of model fine-tuning. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating the code generation model fine-tuning method based on clustering and natural language strategy optimization algorithms provided in this embodiment of the invention.

[0053] Figure 2 This is a framework diagram of policy supervision and fine-tuning using a natural language policy optimization algorithm provided in an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of the structure of the code generation model fine-tuning device based on clustering and natural language strategy optimization algorithms provided in the embodiments of the invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0056] The inventive concept of this invention is as follows: Addressing the problems of high fine-tuning costs and insufficient alignment with user intent in existing code generation models, this invention provides a method and apparatus for fine-tuning a code generation model based on clustering and natural language policy optimization algorithms. First, a clustering algorithm including a K-center greedy algorithm is used to perform preliminary supervised fine-tuning on a pre-trained large language model to improve the accuracy and conformity of code generation. Then, a reward model training process is introduced, using manual sorting to allow the reward model to learn the scoring quality of the code, thereby guiding the learning and improvement of the policy. Finally, the policy model is fine-tuned based on a natural language policy optimization algorithm, including steps such as calculating the reward value, calculating the KL penalty term, calculating the final KL penalty reward value, updating the target policy model parameters, and updating the mask policy model parameters, to iteratively optimize the target policy model and generate code sequences that are more syntactically and semantically consistent.

[0057] Figure 1 This is a flowchart illustrating the code generation model fine-tuning method based on clustering and natural language strategy optimization algorithms provided in this embodiment of the invention. Figure 1 As shown in the embodiment, a method for fine-tuning a code generation model based on clustering and natural language strategy optimization algorithms is provided, including the following steps:

[0058] S1. Clustering algorithm is used to cluster the preprocessed code samples to obtain core code samples. Based on the core code samples, the pre-trained large language model is initially supervised and fine-tuned.

[0059] S1.1, Data Preprocessing. Before fine-tuning the pre-trained large language model, data preprocessing is performed to ensure high-quality code samples. This includes filtering code snippets that are too long or too short, and removing code with ambiguous variable names or syntactic errors. Then, the encoder of the pre-trained large language model encodes the code samples and generates corresponding word vector representations. Finally, dimensionality reduction techniques are used to map the word vectors to a lower-dimensional space to obtain one-dimensional vectors as representation vectors for the code samples.

[0060] S1.2, K-center greedy clustering. The K-center greedy algorithm is used to cluster the preprocessed code samples. K code samples are used as initial centroids, and a core set of code samples is selected from the code samples according to a given ratio. This core set of code samples is selected by minimizing the maximum distance between each sample and its nearest centroid to ensure the representativeness of the samples. Specifically, the steps include:

[0061] S1.2.1, Initialize the non-center point set S1 and the center point set S2, select any point from the non-center point set S1 as the initial center point and add it to the center point set S2, each point represents a code sample;

[0062] S1.2.2 Calculate the minimum distance from each point in the non-center point set S1 to all points in the center point set S2, and select the point with the maximum distance in the non-center point set S1 as a candidate for the next center point;

[0063] S1.2.3, remove the point with the largest distance from the non-center point set S1 and add it to the center point set S2;

[0064] S1.2.4 Repeat steps S1.2.2 and S1.2.3 until K centroids are found, i.e., the centroid set S2 contains K centroids, and return the centroid set S2 as the clustering result;

[0065] S1.2.5: Based on the center point set S2 and the given ratio, select a group of samples that are close to the center point samples as core code samples from the code samples. Select samples that are as close as possible to the center point samples as members of the core code set. By controlling the ratio, it is possible to ensure that the collected core code samples achieve a balance between code quality and diversity.

[0066] S1.3 utilizes core code samples obtained from K-center greedy clustering to perform preliminary supervised fine-tuning of the large model. In each training iteration, the following steps are performed: First, the input code snippet is fed into the model for forward propagation to generate a predicted output. Then, the predicted output is compared with the expected generated code, and the loss value is calculated. Next, the gradient of the loss function is backpropagated from the output layer to the input layer. Finally, the optimizer is used to update the model parameters to minimize the loss function. This series of steps is repeated to gradually optimize the pre-trained large language model and give it basic code generation capabilities, enabling it to generate basic code snippets such as variable declarations, loop structures, and conditional statements, and to more accurately generate code snippets that meet expectations.

[0067] S2: Collect code generation results from different large language models based on the same instruction text, sort the code generation results by quality, construct a fine-tuning dataset, and train the reward model based on the fine-tuning dataset.

[0068] In the training of the reward model, a policy is defined to generate a series of codes. This policy receives a prompt and returns the corresponding code sequence. The policy's action space consists of all tokens corresponding to the vocabulary of the large language model, while the observation space is the possible input token sequences. The reward function combines the preference model and the policy shift constraint. Through the reward function, the quality of the generated codes can be quantified and evaluated, thereby guiding the learning and improvement of the policy.

[0069] Evaluating the quality of code generated by large language models requires considering multiple metrics, including syntactic correctness, semantic consistency, readability, modularity, and error handling. This necessitates collecting code generation results from different language models for the same instruction text and creating a code ranking dataset for reward model training. The specific steps include:

[0070] S2.1, Define Evaluation Metrics: Clearly define the specific standards and measurement methods for metrics such as syntax correctness, semantic consistency, readability, and modularity. For example, use metrics such as code compilation success rate, execution correctness, number of lines of code, and code structure as evaluation criteria.

[0071] S2.2, Collect code generation results from multiple models: Use different code generation models to generate code from the same instruction text input, collect the generated code results as the initial dataset, and ensure that the generation results of multiple models are covered so as to provide diverse samples when training the reward model.

[0072] S2.3, Manual Sorting: The collected code is manually sorted by domain experts or experienced developers. The quality of the code is evaluated and sorted according to the defined evaluation metrics. By comparing the differences between the generated code results, the code is arranged according to its quality, generating an ordered code sorting dataset.

[0073] S2.4, Define the reward model architecture: Select a reward model architecture suitable for the specific task, such as a deep neural network, so as to receive the code representation and output a score that evaluates the quality of the model.

[0074] S2.5, Reward Model Training: The code ranking dataset is used as a label to guide the optimization of the reward model. The reward model is trained under supervision using the initial dataset and the code ranking dataset. Appropriate optimization algorithms and loss functions are used to maximize the consistency with quality ranking, so that the reward model can predict the relative order of code generation quality, thereby further assisting the code generation ability and quality of the large language model, enabling it to generate code sequences that are more in line with human ranking standards.

[0075] S3 utilizes the trained reward model and employs a natural language policy optimization algorithm to perform policy-supervised fine-tuning on the initially supervised fine-tuned large language model to obtain a code generation model.

[0076] By using the large language model fine-tuned under preliminary supervision in step S1 as the code generation model, and introducing it into the reward model trained in step S2, such as... Figure 2 As shown, the large language model after preliminary supervised fine-tuning in step S1 is further fine-tuned using the Natural Language Policy Optimization (NLPO) algorithm. The preliminary supervised fine-tuned large language model serves as a reference policy model to provide initial performance and user intent benchmarks. The final target model of the policy-supervised fine-tuning is used as the target policy model. Based on the target policy model, a masked policy model is generated using masking techniques to address instability during reinforcement learning training. Ultimately, the resulting target policy model aligns with user intent, considers more semantic and logical relationships, and generates correct, efficient, and reliable code. Specifically, the steps include:

[0077] S3.1, Generate Trajectories. Sample a batch of code from dataset D, used for policy-supervised fine-tuning of a large language model. t represents D t The t-th code sample out of a total of T code samples, x t This indicates that the instruction text is being input. The standard code text output is then collected, followed by the code text generated by the reference policy model π0. And the masking strategy model π ψ The generated code text Let it be denoted as the set of trajectories.

[0078] S3.2, Calculate the KL penalty reward value. First, the predicted code y2 generated by the masked policy model is passed to the reward model to obtain the original reward value. The reward model evaluates the generated code and calculates the original reward value based on its learned evaluation criteria. Then, the instruction text x is input into the reference policy model and the masked policy model respectively to obtain the corresponding generated reference code y1 and predicted code y2. The reference code y1 and predicted code y2 are compared, and the KL divergence between them is calculated as the KL penalty term, which is used to regularize the reward function. Finally, the original reward value is combined with the KL penalty term to obtain the final KL penalty reward value. The KL penalty reward value comprehensively considers the quality of code generation and the difference from the reference policy.

[0079] Using the collected trajectory set τ t The formula for calculating the average KL penalty reward value for the t-th trajectory is as follows:

[0080]

[0081] in, R(s) represents the KL penalty / reward value. t ,a t ,y t ) represents the original reward value calculated by the reward model, KL(·) represents the KL divergence, and s t Let x be the state, representing the t-th instruction text input. t a t For the action, y represents the code text output corresponding to the large language model. t Indicates standard code text output π ψ (a t |s t () represents the code text generated by the masking strategy model. The probability, π0(a) t |s t () represents the code text generated by the reference strategy model. The probability, where β represents the KL penalty factor;

[0082] By constraining the reward model with KL penalty reward values, the target policy model can be prevented from deviating from the reference policy model, thus guiding the target policy model to generate code with more accurate syntax and semantics.

[0083] S3.3, based on the V-value function and Q-value function Calculate the advantage function In the NLPO algorithm, the advantage function serves to measure the quality of actions, guide model updates, and provide feedback to reduce variance. The calculation requires two basic functions: the state-action value function (Q-value function) and the state value function (V-value function).

[0084] The dominance function is typically expressed as the difference between the Q-value function and the V-value function, and is calculated using the following formula:

[0085]

[0086] in, Let s represent the advantage function, s represent the state (instruction input), a represent the action (model output), and π represent the model output. θ Represents the target strategy model. Represents the Q-value function, Let V represent the value function, and t represent the set of trajectories τ. t The t-th trajectory among the T trajectories.

[0087] The state-value function (V-value function) describes the expected total reward obtained by acting according to a specific strategy from the current state until the endpoint (terminal state) is reached. The V-value function is used to evaluate the value of the current state, providing the algorithm with a basis for selecting the optimal strategy under different states. The calculation formula is as follows:

[0088]

[0089] Among them, s τ Indicates a state, specifically referring to the instruction input for a particular sample, a τ Indicates that in state s τ The following action specifically refers to the model's code output, where y represents the standard code output, and R(s) τ ,a τ ,y) represents the reward value, and γ represents the discount coefficient.

[0090] The state-action value function (Q-value function) represents the total reward that can be expected to be obtained given a policy and a specific state-action pair. The Q-value function is used to evaluate the value of choosing a particular action in the current state. It provides the basis for choosing the optimal action in a given state, thereby guiding the target model to make an optimal decision. The calculation formula is as follows:

[0091]

[0092] Among them, s t Indicates the state, referring to the t-th instruction text input, a tIndicates the action, referring to the code text output corresponding to the model, y t R(s) represents standard code text output. t ,a t ,y t ) represents the reward value, and γ represents the discount factor. Let V represent the V-value function, and τ represent the set of trajectories.

[0093] S3.4, Update the target policy model parameters. In the NLPO algorithm, the policy network generates actions based on the probability distribution of the state to optimize the policy, while the value network estimates the expected reward of executing the policy in a specific state, providing a basis for evaluating policy performance and guiding policy improvement. NLPO optimizes the target policy model by maximizing the coordination between the policy and value functions, thus achieving a balance between policy optimization and state evaluation. The large model has multi-task versatility, so the policy network (PolicyNet) and value network (ValueNet) can share the same backbone network, reducing computational resources. By updating the model parameters, the model's performance and adaptability can be improved, enabling it to better capture semantic and logical relationships.

[0094] The objective function LOSS is constructed to train the objective policy model, and its calculation formula is as follows:

[0095]

[0096] in, Represents the policy function. Let m+1 represent the value function, m+1 represent the new target policy model, and m represent the current target policy model.

[0097] Policy function Let the objective function of the policy network be the action that guides the code generation model's decision-making, providing a probability distribution for choosing an action given a state. This can be represented as:

[0098]

[0099] Among them, clip(r) t (θ), 1-ε, 1+ε) is the cutoff, indicating that r is cut off. t (θ) is restricted to the range [1-ε, 1+ε], where ε represents the truncation range to prevent the difference between the model update and the truncation range from being too large. Importance weights are used to measure the current target strategy model. The deviation from the reference policy model π0 ensures that the model does not change excessively when it is updated, thus maintaining the stability of training. Describe the dominance function, if This indicates that the value of the code text is above average, maximizing... It will increase rt (θ), but it will not exceed 1+ε; conversely, if... maximize It will reduce r t (θ), but will not allow it to exceed 1-ε.

[0100] value function The objective function of the value network, which evaluates the long-term value of states or actions to help the code generation model learn and optimize its policies, is expressed as:

[0101]

[0102] Among them, s t V represents the t-th instruction text input. φ (s t ) indicates that in state s t The value function under the following conditions This represents the corresponding KL penalty / reward value.

[0103] S3.5, Update the mask policy model. After a certain number of rounds (μ rounds), copy the new target policy model. The parameters are determined, and irrelevant tokens with probabilities less than p are masked out. In other words, tokens with probabilities greater than a certain threshold p are retained, while the probabilities of other tokens are set to 0, resulting in a new masking strategy model. This allows us to retain important and high-probability terms, further optimizing the code generation capabilities of the masking strategy model.

[0104] S3.6 Repeat steps S3.1-S3.5 until the optimal target strategy model is obtained. Through multiple iterations and parameter updates, the model can be gradually optimized and aligned with human intent, thereby generating more accurate, efficient, and reliable code.

[0105] In summary, a code generation model fine-tuning method based on clustering and natural language policy optimization algorithms is proposed. By employing a K-center greedy algorithm to cluster code samples and then fine-tuning the pre-trained large language model, efficient fine-tuning for multiple tasks is achieved, reducing the fine-tuning cost of the large language model. Training a reward model further enhances the code generation capability and quality of the large language model, enabling it to generate code sequences that better conform to human ranking standards. By utilizing natural language policy optimization algorithms to introduce reference policies, masking policies, and target policies to fine-tune the policy of the large language model, a final target policy model is obtained. This fine-tuning process is more stable and the model converges faster, improving the efficiency of model fine-tuning and the accuracy of code generation, and better aligning with user intent.

[0106] Based on the same inventive concept, such as Figure 3 As shown, this embodiment of the invention also provides a code generation model fine-tuning device 300 based on clustering and natural language policy optimization algorithms, comprising a model preliminary fine-tuning module 310, a reward model training module 320, and a model policy fine-tuning module 330.

[0107] Among them, the model preliminary fine-tuning module 310 is used to cluster the preprocessed code samples using a clustering algorithm to obtain core code samples, and to perform preliminary supervised fine-tuning of the pre-trained large language model based on the core code samples;

[0108] The reward model training module 320 is used to collect code generation results of different large language models based on the same instruction text, and to construct a fine-tuning dataset after ranking the code generation results by quality. The reward model is then trained based on the fine-tuning dataset.

[0109] The model policy fine-tuning module 330 is used to perform policy-supervised fine-tuning on the large language model after preliminary supervised fine-tuning using a natural language policy optimization algorithm based on the trained reward model to obtain a code generation model. This includes: using the large language model after preliminary supervised fine-tuning as a reference policy model to provide initial performance and user intent benchmarks; using the final target model of policy-supervised fine-tuning as the target policy model; generating a mask policy model based on the target policy model using masking techniques to address instability during reinforcement learning training; calculating the original reward value using the trained reward model on the predicted code of the mask policy model; calculating the KL divergence between the predicted codes of the reference policy model and the mask policy model; calculating the KL penalty reward value based on the original reward value and KL divergence; constructing an objective function based on the KL penalty reward value to update the parameters of the target policy model and simultaneously update the parameters of the mask policy model; and obtaining the final target policy model after multiple iterations and parameter updates, which serves as the code generation model for generating accurate code aligned with user intent.

[0110] Based on the same inventive concept, this invention also provides a code generation model fine-tuning device based on clustering and natural language policy optimization algorithms, including a memory and a processor. The memory is used to store a computer program, and the processor is used to implement the above-mentioned code generation model fine-tuning method based on clustering and natural language policy optimization algorithms when the computer program is executed.

[0111] It should be noted that the code generation model fine-tuning device and the code generation model fine-tuning equipment based on clustering and natural language strategy optimization algorithms provided in the above embodiments belong to the same inventive concept as the code generation model fine-tuning method based on clustering and natural language strategy optimization algorithms. For details of their implementation process, please refer to the embodiments of the code generation model fine-tuning method based on clustering and natural language strategy optimization algorithms, which will not be repeated here.

[0112] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for fine-tuning a code generation model based on clustering and natural language strategy optimization algorithms, characterized in that, Includes the following steps: Clustering algorithms are used to cluster the preprocessed code samples to obtain core code samples, and the pre-trained large language model is initially supervised and fine-tuned based on the core code samples; We collect code generation results from different large language models based on the same instruction text, sort the code generation results by quality, construct a fine-tuning dataset, and train the reward model based on the fine-tuning dataset. A code generation model is obtained by using a natural language policy optimization algorithm to perform supervised fine-tuning on a large language model after initial supervised fine-tuning, based on a trained reward model. This includes: using the initially supervised fine-tuned large language model as a reference policy model to provide initial performance and user intent benchmarks; using the final target model of the policy-supervised fine-tuning as the target policy model; generating a masked policy model based on the target policy model using masking techniques to address instability during reinforcement learning training; calculating the original reward value for the predicted code of the masked policy model using the trained reward model; calculating the KL divergence between the predicted codes of the reference policy model and the masked policy model; calculating the KL penalty reward value based on the original reward value and KL divergence; constructing an objective function based on the KL penalty reward value to update the parameters of the target policy model and simultaneously updating the parameters of the masked policy model; and obtaining the final target policy model after multiple iterations and parameter updates, which serves as the code generation model for generating accurate code aligned with user intent.

2. The code generation model fine-tuning method based on clustering and natural language strategy optimization algorithms according to claim 1, characterized in that, The process of using a clustering algorithm to cluster the preprocessed code samples to obtain core code samples includes: The acquired code is preprocessed to obtain code samples. The preprocessing includes: filtering code fragments that do not meet the length requirements and deleting code with ambiguous variable names or syntax errors; encoding the code samples using the encoder of a pre-trained large language model and generating corresponding word vector representations; mapping the word vector representations to a low-dimensional space using dimensionality reduction techniques to obtain one-dimensional vectors as the representation vectors of the code samples; and using a clustering algorithm to cluster the code samples based on their representation vectors to obtain core code samples.

3. The code generation model fine-tuning method based on clustering and natural language strategy optimization algorithms according to claim 1 or 2, characterized in that, The clustering algorithm employs a K-center greedy algorithm, including: Initialize the non-center point set S1 and the center point set S2. Select any point from the non-center point set S1 as the initial center point and add it to the center point set S2. Each point represents a code sample. Calculate the minimum distance from each point in the non-center point set S1 to all points in the center point set S2, and select the point with the maximum distance in the non-center point set S1 as a candidate for the next center point; Remove the point with the largest distance from the non-center point set S1 and add it to the center point set S2; Repeat the above distance calculation and point movement steps until the center point set S2 contains K points as the clustering result; Based on the center point set S2 and a given ratio, select a set of samples that are close to the center point samples from the code samples as the core code samples.

4. The code generation model fine-tuning method based on clustering and natural language strategy optimization algorithms according to claim 1, characterized in that, The preliminary supervised fine-tuning of the pre-trained large language model based on core code samples includes: The core code sample is input into the pre-trained large language model for forward propagation to generate predicted code. The predicted code is then compared with the expected generated code and the loss function is calculated. The gradient of the loss function is backpropagated from the output layer to the input layer through backpropagation. An optimizer is used to update the model parameters to minimize the loss function, resulting in a large language model with basic code generation capabilities.

5. The code generation model fine-tuning method based on clustering and natural language strategy optimization algorithms according to claim 1, characterized in that, The training of the reward model based on the fine-tuning dataset includes: The fine-tuning dataset includes an initial dataset constructed from code generation results obtained by different large language models based on the same instruction text, and a code ranking dataset constructed after ranking the code generation results by quality. The code ranking dataset is used as a label, and the reward model is trained under supervision using the initial dataset and the code ranking dataset to maximize consistency with the quality ranking. This enables the reward model to calculate the relative order of predicted code quality and calculate the original reward value of the predicted code based on the relative order.

6. The code generation model fine-tuning method based on clustering and natural language strategy optimization algorithms according to claim 1, characterized in that, The formula for calculating the KL penalty reward value is as follows: in, R(s) represents the KL penalty / reward value. t ,a t ,y t ) represents the original reward value calculated by the reward model, KL(·) represents the KL divergence, and s t Let a be the state, representing the t-th instruction text input. t For actions, y represents the code text output of the large language model. t This represents standard code text output, π. ψ (a t |s t ) represents the probability of the code text generated by the masking strategy model, π0(a t |s t ) represents the probability of the code text generated by the reference policy model, and β represents the KL penalty factor; By constraining the reward model with KL penalty reward values, the target policy model is prevented from deviating from the reference policy model.

7. The code generation model fine-tuning method based on clustering and natural language strategy optimization algorithms according to claim 1, characterized in that, The process of constructing the objective function includes: First, a batch of code samples is sampled from dataset D, which is used for policy-supervised fine-tuning of large language models. t represents D t The t-th code sample out of a total of T code samples, x t This indicates that the instruction text is being input. The standard code text output is then collected, followed by the code text generated by the reference policy model π0. And the masking strategy model π ψ Generated code text Let it be denoted as the set of trajectories. Then, based on the V-value function representing the state value function. and the Q-value function representing the state-action value function. Calculate the advantage function The calculation formula is: in, Let s represent the advantage function, s represent the state (instruction input), a represent the action (model output), and π represent the model output. θ Represents the target strategy model. Let t represent the Q-value function, and t represent the set of trajectories τ. t The t-th trajectory among a total of T trajectories; Next, the objective function LOSS for the natural language policy optimization algorithm is constructed, and its calculation formula is as follows: in, Represents the policy function. Let m+1 represent the new target policy model, and m represent the current target policy model. The strategy function Let the objective function of the policy network be the action that guides the code generation model's decision-making, providing a probability distribution for choosing an action given a state. This can be represented as: Among them, clip(r) t (θ), 1-ε, 1+ε) is the cutoff, indicating that r is cut off. t (θ) is restricted to the range [1-ε, 1+ε], where ε represents the truncation range to prevent the difference between the model update and the truncation range from being too large. Importance weights are used to measure the current target strategy model. The deviation from the reference policy model π0 ensures that the model does not change excessively when it is updated, thus maintaining the stability of training. Denotes the dominance function, if This indicates that the value of the code text is above average, maximizing... It will increase r t (θ), but it will not exceed 1+ε; conversely, if... maximize It will reduce r t (θ), but will not allow it to exceed 1-ε; The value function Let be the objective function of the value network. The value network evaluates the long-term value of a state or action, helping the code generation model learn and optimize its policy, and is expressed as: Among them, s t V represents the t-th instruction text input. φ (s t ) indicates that in state s t The value function under the following conditions This represents the corresponding KL penalty / reward value.

8. The code generation model fine-tuning method based on clustering and natural language strategy optimization algorithms according to claim 1, characterized in that, The parameters of the synchronous update mask strategy model include: After each preset number of updates, the updated target strategy model is copied, retaining the words in the vocabulary with a probability greater than or equal to a preset threshold and masking irrelevant words with a probability less than the preset threshold, thus obtaining a new masking strategy model.

9. A code generation model fine-tuning device based on clustering and natural language strategy optimization algorithms, characterized in that, include: The module includes a preliminary model fine-tuning module, a reward model training module, and a model strategy fine-tuning module. The model preliminary fine-tuning module is used to cluster the preprocessed code samples using a clustering algorithm to obtain core code samples, and to perform preliminary supervised fine-tuning of the pre-trained large language model based on the core code samples. The reward model training module is used to collect code generation results of different large language models based on the same instruction text, and construct a fine-tuning dataset after ranking the code generation results by quality, and train the reward model based on the fine-tuning dataset; The model policy fine-tuning module is used to perform supervised fine-tuning of the large language model after preliminary supervised fine-tuning using a natural language policy optimization algorithm on the trained reward model to obtain a code generation model. This includes: using the pre-trained, fine-tuned large language model as a reference policy model to provide initial performance and user intent benchmarks; using the final target model of the policy-supervised fine-tuning as the target policy model; generating a masked policy model based on the target policy model using masking techniques to address instability during reinforcement learning training; calculating the original reward value for the predicted code of the masked policy model using the trained reward model; calculating the KL divergence between the predicted codes of the reference policy model and the masked policy model; calculating the KL penalty reward value based on the original reward value and KL divergence; constructing an objective function based on the KL penalty reward value to update the parameters of the target policy model and simultaneously update the parameters of the masked policy model; and obtaining the final target policy model after multiple iterations and parameter updates, which serves as the code generation model for generating accurate code aligned with user intent.

10. A code generation model fine-tuning device based on clustering and natural language strategy optimization algorithms, comprising a memory and a processor, wherein the memory is used to store computer programs, characterized in that, The processor is configured to implement, when executing the computer program, the code generation model fine-tuning method based on clustering and natural language strategy optimization algorithms as described in any one of claims 1-8.

Images