Code generation model training method and device, code generation method and device and electronic equipment

Abstract

The invention provides a code generation model training method and device, a code generation method and device and electronic equipment. The method comprises the steps that first sample code data, second sample code data and a pre-training model are obtained; based on the first sample code data, performing autoregression pre-training on a parameter fine tuning module arranged in a key layer of a pre-training model to obtain an initial code generation model; and training a parameter fine tuning module of the initial code generation model based on the second sample code data and a tag code generation result of the second sample code data to obtain the code generation model. The process parameter fine tuning module replaces part of parameters of a pre-training model with fine-tunable low-rank representation through low-rank matrix decomposition, so that the calculation cost during training is reduced while the capacity of the model is kept; through autoregression training, the model can better understand the context information of the code, so that the code conforming to the special code style and logic of the user is more accurately generated, and the readability of the code is improved.

Description

Technical Field

[0001] The present invention relates to the field of code generation technology, and in particular to a code generation model training method, a code generation method, a device and an electronic device. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, the application of large language models in natural language processing is becoming increasingly widespread, and more and more AI products are being integrated into daily work and life. For example, in the work of software developers, much of the coding involves common processing logic. If programming software can automatically generate code that conforms to the current programming style based on context, engineers' work efficiency will be greatly improved. For common coding-related questions, users can ask the large code model questions using natural language, and the model will provide relatively professional answers. In particular, the inclusion of specific prompts in the questions will make the large code model's answers even more accurate.

[0003] Currently, most large-code models rely on training data from open-source code, which is cleaned and processed before training. However, proprietary user code is often not covered, making it a key concern for large-code models to handle this. Furthermore, training models directly at user sites is resource-intensive and time-consuming, making it difficult to implement. Summary of the Invention

[0004] The present invention provides a code generation model training method, a code generation method, an apparatus and an electronic device, which are used to solve the defects in the prior art that the user's proprietary code is not covered in the training data, and that direct model training at the user's site is resource-consuming, time-consuming and difficult to implement.

[0005] The present invention provides a method for training a code generation model, comprising the following steps: Obtaining first sample code data, second sample code data, and a pre-trained model; Based on the first sample code data, performing autoregressive pre-training on a parameter fine-tuning module provided in a key layer of the pre-trained model to obtain an initial code generation model; Based on the second sample code data and the label code generation result of the second sample code data, the parameter fine-tuning module of the initial code generation model is trained to obtain the code generation model.

[0006] According to a code generation model training method provided by the present invention, based on the first sample code data, the parameter fine-tuning module set in the key layer of the pre-trained model is subjected to autoregressive pre-training to obtain an initial code generation model, including: Dividing the first sample code data into a prefix code, a middle tag code and a suffix code; Determining a current intermediate code predicted by the pre-trained model based on the prefix code and the suffix code, and determining a next intermediate code of the current intermediate code predicted by the pre-trained model based on the prefix code, the suffix code, and the current intermediate code, taking the next intermediate code as the current intermediate code, and executing the step of determining the next intermediate code based on the prefix code, the suffix code, and the current intermediate code until an end marker is encountered; Based on the difference between the current intermediate code and the intermediate label code, and the difference between the next intermediate code and the intermediate label code, the autoregressive generation loss is determined, and the parameter fine-tuning module is iterated based on the autoregressive generation loss to obtain the initial code generation model.

[0007] According to a code generation model training method provided by the present invention, the parameter fine-tuning module of the initial code generation model is trained based on the second sample code data and the label code generation result of the second sample code data to obtain the code generation model, including: Inputting the second sample code data into the initial code generation model to obtain a predicted code generation result output by the initial code generation model, and determining a reconstruction loss based on a difference between the predicted code generation result and the labeled code generation result; Based on the reconstruction loss, parameter iteration is performed on a parameter fine-tuning module of the initial code generation model to obtain the code generation model.

[0008] According to a code generation model training method provided by the present invention, the parameter fine-tuning module of the initial code generation model is iterated based on the reconstruction loss to obtain the code generation model, including: Get the standard code generation model; Based on the second sample code data, performing code generation training on the standard code generation model to obtain a first code output distribution; Based on the second sample code data, performing code generation training on the initial code generation model to obtain a second code output distribution; Based on the difference between the first code output distribution and the second code output distribution, a distribution loss is determined, and based on the distribution loss and the reconstruction loss, parameter iteration is performed on a parameter fine-tuning module of the initial code generation model to obtain the code generation model.

[0009] According to a code generation model training method provided by the present invention, determining the distribution loss based on the difference between the first code output distribution and the second code output distribution includes: determining the first code output distribution as a prior distribution and determining the second code output distribution as a latent variable distribution; The distribution loss is determined based on a difference between the prior distribution and the latent variable distribution.

[0010] According to a code generation model training method provided by the present invention, the step of acquiring the second sample code data includes: obtaining second original code data; Parsing the second original code data to obtain an abstract syntax tree corresponding to the second original code data; Traversing the abstract syntax tree, and performing code optimization on the second original code data corresponding to the abstract syntax tree to obtain the second sample code data; The code optimization step includes at least one of static code analysis, deletion of redundant nodes, constant folding, dead code elimination and code conversion.

[0011] The present invention also provides a code generation method, comprising the following steps: Get the context code of the code to be completed; Inputting the context code into the code generation model to obtain the target completion code output by the code generation model; The code generation model is obtained by executing the training method based on the above code generation model.

[0012] The present invention also provides a training device for a code generation model, comprising: A first acquisition unit, configured to acquire first sample code data, second sample code data, and a pre-trained model; A first training unit is configured to perform autoregressive pre-training on the parameter fine-tuning module provided in the key layer of the pre-training model based on the first sample code data to obtain an initial code generation model; The second training unit is used to train the parameter fine-tuning module of the initial code generation model based on the second sample code data and the label code generation result of the second sample code data to obtain the code generation model.

[0013] The present invention also provides a code generating device, comprising: A second acquiring unit is used to acquire the context code of the code to be completed; An input unit, configured to input the context code into the code generation model to obtain a target completion code output by the code generation model; The code generation model is obtained by executing the training method based on the above code generation model.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, it implements the training method of the code generation model as described above, or implements the code generation method.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the training method of the code generation model as described in any one of the above, or implements the code generation method.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the training method of the code generation model as described above, or implements the code generation method.

[0017] The present invention provides a training method, a code generation method, a device and an electronic device for a code generation model, which obtains first sample code data, second sample code data and a pre-trained model, and then performs autoregressive pre-training on a parameter fine-tuning module set at a key layer of the pre-trained model based on the first sample code data to obtain an initial code generation model. Finally, based on the second sample code data and the label code generation result of the second sample code data, the parameter fine-tuning module of the initial code generation model is trained to obtain a code generation model. On the one hand, the parameter fine-tuning module replaces some parameters of the pre-trained model with fine-tunable low-rank representations through low-rank matrix decomposition, thereby reducing the computational cost during training while maintaining the model capacity; on the other hand, through autoregressive training, the model can better understand the contextual information of the code, thereby more accurately generating code that conforms to the user's proprietary code style and logic, improving the readability of the code, and then training the initial code generation model through the second sample code data and its label code generation result. The initial code generation model can more accurately learn the style and logic of the user's proprietary code, thereby generating higher quality code. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a flowchart of the training method of the code generation model provided by the present invention.

[0020] Figure 2 This is one of the flow charts of step 120 in the code generation model training method provided by the present invention.

[0021] Figure 3 This is the second flowchart of step 120 in the code generation model training method provided by the present invention.

[0022] Figure 4 1 is a flow chart of step 130 in the code generation model training method provided by the present invention.

[0023] Figure 5 This is one of the flow charts of step 132 in the code generation model training method provided by the present invention.

[0024] Figure 6 This is the second flowchart of step 132 in the code generation model training method provided by the present invention.

[0025] Figure 7 It is a flowchart of the code generation method provided by the present invention.

[0026] Figure 8 It is a structural diagram of the training device of the code generation model provided by the present invention.

[0027] Figure 9 It is a structural diagram of the code generating device provided by the present invention.

[0028] Figure 10 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0030] The terms "first," "second," and the like in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type.

[0031] Among existing model fine-tuning techniques, LoRA (Low-Rank Adaptation) and full-parameter fine-tuning are two common approaches. LoRA is an efficient fine-tuning technique particularly suitable for scenarios with limited computing resources and scarce labeled data. It decomposes the parameters of a large pre-trained model into tunable and non-tunable components by introducing a low-rank matrix. Specifically, LoRA uses the low-rank matrix to capture the latent representation of the input data, which is then used to guide the pre-trained model to learn unique features for a specific task. This enables LoRA to effectively extract useful features from unlabeled data without requiring large amounts of labeled data. This feature is particularly suitable for fields such as natural language processing and code generation, where data sources are diverse but difficult to label. By automatically extracting patterns and features from the data, LoRA can effectively enhance the performance of pre-trained models for specific tasks, thereby improving the model's task adaptability and generalization capabilities. Compared to traditional full-parameter fine-tuning, LoRA achieves task adaptation by adjusting a small number of model parameters, reducing the demand for computing resources and storage. This makes LoRA ideal for scenarios with scarce labeled data, large models, or limited computing resources.

[0032] Full parameter fine-tuning is another commonly used method for fine-tuning large models. Unlike LoRA, full parameter fine-tuning involves adjusting all parameters of the pre-trained model. This means the model can be fully optimized for a specific task, fully capturing subtle differences in the data. This is particularly suitable for large-scale scenarios with sufficient labeled data. In this case, the model has sufficient training data and computing resources, and the full potential of the pre-trained model can be fully realized by adjusting all parameters.

[0033] LoRA fine-tunes some parameters of the pre-trained model by introducing a low-rank matrix. This method can effectively adjust model parameters while reducing computing resources, but its constrained nature may not be sufficient when dealing with certain complex tasks or tasks that require fine-grained adjustments. In contrast, full parameter fine-tuning is more flexible and allows all parameters of the model to be adjusted, thereby more finely capturing subtle differences in the task. However, the fine-tuning effect of LoRA depends largely on the basic capabilities of the pre-trained model. If the pre-trained model performs poorly in a specific task or field, LoRA may find it difficult to significantly improve performance because it can only make limited adjustments to the existing model representation.

[0034] Full parameter fine-tuning requires updating all parameters in the model, which imposes enormous computational and memory requirements. This can dramatically increase resource consumption, especially when working with large pre-trained models. Because all model parameters are adjusted, full parameter fine-tuning can cause the model to lose its general knowledge in some tasks, reducing its performance in other tasks.

[0035] In summary, LoRA and full-parameter fine-tuning each have their own applicable scenarios. LoRA, with its efficiency and resource-friendliness, is suitable for handling scarce labeled data and multi-task environments, while full-parameter fine-tuning provides greater task adaptability and accuracy when labeled data is sufficient and computing resources are abundant. Large code models are typically trained based on LoRA or full-parameter fine-tuning methods. However, user-specific code is often not covered, so how to enable large code models to handle user-specific code has become a core concern for users. Furthermore, direct model training at the user's site is resource-intensive and time-consuming, making it difficult to implement.

[0036] Based on the above problems, the present invention provides a training method for a code generation model. Figure 1 This is a flow chart of the training method of the code generation model provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0037] Step 110: Obtain first sample code data, second sample code data, and a pre-trained model.

[0038] Specifically, first sample code data, second sample code data and a pre-trained model are obtained, wherein the first sample code data and the second sample code data are both code data to be trained by the user, and the first sample code data can be the same as the second sample code data or different from the second sample code data, which is not specifically limited in this embodiment of the present invention.

[0039] The pre-trained model may be a large language model, and the large language model may be a pre-trained language model such as the iFlytek Spark model (IFlytekSpark), the XLNet model (extreme Multi-label Learning Network), or the ROBERTa model (RobustlyOptimized BERT approach), which is not specifically limited in the embodiment of the present invention.

[0040] Here, the first sample code data can be obtained by cleaning, standardizing and data enhancement of the original code data. Its core purpose is to remove duplicate data, outliers and noise in the original code data, and improve data quality through data correction and normalization to ensure data consistency. This step usually combines automated data cleaning techniques and statistical methods, such as denoising, anomaly detection, and duplicate record elimination based on distance metrics. For outliers, density estimation-based algorithms can be used for detection and automatic filling or interpolation techniques can be used for correction. This process ensures that subsequent model training is based on high-quality sample code data, thereby improving the accuracy and reliability of code generation.

[0041] The second sample code data can be a high-quality supervised dataset generated based on a specific task, so as to ensure that the training data covers the needs of a specific field and meets the input-output structural requirements of the model in subsequent training. For specific tasks, such as code generation or text classification, a supervised dataset is generated through automatic annotation or semi-automatic annotation tools in the format of {"Input": xxx, "Target": xxx}. The annotation process is usually combined with knowledge in a specific field to ensure that the data has sufficient semantic information and contextual relevance. Furthermore, for the original code data, the second sample code data can be obtained by parsing the Abstract Syntax Tree (AST) and annotating with specific rules.

[0042] Step 120 : Based on the first sample code data, perform autoregressive pre-training on the parameter fine-tuning module set in the key layer of the pre-training model to obtain an initial code generation model.

[0043] Specifically, after obtaining the first sample code data, autoregressive pre-training can be performed on the parameter fine-tuning module set in the key layer of the pre-trained model based on the first sample code data to obtain an initial code generation model. The parameter fine-tuning module can be LoRA, QLoRA (Quantized Low-Rank Adaptation), or Adapter Tuning, etc., which are not specifically limited in this embodiment of the present invention.

[0044] It should be noted that the key layer of the pre-training model can be an attention layer, a feed-forward network layer (FFN), or a specific task layer, etc., and the embodiment of the present invention does not specifically limit this.

[0045] In addition, a parameter fine-tuning module can be set for each layer of the pre-trained model, so that the parameters can be efficiently fine-tuned in subsequent model training. The embodiment of the present invention does not make specific limitations on this.

[0046] It should be noted that the core idea of LoRA is to replace some parameters of the pre-trained model with fine-tunable low-rank representations through low-rank matrix decomposition, thereby reducing the computational cost during training while maintaining the capacity of the pre-trained model.

[0047] In this phase, the pre-trained model is further optimized using autoregressive pre-training methods, and multiple LoRA modules are constructed. These modules capture common task invariants and provide a foundation for specific tuning in subsequent tasks. This step aims to prepare LoRA parameters that are universal and transferable for downstream tasks. During this phase, data is combed using the Fill-In-The-Middle (FIM) technique, which allows for filling in the middle of a complete code segment. The generation of each intermediate code segment depends not only on the prefix and suffix codes, but also on previously generated intermediate codes.

[0048] It can be understood that based on the first sample code data, the parameter fine-tuning module set in the key layer of the pre-trained model is autoregressively pre-trained to obtain an initial code generation model. On the one hand, the parameter fine-tuning module replaces part of the parameters of the pre-trained model with fine-tunable low-rank representations through low-rank matrix decomposition, thereby reducing the computational cost during training while maintaining the model capacity; on the other hand, through autoregressive training, the model can better understand the contextual information of the code, thereby more accurately generating code that conforms to the user's proprietary code style and logic, thereby improving the readability of the code.

[0049] Step 130 : Based on the second sample code data and the label code generation result of the second sample code data, the parameter fine-tuning module of the initial code generation model is trained to obtain the code generation model.

[0050] Specifically, after obtaining the second sample code data, the parameter fine-tuning module of the initial code generation model may be trained based on the second sample code data and the label code generation result of the second sample code data to obtain the code generation model.

[0051] Here, based on the second sample code data and the label code generation result of the second sample code data, the parameter fine-tuning module of the initial code generation model is trained to obtain the code generation model. Specifically, the second sample code data can be input into the initial code generation model to obtain the predicted code generation result output by the initial code generation model, and then the reconstruction loss is determined based on the difference between the predicted code generation result and the label code generation result. The parameter fine-tuning module of the initial code generation model is iterated to obtain the code generation model. Alternatively, the base model can be used as the teacher model and the initial code generation model as the student model. The distribution loss is determined based on the difference between the first code output distribution output by the teacher model and the second code output distribution output by the student model. Then, based on the reconstruction loss and the distribution loss, the parameter fine-tuning module of the initial code generation model is iterated to obtain the code generation model. The embodiment of the present invention does not make any specific limitations on this.

[0052] It can be understood that, on the one hand, the parameter fine-tuning module replaces some parameters of the pre-trained model with fine-tunable low-rank representations through low-rank matrix decomposition, thereby once again reducing the computational cost during training while maintaining the model capacity; on the other hand, the initial code generation model is trained by the second sample code data and its labeled code generation results. The initial code generation model can learn the style and logic of the user's proprietary code more accurately, thereby generating higher quality code.

[0053] The method provided by the embodiment of the present invention obtains first sample code data, second sample code data and a pre-trained model, and then, based on the first sample code data, performs autoregressive pre-training on a parameter fine-tuning module set in a key layer of the pre-trained model to obtain an initial code generation model. Finally, based on the second sample code data and the label code generation result of the second sample code data, the parameter fine-tuning module of the initial code generation model is trained to obtain a code generation model. On the one hand, the parameter fine-tuning module replaces some parameters of the pre-trained model with fine-tunable low-rank representations through low-rank matrix decomposition, thereby reducing the computational cost during training while maintaining the model capacity; on the other hand, through autoregressive training, the model can better understand the contextual information of the code, thereby more accurately generating code that conforms to the user's proprietary code style and logic, improving the readability of the code, and then the initial code generation model is trained by the second sample code data and its label code generation result. The initial code generation model can more accurately learn the style and logic of the user's proprietary code, thereby generating higher quality code.

[0054] Based on the above embodiments, Figure 2 This is one of the flow charts of step 120 in the code generation model training method provided by the present invention. Figure 3This is the second flow chart of step 120 in the code generation model training method provided by the present invention, as shown in FIG. Figure 2 、 Figure 3 As shown, step 120 includes: Step 121: Divide the first sample code data into a prefix code, an intermediate label code, and a suffix code; Step 122: Determine the current intermediate code predicted by the pre-trained model based on the prefix code and the suffix code, and determine the next intermediate code of the current intermediate code predicted by the pre-trained model based on the prefix code, the suffix code, and the current intermediate code, use the next intermediate code as the current intermediate code, and execute the step of determining the next intermediate code based on the prefix code, the suffix code, and the current intermediate code until an end marker is encountered; Step 123: Determine the autoregressive generation loss based on the difference between the current intermediate code and the intermediate label code, and the difference between the next intermediate code and the intermediate label code, and perform parameter iteration on the parameter fine-tuning module based on the autoregressive generation loss to obtain the initial code generation model.

[0055] Specifically, the first sample code data can be obtained after the user's code data to be trained undergoes unsupervised data cleaning services. The base model is equivalent to a pre-trained model. The pre-trained model can be pre-trained with a parameter fine-tuning module (LoRA module) based on the first sample code data to obtain the pre-trained weight parameters of LoRA in the pre-trained model, as follows: The first sample code data is divided into prefix code, middle tag code and suffix code. For example, the input sequence X is divided into three parts: Prefix: The missing part in the middle: Suffix: .

[0056] The above corresponds to the embodiment of the present invention. Accordingly, the first sample code data corresponds to the input sequence X, the prefix code corresponds to the pre-context, and the suffix code corresponds to the post-context.

[0057] Then, based on the prefix code and suffix code, the current intermediate code predicted by the pre-trained model is determined, whose goal is to pass the given prefix code and suffix code , to predict the intermediate code , formally expressed as: Furthermore, based on the prefix code, suffix code, and current intermediate code, the next intermediate code of the current intermediate code predicted by the pre-trained model is determined, the next intermediate code is used as the current intermediate code, and the steps of determining the next intermediate code based on the prefix code, suffix code, and current intermediate code are performed until the end marker is encountered. That is, the generated intermediate code depends on the code data generated step by step from left to right by the pre-trained model, and its form is: In summary, the above process means that the generation of each intermediate code depends not only on the prefix code and the suffix code, but also on the intermediate codes that have been generated previously.

[0058] Finally, based on the difference between the current intermediate code and the intermediate label code, as well as the difference between the next intermediate code and the intermediate label code, the autoregressive generation loss is determined, and the parameter fine-tuning module is iterated based on the autoregressive generation loss. The pre-trained model that completes the parameter iteration is used as the initial code generation model.

[0059] It can be understood that the greater the difference between the current intermediate code and the intermediate label code, the greater the autoregressive generation loss; the smaller the difference between the current intermediate code and the intermediate label code, the smaller the autoregressive generation loss.

[0060] Similarly, the greater the difference between the next intermediate code and the intermediate label code, the greater the autoregressive generation loss; the smaller the difference between the next intermediate code and the intermediate label code, the smaller the autoregressive generation loss.

[0061] It is understandable that this process uses the first sample code data for training. In order to better support the code completion task, the model is allowed to fill in the "middle position", which is more in line with the developer's actual programming scenario.

[0062] The method provided by the embodiment of the present invention, on the one hand, by calculating the difference between the current intermediate code and the label intermediate code, the pre-trained model can accurately identify the deviation between the generated code content and the target, thereby optimizing the generated content in a targeted manner in subsequent iterations; on the other hand, in the autoregressive generation process, the pre-trained model not only considers the prefix code and the suffix code, but also refers to the previously generated intermediate code, so that the generated code content gradually approaches the target code, thereby improving the accuracy and coherence of code generation.

[0063] Based on the above embodiments, Figure 4 1 is a flow chart of step 130 in the code generation model training method provided by the present invention, as shown in FIG. Figure 4 As shown, step 130 includes: Step 131: input the second sample code data into the initial code generation model to obtain a predicted code generation result output by the initial code generation model, and determine a reconstruction loss based on a difference between the predicted code generation result and the labeled code generation result; Step 132 : Based on the reconstruction loss, perform parameter iteration on the parameter fine-tuning module of the initial code generation model to obtain the code generation model.

[0064] Specifically, the second sample code data may be input into the initial code generation model to obtain a predicted code generation result output by the initial code generation model, and the reconstruction loss may be determined based on the difference between the predicted code generation result and the label code generation result.

[0065] It can be understood that the greater the difference between the predicted code generation result and the labeled code generation result, the greater the reconstruction loss; the smaller the difference between the predicted code generation result and the labeled code generation result, the smaller the reconstruction loss.

[0066] After determining the reconstruction loss, the parameter fine-tuning module of the initial code generation model can be iterated based on the reconstruction loss, and the initial code generation model after the parameter iteration is completed is used as the code generation model.

[0067] Among them, the formula of reconstruction loss is as follows: in, represents the reconstruction loss, Represents the predicted code generation result, Indicates the label code generation result, represents the number of samples, Represents the mean squared error between the predicted code generation result and the labeled code generation result.

[0068] Based on the above embodiments, Figure 5 This is one of the flow charts of step 132 in the code generation model training method provided by the present invention. Figure 6 This is the second flow chart of step 132 in the code generation model training method provided by the present invention, as shown in FIG. Figure 5 、 Figure 6 As shown, step 132 includes: Step 1321, obtaining a standard code generation model; Step 1322: Perform code generation training on the standard code generation model based on the second sample code data to obtain a first code output distribution; Step 1323: Perform code generation training on the initial code generation model based on the second sample code data to obtain a second code output distribution; Step 1324: Determine the distribution loss based on the difference between the first code output distribution and the second code output distribution, and perform parameter iteration on the parameter fine-tuning module of the initial code generation model based on the distribution loss and the reconstruction loss to obtain the code generation model.

[0069] Specifically, the second sample code data may be supervised data obtained by performing a supervised data cleaning service on the user's code data to be trained, and performing a packaging service on the supervised data, that is, the second sample code data may be packaged supervised data. The base model in the figure is the initial code generation model, which includes the LoRA pre-training weight parameters obtained by training in step 123. Based on the LoRA pre-training weight parameters in the initial code generation model, the LoRA SFT training service is performed to obtain a LoRA fine-tuned model, that is, a code generation model, as follows: The embodiment of the present invention performs training by minimizing the difference between the student model and the teacher model. The student model fine-tuned using LoRA can guide the optimization process by adjusting only a small number of low-rank matrix parameters, while relying on the output distribution of the student model and the teacher model, thereby improving the model effect of special scenarios and significantly reducing computational overhead and memory usage.

[0070] LoRA supervised fine-tuning, such as LoRA SFT (Supervised Fine-Tuning), aims to fine-tune LoRA modules using supervised data to better adapt the model to specific tasks. During LoRA SFT, the model parameters are divided into two parts: non-adjustable and adjustable. The non-adjustable part retains the knowledge inherent in the pre-trained model, while the low-rank matrix in the LoRA module is fine-tuned using supervised data. Using LoRA for fine-tuning allows for lightweight task adaptation based on existing large pre-trained models. This approach enables rapid adaptation to new tasks through local adjustments, building on the strength of the pre-trained model.

[0071] Specifically, a standard code generation model can be obtained. The standard code generation model can be a pre-trained language model such as the iFlytek Spark model, the XLNet model, the ROBERTa model, etc., which is not specifically limited in this embodiment of the present invention.

[0072] It's understandable that the standard code generation model is equivalent to the teacher model, and the initial code generation model is equivalent to the student model. Combined with knowledge distillation, the smaller student model (the initial code generation model) can learn from the teacher model (the standard code generation model) and then be further fine-tuned through LoRA to optimize its performance on specific tasks. This combined approach not only improves the model's efficiency but also its task adaptability.

[0073] Specifically, based on the second sample code data, the standard code generation model can be trained for code generation to obtain a first code output distribution, and then based on the second sample code data, the initial code generation model can be trained for code generation to obtain a second code output distribution.

[0074] Then, based on the difference between the first code output distribution and the second code output distribution, the distribution loss is determined, and the formula for the distribution loss is as follows: in, represents the distribution loss, represents the first code output distribution, represents the second code output distribution.

[0075] It should be noted that the distribution loss can be KL divergence loss (Kullback-Leibler DivergenceLoss), given two probability distributions and , KL divergence is used to measure the "information loss" from one probability distribution to another. KL divergence can be used to measure the difference between the data distribution generated by the model and the true data distribution, that is, the first code output distribution is determined as the prior distribution (true data distribution), and the second code output distribution is determined as the latent variable distribution. Based on the difference between the prior distribution and the latent variable distribution, the distribution loss is determined.

[0076] It can be understood that the greater the difference between the first code output distribution and the second code output distribution, the greater the distribution loss; and the smaller the difference between the first code output distribution and the second code output distribution, the smaller the distribution loss.

[0077] KL divergence is an important part of the loss function. When training the initial code generation model, the goal of the model is to minimize the KL divergence so that the distribution in the latent space (latent variable distribution) is close to the prior distribution.

[0078] It should be noted that the proportion of the student model updated from the teacher model can also be determined by setting the KL divergence update ratio.

[0079] After obtaining the distribution loss and reconstruction loss, the target loss can be determined based on the distribution loss and reconstruction loss, and the parameter fine-tuning module of the initial code generation model can be iterated based on the target loss. The initial code generation model after the parameter iteration is used as the code generation model.

[0080] Among them, the formula of target loss is as follows: in, represents the target loss, represents the reconstruction loss, represents the distribution loss, i.e., the KL divergence between the latent variable distribution and the prior distribution, is a hyperparameter that balances the reconstruction loss and the distribution loss.

[0081] The method provided by the embodiments of this invention, on the one hand, reduces the size of the initial code generation model, but through distillation learning, the small model retains most of the reasoning capabilities of the standard code generation model, maintaining high accuracy. On the other hand, the KL divergence can be used to measure the difference between the data distribution generated by the model and the real data distribution. During training tasks, the KL divergence is used to measure the information loss between the two distributions, thereby guiding the initial code generation model to learn better.

[0082] Based on the above embodiment, the step of obtaining the second sample code data includes: Step 310, obtaining second original code data; Step 320: Parse the second original code data to obtain an abstract syntax tree corresponding to the second original code data; Step 330: traverse the abstract syntax tree and perform code optimization on the second original code data corresponding to the abstract syntax tree to obtain the second sample code data; The code optimization step includes at least one of static code analysis, deletion of redundant nodes, constant folding, dead code elimination and code conversion.

[0083] Specifically, the second original code data may be acquired, and then the second original code data may be parsed to obtain an abstract syntax tree corresponding to the second original code data.

[0084] After obtaining the abstract syntax tree, the abstract syntax tree may be traversed, and code optimization may be performed on the second original code data corresponding to the abstract syntax tree to obtain second sample code data.

[0085] The code optimization step includes at least one of static code analysis, deletion of redundant nodes, constant folding, dead code elimination and code conversion.

[0086] Static code analysis is the process of checking and analyzing code without executing it to identify potential errors, defects, security vulnerabilities, or non-compliance with programming standards. It is typically implemented through syntax checking, semantic checking, code style checking, and complexity analysis.

[0087] Syntax checking checks whether the code complies with the language's grammatical rules. Semantic checking checks whether the code's logic is sound, for example, whether variables are uninitialized or whether types match. Code style checking checks whether the code complies with specific coding standards, such as indentation and naming conventions. Complexity analysis assesses the complexity of the code, such as the nesting depth of functions and the complexity of loops.

[0088] Redundant node removal refers to removing nodes in the abstract syntax tree that have no practical impact on the program's functionality. These nodes may include duplicate variable declarations, redundant parentheses (such as unnecessary nested parentheses), and redundant expressions.

[0089] Constant folding refers to precalculating the values of constants in expressions during compilation or code optimization and replacing the resulting values in the code. Dead code is code that is never executed during program execution. Dead code elimination removes these unexecuted sections from the code. Code transformation refers to structural adjustments or rewriting of code to improve performance, readability, or compatibility.

[0090] After the second sample code data, the second sample code data can be converted into an input format that can be directly processed by the model, including vocabulary generation, data serialization, tokenization, and feature conversion.

[0091] This step includes vocabulary conversion, tokenization, subword segmentation processing based on the pre-trained model, and serializing the text or code data into a format suitable for neural network input. The code data is then enhanced, randomly shuffled, batched, and packaged to improve training efficiency and prevent overfitting.

[0092] The method provided by the embodiment of the present invention can significantly improve the overall quality of the second sample code data through static code analysis, deletion of redundant nodes, constant folding, dead code elimination and code conversion.

[0093] Considering that after the code generation model is trained, the code generation model can be applied to generate code.

[0094] Based on any of the above embodiments, the present invention provides a code generation method, Figure 7 It is a flow chart of the code generation method provided by the present invention, such as Figure 7 As shown, the method includes: Step 710: Obtain the context code of the code to be completed; Step 720: Input the context code into the code generation model to obtain the target completion code output by the code generation model; The code generation model is obtained by executing the training method based on the above code generation model.

[0095] Specifically, you can obtain the context code for the code to be completed. The code to be completed refers to code snippets that are unfinished, have missing parts, or require further supplementation. The context code for the code to be completed refers to the existing code sections associated with the code to be completed, providing background information and the runtime environment for the code to be completed. This context code helps developers understand the purpose, location, and dependencies of the code to be completed, allowing them to complete the code more accurately.

[0096] After obtaining the context code, it can be input into the code generation model to generate the target completion code. The target completion code refers to the code snippet that the developer expects to generate in the code completion task, which meets specific functional or logical requirements.

[0097] Here, the training steps of the code generation model are as follows: Obtain first sample code data, second sample code data, and a pre-trained model, wherein the first sample code data and the second sample code data are both code data to be trained by the user, and the first sample code data may be the same as or different from the second sample code data, which is not specifically limited in this embodiment of the present invention.

[0098] The pre-trained model may be a large language model, and the large language model may be a pre-trained language model such as the iFlytek Spark large model, the XLNet model, or the ROBERTa model. This embodiment of the present invention does not specifically limit this.

[0099] Here, the first sample code data can be obtained by cleaning, standardizing and data enhancement of the original code data. Its core purpose is to remove duplicate data, outliers and noise in the original code data, and improve data quality through data correction and normalization to ensure data consistency. This step usually combines automated data cleaning techniques and statistical methods, such as denoising, anomaly detection, and duplicate record elimination based on distance metrics. For outliers, density estimation-based algorithms can be used for detection and automatic filling or interpolation techniques can be used for correction. This process ensures that subsequent model training is based on high-quality sample code data, thereby improving the accuracy and reliability of code generation.

[0100] The second sample code data can be a high-quality supervised data set generated based on a specific task, so as to ensure that the training data covers the needs of a specific field and meets the input-output structural requirements of the model in subsequent training. For specific tasks, such as code generation or text classification, a supervised data set is generated through automatic annotation or semi-automatic annotation tools in the format of {"Input": xxx, "Target": xxx}. The annotation process is usually combined with knowledge in a specific field to ensure that the data has sufficient semantic information and contextual relevance. Furthermore, for the original code data, the second sample code data can be obtained by parsing the abstract syntax tree and annotating with specific rules.

[0101] After obtaining the first sample code data, autoregressive pre-training can be performed on the parameter fine-tuning module set in the key layer of the pre-trained model based on the first sample code data to obtain an initial code generation model. The parameter fine-tuning module can be LoRA, QLoRA, or Adapter Tuning, etc., which is not specifically limited in this embodiment of the present invention.

[0102] It should be noted that the key layer of the pre-training model can be an attention layer, a feedforward network layer, or a specific task layer, etc., and the embodiment of the present invention does not make specific limitations on this.

[0103] In addition, a parameter fine-tuning module can be set for each layer of the pre-trained model, so that the parameters can be efficiently fine-tuned in subsequent model training. The embodiment of the present invention does not make specific limitations on this.

[0104] It should be noted that the core idea of LoRA is to replace some parameters of the pre-trained model with fine-tunable low-rank representations through low-rank matrix decomposition, thereby reducing the computational cost during training while maintaining the capacity of the pre-trained model.

[0105] In this phase, the pre-trained model is further optimized using autoregressive pre-training methods, and multiple LoRa modules are constructed. These modules capture common task invariants and provide a foundation for specific tuning in subsequent tasks. This step aims to prepare LoRa parameters that are universal and transferable for downstream tasks. During this phase, data is combed using FIM technology, which allows for filling in the middle of a complete code segment. The generation of each intermediate code segment depends not only on the prefix and suffix codes, but also on previously generated intermediate codes.

[0106] It can be understood that based on the first sample code data, the parameter fine-tuning module set in the key layer of the pre-trained model is autoregressively pre-trained to obtain an initial code generation model. On the one hand, the parameter fine-tuning module replaces part of the parameters of the pre-trained model with fine-tunable low-rank representations through low-rank matrix decomposition, thereby reducing the computational cost during training while maintaining the model capacity; on the other hand, through autoregressive training, the model can better understand the contextual information of the code, thereby more accurately generating code that conforms to the user's proprietary code style and logic, thereby improving the readability of the code.

[0107] After obtaining the second sample code data, the parameter fine-tuning module of the initial code generation model may be trained based on the second sample code data and the label code generation result of the second sample code data to obtain a code generation model.

[0108] Here, based on the second sample code data and the label code generation result of the second sample code data, the parameter fine-tuning module of the initial code generation model is trained to obtain the code generation model. Specifically, the second sample code data can be input into the initial code generation model to obtain the predicted code generation result output by the initial code generation model, and then the reconstruction loss is determined based on the difference between the predicted code generation result and the label code generation result. The parameter fine-tuning module of the initial code generation model is iterated to obtain the code generation model. Alternatively, the base model can be used as the teacher model and the initial code generation model as the student model. The distribution loss is determined based on the difference between the first code output distribution output by the teacher model and the second code output distribution output by the student model. Then, based on the reconstruction loss and the distribution loss, the parameter fine-tuning module of the initial code generation model is iterated to obtain the code generation model. The embodiment of the present invention does not make any specific limitations on this.

[0109] It can be understood that, on the one hand, the parameter fine-tuning module replaces some parameters of the pre-trained model with fine-tunable low-rank representations through low-rank matrix decomposition, thereby once again reducing the computational cost during training while maintaining the model capacity; on the other hand, the initial code generation model is trained by the second sample code data and its labeled code generation results. The initial code generation model can learn the style and logic of the user's proprietary code more accurately, thereby generating higher quality code.

[0110] The method provided by an embodiment of the present invention obtains the context code of the code to be completed, and then inputs the context code into a code generation model to obtain the target completion code output by the code generation model. The code generation model is obtained by executing a training method based on the code generation model, thereby improving the accuracy and reliability of the target completion code generation.

[0111] The following describes the training device for the code generation model provided by the present invention. The training device for the code generation model described below and the training method for the code generation model described above can refer to each other.

[0112] Based on any of the above embodiments, the present invention provides a training device for a code generation model. Figure 8 This is a schematic diagram of the structure of the training device for the code generation model provided by the present invention. Figure 8 As shown, the device includes: A first acquisition unit 810 is configured to acquire first sample code data, second sample code data, and a pre-trained model; A first training unit 820 is configured to perform autoregressive pre-training on the parameter fine-tuning module provided in the key layer of the pre-training model based on the first sample code data to obtain an initial code generation model; The second training unit 830 is configured to train the parameter fine-tuning module of the initial code generation model based on the second sample code data and the label code generation result of the second sample code data to obtain the code generation model.

[0113] The device provided by the embodiment of the present invention obtains first sample code data, second sample code data and a pre-trained model, and then performs autoregressive pre-training on the parameter fine-tuning module set in the key layer of the pre-trained model based on the first sample code data to obtain an initial code generation model. Finally, based on the second sample code data and the label code generation result of the second sample code data, the parameter fine-tuning module of the initial code generation model is trained to obtain a code generation model. On the one hand, the parameter fine-tuning module replaces some parameters of the pre-trained model with fine-tunable low-rank representations through low-rank matrix decomposition, thereby reducing the computational cost during training while maintaining the model capacity; on the other hand, through autoregressive training, the model can better understand the contextual information of the code, thereby more accurately generating code that conforms to the user's proprietary code style and logic, improving the readability of the code, and then the initial code generation model is trained by the second sample code data and its label code generation result. The initial code generation model can more accurately learn the style and logic of the user's proprietary code, thereby generating higher quality code.

[0114] Based on any of the above embodiments, the first training unit 820 is specifically configured to: Dividing the first sample code data into a prefix code, a middle tag code and a suffix code; Determining a current intermediate code predicted by the pre-trained model based on the prefix code and the suffix code, and determining a next intermediate code of the current intermediate code predicted by the pre-trained model based on the prefix code, the suffix code, and the current intermediate code, taking the next intermediate code as the current intermediate code, and executing the step of determining the next intermediate code based on the prefix code, the suffix code, and the current intermediate code until an end marker is encountered; Based on the difference between the current intermediate code and the intermediate label code, and the difference between the next intermediate code and the intermediate label code, the autoregressive generation loss is determined, and the parameter fine-tuning module is iterated based on the autoregressive generation loss to obtain the initial code generation model.

[0115] Based on any of the above embodiments, the second training unit 830 specifically includes: a reconstruction loss determining unit, configured to input the second sample code data into the initial code generation model, obtain a predicted code generation result output by the initial code generation model, and determine a reconstruction loss based on a difference between the predicted code generation result and the label code generation result; A parameter iteration unit is used to perform parameter iteration on a parameter fine-tuning module of the initial code generation model based on the reconstruction loss to obtain the code generation model.

[0116] Based on any of the above embodiments, the parameter iteration unit specifically includes: An acquisition model unit is used to acquire a standard code generation model; determining a first output distribution unit, configured to perform code generation training on the standard code generation model based on the second sample code data to obtain a first code output distribution; determining a second output distribution unit, configured to perform code generation training on the initial code generation model based on the second sample code data to obtain a second code output distribution; A parameter iteration subunit is used to determine a distribution loss based on the difference between the first code output distribution and the second code output distribution, and to perform parameter iteration on a parameter fine-tuning module of the initial code generation model based on the distribution loss and the reconstruction loss to obtain the code generation model.

[0117] Based on any of the above embodiments, the parameter iteration subunit is specifically configured to: determining the first code output distribution as a prior distribution and determining the second code output distribution as a latent variable distribution; The distribution loss is determined based on a difference between the prior distribution and the latent variable distribution.

[0118] Based on any of the above embodiments, the further embodiment includes a sample code data acquisition unit, wherein the sample code data acquisition unit is specifically configured to: obtaining second original code data; Parsing the second original code data to obtain an abstract syntax tree corresponding to the second original code data; Traversing the abstract syntax tree, and performing code optimization on the second original code data corresponding to the abstract syntax tree to obtain the second sample code data; The code optimization step includes at least one of static code analysis, deletion of redundant nodes, constant folding, dead code elimination and code conversion.

[0119] The code generation device provided by the present invention is described below. The code generation device described below and the code generation method described above can be referenced to each other.

[0120] Based on any of the above embodiments, the present invention provides a code generating device, Figure 9 It is a structural diagram of the code generating device provided by the present invention, such as Figure 9 As shown, the device includes: The second acquiring unit 910 is used to acquire the context code of the code to be completed; An input unit 920 is configured to input the context code into the code generation model to obtain a target completion code output by the code generation model; The code generation model is obtained by executing the training method based on the above code generation model.

[0121] The device provided by the embodiment of the present invention obtains the context code of the code to be completed, and then inputs the context code into the code generation model to obtain the target completion code output by the code generation model. The code generation model is obtained by executing a training method based on the code generation model, thereby improving the accuracy and reliability of the target completion code generation.

[0122] Figure 10 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 10As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, the communication interface 1020, and the memory 1030 communicate with each other via the communication bus 1040. The processor 1010 may call logic instructions in the memory 1030 to execute a method for training a code generation model, the method comprising: obtaining first sample code data, second sample code data, and a pre-trained model; based on the first sample code data, performing autoregressive pre-training on a parameter fine-tuning module set at a key layer of the pre-trained model to obtain an initial code generation model; based on the second sample code data and the label code generation result of the second sample code data, training the parameter fine-tuning module of the initial code generation model to obtain the code generation model.

[0123] The processor 1010 can also call the logic instructions in the memory 1030 to execute the code generation method, which includes: obtaining the context code of the code to be completed; inputting the context code into the code generation model to obtain the target completion code output by the code generation model; the code generation model is obtained by executing the training method based on the above-mentioned code generation model.

[0124] Furthermore, the logic instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0125] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the code generation model provided by the above methods, the method including: obtaining first sample code data, second sample code data and a pre-trained model; based on the first sample code data, performing autoregressive pre-training on a parameter fine-tuning module set in a key layer of the pre-trained model to obtain an initial code generation model; based on the second sample code data and the label code generation result of the second sample code data, training the parameter fine-tuning module of the initial code generation model to obtain the code generation model.

[0126] When the computer program is executed by a processor, the computer can execute the code generation method provided by the above methods, which includes: obtaining context code of the code to be completed; inputting the context code into the code generation model to obtain the target completion code output by the code generation model; the code generation model is obtained by executing the training method based on the above code generation model.

[0127] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a code generation model provided by the above-mentioned methods, the method comprising: obtaining first sample code data, second sample code data and a pre-trained model; based on the first sample code data, performing autoregressive pre-training on a parameter fine-tuning module provided in a key layer of the pre-trained model to obtain an initial code generation model; based on the second sample code data and the label code generation result of the second sample code data, training the parameter fine-tuning module of the initial code generation model to obtain the code generation model.

[0128] When the computer program is executed by a processor, it is implemented to execute the code generation method provided by the above methods, which includes: obtaining context code of the code to be completed; inputting the context code into the code generation model to obtain the target completion code output by the code generation model; the code generation model is obtained by executing the training method based on the above code generation model.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0130] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for training a code generation model, characterized in that: include: Obtaining first sample code data, second sample code data, and a pre-trained model; Based on the first sample code data, performing autoregressive pre-training on a parameter fine-tuning module provided in a key layer of the pre-trained model to obtain an initial code generation model; Based on the second sample code data and the label code generation result of the second sample code data, the parameter fine-tuning module of the initial code generation model is trained to obtain the code generation model.

2. The code generation model training method according to claim 1, characterized in that: The method of performing autoregressive pre-training on a parameter fine-tuning module provided at a key layer of the pre-training model based on the first sample code data to obtain an initial code generation model includes: Dividing the first sample code data into a prefix code, a middle tag code and a suffix code; Determining a current intermediate code predicted by the pre-trained model based on the prefix code and the suffix code, and determining a next intermediate code of the current intermediate code predicted by the pre-trained model based on the prefix code, the suffix code, and the current intermediate code, taking the next intermediate code as the current intermediate code, and executing the step of determining the next intermediate code based on the prefix code, the suffix code, and the current intermediate code until an end marker is encountered; Based on the difference between the current intermediate code and the intermediate label code, and the difference between the next intermediate code and the intermediate label code, the autoregressive generation loss is determined, and the parameter fine-tuning module is iterated based on the autoregressive generation loss to obtain the initial code generation model.

3. The code generation model training method according to claim 1, characterized in that: The step of training the parameter fine-tuning module of the initial code generation model based on the second sample code data and the label code generation result of the second sample code data to obtain the code generation model includes: Inputting the second sample code data into the initial code generation model to obtain a predicted code generation result output by the initial code generation model, and determining a reconstruction loss based on a difference between the predicted code generation result and the labeled code generation result; Based on the reconstruction loss, parameter iteration is performed on a parameter fine-tuning module of the initial code generation model to obtain the code generation model.

4. The code generation model training method according to claim 3, characterized in that: The step of performing parameter iteration on a parameter fine-tuning module of the initial code generation model based on the reconstruction loss to obtain the code generation model includes: Get the standard code generation model; Based on the second sample code data, performing code generation training on the standard code generation model to obtain a first code output distribution; Based on the second sample code data, performing code generation training on the initial code generation model to obtain a second code output distribution; Based on the difference between the first code output distribution and the second code output distribution, a distribution loss is determined, and based on the distribution loss and the reconstruction loss, parameter iteration is performed on a parameter fine-tuning module of the initial code generation model to obtain the code generation model.

5. The code generation model training method according to claim 4, characterized in that: The determining of the distribution loss based on the difference between the first code output distribution and the second code output distribution includes: determining the first code output distribution as a prior distribution and determining the second code output distribution as a latent variable distribution; The distribution loss is determined based on a difference between the prior distribution and the latent variable distribution.

6. The method for training a code generation model according to any one of claims 1 to 5, characterized in that: The step of acquiring the second sample code data includes: obtaining second original code data; Parsing the second original code data to obtain an abstract syntax tree corresponding to the second original code data; Traversing the abstract syntax tree, and performing code optimization on the second original code data corresponding to the abstract syntax tree to obtain the second sample code data; The code optimization step includes at least one of static code analysis, deletion of redundant nodes, constant folding, dead code elimination and code conversion.

7. A code generation method, characterized in that: include: Get the context code of the code to be completed; Inputting the context code into the code generation model to obtain the target completion code output by the code generation model; The code generation model is obtained by executing a training method for a code generation model according to any one of claims 1 to 6.

8. A training device for a code generation model, characterized in that: include: A first acquisition unit, configured to acquire first sample code data, second sample code data, and a pre-trained model; A first training unit is configured to perform autoregressive pre-training on the parameter fine-tuning module provided at the key layer of the pre-training model based on the first sample code data to obtain an initial code generation model; The second training unit is used to train the parameter fine-tuning module of the initial code generation model based on the second sample code data and the label code generation result of the second sample code data to obtain the code generation model.

9. A code generating device, characterized in that: include: A second acquiring unit is used to acquire the context code of the code to be completed; An input unit, configured to input the context code into the code generation model to obtain a target completion code output by the code generation model; The code generation model is obtained by executing a training method for a code generation model according to any one of claims 1 to 6.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, it implements the training method of the code generation model according to any one of claims 1 to 6, or implements the code generation method according to claim 7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the training method of the code generation model according to any one of claims 1 to 6, or implements the code generation method according to claim 7.

Images