Code abstract generation method, system and equipment based on improved Transform model
By improving the Transformer model and deep attention mechanism, combining knowledge distillation and cross-entropy loss function, the problem of insufficient utilization of different representation patterns in code summary generation is solved, and a more accurate and concise code summary generation is achieved.
Patent Information
- Application Number
- CN202411088917.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-09
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The prior art is difficult to fully utilize different representation patterns of the code in code digest generation, resulting in inaccurate and comprehensive enough in the generated digest, especially when processing long sequences.
Using the improved Transformer model, the source token is input to the prediction network and partner network through data cleaning and deep attention mechanism modules for training, and the model is optimized by using knowledge distillation and cross-entropy loss function to enhance the correlation feature learning between codes and generate more accurate generation. code summary.
Through mutual learning and cross-attention mechanisms, the accuracy and completeness of code summary are improved, and the generated summary is more insightful and simplified, significantly improving the performance of code summary generation.
Smart Images

Figure CN118963825A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of code semantic analysis, and in particular to a code summary generation method, system and device based on an improved Transformer model. Background Art
[0002] Code summaries can provide accurate natural language descriptions of source code, greatly saving developers time and energy in understanding code functionality. Since manual document writing is time-consuming and labor-intensive, models for automatically generating code summaries have become an important tool, which regards the generation of code summaries as a language generation problem.
[0003] In order to improve the accuracy of code summary generation, the model must understand the key characteristics of the code; initially, Haiduc et al. used traditional information retrieval (IR) technology to obtain code summaries by matching code snippets with pre-labeled summaries. Subsequently, researchers such as Sridhara and Moreno proposed creating manual templates or heuristic criteria to facilitate the generation of code comments.
[0004] Early implementations demonstrated the effectiveness of Seq2Seq models, such as recurrent neural networks (RNNs), long short-term memory (LSTM) networks, and Tree-LSTM, which are able to skillfully model the semantic relationship between source code and summary. Hu et al. continued to innovate by introducing a structure-based traversal method for processing abstract syntax trees (ASTs) using RNN encoders, while Fernandes et al. used gated graph neural networks to extract features from code graphs. Shido et al. used Tree-LSTM to effectively capture the structural properties of tree-structured data, while Iyer et al. utilized LSTM networks with attention mechanisms to summarize code snippets. Although Seq2Seq-based models have shown superior performance over traditional methods, they often encounter bottlenecks on long sequences, resulting in performance degradation.
[0005] To address these limitations, Ahmad et al. introduced an enhanced Transformer-based model that significantly outperformed previous RNN-based models. Building on this, Wu et al. proposed a structure-induced Transformer that combines structure-based traversal with self-attention to capture the hierarchical structure of code. Nguyen et al. developed a heterogeneous code representation network that leverages AST and code graphs to capture lexical, syntactic, and semantic code properties. Gong et al. introduced the SCRIPT method to extract and fuse semantic parameters from AST.
[0006] These advances highlight the transformative role of Transformer-based models in code summary generation. Although previous studies have successfully leveraged AST, code graphs, and SBT to generate code summaries, there is still much room for improvement to enhance the way these models handle and integrate different code representation modes to obtain more accurate and insightful code summaries. Summary of the invention
[0007] In order to solve the problems existing in the prior art, the present invention provides a code summary generation method, system and device based on an improved Transformer model, which can link discrete code knowledge, enhance the correlation between source code tokens, and more completely learn the correlation features between codes, thereby generating a more accurate code summary.
[0008] To achieve the above object, the present invention provides a code summary generation method based on an improved Transformer model, comprising:
[0009] Extract the source token after cleaning the corpus to be processed;
[0010] Input the source Token into the prediction network and the partner network in the improved Transformer model for model training; the prediction network and the partner network both include a deep attention mechanism module;
[0011] During the model training process, based on the principle of knowledge distillation, the prediction network and the partner network learn from each other and optimize the prediction network;
[0012] The optimized prediction network is tuned using the cross entropy loss function to obtain a trained improved Transformer model;
[0013] Generate code summaries using the prediction network from the trained improved Transformer model.
[0014] Optionally, the data cleaning includes: lowercase conversion, underscore splitting, camel case splitting and word form restoration.
[0015] Optionally, the deep attention mechanism module includes two encoders and one decoder, and the two encoders implement a superimposed cross-attention mechanism at the decoder.
[0016] Optionally, the two encoders implement a superimposed cross-attention mechanism at the decoder, including:
[0017] The result of the intermediate features output by one encoder after the multi-head cross-attention mechanism and the result of the intermediate features output by another encoder after the multi-head cross-attention mechanism are superimposed at the decoder to perform a cross-self-attention operation.
[0018] Optionally, the prediction network and the partner network learn from each other and optimize the prediction network, including:
[0019] Based on the output result of the prediction network and the output result of the partner network, the KL divergence loss function is used to reduce the difference between the prediction score of the prediction network and the prediction score of the partner network to optimize the prediction network; the expression of the KL divergence loss function is:
[0020]
[0021] Where D KL () represents the difference in prediction distribution between the prediction network and the partner network D KL (P2||P1) represents the KL distance from P1 to P2; i represents a sample, N is the number of samples; m represents a category, M is the number of categories; and represents the sample x predicted by the partner network and the prediction network in N samples and M categories respectively i The probability of belonging to class m.
[0022] Optionally, the optimized prediction network is tuned using the cross entropy loss function to obtain a trained improved Transformer model, including:
[0023] S1, using the cross entropy loss function to calculate the loss value of the output result of the prediction network;
[0024] S2, based on the loss value, updating and improving the parameters in the Transformer model by back-propagating the output result;
[0025] S1-S2 are repeatedly executed until the parameters converge to obtain a trained improved Transformer model.
[0026] Optionally, the expression of the cross entropy loss function is as follows:
[0027]
[0028]
[0029] In the formula, y i,c Indicates the value of sample i corresponding to the true label c, p i,c Indicates the probability of whether it is a true label; Li represents the cross entropy loss of a batch, C represents the total number of samples in a batch, L represents the average cross entropy loss of a training, and N represents the number of batches required for a training.
[0030] Optionally, the method further includes:
[0031] The trained improved Transformer model is evaluated and optimized according to the evaluation indicators; the evaluation indicators include at least one of BLEU, ROUGE-L, METEOR and F1-SCORE.
[0032] The present invention also provides a code summary generation system based on an improved Transformer model, comprising:
[0033] The data processing module is used to extract the source token after cleaning the corpus to be processed;
[0034] Code summary generation module for:
[0035] Input the source token into the prediction network and the partner network in the improved Transformer model for model training; the prediction network and the partner network both include a deep attention mechanism module;
[0036] During the model training process, based on the principle of knowledge distillation, the prediction network and the partner network learn from each other and optimize the prediction network;
[0037] The optimized prediction network is tuned using the cross entropy loss function to obtain a trained improved Transformer model;
[0038] Generate code summaries using the prediction network from the trained improved Transformer model.
[0039] The present invention also provides an electronic device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the above method when executing the computer program.
[0040] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0041] The code summary generation method based on the improved Transformer model provided by the present invention is that the tokens extracted from the source code (corpus to be processed) and its corresponding summary are preprocessed and then input into two specialized networks (prediction network and partner network); dual encoder encoding is implemented in each network, aiming to guide and improve the attention head through a complex deep cross-attention strategy. Finally, the trained prediction network is used to perform the summary, and its effectiveness is enhanced through mutual learning technology.
[0042] The existing methods for code summary generation tasks directly train the model based on the existing bimodal data (NL-PL pairs). These bimodal data samples contain rich connections, but this is not fully utilized. The present invention connects the discrete code knowledge and enhances the connection between samples in the data set. Compared with the traditional method, it can learn the correlation features between codes more completely.
[0043] The existing technology often does not fully utilize the key Token mode, while the present invention emphasizes the ultimate use of the Token mode and the cross-attention layer to promote method summarization, thereby enhancing the learning of source code features (especially method parameters) to generate concise and accurate code summaries. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.
[0045] Figure 1 A schematic diagram of a method flow of a code summary generation method based on an improved Transformer model shown in an embodiment of the present invention;
[0046] Figure 2 A schematic diagram of model training steps shown in an embodiment of the present invention;
[0047] Figure 3 A model architecture diagram of an improved Transformer model shown in an embodiment of the present invention;
[0048] Figure 4 Schematic diagram of a deep cross-attention mechanism according to an embodiment of the present invention; wherein FIG. (4-1) is a cross-attention pattern diagram of a native Transformer, and FIG. (4-2) is a cross-attention pattern diagram of an improved Transformer;
[0049] Figure 5A schematic diagram of mutual learning shown in an embodiment of the present invention; wherein Figure (a) is an attention feature map learned by the prediction network during the training process; Figure (b) is an attention feature map learned by the partner network during the training process;
[0050] Figure 6 A schematic diagram of the code summary generation effect shown in an embodiment of the present invention;
[0051] Figure 7 A schematic diagram of the module structure of a code summary generation system based on an improved Transformer model shown in an embodiment of the present invention;
[0052] Figure 8 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] See also Figure 1 , Figure 1 Flowchart of the method for code summary generation based on the improved Transformer model.
[0055] The above code summary generation method comprises the following steps:
[0056] Step 101: Extract source tokens after cleaning the corpus to be processed.
[0057] In the application, Python toolkits and specific cleaning methods can be used to clean the data of the corpus to be processed and extract its tokens. Examples of data cleaning include lowercase, underscore splitting, camel case splitting, and word form restoration.
[0058] Data cleaning can remove noise data such as comments, blank lines, non-ASCII characters, etc. in the code to ensure the purity of the source token. The cleaned code is segmented to generate a source token sequence. At the same time, the target token sequence in the corresponding target language (such as natural language) can be prepared as a summary reference.
[0059] During the training process, the corpus to be processed may include Java datasets and Python datasets; the Java dataset contains a total of 87,136 data instances, 69,708 of which are used for training and 8,714 for validation. The Python dataset contains 92,545 data instances, 55,538 of which are used for training and 18,505 for validation. The cleaning process can be found in Figure 2 .
[0060] In one embodiment, the following operations may be performed when data cleaning is performed on the corpus to be processed:
[0061] Tokenize the code snippets in the corpus to be processed and convert each token to lowercase;
[0062] Use the NLTK library to split each identifier based on camel case and underscores; for example, the identifier "IndexOutOfBoundError" will be broken down into "index", "out", "of", "bound", and "error";
[0063] Each word was lemmatized using spaCy2, standardizing the form to improve uniformity.
[0064] Step 102: Input the source token into the prediction network and partner network in the improved Transformer model for model training. Both the prediction network and the partner network contain deep attention mechanism modules.
[0065] like Figure 3 As shown in the lower part, the deep attention mechanism module is Figure 3 The dual-Token model shown in Figure 2; dual-Token means that the source Token is used twice in the encoding layer to generate two encoding results, which are then input into the decoder for cross attention, thus forming a deep (two-layer) attention mechanism module.
[0066] The deep attention mechanism module includes two encoders and a decoder. The two encoders implement a superimposed cross-attention mechanism at the decoder. Specifically, the intermediate features output by one encoder after the multi-head cross-attention mechanism and the intermediate features output by the other encoder after the multi-head cross-attention mechanism are cross-attended at the decoder for multi-head self-attention operations.
[0067] The input to the improved Transformer model includes tokenized modalities of code and summary; SeTransformer aims to learn lexical information from source code tokens, which usually consist of keywords, identifiers, and various operators. Challenges associated with large vocabularies lead to increased computational requirements and reduced performance due to different word forms and irregularly defined identifiers. Considering that semantically similar words should keep similar representations, pure representations are extracted from each code token despite different forms or identifier names.
[0068] The source tokens extracted after data cleaning are input into the prediction network and the partner network respectively. The two networks have the same architecture, and each network is equipped with dual encoders and a shared decoder.
[0069] In applications, a mutual learning strategy can be used to synthesize the knowledge obtained from the two encoders and improve the attention mechanism in the decoder.
[0070] Step 103: During the model training process, based on the knowledge distillation principle, the prediction network and the partner network learn from each other and optimize the prediction network.
[0071] See also Figure 3 In the upper part, the training architecture includes two networks: a prediction network for prediction and a partner network for mutual learning. In the mutual learning mechanism, a knowledge distillation strategy can be adopted to transfer knowledge from one model to another. This method helps to train a more stable and efficient network. Two equivalent deep cross-attention networks can be configured in the application to learn from each other during the training phase. Based on this, during the model training process, the present invention allows the partner network and the prediction network to learn from each other to optimize the prediction network. The optimized prediction network is used to generate the final summary.
[0072] Contrary to traditional approaches that might suggest that larger, more complex teacher networks are needed to guide the learning process, experiments show that mutual learning between simpler student networks is not only feasible, but can surpass the performance achieved by distilling from a more powerful but static teacher network.
[0073] like Figure 4As shown, Figure (4-1) is the cross-attention pattern diagram of the native Transformer, and Figure (4-2) is the cross-attention pattern diagram of the improved Transformer; by comparison, it can be found that the improved Transformer can learn more key features, which shows that the information learned from the encoder is crucial, and the previous native Transformer did not make full use of this information. Based on the native Transformer, the present invention introduces an additional cross-attention layer to enhance the key features learned from the encoder.
[0074] During the training process, there are differences in the word probability estimates for each training sample between the sibling networks (including the prediction network and the partner network). These differences provide additional knowledge that can be used to enhance the prediction ability of the network. The outputs of the last layer of the prediction network and the partner network are represented as q∈R BLS and t∈R BLS , B, L and S represent the batch size, maximum sequence length and target vocabulary size respectively; q and t represent the prediction scores of the two networks.
[0075] In one embodiment, based on the output results of the prediction network and the output results of the partner network, the KL divergence loss function can be used to reduce the difference in prediction scores between the prediction network and the partner network (the difference between q and t); wherein the expression of the KL (Kullback-Leibler) divergence loss function is:
[0076]
[0077] Where D KL () represents the difference in prediction distribution between the prediction network and the partner network, D KL (P2||P1) represents the KL distance from P1 to P2; i represents a sample, N is the number of samples; m represents a category, M is the number of categories; and represents the sample x predicted by the partner network and the prediction network in N samples and M categories respectively i The probability of belonging to class m.
[0078] See also Figure 5 , where (a) is the attention feature map learned by the prediction network during training; (b) is the attention feature map learned by the partner network during training; it can be found that the prediction network and the partner network can indeed learn different features. This mutual learning framework not only enhances the individual capabilities of each network, but also ensures a unified learning experience, thereby significantly improving the overall model performance in the code summarization task.
[0079] In the application, each encoder follows the basic architecture of the Transformer network. For a batch of code tag sequences X∈R BL , where B represents the training batch size and L is the maximum length of the sequence in this batch. The encoder embeds the sequence X with an embedding size d=512. It is worth noting that the source code and summary share the same embedding layer.
[0080] The embedding outputs of the source token are utilized twice: first, the encoder processes them to generate intermediate representations for prediction, and then they are encoded so that the partner network can learn alternative representations. In the encoding layer, tokens with important semantic importance are attended to in the Multihead self-attention layer, and then the output is transformed by layer normalization after forming a residual connection with the input. Finally, the layerNorm function of the Transformer network is reused, and the last layer output is regarded as the encoder result. The output results of both encoders are input to the same decoder for cross attention, that is, forming a double cross attention layer.
[0081] After encoding, a double cross attention layer is implemented to decode the token information of the two encoders. The decoder model uses the source token sequence as the source sequence and the comment sequence as the target sequence. Residual connection, batch normalization and full connection are also used. Finally, the Softmax layer outputs the probability of each word.
[0082] Step 104: Use the cross entropy loss function to tune the optimized prediction network to obtain a trained improved Transformer model.
[0083] In one embodiment, the above step 104 specifically includes:
[0084] S1, using the cross entropy loss function to calculate the loss value of the output result of the prediction network;
[0085] S2, based on the loss value, improves the parameters in the Transformer model by back-propagating the output results;
[0086] Repeat S1-S2 until the parameters converge to obtain the trained improved Transformer model.
[0087] Among them, the expression of the cross entropy loss function is as follows:
[0088]
[0089]
[0090] In the formula, y i,cIndicates the value of sample i corresponding to the true label c, p i,c Indicates the probability of whether it is a true label; L i represents the cross entropy loss of a batch, C represents the total number of samples in a batch, L represents the average cross entropy loss of a training, and N represents the number of batches required for a training.
[0091] like Figure 3 As shown in the upper part, after the forward propagation of the neural network, the cross entropy loss function is calculated for the input features, and finally added to the divergence loss to obtain the total loss. Further, the total loss can be used to train and optimize the model. It should be noted that during the optimization process, the parameters of the encoder and decoder in the prediction network are not only adjusted to optimize summary generation, but also aligned with the partner network. This alignment enhances the stability and robustness of the model performance.
[0092] Furthermore, in one embodiment, the trained improved Transformer model can also be evaluated and optimized according to evaluation indicators; wherein the evaluation indicators include at least one of BLEU, ROUGE-L, METEOR and F1-SCORE.
[0093] Step 105: Generate a code summary using the prediction network in the trained improved Transformer model.
[0094] See also Figure 6 , Figure 6 Generate a schematic diagram for the code summary effect; in the application, input a piece of code into the improved Transformer model, and the trained model generates the corresponding natural language describing the program function based on the code, such as Figure 6 As shown, the natural language predicted by the model (Ours) and the standard answer (Ground-truth) are basically consistent, meeting the requirements of natural language; this also proves the feasibility and accuracy of the code summary generation method provided by the present invention.
[0095] Corresponding to the aforementioned application function implementation method embodiment, the present invention also provides a code summary generation system and device based on an improved Transformer model and corresponding embodiments.
[0096] See also Figure 7 , Figure 7 This is a schematic diagram of the module structure of the code summary generation system based on the improved Transformer model. The code summary generation system includes:
[0097] The data processing module 71 is used to extract source tokens after performing data cleaning on the corpus to be processed;
[0098] The code summary generation module 72 is used to:
[0099] Input the source token into the prediction network and partner network in the improved Transformer model for model training; both the prediction network and the partner network contain deep attention mechanism modules;
[0100] During the model training process, based on the principle of knowledge distillation, the prediction network and the partner network learn from each other and optimize the prediction network;
[0101] The optimized prediction network is tuned using the cross entropy loss function to obtain a trained improved Transformer model;
[0102] Generate code summaries using the prediction network from the trained improved Transformer model.
[0103] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.
[0104] An electronic device is also provided in an embodiment of the present invention. Figure 8 , the electronic device 800 includes a memory 810 and a processor 820 .
[0105] The processor 820 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0106] The memory 810 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage. Among them, ROM can store static data or instructions required by the processor 820 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 810 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 810 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.
[0107] The memory 810 stores executable codes, and when the executable codes are processed by the processor 820 , the processor 820 can execute part or all of the methods described above.
[0108] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A code summary generation method based on an improved Transformer model, characterized in that: include: Extract the source token after cleaning the corpus to be processed; Input the source Token into the prediction network and the partner network in the improved Transformer model for model training; the prediction network and the partner network both include a deep attention mechanism module; During the model training process, based on the principle of knowledge distillation, the prediction network and the partner network learn from each other and optimize the prediction network; The optimized prediction network is tuned using the cross entropy loss function to obtain a trained improved Transformer model; Generate code summaries using the prediction network from the trained improved Transformer model.
2. The code summary generation method based on the improved Transformer model according to claim 1 is characterized in that: The data cleaning includes: lowercase, underline splitting, camel case splitting and word form restoration.
3. The code summary generation method based on the improved Transformer model according to claim 1 is characterized in that: The deep attention mechanism module includes two encoders and a decoder, and the two encoders implement a superimposed cross-attention mechanism at the decoder.
4. The code summary generation method based on the improved Transformer model according to claim 3 is characterized in that: The two encoders implement a superimposed cross-attention mechanism at the decoder, including: The result of the intermediate features output by one encoder after passing through the multi-head cross-attention mechanism and the result of the intermediate features output by another encoder after passing through the multi-head cross-attention mechanism are superimposed at the decoder to perform a cross-self-attention operation.
5. The code summary generation method based on the improved Transformer model according to claim 1 is characterized in that: The prediction network and the partner network learn from each other and optimize the prediction network, including: Based on the output result of the prediction network and the output result of the partner network, the KL divergence loss function is used to reduce the difference between the prediction score of the prediction network and the prediction score of the partner network to optimize the prediction network; the expression of the KL divergence loss function is: Where D KL () represents the difference in prediction distribution between the prediction network and the partner network, D KL (P2||P1) represents the KL distance from P1 to P2; i represents a sample, N is the number of samples; m represents a category, M is the number of categories; and represents the sample x predicted by the partner network and the prediction network in N samples and M categories respectively i The probability of belonging to class m.
6. The code summary generation method based on the improved Transformer model according to claim 1, characterized in that: The optimized prediction network is tuned using the cross entropy loss function to obtain a trained improved Transformer model, including: S1, using the cross entropy loss function to calculate the loss value of the output result of the prediction network; S2, based on the loss value, updating and improving the parameters in the Transformer model by back-propagating the output result; S1-S2 are repeatedly executed until the parameters converge to obtain a trained improved Transformer model.
7. The code summary generation method based on the improved Transformer model according to claim 6 is characterized in that: The expression of the cross entropy loss function is as follows: In the formula, y i,c Indicates the value of sample i corresponding to the true label c, p i,c Indicates the probability of whether it is a true label; L i represents the cross entropy loss of a batch, C represents the total number of samples in a batch, L represents the average cross entropy loss of a training, and N represents the number of batches required for a training.
8. The method for generating a double-Token code digest based on an improved converter model according to claim 6, characterized in that: The method further comprises: The trained improved Transformer model is evaluated and optimized according to the evaluation indicators; the evaluation indicators include at least one of BLEU, ROUGE-L, METEOR and F1-SCORE.
9. A code summary generation system based on an improved Transformer model, characterized in that: The method according to any one of claims 1 to 8, wherein the system comprises: The data processing module is used to extract the source token after cleaning the corpus to be processed; Code summary generation module for: Input the source token into the prediction network and the partner network in the improved Transformer model for model training; the prediction network and the partner network both include a deep attention mechanism module; During the model training process, based on the principle of knowledge distillation, the prediction network and the partner network learn from each other and optimize the prediction network; The optimized prediction network is tuned using the cross entropy loss function to obtain a trained improved Transformer model; Generate code summaries using the prediction network from the trained improved Transformer model.
10. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the computer program.
Citation Information
Patent Citations
Semantic segmentation model training method, image semantic segmentation method and related device
CN115294337A
Online knowledge distillation method based on hybrid network structure
CN116108921A
Code parameter abstract generation method and system based on improved converter model
CN117093222A
ResNet model optimization algorithm based on mutual learning knowledge distillation
CN117333706A
Method and system for fine-tuning neural conditional language models using constraints
US20240054338A1
Cited By
Code search method, system and equipment based on large language model classification and prototype integration
CN119961381A
Source code abstract generation method and system based on enhanced prompt learning framework
CN120045226A
Self-supervised abstract generation method, system, equipment and medium
CN120196750A
