A method, device, computer equipment and storage medium for generating code comments based on language processing

Abstract

This application relates to a method, apparatus, computer device, and storage medium for generating code annotations based on language processing. The method comprises: inputting pre-interpreted code to be annotated into a pre-trained model; parsing the pre-interpreted code with the pre-trained model to obtain a pre-interpreted code fusion feature vector; and decoding the pre-interpreted code fusion feature vector using a decoder to generate code annotations. This method enables more precise analysis and understanding of the code's structure and semantics through language processing, automatically generating high-quality code annotations.

Description

Technical Field

[0001] The present invention relates to the field of language processing, and in particular to a code comment generation method based on language processing. Background Art

[0002] With the rapid development of software development, code readability and maintainability have become important criteria for measuring software quality. Comments, as a key means of improving code readability and maintainability, are becoming increasingly important. However, traditional commenting methods rely primarily on manual compilation based on developer experience and specifications. This is not only inefficient but also difficult to ensure the accuracy and consistency of comments. Therefore, how to automatically generate high-quality code comments has become an important research direction in computer technology.

[0003] Current methods for generating code comments include template-based, statistical, and machine learning-based methods. The closest existing technology uses machine learning and natural language processing techniques to generate code comments. Machine learning and natural language processing techniques can parse and process natural language text to extract semantic and structural information. While this has improved the automation level of code commenting to a certain extent, it also faces certain challenges. The syntax and semantics of code are more complex than natural language, requiring more precise analysis and understanding. Existing methods have limited ability to understand the semantics of code, and the generated comments may not be accurate and comprehensive. In addition, elements such as variables, functions, and classes in the code have specific semantics and uses. When dealing with such complex code structures, existing methods struggle to accurately analyze the code semantics in the context of the code. Summary of the Invention

[0004] Based on this, it is necessary to provide a code comment generation method, device, computer equipment and storage medium based on language processing that can accurately analyze and understand the code semantics in combination with the context to address the above technical problems.

[0005] In one aspect, a method for generating code comments based on language processing is provided, the method comprising:

[0006] Input the pre-interpreted code to be annotated into the pre-trained model;

[0007] The pre-trained model parses the pre-interpretation code to obtain a pre-interpretation code fusion feature vector;

[0008] Decoding the pre-interpreted code fusion feature vector according to the decoder and generating code annotations;

[0009] The step of parsing the pre-interpretation code to obtain a pre-interpretation code fusion feature vector includes:

[0010] Parsing the pre-release code to obtain the pre-release code feature;

[0011] Encoding the pre-interpretation code feature into a pre-interpretation code feature vector, wherein the pre-interpretation code feature vector includes a query vector, a key vector, and a value vector;

[0012] Performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector;

[0013] The self-attention calculation includes:

[0014] Calculate the dot product of the query vector and the key vector to obtain a first attention score;

[0015] Calculating a second attention score based on the first attention score and the scaling factor;

[0016] Normalizing the second attention score to a third attention score according to a function;

[0017] The pre-code fusion feature vector is obtained based on the third attention score and the value vector.

[0018] In one embodiment, the pre-training model is also constructed:

[0019] Obtaining a data sample including codes and corresponding code comments, and cleaning the data sample to obtain a data set;

[0020] Obtaining features of the code and the corresponding code annotations according to the data set, and encoding the features of the code and the corresponding code annotations into feature vectors;

[0021] Dividing the data set containing the feature vector into training set data and test set data;

[0022] Select a pre-trained initial model;

[0023] Construct the encoder-decoder structure of the pre-trained initial model, set the number of encoder-decoder layers, activation function, and set initial parameter values;

[0024] The training set data is input into the pre-trained initial model, and the code semantic analysis training is completed according to the self-attention mechanism in the pre-trained initial model to obtain a pre-trained model.

[0025] In one embodiment, the features of obtaining the code and the corresponding code annotations include:

[0026] Parse the code through a word segmenter and generate an abstract syntax tree;

[0027] Traverse each node of the abstract syntax tree according to the breadth-first search algorithm;

[0028] Obtaining node type, node content, and relationships between nodes according to each node;

[0029] The node type, node content, and the relationship between nodes are encoded into a feature vector.

[0030] In one embodiment, the feature vector may be combined with a subcode address sequence, including:

[0031] Decomposing the codes in the data set into preliminary sub-codes based on characters;

[0032] In response to the preliminary subcode being included in the pre-trained model dictionary, the preliminary subcode is directly used,

[0033] In response to the preliminary subcode not being included in the pre-trained model dictionary, further decomposing the preliminary subcode until a final decomposed subcode is included in the pre-trained model dictionary;

[0034] Add special identifiers at the beginning and end of the final subcode;

[0035] The final subcode is mapped one-to-one with the predefined addresses in the pre-trained model dictionary to obtain a subcode address sequence.

[0036] In one embodiment, the code semantic analysis training includes the tasks of predicting the final sub-code to be blocked and judging whether two code snippets are continuous:

[0037] The final sub-code task of predicting the shielding includes:

[0038] randomly masking at least one final sub-code according to the sub-code address sequence;

[0039] inputting the subcode address sequence and the subcode address sequence of the masked final subcode into a pre-trained model simultaneously;

[0040] The pre-trained model outputs a predicted subcode;

[0041] Calculating a first loss between the predicted subcode and the masked final subcode;

[0042] The step of determining whether two code snippets are consecutive tasks includes:

[0043] Creating code snippet pairs based on the training set data;

[0044] The code snippet pair includes a continuous code snippet and a code snippet unrelated to the continuous code snippet;

[0045] Merging the code snippet pairs into a pre-trained model;

[0046] The pre-trained model outputs a second loss of whether two code snippets are continuous.

[0047] In one embodiment, the first loss and the second loss are obtained according to a cross entropy function:

[0048] The cross entropy function H(P,Q)=-∑P(i)log(Q(i)),

[0049] Among them, P(i) represents the probability distribution of the true result, and Q(i) represents the probability distribution of the predicted result;

[0050] The closer the value of the cross entropy function is to 0, the better the performance of the pre-trained model is;

[0051] The pre-trained model parameters are updated according to the value of the cross entropy function.

[0052] In one embodiment, the construction of the pre-training model further includes obtaining evaluation performance data:

[0053] Inputting the test set data into the pre-training model;

[0054] according to Get the accuracy of code comments;

[0055] according to Get the accuracy of code comments;

[0056] according to Get the recall rate of code comments;

[0057] According to F1 Get the F1 score of code comments;

[0058] Updating the pre-training model parameters according to the accuracy, precision, and recall of the code annotations;

[0059] Among them, TP represents the true positive class, that is, the number of samples predicted by the pre-trained model as positive and actually positive;

[0060] FP stands for false positive, which is the number of samples that the pre-trained model predicts as positive but is actually negative;

[0061] TN stands for true negative, which is the number of samples that the pre-trained model predicts as negative and is actually negative;

[0062] FN stands for false negatives, which is the number of samples that the pre-trained model predicts as negative but is actually positive.

[0063] On the other hand, a device for generating code comments based on language processing is provided, the device comprising:

[0064] An input module, used to input the pre-interpreted code to be annotated into the pre-trained model;

[0065] An acquisition module is used for parsing the pre-interpreted code by the pre-trained model to obtain a pre-interpreted code fusion feature vector;

[0066] The output module is used to decode the pre-interpreted code fusion feature vector according to the decoder and generate code annotations.

[0067] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0068] Input the pre-interpreted code to be annotated into the pre-trained model;

[0069] The pre-trained model parses the pre-interpretation code to obtain a pre-interpretation code fusion feature vector;

[0070] Decoding the pre-interpreted code fusion feature vector according to the decoder and generating code annotations;

[0071] The step of parsing the pre-interpretation code to obtain a pre-interpretation code fusion feature vector includes:

[0072] Parsing the pre-release code to obtain the pre-release code feature;

[0073] Encoding the pre-interpretation code feature into a pre-interpretation code feature vector, wherein the pre-interpretation code feature vector includes a query vector, a key vector, and a value vector;

[0074] Performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector;

[0075] The self-attention calculation includes:

[0076] Calculate the dot product of the query vector and the key vector to obtain a first attention score;

[0077] Calculating a second attention score based on the first attention score and the scaling factor;

[0078] Normalizing the second attention score to a third attention score according to a function;

[0079] The pre-code fusion feature vector is obtained based on the third attention score and the value vector.

[0080] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0081] Input the pre-interpreted code to be annotated into the pre-trained model;

[0082] The pre-trained model parses the pre-interpretation code to obtain a pre-interpretation code fusion feature vector;

[0083] Decoding the pre-interpreted code fusion feature vector according to the decoder and generating code annotations;

[0084] The step of parsing the pre-interpretation code to obtain a pre-interpretation code fusion feature vector includes:

[0085] Parsing the pre-release code to obtain the pre-release code feature;

[0086] Encoding the pre-interpretation code feature into a pre-interpretation code feature vector, wherein the pre-interpretation code feature vector includes a query vector, a key vector, and a value vector;

[0087] Performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector;

[0088] The self-attention calculation includes:

[0089] Calculate the dot product of the query vector and the key vector to obtain a first attention score;

[0090] Calculating a second attention score based on the first attention score and the scaling factor;

[0091] Normalizing the second attention score to a third attention score according to a function;

[0092] The pre-code fusion feature vector is obtained based on the third attention score and the value vector.

[0093] The aforementioned code comment generation method, apparatus, computer device, and storage medium based on language processing leverage the parsing capabilities of a pre-trained model to parse and extract features from pre-interpreted code, generating a pre-interpreted code feature vector. This process ensures the model captures key code information, including important details such as variable scopes, function parameters, and return values. The construction of the feature vector quantifies the complexity of the code, facilitating subsequent processing. The self-attention mechanism further enhances the model's understanding of local and global relationships within the code. The resulting pre-interpreted code fusion feature vector more accurately reflects the interactions between code elements, ensuring that the generated comments not only conform to grammatical specifications but also accurately describe the functional intent of the code. This approach is particularly suitable for code snippets containing complex logic and multi-level nested structures, improving the quality of comment generation. Finally, a decoder generates code comments based on the pre-interpreted code fusion feature vector. The model structure is optimized using a cross-entropy function and performance evaluation, ensuring the accuracy of the comment content while maintaining the fluency and consistency of the natural language description. This further enhances the reliability and professionalism of software development, enabling more precise analysis and understanding of the code structure and semantics through language processing, and automatically generating high-quality code comments. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 An application environment diagram of a code comment generation method based on language processing in one embodiment;

[0095] Figure 2 A flowchart of a method for generating code comments based on language processing in one embodiment is shown;

[0096] Figure 3 A schematic diagram of the steps of self-attention calculation according to one embodiment;

[0097] Figure 4 A structural block diagram of a code comment generation device based on language processing in one embodiment;

[0098] Figure 5 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0099] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0100] The code comment generation method based on language processing provided by this application can be applied to Figure 1In the application environment shown, the terminal 102 communicates with the server 104 via a network. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0101] In one embodiment, Figure 2 As shown, a code comment generation method based on language processing is provided, which is applied to Figure 1 The server in the example is used as an example to illustrate the following steps:

[0102] Step 110, inputting the pre-interpreted code to be annotated into the pre-trained model;

[0103] The pre-interpreted code to be annotated may be in any programming language, including but not limited to Python, Java, JavaScript, C, C++, and PHP.

[0104] In one embodiment, before the pre-trained model parses the pre-interpreted code, data is collected and cleaned.

[0105] Data collection involves collecting data samples containing code and corresponding code annotations from sources such as open source projects, code repositories, and online platforms.

[0106] Data cleaning involves removing noise, errors, and inconsistencies from the data, such as formatting errors and non-standard comments. It also involves breaking the code into basic units, such as identifiers, keywords, and operators. It then removes or replaces certain identifiers, such as variable and function names, with generic tags to improve the model's generalization capabilities.

[0107] Step 120: The pre-trained model parses the pre-interpreted code to obtain a pre-interpreted code fusion feature vector;

[0108] In one embodiment, obtaining pre-interpreted code features includes obtaining grammatical features, word embedding features, and other features from different code layers. The acquisition of these code features may utilize an abstract syntax tree to analyze the code structure, or a control flow graph or program dependency graph to capture code dependencies.

[0109] In one embodiment, encoding into a feature vector refers to converting the processed code into a numerical representation for easier model processing. This may include using word embedding techniques, such as Word2Vec, GloVe, or BERT embedding layers, to map each word in the code to a vector in a high-dimensional space.

[0110] Step S130, performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector;

[0111] In one embodiment, Figure 3 The method of performing self-attention calculation on the pre-interpretation code feature vector to obtain the pre-interpretation code fusion feature vector includes the following steps:

[0112] Step S131, calculating the dot product between the query vector and the key vector to obtain a first attention score;

[0113] Step S132, calculating a second attention score based on the first attention score and a scaling factor;

[0114] Step S133, normalizing the second attention score to a third attention score according to a function;

[0115] Step S134: Obtain a pre-code fusion feature vector based on the third attention score and the value vector.

[0116] The query vector represents the information the model is "querying" for. The query vector at each position attempts to find relevant information from other positions. The key vector represents a summary or representation of the information at each position in the model. It is used to compare with the query vectors at other positions to determine their relevance. The value vector contains the actual content information. When the query vector successfully matches the key vector at a certain position, the value vector at that position is used as part of the output. The above three vectors are obtained by multiplying the input embedding vector by the weight matrix W_Q, the weight matrix W_k, and the weight matrix W_v, respectively. The weight matrix is ​​part of the pre-trained model.

[0117] Specifically, for each position i in a given input sequence, we first obtain the input embedding vector E at that position: i , calculate the query vector Q i =E i W Q , calculate the key vector K i =E i W K , calculate the query vector V i =E i W V .

[0118] In one embodiment, a first attention score is obtained by calculating the dot product between the query vector and the key vector, a second attention score is obtained by calculating the first attention score and a scaling factor, and the second attention score is normalized to a third attention score according to a function; a pre-code fusion feature vector is obtained by calculating the third attention score and the value vector.

[0119]

[0120] Specifically,

[0121] Calculate the dot product of the query vector q1 and all key vectors k1, k2, k3 to get the first attention score q 1· k1

[0122] q 1· k2, q 1· k3.

[0123]

[0124] Divide the above first attention scores by the scaling factor to obtain the second attention score q 1· k1 / 8,q 1· k2 / 8,q 1· k3 / 8. The scaling factor is the square root of the dimension of the key vector.

[0125]

[0126] Use the activation function to normalize the second attention score to get the third attention score X 11 、X 12 、X 13 .

[0127]

[0128] Multiply these normalized scores by the value vectors V1, V2, and V3, and sum the resulting vectors to obtain the pre-interpreted code feature vector Z1. Z1 is the fused feature vector for the first code segment in the pre-interpreted code. The vectors for the remaining code segments in the pre-interpreted code can be obtained in the same way.

[0129] In one embodiment, the decoder can be a recurrent neural network (RNN) decoder, a Transformer decoder, a convolutional neural network (CNN) decoder, a conditional variational autoencoder (CVAE) decoder, etc.

[0130] The above-mentioned language processing-based code comment generation method uses a pre-trained model and a self-attention mechanism to improve the quality and accuracy of code comments. The pre-interpreted code to be annotated is input into the pre-trained model, which parses the code and extracts features reflecting the code's syntactic and semantic information. These extracted features are then encoded into pre-interpreted code feature vectors, which contain key information from the code. Self-attention is then performed on these feature vectors to better understand the code's context. Specifically, the self-attention mechanism calculates the dot product of the query vector and the key vector to obtain preliminary attention scores. These scores are then converted to more precise attention scores through scaling and normalization. These final attention scores are combined with the value vector to generate a fused feature vector representing the comprehensive information of each part of the code. Finally, the decoder uses these fused feature vectors to generate code comments. This process not only provides a deep understanding of the code's logic and functionality but also generates more contextually relevant comments, thereby improving code readability and maintainability. In this way, the solution can automatically generate high-quality code comments, reduce the workload of manual comment writing, and improve the accuracy and consistency of code documentation.

[0131] In one embodiment, the pre-training model is also constructed:

[0132] Obtaining a data sample including codes and corresponding code comments, and cleaning the data sample to obtain a data set;

[0133] Obtaining features of the code and the corresponding code annotations according to the data set, and encoding the features of the code and the corresponding code annotations into feature vectors;

[0134] Dividing the data set containing the feature vector into training set data and test set data;

[0135] Select a pre-trained initial model;

[0136] Construct the encoder-decoder structure of the pre-trained initial model, set the number of encoder-decoder layers, activation function, and set initial parameter values;

[0137] The training set data is input into the pre-trained initial model, and the code semantic analysis training is completed according to the self-attention mechanism in the pre-trained initial model to obtain a pre-trained model.

[0138] The dataset includes training set, validation set, and test set.

[0139] The training set is used to train the model. The model learns the characteristics and patterns of the data on this dataset. The training set usually occupies a major portion of the total dataset, such as 70% to 80%.

[0140] The validation set is used to tune model hyperparameters and perform model selection. It helps assess the generalization ability of the model during training. The validation set typically ranges from 10% to 20%. In some cases, the validation set may also be used as part of cross-validation.

[0141] The test set is used to evaluate the performance of the final model. It is used after model training and hyperparameter tuning to verify the model's generalization ability on unseen data. The test set usually occupies the remaining portion of the total dataset, such as 10% to 20%.

[0142] Among them, the pre-trained initial model can be a Transformer-based model, such as GPT-4, CodeBERT, Codex, etc.

[0143] In one embodiment, the encoder includes an input layer, a coding layer, and an output layer.

[0144] The input layer refers to input embeddings, including Token Embeddings, which are used to convert input word tokens into word vectors; Segment Embeddings are used to distinguish code pairs in the input, usually using identifiers of 0 and 1; Position Embeddings: used to add position information to each position in the input sequence to capture the order of words. These three embeddings are added together to form the final input embedding.

[0145] The encoding layer structure includes a self-attention mechanism, which calculates the weights of each word at all positions in the sequence and uses these weighted sums to obtain contextual word representations. A feedforward neural network processes the output of the self-attention layer through two fully connected layers to increase the model's expressive power. Layer normalization and residual connections facilitate model training and stability.

[0146] The output layer consists of an MLM output layer and an NSP output layer. The MLM output layer is used to predict masked codes. During training, the model randomly masks some codes in the input sequence and then trains the model to predict the codes at these masked locations. The NSP output layer determines whether two sentences are adjacent code segments, helping the model understand the relationship between the sentences.

[0147] The construction of this pre-trained model, through steps such as data cleaning, feature encoding, and model training, effectively improves code comprehension and generation capabilities. Through its adaptable encoder-decoder architecture and self-attention mechanism, the model can more accurately capture the semantics and structure of code, improving the automation and efficiency of code processing.

[0148] In one embodiment, the features of the acquired code and corresponding code annotations include:

[0149] Parse the code through a word segmenter and generate an abstract syntax tree;

[0150] Traverse each node of the abstract syntax tree according to the breadth-first search algorithm;

[0151] Obtaining node type, node content, and relationships between nodes according to each node;

[0152] The node type, node content, and the relationship between nodes are encoded into a feature vector.

[0153] The feature vector may be combined with a subcode address sequence, including:

[0154] Decomposing the codes in the data set into preliminary sub-codes based on characters;

[0155] In response to the preliminary subcode being included in the pre-trained model dictionary, the preliminary subcode is directly used,

[0156] In response to the preliminary subcode not being included in the pre-trained model dictionary, further decomposing the preliminary subcode until a final decomposed subcode is included in the pre-trained model dictionary;

[0157] Add special identifiers at the beginning and end of the final subcode;

[0158] The final subcode is mapped one-to-one with the predefined addresses in the pre-trained model dictionary to obtain a subcode address sequence.

[0159] Tokenizers are used to break code into meaningful units, such as subcode or characters. Different tokenizers can be used for code and comments. For example, for code, you can use tokenizers from codeBERT or graph-based models; for comments, use a regular text tokenizer.

[0160] An abstract syntax tree is a tree diagram that represents the structure of code. It can show the grammatical structure and hierarchical relationships of the code. It breaks the code into nodes, each representing a grammatical unit, helping the model parse and understand the logical structure of the code.

[0161] The acquisition of the subcode address sequence includes: splitting the code characters in the data set into preliminary subcodes at the character level; if the preliminary subcode is in the pre-trained model dictionary, directly using the preliminary subcode; if the preliminary subcode is not in the pre-trained model dictionary, further decomposing it until the final subcode is included in the dictionary; adding special identifiers at the beginning and end of the final subcode to mark its boundaries; establishing a one-to-one correspondence between the final subcode and the predefined address in the pre-trained model dictionary to form a subcode address sequence.

[0162] By constructing an abstract syntax tree and obtaining the sub-code address sequence, the code structure and surface meaning can be parsed. Combining surface and deep analysis can convert complex code structures into a format that the model can understand, and deeply understand the meaning of the code, thereby improving the model's code processing capabilities and accuracy.

[0163] In one embodiment, the code semantic analysis training includes the tasks of predicting the final sub-code to be blocked and determining whether two code snippets are continuous:

[0164] said randomly masking at least one final subcode;

[0165] inputting the subcode address sequence and the subcode address sequence of the masked final subcode into a pre-trained model simultaneously;

[0166] The pre-trained model outputs a predicted subcode;

[0167] Calculating a first loss between the predicted subcode and the masked final subcode;

[0168] Determining whether two code snippets are consecutive tasks includes:

[0169] Creating code snippet pairs based on the training set data;

[0170] The code snippet pair includes a continuous code snippet and a code snippet unrelated to the continuous code snippet;

[0171] Merging the code snippet pairs into a pre-trained model;

[0172] The pre-trained model outputs a second loss of whether two code snippets are continuous.

[0173] Specifically, taking Python language as an example, the task of predicting the masked final sub-code is as follows:

[0174] Among them, tokenizer is the final subcode.

[0175] #Start training

[0176] trainer.train()

[0177] #Mask prediction

[0178] def predict_masked_word(text):

[0179] inputs=tokenizer(text,return_tensors='pt')

[0180] input_ids=inputs['input_ids']

[0181] attention_mask=inputs['attention_mask']

[0182] # Find the location of the mask

[0183] mask_token_index=torch.where(input_ids==

[0184] tokenizer.mask_token_id)[1]

[0185] #Model makes predictions

[0186] with torch.no_grad():

[0187] outputs=model(input_ids, attention_mask=attention_mask)

[0188] predictions = outputs.logits

[0189] #Predict the vocabulary of the mask position

[0190] predicted_ids=torch.argmax(predictions[0,mask_token_index],dim=-1)

[0191] predicted_tokens=tokenizer.convert_ids_to_tokens(predicted_ids)

[0192] return predicted_tokens[0]

[0193] # Example prediction

[0194] text="def add(a,b):return a+[MASK]"

[0195] predicted_word=predict_masked_word(text)

[0196] print(f"Predicted word: {predicted_word}")

[0197] Specifically, taking Python as an example, the following example shows how to determine whether two code snippets are continuous tasks:

[0198] #Start training

[0199] trainer.train()

[0200] # Make predictions

[0201] def predict_next_sentence(sentence_a,sentence_b):

[0202] inputs=tokenizer.encode_plus(sentence_a, sentence_b, return_tensors='pt', max_length=512, truncation=True)

[0203] input_ids=inputs['input_ids']

[0204] attention_mask=inputs['attention_mask']

[0205] with torch.no_grad():

[0206] outputs=model(input_ids, attention_mask=attention_mask)

[0207] logits=outputs.logits

[0208] prediction=torch.argmax(logits,dim=1).item()

[0209] return prediction==0

[0210] # Example prediction

[0211] sentence_a="def add(a,b):"

[0212] sentence_b="return a+b"

[0213] is_next_sentence=predict_next_sentence(sentence_a,sentence_b)

[0214] print(f"Is sentence B the next sentence of sentence A? {'yes' if is_next_sentence else 'no'}")

[0215] In this example, the tasks of predicting the masked final subcode and determining whether two code snippets are continuous significantly improve the model's understanding of code and the accuracy of annotation generation. The task of predicting the masked final subcode helps the model learn the structure and semantics of the code by training the model to predict masked code vocabulary, thereby generating more relevant annotations. The task of determining whether two code snippets are continuous enhances the model's understanding of the logical relationship between code snippets and annotations, helping the model identify which code snippets typically follow specific annotations.

[0216] In one embodiment, an adaptive masking strategy may be defined, including:

[0217] Set a masking strategy to dynamically determine the number and location of masks. Adjust the mask ratio based on code complexity: for example, for complex code segments, increase the mask ratio to ensure the model learns more contextual information. For simple code segments, decrease the mask ratio to avoid overtraining.

[0218] Context-sensitive masking is also possible, where mask locations are dynamically selected based on the frequency and importance of functions, variables, or classes in the code.

[0219] Compared to fixed-ratio masks, this strategy adjusts the mask settings based on the specific content and context of the code snippet.

[0220] Specifically, when processing code data, the code is first segmented and encoded.

[0221] Count the frequency of different words or symbols in the code snippet and determine the location of the mask based on these statistics. For example, frequently occurring words may be masked out to force the model to focus on uncommon words.

[0222] Use pre-trained models or heuristics to assess the importance of code elements and select the more important parts to mask. For example, masking function names and parameters can encourage the model to generate more accurate annotations.

[0223] During each training session, we randomly mask a portion of the vocabulary in the code snippet according to the above strategy. The model will process different masks in each training session, enhancing its ability to understand the code context.

[0224] The adaptive masking strategy significantly improves the model's generalization and annotation quality during training. By dynamically adjusting the position and scale of the mask, the model can better understand the code context, capture key information, and generate more accurate and informative annotations. This strategy not only helps reduce overfitting and improves model robustness, but also optimizes its ability to handle complex code, making the model more stable and reliable in real-world applications.

[0225] In one embodiment, the first loss and the second loss are obtained according to the cross entropy function:

[0226] The cross entropy function H(P,Q)=-∑P(i)log(Q(i)),

[0227] Among them, P(i) represents the probability distribution of the true result, and Q(i) represents the probability distribution of the predicted result;

[0228] The closer the value of the cross entropy function is to 0, the better the performance of the pre-trained model is;

[0229] The pre-trained model parameters are updated according to the value of the cross entropy function.

[0230] In one embodiment, the construction of the pre-training model further includes obtaining evaluation performance data:

[0231] Inputting the test set data into the pre-training model;

[0232] according to Get the accuracy of code comments;

[0233] according to Get the accuracy of code comments;

[0234] according to Get the recall rate of code comments;

[0235] According to F1 Get the F1 score of code comments;

[0236] Updating the pre-training model parameters according to the accuracy, precision, and recall of the code annotations;

[0237] Among them, TP represents the true positive class, that is, the number of samples predicted by the pre-trained model as positive and actually positive;

[0238] FP stands for false positive, which is the number of samples that the pre-trained model predicts as positive but is actually negative;

[0239] TN stands for true negative, which is the number of samples that the pre-trained model predicts as negative and is actually negative;

[0240] FN stands for false negatives, which is the number of samples that the pre-trained model predicts as negative but is actually positive.

[0241] Specifically, the cross entropy function obtains the loss as follows:

[0242] #Suppose there are two class labels

[0243] #Probability distribution of true labels (real results)

[0244] P = torch.tensor([0.0,1.0])#For example, the probability distribution of the real annotation category

[0245] #Model's predicted probability distribution (prediction results)

[0246] Q = torch.tensor([0.2,0.8]) #Model's predicted probability distribution for annotation categories

[0247] Where P is the probability distribution of the true annotation category. For example, the true category is the second category, and the probability distribution is [0.0, 1.0].

[0248] Q is the probability distribution of the annotation categories predicted by the model. For example, the probability predicted by the model is [0.2, 0.8].

[0249] The cross entropy loss function is calculated as follows:

[0250] H(P,Q)=-∑iP(i)log(Q(i))H(P,Q)=-i∑P(i)log(Q(i))

[0251] Where P(i) is the probability distribution of the true label and Q(i) is the probability distribution of the model prediction. In this example:

[0252] H(P,Q)=-(0.0·log(0.2)+1.0·log(0.8))

[0253] = -log(0.8)H(P,Q)

[0254] =-(0.0·log(0.2)+1.0·log(0.8))

[0255] =-log(0.8)

[0256] The closer the cross-entropy loss value is to 0, the closer the predicted probability distribution is to the true probability distribution. In the above example, if the model's predicted probability distribution is close to the true [0.0, 1.0] distribution, the cross-entropy loss value will be small, indicating good model performance. Conversely, a large loss value indicates that the model's predictions do not match the true annotations and the model needs improvement.

[0257] In this embodiment, the cross-entropy loss function is used to calculate the error between the model prediction and the true annotation, which helps to accurately adjust the model parameters and thus improve the quality of the generated annotations. Through the cross-entropy loss function, the model's errors in annotation generation can be quantified and its parameters can be optimized so that the predicted distribution is closer to the true label. In addition, combining the accuracy, precision, and recall of code annotations to evaluate model performance can more comprehensively measure the effect and quality of annotation generation. Accuracy ensures that the generated annotations correctly match the true annotations, precision measures the proportion of relevant content in the generated annotations, and recall evaluates the ability of the generated annotations to cover the true annotations. Combining these indicators, the model can better capture the relationship between code and annotations, thereby improving the accuracy and effectiveness of annotation generation.

[0258] It should be understood that although Figure 2-3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2-3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0259] In one embodiment, Figure 4 As shown, a code comment generation device based on language processing is provided, including: an input module, an acquisition module and an output module, wherein:

[0260] An input module 21 is used to input the pre-interpreted code to be annotated into the pre-trained model;

[0261] An acquisition module 22 is configured to parse the pre-interpreted code using the pre-trained model to obtain a pre-interpreted code fusion feature vector;

[0262] The output module 23 is configured to decode the pre-interpreted code fusion feature vector according to the decoder and generate a code annotation.

[0263] In one embodiment, the device acquisition module 22 is used for parsing the pre-interpreted code with the pre-trained model to obtain the pre-interpreted code fusion feature vector, including:

[0264] Parse the code through a word segmenter and generate an abstract syntax tree;

[0265] Traverse each node of the abstract syntax tree according to the breadth-first search algorithm;

[0266] Obtaining node type, node content, and relationships between nodes according to each node;

[0267] The node type, node content, and the relationship between nodes are encoded into a feature vector.

[0268] Decomposing the codes in the data set into preliminary sub-codes based on characters;

[0269] In response to the preliminary subcode being included in the pre-trained model dictionary, the preliminary subcode is directly used,

[0270] In response to the preliminary subcode not being included in the pre-trained model dictionary, further decomposing the preliminary subcode until a final decomposed subcode is included in the pre-trained model dictionary;

[0271] Add special identifiers at the beginning and end of the final subcode;

[0272] The final subcode is mapped one-to-one with the predefined addresses in the pre-trained model dictionary to obtain a subcode address sequence.

[0273] The task of predicting the final sub-code to be blocked and judging whether two code fragments are continuous:

[0274] The final sub-code task of predicting the shielding includes:

[0275] randomly masking at least one final sub-code according to the sub-code address sequence;

[0276] inputting the subcode address sequence and the subcode address sequence of the masked final subcode into a pre-trained model simultaneously;

[0277] The pre-trained model outputs a predicted subcode;

[0278] Calculating a first loss between the predicted subcode and the masked final subcode;

[0279] The step of determining whether two code snippets are consecutive tasks includes:

[0280] Creating code snippet pairs based on the training set data;

[0281] The code snippet pair includes a continuous code snippet and a code snippet unrelated to the continuous code snippet;

[0282] Merging the code snippet pairs into a pre-trained model;

[0283] The pre-trained model outputs a second loss of whether two code snippets are continuous.

[0284] For the specific definition of the code comment generation method device based on language processing, please refer to the definition of the code comment generation method based on language processing above, which will not be repeated here. The various modules in the above-mentioned code comment generation method device based on language processing can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0285] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a code comment generation method based on language processing is implemented.

[0286] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0287] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0288] Input the pre-interpreted code to be annotated into the pre-trained model;

[0289] The pre-trained model parses the pre-interpretation code to obtain a pre-interpretation code fusion feature vector;

[0290] Decoding the pre-interpreted code fusion feature vector according to the decoder and generating code annotations;

[0291] The step of parsing the pre-interpretation code to obtain a pre-interpretation code fusion feature vector includes:

[0292] Parsing the pre-release code to obtain the pre-release code feature;

[0293] Encoding the pre-interpretation code feature into a pre-interpretation code feature vector, wherein the pre-interpretation code feature vector includes a query vector, a key vector, and a value vector;

[0294] Performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector;

[0295] The self-attention calculation includes:

[0296] Calculate the dot product of the query vector and the key vector to obtain a first attention score;

[0297] Calculating a second attention score based on the first attention score and the scaling factor;

[0298] Normalizing the second attention score to a third attention score according to a function;

[0299] The pre-code fusion feature vector is obtained based on the third attention score and the value vector.

[0300] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0301] Construction of pre-trained model:

[0302] Obtaining a data sample including codes and corresponding code comments, and cleaning the data sample to obtain a data set;

[0303] Obtaining features of the code and the corresponding code annotations according to the data set, and encoding the features of the code and the corresponding code annotations into feature vectors;

[0304] Dividing the data set containing the feature vector into training set data and test set data;

[0305] Select a pre-trained initial model;

[0306] Construct the encoder-decoder structure of the pre-trained initial model, set the number of encoder-decoder layers, activation function, and set initial parameter values;

[0307] The training set data is input into the pre-trained initial model, and the code semantic analysis training is completed according to the self-attention mechanism in the pre-trained initial model to obtain a pre-trained model.

[0308] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0309] Features of code and corresponding code comments are obtained, including:

[0310] Parse the code through a word segmenter and generate an abstract syntax tree;

[0311] Traverse each node of the abstract syntax tree according to the breadth-first search algorithm;

[0312] Obtaining node type, node content, and relationships between nodes according to each node;

[0313] The node type, node content, and the relationship between nodes are encoded into a feature vector.

[0314] The feature vector may be combined with a subcode address sequence, including:

[0315] Decomposing the codes in the data set into preliminary sub-codes based on characters;

[0316] In response to the preliminary subcode being included in the pre-trained model dictionary, the preliminary subcode is directly used,

[0317] In response to the preliminary subcode not being included in the pre-trained model dictionary, further decomposing the preliminary subcode until a final decomposed subcode is included in the pre-trained model dictionary;

[0318] Add special identifiers at the beginning and end of the final subcode;

[0319] The final subcode is mapped one-to-one with the predefined addresses in the pre-trained model dictionary to obtain a subcode address sequence.

[0320] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0321] Code semantic analysis training, including the task of predicting the final blocked sub-code and determining whether two code snippets are continuous:

[0322] The final sub-code task of predicting the shielding includes:

[0323] randomly masking at least one final sub-code according to the sub-code address sequence;

[0324] inputting the subcode address sequence and the subcode address sequence of the masked final subcode into a pre-trained model simultaneously;

[0325] The pre-trained model outputs a predicted subcode;

[0326] Calculating a first loss between the predicted subcode and the masked final subcode;

[0327] The step of determining whether two code snippets are consecutive tasks includes:

[0328] Creating code snippet pairs based on the training set data;

[0329] The code snippet pair includes a continuous code snippet and a code snippet unrelated to the continuous code snippet;

[0330] Merging the code snippet pairs into a pre-trained model;

[0331] The pre-trained model outputs a second loss of whether two code snippets are continuous.

[0332] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0333] The first and second losses are obtained according to the cross entropy function:

[0334] The cross entropy function H(P,Q)=-∑P(i)log(Q(i)),

[0335] Among them, P(i) represents the probability distribution of the true result, and Q(i) represents the probability distribution of the predicted result;

[0336] The closer the value of the cross entropy function is to 0, the better the performance of the pre-trained model is;

[0337] The pre-trained model parameters are updated according to the value of the cross entropy function.

[0338] The construction of the pre-trained model also includes obtaining evaluation performance data:

[0339] Inputting the test set data into the pre-training model;

[0340] according to Get the accuracy of code comments;

[0341] according to Get the accuracy of code comments;

[0342] according to Get the recall rate of code comments;

[0343] According to F1 Get the F1 score of code comments;

[0344] Updating the pre-training model parameters according to the accuracy, precision, and recall of the code annotations;

[0345] Among them, TP represents the true positive class, that is, the number of samples predicted by the pre-trained model as positive and actually positive;

[0346] FP stands for false positive, which is the number of samples that the pre-trained model predicts as positive but is actually negative;

[0347] TN stands for true negative, which is the number of samples that the pre-trained model predicts as negative and is actually negative;

[0348] FN stands for false negatives, which is the number of samples that the pre-trained model predicts as negative but is actually positive.

[0349] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0350] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0351] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A code comment generation method based on language processing, characterized in that: include: Input the pre-interpreted code to be annotated into the pre-trained model that has completed the code semantic analysis training; The code semantic analysis training includes determining whether two code snippets are continuous tasks; The task of determining whether two code snippets are continuous includes: creating a pair of code snippets based on training set data; merging the pair of code snippets and inputting them into a pre-training model; the pre-training model outputting a second loss of whether the two code snippets are continuous; The code snippet pair includes a continuous code snippet and a code snippet unrelated to the continuous code snippet; The pre-trained model parses the pre-interpretation code to obtain a pre-interpretation code fusion feature vector; Decoding the pre-interpreted code fusion feature vector according to the decoder and generating code annotations; The step of parsing the pre-interpretation code to obtain a pre-interpretation code fusion feature vector includes: Parsing the pre-release code to obtain the pre-release code feature; Encoding the pre-interpretation code feature into a pre-interpretation code feature vector, wherein the pre-interpretation code feature vector includes a query vector, a key vector, and a value vector; Performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector; The self-attention calculation includes: Calculate the dot product of the query vector and the key vector to obtain a first attention score; Calculating a second attention score based on the first attention score and the scaling factor; Normalizing the second attention score to a third attention score according to a function; The pre-code fusion feature vector is obtained based on the third attention score and the value vector.

2. The method according to claim 1, characterized in that The method also includes constructing the pre-trained model: obtaining a data sample containing code and corresponding code annotations, and cleaning the data sample to obtain a data set; obtaining features of the code and the corresponding code annotations based on the data set, and encoding the features of the code and the corresponding code annotations into feature vectors; Dividing the data set containing the feature vector into training set data and test set data; Select a pre-trained initial model; Construct the encoder-decoder structure of the pre-trained initial model, set the number of encoder-decoder layers, activation function, and set initial parameter values; The training set data is input into the pre-trained initial model, and the code semantic analysis training is completed according to the self-attention mechanism in the pre-trained initial model to obtain a pre-trained model.

3. The method according to claim 2, characterized in that Features of code and corresponding code comments are obtained, including: Parse the code through a word segmenter and generate an abstract syntax tree; Traverse each node of the abstract syntax tree according to the breadth-first search algorithm; Obtaining node type, node content, and relationships between nodes according to each node; The node type, node content, and the relationship between nodes are encoded into a feature vector.

4. The method according to claim 2, characterized in that The feature vector is combined with the subcode address sequence, including: Decomposing the codes in the data set into preliminary sub-codes based on characters; In response to the preliminary subcode being included in the pre-trained model dictionary, directly using the preliminary subcode; in response to the preliminary subcode not being included in the pre-trained model dictionary, further decomposing the preliminary subcode until a final decomposed subcode is included in the pre-trained model dictionary; Add special identifiers at the beginning and end of the final subcode; The final subcode is mapped one-to-one with the predefined addresses in the pre-trained model dictionary to obtain a subcode address sequence.

5. The method according to claim 4, characterized in that The code semantic analysis training includes the task of predicting the final sub-code that is blocked and the task of determining whether two code fragments are continuous: The final sub-code task of predicting the shielding includes: randomly masking at least one final sub-code according to the sub-code address sequence; inputting the subcode address sequence and the subcode address sequence of the masked final subcode into a pre-trained model simultaneously; The pre-trained model outputs a predicted subcode; A first loss between the predicted subcode and the masked final subcode is calculated according to the predicted subcode.

6. The method according to claim 5, characterized in that The first and second losses are obtained according to the cross entropy function: The cross entropy function H(P,Q)=-∑P(i)log(Q(i)), Among them, P(i) represents the probability distribution of the true result, and Q(i) represents the probability distribution of the predicted result; The closer the value of the cross entropy function is to 0, the better the performance of the pre-trained model is; The pre-trained model parameters are updated according to the value of the cross entropy function.

7. The method according to claim 2, characterized in that The construction of the pre-trained model also includes obtaining evaluation performance data: Inputting the test set data into the pre-training model; according to Get the accuracy of code comments; according to Get the accuracy of code comments; according to Get the recall rate of code comments; according to Get the F1 score of code comments; Updating the pre-training model parameters according to the accuracy, precision, and recall of the code annotations; Among them, TP represents the true positive class, that is, the number of samples predicted by the pre-trained model as positive and actually positive; FP stands for false positive, which is the number of samples that the pre-trained model predicts as positive but is actually negative; TN stands for true negative, which is the number of samples that the pre-trained model predicts as negative and is actually negative; FN stands for false negatives, which is the number of samples that the pre-trained model predicts as negative but is actually positive.

8. A device for implementing the method for generating code comments based on language processing according to any one of claims 1 to 7, characterized in that: The device comprises: An input module, used to input the pre-interpreted code to be annotated into the pre-trained model; A parsing module, configured to parse the pre-interpreted code according to the pre-trained model to obtain features of the pre-interpreted code; An encoding module, configured to encode the pre-interpretation code feature into a pre-interpretation code feature vector; A calculation module, configured to perform self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector; The generating module is used for decoding the pre-interpreted code fusion feature vector according to the decoder and generating code annotations.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images