Code annotation generation method and device based on language processing, computer equipment and storage medium

Abstract

The invention relates to a code annotation generation method and device based on language processing, computer equipment and a storage medium. The method comprises the following steps: inputting a pre-release code to be annotated into a pre-training model; the pre-training model analyzes the pre-release code to obtain a pre-release code fusion feature vector; and decoding the pre-released code fusion feature vector according to a decoder, and generating a code annotation. By adopting the method, the structure and semanteme of the code can be more accurately analyzed and understood through language processing, and high-quality code annotations are automatically generated.

Description

Technical Field

[0001] The present invention relates to the field of language processing, and in particular to a code comment generation method based on language processing. Background Art

[0002] With the rapid development of software development, code readability and maintainability have become important criteria for measuring software quality. As an important means to improve code readability and maintainability, comments are becoming increasingly important. However, traditional comment methods mainly rely on developers' experience and specifications for manual writing, which is not only inefficient, but also difficult to ensure the accuracy and consistency of comments. Therefore, how to automatically generate high-quality code comments has become an important research direction in the field of computer technology.

[0003] At present, the methods for generating code comments include template-based methods, statistics-based methods, machine learning-based methods, etc. The closest existing technology is to use machine learning and natural language processing technology to generate code comments. Machine learning and natural language processing technology can parse and process natural language text to extract semantic information and structural information. Although the automation level of code comments has been improved to a certain extent, it also faces some challenges. The syntax and semantics of code are more complex than natural language and require more precise analysis and understanding. Existing methods have limited ability to understand the semantics of code, and the generated comments may not be accurate and comprehensive enough. In addition, elements such as variables, functions, and classes in the code have specific semantics and uses. When dealing with such complex code structures, existing methods find it difficult to accurately analyze the semantics of the code in context. Summary of the invention

[0004] Based on this, it is necessary to provide a language processing-based code comment generation method, device, computer equipment and storage medium that can accurately analyze and understand the code semantics in combination with the context to address the above technical problems.

[0005] On the one hand, a method for generating code comments based on language processing is provided, the method comprising:

[0006] Input the pre-interpreted code to be annotated into the pre-trained model;

[0007] The pre-trained model parses the pre-interpretation code to obtain a pre-interpretation code fusion feature vector;

[0008] Decoding the pre-interpreted code fusion feature vector according to the decoder and generating code annotations;

[0009] The pre-interpretation code is parsed to obtain a pre-interpretation code fusion feature vector, including:

[0010] Parsing the pre-release code to obtain the pre-release code feature;

[0011] Encoding the pre-interpretation code feature into a pre-interpretation code feature vector, wherein the pre-interpretation code feature vector includes a query vector, a key vector, and a value vector;

[0012] Performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector;

[0013] The self-attention calculation includes:

[0014] Calculate the dot product between the query vector and the key vector to obtain a first attention score;

[0015] Calculate a second attention score based on the first attention score and the scaling factor;

[0016] Normalizing the second attention score to a third attention score according to a function;

[0017] The pre-code fusion feature vector is obtained based on the third attention score and the value vector.

[0018] In one embodiment, the pre-training model is also constructed:

[0019] Obtaining a data sample including codes and corresponding code comments, and cleaning the data sample to obtain a data set;

[0020] According to the data set, features of the code and the corresponding code annotations are obtained, and the features of the code and the corresponding code annotations are encoded into feature vectors;

[0021] Dividing the data set containing the feature vector into training set data and test set data;

[0022] Select the pre-trained initial model;

[0023] Construct the encoder-decoder structure of the pre-trained initial model, set the number of encoder-decoder layers, activation function, and set initial parameter values;

[0024] The training set data is input into the pre-trained initial model, and the code semantic analysis training is completed according to the self-attention mechanism in the pre-trained initial model to obtain a pre-trained model.

[0025] In one embodiment, the features of acquiring the code and the corresponding code annotations include:

[0026] Parse the code through a tokenizer and generate an abstract syntax tree;

[0027] Traverse each node of the abstract syntax tree according to the breadth-first search algorithm;

[0028] According to each node, obtain the node type, node content, and the relationship between the nodes;

[0029] The node type, node content, and the relationship between nodes are encoded into a feature vector.

[0030] In one embodiment, the feature vector may be combined with a subcode address sequence, including:

[0031] Decomposing the codes in the data set into preliminary sub-codes based on characters;

[0032] In response to the preliminary subcode being included in the pre-trained model dictionary, directly using the preliminary subcode,

[0033] In response to the preliminary subcode not being included in the pre-trained model dictionary, further decomposing the preliminary subcode until the decomposed final subcode is included in the pre-trained model dictionary;

[0034] Add special identifiers at the beginning and end of the final subcode;

[0035] The final subcode is mapped one-to-one with the predefined addresses in the pre-trained model dictionary to obtain a subcode address sequence.

[0036] In one embodiment, the code semantic analysis training includes the tasks of predicting the final sub-code to be blocked and judging whether two code fragments are continuous:

[0037] The prediction of the final sub-code task being shielded includes:

[0038] randomly masking at least one final sub-code according to the sub-code address sequence;

[0039] inputting the subcode address sequence and the subcode address sequence of the masked final subcode into a pre-trained model simultaneously;

[0040] The pre-trained model outputs a predicted subcode;

[0041] Calculate a first loss between the predicted subcode and the masked final subcode according to the predicted subcode;

[0042] The step of determining whether two code snippets are continuous tasks includes:

[0043] Creating code snippet pairs according to the training set data;

[0044] Wherein, the code snippet pair includes a continuous code snippet and a code snippet unrelated to the continuous code snippet;

[0045] Merging the code snippet pairs and inputting them into a pre-trained model;

[0046] The pre-trained model outputs a second loss of whether two code snippets are consecutive.

[0047] In one embodiment, the first loss and the second loss are obtained according to the cross entropy function:

[0048] The cross entropy function H(P,Q)=-∑P(i)log(Q(i)),

[0049] Among them, P(i) represents the probability distribution of the true result, and Q(i) represents the probability distribution of the predicted result;

[0050] The closer the value of the cross entropy function is to 0, the better the performance of the pre-trained model is;

[0051] The pre-trained model parameters are updated according to the value of the cross entropy function.

[0052] In one embodiment, the construction of the pre-training model further includes obtaining evaluation performance data:

[0053] Inputting the test set data into the pre-trained model;

[0054] according to Get the accuracy of code comments;

[0055] according to Get the accuracy of code comments;

[0056] according to Get the recall rate of code comments;

[0057] According to F1 Get the F1 score of the code comments;

[0058] Updating the pre-trained model parameters according to the accuracy, precision, and recall of the code annotations;

[0059] Among them, TP represents the true positive class, that is, the number of samples predicted by the pre-trained model as positive and actually positive;

[0060] FP stands for false positive, which is the number of samples that the pre-trained model predicts to be positive but is actually negative;

[0061] TN stands for true negatives, which is the number of samples that the pre-trained model predicts to be negative and are actually negative;

[0062] FN stands for false negatives, which is the number of samples that the pre-trained model predicts to be negative but are actually positive.

[0063] On the other hand, a device for generating code comments based on language processing is provided, the device comprising:

[0064] An input module, used for inputting the pre-interpreted code to be annotated into the pre-trained model;

[0065] An acquisition module is used for the pre-trained model to parse the pre-release code and obtain a pre-release code fusion feature vector;

[0066] The output module is used to decode the pre-interpreted code fusion feature vector according to the decoder and generate code annotations.

[0067] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0068] Input the pre-interpreted code to be annotated into the pre-trained model;

[0069] The pre-trained model parses the pre-interpretation code to obtain a pre-interpretation code fusion feature vector;

[0070] Decoding the pre-interpreted code fusion feature vector according to the decoder and generating code annotations;

[0071] The pre-interpretation code is parsed to obtain a pre-interpretation code fusion feature vector, including:

[0072] Parsing the pre-release code to obtain the pre-release code feature;

[0073] Encoding the pre-interpretation code feature into a pre-interpretation code feature vector, wherein the pre-interpretation code feature vector includes a query vector, a key vector, and a value vector;

[0074] Performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector;

[0075] The self-attention calculation includes:

[0076] Calculate the dot product between the query vector and the key vector to obtain a first attention score;

[0077] Calculate a second attention score based on the first attention score and the scaling factor;

[0078] Normalizing the second attention score to a third attention score according to a function;

[0079] The pre-code fusion feature vector is obtained based on the third attention score and the value vector.

[0080] In another aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0081] Input the pre-interpreted code to be annotated into the pre-trained model;

[0082] The pre-trained model parses the pre-interpretation code to obtain a pre-interpretation code fusion feature vector;

[0083] Decoding the pre-interpreted code fusion feature vector according to the decoder and generating code annotations;

[0084] The pre-interpretation code is parsed to obtain a pre-interpretation code fusion feature vector, including:

[0085] Parsing the pre-release code to obtain the pre-release code feature;

[0086] Encoding the pre-interpretation code feature into a pre-interpretation code feature vector, wherein the pre-interpretation code feature vector includes a query vector, a key vector, and a value vector;

[0087] Performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector;

[0088] The self-attention calculation includes:

[0089] Calculate the dot product between the query vector and the key vector to obtain a first attention score;

[0090] Calculate a second attention score based on the first attention score and the scaling factor;

[0091] Normalizing the second attention score to a third attention score according to a function;

[0092] The pre-code fusion feature vector is obtained based on the third attention score and the value vector.

[0093] The above-mentioned method, device, computer equipment and storage medium for generating code comments based on language processing utilize the parsing ability of the pre-trained model, parse the pre-interpreted code and extract its features to generate the pre-interpreted code feature vector. This process ensures that the model can capture the key information points of the code, including important details such as the scope of variables, parameters and return values ​​of functions. The construction of the feature vector enables the complexity of the code to be quantified and expressed, which is convenient for subsequent processing. Among them, the self-attention mechanism further enhances the model's understanding of the local and global relationships of the code. The obtained pre-interpreted code fusion feature vector can more accurately reflect the interaction between code elements, ensuring that the generated comments not only conform to the grammatical specifications, but also accurately describe the functional intent of the code. This method is particularly suitable for code snippets containing complex logic and multi-level nested structures, and improves the quality of comment generation. Finally, the decoder generates code comments based on the pre-interpreted code fusion feature vector, and optimizes the model structure through the cross entropy function and performance evaluation, ensuring that the annotation content is accurate while maintaining the fluency of the natural language description and the consistency of the annotation, further improving the reliability and professionalism of software development, so that the structure and semantics of the code can be more accurately analyzed and understood through language processing, and high-quality code comments can be automatically generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 An application environment diagram of a method for generating code comments based on language processing in one embodiment;

[0095] Figure 2 A schematic diagram of a flow chart of a method for generating code comments based on language processing in one embodiment;

[0096] Figure 3 A schematic diagram of the steps of self-attention calculation according to an embodiment;

[0097] Figure 4 It is a structural block diagram of a code comment generation device based on language processing in one embodiment;

[0098] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0099] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0100] The code comment generation method based on language processing provided in this application can be applied to Figure 1In the application environment shown, the terminal 102 communicates with the server 104 through the network. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0101] In one embodiment, Figure 2 As shown, a code comment generation method based on language processing is provided, and the method is applied to Figure 1 The server in the example is used as an example to illustrate, including the following steps:

[0102] Step 110, inputting the pre-interpreted code to be annotated into the pre-trained model;

[0103] The pre-annotated code may be in any programming language, including but not limited to Python, Java, JavaScript, C, C++, and PHP.

[0104] In one embodiment, before the pre-trained model parses the pre-interpreted code, data is also collected and cleaned.

[0105] Data collection includes collecting data samples containing code and corresponding code annotations from sources such as open source projects, code repositories, and online platforms.

[0106] Data cleaning refers to removing noise, errors and inconsistencies in the data, such as format errors, non-standard comments, etc. The code is divided into basic units, such as identifiers, keywords, operators, etc. Then some identifiers, such as variable names, function names, etc. are removed or replaced with common tags to improve the generalization ability of the model.

[0107] Step 120, the pre-trained model parses the pre-interpretation code to obtain a pre-interpretation code fusion feature vector;

[0108] In one embodiment, obtaining the pre-interpreted code features includes obtaining the grammatical features, word embedding features, etc. of the code, which may be features from different levels of the code. In obtaining the above code features, an abstract syntax tree may be used to analyze the code structure, or a control flow graph or a program dependency graph may be used to capture the code dependency.

[0109] In one embodiment, encoding into a feature vector refers to converting the processed code into a numerical representation for model processing, which may include using word embedding techniques, such as Word2Vec, GloVe, or BERT embedding layers, to map each word in the code to a vector in a high-dimensional space.

[0110] Step S130, performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector;

[0111] In one embodiment, Figure 3 The method of performing self-attention calculation on the pre-interpretation code feature vector to obtain the pre-interpretation code fusion feature vector comprises the following steps:

[0112] Step S131, calculating the dot product between the query vector and the key vector to obtain a first attention score;

[0113] Step S132, calculating according to the first attention score and the scaling factor to obtain a second attention score;

[0114] Step S133, normalizing the second attention score to a third attention score according to a function;

[0115] Step S134, obtaining a pre-code fusion feature vector based on the third attention score and value vector calculation.

[0116] Among them, the query vector represents the information that the model is "querying". The query vector at each position tries to find relevant information from other positions; the key vector represents the summary or representation of the information at each position in the model, which is used to compare with the query vectors at other positions to determine the correlation between them; the value vector contains the actual content information. When the query vector successfully matches the key vector at a certain position, the value vector at that position will be used as part of the output. The above three vectors are obtained by multiplying the input embedding vector with the weight matrix W_Q, the weight matrix W_k, and the weight matrix W_v respectively. The weight matrix is ​​part of the pre-trained model.

[0117] Specifically, for each position i in a given input sequence, we first obtain the input embedding vector E at that position: i , calculate the query vector Q i =E i ·W Q , calculate the key vector K i =E i ·W K , calculate the query vector V i =E i ·W V .

[0118] In one embodiment, a first attention score is obtained by calculating the dot product between the query vector and the key vector, a second attention score is obtained by calculating the first attention score and a scaling factor, and the second attention score is normalized to a third attention score according to a function; a pre-code fusion feature vector is obtained by calculating the third attention score and a value vector.

[0119]

[0120] Specifically,

[0121] Calculate the query vector q 1 and all key vectors k 1 ,k 2 ,k 3 The dot product of 1· k 1

[0122] q 1· k 2 ,q 1· k 3 .

[0123]

[0124] Divide the above first attention scores by the scaling factor to get the second attention score q 1· k 1 / 8,q 1· k 2 / 8,q 1· k 3 / 8. The scaling factor is the square root of the dimension of the key vector.

[0125]

[0126] Use the activation function to normalize the second attention score to get the third attention score X 11 , X 12 , X 13 .

[0127]

[0128] Multiply these normalized scores by the value vector V 1 、V 2 、V 3 , and sum the obtained vectors to obtain the pre-code feature vector Z 1 . Z 1 is the decompression fusion feature vector of the first code fragment of the pre-decompression code. The vectors of the remaining code fragments in the pre-decompression code can be obtained in the same way.

[0129] In one embodiment, the decoder can be a recurrent neural network (RNN) decoder, a Transformer decoder, a convolutional neural network (CNN) decoder, a conditional variational autoencoder (CVAE) decoder, etc.

[0130] The above-mentioned code comment generation method based on language processing improves the quality and accuracy of code comments through a pre-trained model and a self-attention mechanism. The pre-interpreted code to be annotated is input into the pre-trained model, which parses the code and extracts its features, which reflect the grammatical and semantic information of the code. The extracted features are then encoded into pre-interpreted code feature vectors, which contain the key information in the code. Self-attention calculations are performed on these feature vectors to better understand the contextual relationship of the code. Specifically, the self-attention mechanism obtains preliminary attention scores by calculating the dot product of the query vector and the key vector, and converts these scores into more accurate attention scores through scaling and normalization. These final attention scores are combined with the value vector to generate a fused feature vector, which represents the comprehensive information of each part of the code. Finally, the decoder uses these fused feature vectors to generate code comments. This process can not only deeply understand the logic and function of the code, but also generate more contextual comments, thereby improving the readability and maintainability of the code. In this way, the scheme can automatically generate high-quality code comments, reduce the workload of manual comment writing, and improve the accuracy and consistency of code documentation.

[0131] In one embodiment, the construction of the pre-training model is also included:

[0132] Obtaining a data sample including codes and corresponding code comments, and cleaning the data sample to obtain a data set;

[0133] According to the data set, features of the code and the corresponding code annotations are obtained, and the features of the code and the corresponding code annotations are encoded into feature vectors;

[0134] Dividing the data set containing the feature vector into training set data and test set data;

[0135] Select the pre-trained initial model;

[0136] Construct the encoder-decoder structure of the pre-trained initial model, set the number of encoder-decoder layers, activation function, and set initial parameter values;

[0137] The training set data is input into the pre-trained initial model, and the code semantic analysis training is completed according to the self-attention mechanism in the pre-trained initial model to obtain a pre-trained model.

[0138] The data set includes training set, validation set and test set.

[0139] The training set is used to train the model. The model learns the characteristics and patterns of the data on this dataset. The training set usually occupies a major part of the total dataset, such as 70% to 80%.

[0140] The validation set is used to adjust the model's hyperparameters and perform model selection. The existence of the validation set helps to evaluate the generalization ability of the model during the training process. The proportion of the validation set is usually 10% to 20%. In some cases, the validation set may also be used as part of cross-validation.

[0141] The test set is used to evaluate the performance of the final model. The test set is used after model training and hyperparameter adjustment to test the generalization ability of the model on unseen data. The test set usually occupies the remaining part of the total dataset, such as 10% to 20%.

[0142] Among them, the pre-trained initial model can be a Transformer-based model, such as GPT-4, CodeBERT, Codex, etc.

[0143] In one embodiment, the encoder includes an input layer, an encoding layer, and an output layer.

[0144] The input layer refers to input embeddings, including Token Embeddings, which are used to convert input word tokens into word vectors; Segment Embeddings are used to distinguish code pairs in the input, usually using identifiers of 0 and 1; Position Embeddings: used to add position information to each position in the input sequence to capture the order of words. These three embeddings are added together to form the final input embedding.

[0145] The encoding layer structure includes a self-attention mechanism that calculates the weights of each word at all positions in the sequence and uses these weights to sum up to get context-related word representations; a feedforward neural network, where the output of the self-attention layer is processed by two layers of fully connected networks to increase the model's expressive power. Layer normalization and residual connections help with model training and stability.

[0146] The output layer includes the MLM output layer and the NSP output layer. The MLM output layer is used to predict the masked codes. During the training process, the model randomly masks some codes in the input sequence, and then trains the model to predict the codes at these masked positions. The NSP output layer is used to determine whether two sentences are adjacent code segments, helping the model to understand the relationship between sentences.

[0147] The construction of the above pre-trained model effectively improves the code understanding and generation capabilities through steps such as data cleaning, feature encoding, and model training. Through the highly adaptable encoder-decoder architecture and self-attention mechanism, the model can more accurately capture the semantics and structure of the code, improving the automation level and efficiency of code processing.

[0148] In one embodiment, the features of acquiring the code and the corresponding code annotations include:

[0149] Parse the code through a tokenizer and generate an abstract syntax tree;

[0150] Traverse each node of the abstract syntax tree according to the breadth-first search algorithm;

[0151] According to each node, obtain the node type, node content, and the relationship between the nodes;

[0152] The node type, node content, and the relationship between nodes are encoded into a feature vector.

[0153] The feature vector may be combined with a subcode address sequence, including:

[0154] Decomposing the codes in the data set into preliminary sub-codes based on characters;

[0155] In response to the preliminary subcode being included in the pre-trained model dictionary, directly using the preliminary subcode,

[0156] In response to the preliminary subcode not being included in the pre-trained model dictionary, further decomposing the preliminary subcode until the decomposed final subcode is included in the pre-trained model dictionary;

[0157] Add special identifiers at the beginning and end of the final subcode;

[0158] The final subcode is mapped one-to-one with the predefined addresses in the pre-trained model dictionary to obtain a subcode address sequence.

[0159] Among them, the tokenizer is used to split the code into meaningful units, such as subcodes or characters. Different tokenizers can be used for code and comments. For example, for code, you can use the tokenizer in codeBERT or graph-based models; for comments, use a normal text tokenizer.

[0160] The abstract syntax tree is a tree diagram that represents the structure of the code, which can show the grammatical structure and hierarchical relationship of the code. It breaks down the code into nodes, each node represents a grammatical unit, and helps the model parse and understand the logical structure of the code.

[0161] The acquisition of the subcode address sequence includes: splitting the codes in the data set into preliminary subcodes at the character level; if the preliminary subcode is in the pre-trained model dictionary, directly using the preliminary subcode; if the preliminary subcode is not in the pre-trained model dictionary, further decomposing it until the final subcode is included in the dictionary; adding special identifiers at the beginning and end of the final subcode to mark its boundaries; establishing a one-to-one correspondence between the final subcode and the predefined address in the pre-trained model dictionary to form a subcode address sequence.

[0162] By constructing an abstract syntax tree and obtaining the sub-code address sequence, the code structure and surface meaning can be parsed. By combining surface and deep analysis, the complex code structure can be converted into a format that the model can understand, and the code meaning can be deeply understood, thereby improving the model's code processing capabilities and accuracy.

[0163] In one embodiment, the code semantic analysis training includes the tasks of predicting the shielded final sub-code and judging whether two code fragments are continuous:

[0164] The randomly masking at least one final subcode;

[0165] inputting the subcode address sequence and the subcode address sequence of the masked final subcode into a pre-trained model simultaneously;

[0166] The pre-trained model outputs a predicted subcode;

[0167] Calculate a first loss between the predicted subcode and the masked final subcode according to the predicted subcode;

[0168] The step of determining whether two code snippets are continuous tasks includes:

[0169] Creating code snippet pairs according to the training set data;

[0170] Wherein, the code snippet pair includes a continuous code snippet and a code snippet unrelated to the continuous code snippet;

[0171] Merging the code snippet pairs and inputting them into a pre-trained model;

[0172] The pre-trained model outputs a second loss of whether two code snippets are consecutive.

[0173] Specifically, taking Python language as an example, the task of predicting the final sub-code that is blocked is as follows:

[0174] Among them, tokenizer is the final subcode.

[0175] #Start training

[0176] trainer.train()

[0177] #Mask prediction

[0178] def predict_masked_word(text):

[0179] inputs=tokenizer(text,return_tensors='pt')

[0180] input_ids=inputs['input_ids']

[0181] attention_mask=inputs['attention_mask']

[0182] # Find the location of the mask

[0183] mask_token_index=torch.where(input_ids==

[0184] tokenizer.mask_token_id)[1]

[0185] #Model makes predictions

[0186] with torch.no_grad():

[0187] outputs=model(input_ids, attention_mask=attention_mask)

[0188] predictions = outputs.logits

[0189] #Predict the vocabulary of the mask position

[0190] predicted_ids=torch.argmax(predictions[0,mask_token_index],dim=-1)

[0191] predicted_tokens=tokenizer.convert_ids_to_tokens(predicted_ids)

[0192] return predicted_tokens[0]

[0193] # Example prediction

[0194] text="def add(a,b):return a+[MASK]"

[0195] predicted_word=predict_masked_word(text)

[0196] print(f"Predicted word: {predicted_word}")

[0197] Specifically, taking Python language as an example, the following example is used to determine whether two code snippets are continuous tasks:

[0198] #Start training

[0199] trainer.train()

[0200] # Make predictions

[0201] def predict_next_sentence(sentence_a,sentence_b):

[0202] inputs=tokenizer.encode_plus(sentence_a, sentence_b, return_tensors='pt', max_length=512, truncation=True)

[0203] input_ids=inputs['input_ids']

[0204] attention_mask=inputs['attention_mask']

[0205] with torch.no_grad():

[0206] outputs=model(input_ids, attention_mask=attention_mask)

[0207] logits = outputs.logits

[0208] prediction=torch.argmax(logits,dim=1).item()

[0209] return prediction==0

[0210] # Example prediction

[0211] sentence_a = "def add(a,b):"

[0212] sentence_b="return a+b"

[0213] is_next_sentence=predict_next_sentence(sentence_a,sentence_b)

[0214] print(f"Is sentence B the next sentence of sentence A? {'Yes' if is_next_sentence else 'No'}")

[0215] In this embodiment, the task of predicting the masked final sub-code and the task of judging whether two code snippets are continuous can significantly improve the accuracy of the model's understanding of the code and the generation of comments. The task of predicting the masked final sub-code helps the model learn the structure and semantics of the code by training the model to predict the masked code vocabulary, thereby generating more relevant comments. The task of judging whether two code snippets are continuous enhances the model's understanding of the logical relationship between code snippets and comments, helping the model identify which code snippets are usually followed by specific comments.

[0216] In one embodiment, an adaptive masking strategy may be defined, including:

[0217] Set a masking strategy to dynamically determine the number and location of masks. Adjust the mask ratio based on code complexity: for example, for complex code snippets, you can increase the mask ratio to ensure that the model learns more context information. For simple code snippets, reduce the mask ratio to avoid overtraining.

[0218] It is also possible to do context-sensitive masking, where mask locations are chosen dynamically based on the frequency and importance of functions, variables, or classes in the code.

[0219] Compared to fixed-ratio masks, this strategy adjusts the mask settings based on the specific content and context of the code snippet.

[0220] Specifically, when processing code data, the code is first segmented and encoded.

[0221] Count the frequency of different words or symbols in the code snippet and determine the location of the mask based on these statistics. For example, frequently occurring words may be masked out to force the model to focus on uncommon words.

[0222] Use a pre-trained model or heuristics to assess the importance of code elements and select the more important parts to mask. For example, masking function names and parameters can encourage the model to generate more accurate comments.

[0223] During each training, a portion of the words in the code snippet is randomly masked according to the above strategy. The model will process different masks in each training, enhancing its ability to understand the code context.

[0224] The adaptive mask strategy can significantly improve the generalization ability of the model and the quality of annotation generation during training. By dynamically adjusting the position and scale of the mask, the model can better understand the code context and capture key information, thereby generating more accurate and informative annotations. This strategy not only helps to reduce overfitting and improve the robustness of the model, but also optimizes the ability to handle complex code, making the model more stable and reliable in practical applications.

[0225] In one embodiment, the first loss and the second loss are obtained according to the cross entropy function:

[0226] The cross entropy function H(P,Q)=-∑P(i)log(Q(i)),

[0227] Among them, P(i) represents the probability distribution of the true result, and Q(i) represents the probability distribution of the predicted result;

[0228] The closer the value of the cross entropy function is to 0, the better the performance of the pre-trained model is;

[0229] The pre-trained model parameters are updated according to the value of the cross entropy function.

[0230] In one embodiment, the construction of the pre-training model further includes obtaining evaluation performance data:

[0231] Inputting the test set data into the pre-trained model;

[0232] according to Get the accuracy of code comments;

[0233] according to Get the accuracy of code comments;

[0234] according to Get the recall rate of code comments;

[0235] According to F1 Get the F1 score of the code comments;

[0236] Updating the pre-trained model parameters according to the accuracy, precision, and recall of the code annotations;

[0237] Among them, TP represents the true positive class, that is, the number of samples predicted by the pre-trained model as positive and actually positive;

[0238] FP stands for false positive, which is the number of samples that the pre-trained model predicts to be positive but is actually negative;

[0239] TN stands for true negatives, which is the number of samples that the pre-trained model predicts to be negative and are actually negative;

[0240] FN stands for false negatives, which is the number of samples that the pre-trained model predicts to be negative but are actually positive.

[0241] Specifically, the cross entropy function obtains the loss as follows:

[0242] #Assume there are two class labels

[0243] #Probability distribution of true labels (true results)

[0244] P = torch.tensor([0.0,1.0])#For example, the probability distribution of the real annotation category

[0245] #Model's predicted probability distribution (prediction results)

[0246] Q = torch.tensor([0.2,0.8]) #Model's predicted probability distribution for annotation categories

[0247] Where P is the probability distribution of the true annotation category. For example, the true category is the second category, and the probability distribution is [0.0, 1.0].

[0248] Q is the probability distribution of annotation categories predicted by the model. For example, the probability predicted by the model is [0.2, 0.8].

[0249] The cross entropy loss function is calculated as follows:

[0250] H(P,Q)=-∑iP(i)log(Q(i))H(P,Q)=-i∑P(i)log(Q(i))

[0251] Where P(i) is the probability distribution of the true label and Q(i) is the probability distribution of the model's prediction. In this example:

[0252] H(P,Q)=-(0.0·log(0.2)+1.0·log(0.8))

[0253] = -log(0.8)H(P,Q)

[0254] =-(0.0·log(0.2)+1.0·log(0.8))

[0255] = -log(0.8)

[0256] The closer the cross entropy loss value is to 0, the closer the predicted probability distribution is to the true probability distribution. In the above example, if the model's predicted probability distribution is close to the true [0.0, 1.0] distribution, the cross entropy loss value will be small, indicating that the model performs well. Conversely, a large loss value indicates that the model's predictions do not match the true annotations and the model needs to be improved.

[0257] In this embodiment, the cross entropy loss function is used to calculate the error between the model prediction and the true annotation, which helps to accurately adjust the model parameters, thereby improving the quality of the generated annotations. Through the cross entropy loss function, the model's errors in annotation generation can be quantified and its parameters can be optimized so that the predicted distribution is closer to the true label. In addition, combining the accuracy, precision, and recall of code annotations to evaluate model performance can more comprehensively measure the effect and quality of annotation generation. The accuracy ensures that the generated annotations correctly match the real annotations, the precision measures the proportion of relevant content in the generated annotations, and the recall evaluates the ability of the generated annotations to cover the real annotations. Combining these indicators, the model can better capture the relationship between code and annotations, thereby improving the accuracy and effectiveness of annotation generation.

[0258] It should be understood that although Figure 2-3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0259] In one embodiment, Figure 4 As shown, a code comment generation device based on language processing is provided, comprising: an input module, an acquisition module and an output module, wherein:

[0260] An input module 21, used for inputting the pre-interpreted code to be annotated into the pre-trained model;

[0261] An acquisition module 22 is used for the pre-trained model to parse the pre-interpreted code and obtain a pre-interpreted code fusion feature vector;

[0262] The output module 23 is used to decode the pre-interpreted code fusion feature vector according to the decoder and generate code annotations.

[0263] In one embodiment, the device acquisition module 22 is used for the pre-trained model to parse the pre-interpreted code and obtain the pre-interpreted code fusion feature vector, including:

[0264] Parse the code through a tokenizer and generate an abstract syntax tree;

[0265] Traverse each node of the abstract syntax tree according to the breadth-first search algorithm;

[0266] According to each node, obtain the node type, node content, and the relationship between the nodes;

[0267] The node type, node content, and the relationship between nodes are encoded into a feature vector.

[0268] Decomposing the codes in the data set into preliminary sub-codes based on characters;

[0269] In response to the preliminary subcode being included in the pre-trained model dictionary, directly using the preliminary subcode,

[0270] In response to the preliminary subcode not being included in the pre-trained model dictionary, further decomposing the preliminary subcode until the decomposed final subcode is included in the pre-trained model dictionary;

[0271] Add special identifiers at the beginning and end of the final subcode;

[0272] The final subcode is mapped one-to-one with the predefined addresses in the pre-trained model dictionary to obtain a subcode address sequence.

[0273] Tasks for predicting the final sub-code to be blocked and judging whether two code snippets are continuous:

[0274] The prediction of the final sub-code task being shielded includes:

[0275] randomly masking at least one final sub-code according to the sub-code address sequence;

[0276] inputting the subcode address sequence and the subcode address sequence of the masked final subcode into a pre-trained model simultaneously;

[0277] The pre-trained model outputs a predicted subcode;

[0278] Calculate a first loss between the predicted subcode and the masked final subcode according to the predicted subcode;

[0279] The step of determining whether two code snippets are continuous tasks includes:

[0280] Creating code snippet pairs according to the training set data;

[0281] Wherein, the code snippet pair includes a continuous code snippet and a code snippet unrelated to the continuous code snippet;

[0282] Merging the code snippet pairs and inputting them into a pre-trained model;

[0283] The pre-trained model outputs a second loss of whether two code snippets are consecutive.

[0284] For the specific limitations of the code comment generation method device based on language processing, please refer to the limitations of the code comment generation method based on language processing above, which will not be repeated here. The various modules in the above-mentioned code comment generation method device based on language processing can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0285] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a code annotation generation method based on language processing is implemented.

[0286] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0287] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0288] Input the pre-interpreted code to be annotated into the pre-trained model;

[0289] The pre-trained model parses the pre-interpretation code to obtain a pre-interpretation code fusion feature vector;

[0290] Decoding the pre-interpreted code fusion feature vector according to the decoder and generating code annotations;

[0291] The pre-interpretation code is parsed to obtain a pre-interpretation code fusion feature vector, including:

[0292] Parsing the pre-release code to obtain the pre-release code feature;

[0293] Encoding the pre-interpretation code feature into a pre-interpretation code feature vector, wherein the pre-interpretation code feature vector includes a query vector, a key vector, and a value vector;

[0294] Performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector;

[0295] The self-attention calculation includes:

[0296] Calculate the dot product between the query vector and the key vector to obtain a first attention score;

[0297] Calculate a second attention score based on the first attention score and the scaling factor;

[0298] Normalizing the second attention score to a third attention score according to a function;

[0299] The pre-code fusion feature vector is obtained based on the third attention score and the value vector.

[0300] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0301] Construction of pre-trained model:

[0302] Obtaining a data sample including codes and corresponding code comments, and cleaning the data sample to obtain a data set;

[0303] According to the data set, features of the code and the corresponding code annotations are obtained, and the features of the code and the corresponding code annotations are encoded into feature vectors;

[0304] Dividing the data set containing the feature vector into training set data and test set data;

[0305] Select the pre-trained initial model;

[0306] Construct the encoder-decoder structure of the pre-trained initial model, set the number of encoder-decoder layers, activation function, and set initial parameter values;

[0307] The training set data is input into the pre-trained initial model, and the code semantic analysis training is completed according to the self-attention mechanism in the pre-trained initial model to obtain a pre-trained model.

[0308] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0309] Features of obtaining code and corresponding code comments include:

[0310] Parse the code through a tokenizer and generate an abstract syntax tree;

[0311] Traverse each node of the abstract syntax tree according to the breadth-first search algorithm;

[0312] According to each node, obtain the node type, node content, and the relationship between the nodes;

[0313] The node type, node content, and the relationship between nodes are encoded into a feature vector.

[0314] The feature vector may be combined with a subcode address sequence, including:

[0315] Decomposing the codes in the data set into preliminary sub-codes based on characters;

[0316] In response to the preliminary subcode being included in the pre-trained model dictionary, directly using the preliminary subcode,

[0317] In response to the preliminary subcode not being included in the pre-trained model dictionary, further decomposing the preliminary subcode until the decomposed final subcode is included in the pre-trained model dictionary;

[0318] Add special identifiers at the beginning and end of the final subcode;

[0319] The final subcode is mapped one-to-one with the predefined addresses in the pre-trained model dictionary to obtain a subcode address sequence.

[0320] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0321] Code semantic analysis training, including the task of predicting the final sub-code that is blocked and the task of judging whether two code fragments are continuous:

[0322] The prediction of the final sub-code task being shielded includes:

[0323] randomly masking at least one final sub-code according to the sub-code address sequence;

[0324] inputting the subcode address sequence and the subcode address sequence of the masked final subcode into a pre-trained model simultaneously;

[0325] The pre-trained model outputs a predicted subcode;

[0326] Calculate a first loss between the predicted subcode and the masked final subcode according to the predicted subcode;

[0327] The step of determining whether two code snippets are continuous tasks includes:

[0328] Creating code snippet pairs according to the training set data;

[0329] Wherein, the code snippet pair includes a continuous code snippet and a code snippet unrelated to the continuous code snippet;

[0330] Merging the code snippet pairs and inputting them into a pre-trained model;

[0331] The pre-trained model outputs a second loss of whether two code snippets are consecutive.

[0332] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0333] The first and second losses are obtained according to the cross entropy function:

[0334] The cross entropy function H(P,Q)=-∑P(i)log(Q(i)),

[0335] Among them, P(i) represents the probability distribution of the true result, and Q(i) represents the probability distribution of the predicted result;

[0336] The closer the value of the cross entropy function is to 0, the better the performance of the pre-trained model is;

[0337] The pre-trained model parameters are updated according to the value of the cross entropy function.

[0338] The construction of the pre-trained model also includes obtaining evaluation performance data:

[0339] Inputting the test set data into the pre-trained model;

[0340] according to Get the accuracy of code comments;

[0341] according to Get the accuracy of code comments;

[0342] according to Get the recall rate of code comments;

[0343] According to F1 Get the F1 score of the code comments;

[0344] Updating the pre-trained model parameters according to the accuracy, precision, and recall of the code annotations;

[0345] Among them, TP represents the true positive class, that is, the number of samples predicted by the pre-trained model as positive and actually positive;

[0346] FP stands for false positive, which is the number of samples that the pre-trained model predicts to be positive but is actually negative;

[0347] TN stands for true negatives, which is the number of samples that the pre-trained model predicts to be negative and are actually negative;

[0348] FN stands for false negatives, which is the number of samples that the pre-trained model predicts as negative but is actually positive.

[0349] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0350] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0351] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A code comment generation method based on language processing, characterized in that: include: Input the pre-interpreted code to be annotated into the pre-trained model; The pre-trained model parses the pre-interpretation code to obtain a pre-interpretation code fusion feature vector; Decoding the pre-interpreted code fusion feature vector according to the decoder and generating code annotations; The pre-interpretation code is parsed to obtain a pre-interpretation code fusion feature vector, including: Parsing the pre-release code to obtain the pre-release code feature; Encoding the pre-interpretation code feature into a pre-interpretation code feature vector, wherein the pre-interpretation code feature vector includes a query vector, a key vector, and a value vector; Performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector; The self-attention calculation includes: Calculate the dot product between the query vector and the key vector to obtain a first attention score; Calculate a second attention score based on the first attention score and the scaling factor; Normalizing the second attention score to a third attention score according to a function; The pre-code fusion feature vector is obtained based on the third attention score and the value vector.

2. The method according to claim 1, characterized in that It also includes the construction of the pre-trained model: Obtaining a data sample including codes and corresponding code comments, and cleaning the data sample to obtain a data set; According to the data set, features of the code and the corresponding code annotations are obtained, and the features of the code and the corresponding code annotations are encoded into feature vectors; Dividing the data set containing the feature vector into training set data and test set data; Select the pre-trained initial model; Construct the encoder-decoder structure of the pre-trained initial model, set the number of encoder-decoder layers, activation function, and set initial parameter values; The training set data is input into the pre-trained initial model, and the code semantic analysis training is completed according to the self-attention mechanism in the pre-trained initial model to obtain a pre-trained model.

3. The method according to claim 2, characterized in that The features of the acquisition code and the corresponding code annotations include: Parse the code through a tokenizer and generate an abstract syntax tree; Traverse each node of the abstract syntax tree according to the breadth-first search algorithm; According to each node, obtain the node type, node content, and the relationship between the nodes; The node type, node content, and the relationship between nodes are encoded into a feature vector.

4. The method according to claim 2, characterized in that: The feature vector may be combined with a subcode address sequence, including: Decomposing the codes in the data set into preliminary sub-codes based on characters; In response to the preliminary subcode being included in the pre-trained model dictionary, directly using the preliminary subcode, In response to the preliminary subcode not being included in the pre-trained model dictionary, further decomposing the preliminary subcode until the decomposed final subcode is included in the pre-trained model dictionary; Add special identifiers at the beginning and end of the final subcode; The final subcode is mapped one-to-one with the predefined addresses in the pre-trained model dictionary to obtain a subcode address sequence.

5. The method according to claim 4, characterized in that The code semantic analysis training includes the task of predicting the final sub-code to be blocked and the task of judging whether two code fragments are continuous: The prediction of the final sub-code task being shielded includes: randomly masking at least one final sub-code according to the sub-code address sequence; inputting the subcode address sequence and the subcode address sequence of the masked final subcode into a pre-trained model simultaneously; The pre-trained model outputs a predicted subcode; Calculate a first loss between the predicted subcode and the masked final subcode according to the predicted subcode; The step of determining whether two code snippets are continuous tasks includes: Creating code snippet pairs according to the training set data; Wherein, the code snippet pair includes a continuous code snippet and a code snippet unrelated to the continuous code snippet; Merging the code snippet pairs and inputting them into a pre-trained model; The pre-trained model outputs a second loss of whether two code snippets are consecutive.

6. The method according to claim 5, characterized in that The first and second losses are obtained according to the cross entropy function: The cross entropy function H(P,Q)=-∑P(i)log(Q(i)), Among them, P(i) represents the probability distribution of the true result, and Q(i) represents the probability distribution of the predicted result; The closer the value of the cross entropy function is to 0, the better the performance of the pre-trained model is; The pre-trained model parameters are updated according to the value of the cross entropy function.

7. The method according to claim 2, characterized in that The construction of the pre-trained model also includes obtaining evaluation performance data: Inputting the test set data into the pre-trained model; according to Get the accuracy of code comments; according to Get the accuracy of code comments; according to Get the recall rate of code comments; according to Get the F1 score of the code comments; Updating the pre-trained model parameters according to the accuracy, precision, and recall of the code annotations; Among them, TP represents the true positive class, that is, the number of samples predicted by the pre-trained model as positive and actually positive; FP stands for false positive, which is the number of samples that the pre-trained model predicts to be positive but is actually negative; TN stands for true negatives, which is the number of samples that the pre-trained model predicts to be negative and are actually negative; FN stands for false negatives, which is the number of samples that the pre-trained model predicts as negative but is actually positive.

8. A method and device for generating code comments based on language processing, characterized in that: The device comprises: An input module, used for inputting the pre-interpreted code to be annotated into the pre-trained model; A parsing module, used for parsing the pre-interpreted code according to the pre-trained model to obtain the pre-interpreted code features; An encoding module, used for encoding the pre-interpretation code feature into a pre-interpretation code feature vector; A calculation module, used for performing self-attention calculation on the pre-interpretation code feature vector to obtain a pre-interpretation code fusion feature vector; The generating module is used for decoding the pre-interpreted code fusion feature vector according to the decoder and generating code annotations.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images