A code submission automatic classification method based on deep learning

Abstract

The present application relates to a kind of code submission automatic classification method based on deep learning method, belong to the technical field of automated software engineering.The method uses the bidirectional Transform encoder to express the text feature extracted from the change description of code submission, and extracts the fine-grained code line change feature of the modified module from the code change of code submission.The text feature and code feature are fused by multi-modal adaptive gate mechanism and other methods, and the fused code feature is classified using multi-layer perceptron.The classification performance (accuracy and macro average F1) of the present application is higher than that of the existing method under the same data set;It can work across different programming languages, and can also work in complex multilingual projects.

Description

TECHNICAL FIELD

[0001] The application relates to a code commit classification method based on deep learning and belongs to the technical field of automatic software engineering. BACKGROUND

[0002] Code commit refers to the process of saving the modification of the current working directory and submitting it to the version control system (such as Git) to form a new project version when using the version control system. This is a basic process for software developers to develop and maintain software projects. A software project is continuously evolved by a code commit to adapt to changes in various external requirements.

[0003] Understanding the motivation of software iteration can help developers better manage requirement changes, control risks and improve development efficiency, thereby reducing maintenance costs. In order to better study the process of software iteration evolution and improve the quality of software, many works have been done in recent years to study the automatic classification of code commits.

[0004] The application is improved based on the method of Ghadhab et al. Their method is the best in the current code commit automatic classification technology, but still has certain limitations.

[0005] 1. Although their method uses bidirectional encoding representation (BERT) based on Transformer to extract features from the commit message in the code commit, BERT is not fine-tuned in the downstream task, and the encoding vector of BERT cannot fully capture the features required in the code commit automatic classification task, so the classification performance is not high.

[0006] 2. Their method is based on tools such as fixminer to mine code features from source code changes, but due to the limitations of these tools, their method is limited to Java language and cannot be used across languages, so it cannot be used in complex multi-language software projects.

[0007] 3. Their method directly splices the feature vector of BERT and the code features extracted by tools such as fixminer, which will bring redundancy and noise to the feature representation, and thus reduce the performance of the model.

[0008] To solve the above problems, the application designs a neural network, extracts features of change description in code submission based on BERT, and adds BERT to the training of the model to fine-tune BERT in the downstream task. Meanwhile, the application designs a fine-grained code line change feature to model the code change in the code submission. This feature is language-independent and can be used across languages. In addition, the application uses a multi-modal adaptive gate feature fusion method to fuse the features of the two modalities, further improving the classification performance of the model. SUMMARY

[0009] The application aims to solve the problem that the current code submission automatic classification accuracy is poor and cannot work across different programming languages, and proposes a code submission automatic classification method based on deep learning.

[0010] The purpose of the application is achieved through the following technical solutions.

[0011] A code submission automatic classification method based on deep learning comprises:

[0012] Step 1: Data collection

[0013] Data collection includes two sub-steps of manually annotating the fine-tuning data set and automatically generating the pre-training data set from GitHub.

[0014] Step 2.1: Manually annotate the fine-tuning data set

[0015] Randomly sample code submission records from GitHub. To avoid subjective bias caused by manual annotation, two developers familiar with Git tools and code submission categories independently annotate the sampled code submission records according to the rules proposed by Mockus et al. in the definition of code submission automatic classification. During their annotation process, a code submission record can be annotated as one of Corrective, Perfective and Adaptive. When they encounter uncertain code submission records, they are labeled with the #uncertain label. After the two developers complete their annotation, the consistency Kappa coefficient of their annotation results is calculated to determine the consistency of the annotation of the two developers. The calculation result is 0.65. Industry believes that when the consistency Kappa coefficient is between 0.61 and 0.80, the results annotated by the two annotators have high consistency, and when the consistency Kappa coefficient is between 0.81 and 1.00, the results annotated by the two annotators have complete consistency. Finally, the two developers discuss and modify the different parts (including the #uncertain label) of the annotation results, and eliminate the data that cannot be unified, and finally obtain the fine-tuning data set.

[0016] Step 1.2: Automatic generation of pre-training dataset

[0017] The current mainstream convention of the open source community requires that the category to which the modification belongs be written at the beginning of the code submission description. These categories include: build, chore, ci, docs, style, refactor, perf, fix, feat, test, and revert. Except for the revert category, the definitions of these categories can be explicitly reduced to the three categories of the code submission automatic classification task. The chore category is a general term for non-functional modifications, corresponding to some trivial tasks in software development, such as formatting code, modifying comments, cleaning up code, and other activities. These modifications can improve the readability and maintainability of the software system, and belong to the improvement (Perfective) category. The build category represents changes related to code compilation and library dependencies, usually to generate executable files or library files. These modifications correspond to activities such as code packaging and document generation in software development, which can improve the deployability and repeatability of the software system, and belong to the improvement (Perfective) category. The ci category represents changes related to continuous integration, which is used to ensure that the code is correctly built, tested, and deployed in the continuous integration system. These modifications can improve the stability and reliability of the software system, and belong to the improvement (Perfective) category. The docs category represents modifications to documentation files, such as API documentation, user manuals, or other technical documentation. These modifications can help users better understand how to use the software and help developers better collaborate on development, and belong to the improvement (Perfective) category. The style category represents modifications to code style. These modifications do not change the semantics of the code, and these modifications can provide code readability and style consistency, and belong to the improvement (Perfective) category. The refactor category represents code refactoring changes, i.e., modifications to the structure, design, or implementation of the code to improve its readability, maintainability, or other aspects of quality, and belongs to the improvement (Perfective) category. The perf category represents improvements in code performance, such as improving algorithms, reducing resource usage, etc. Because these modifications do not involve the introduction of functionality, they belong to the improvement (Perfective) category. The test category represents changes related to software testing, such as adding new test cases, fixing test cases, etc., to improve the stability and reliability of the software, and belongs to the improvement (Perfective) category. The revert category represents a rollback of the software project. Usually, due to the introduction of some errors or unnecessary code submissions in the previous development history, these changes are undone through rollback. This is a special category, and the invention does not consider the revert category. The feat category represents the addition of new features, which belongs to the adaptive (Adaptive) category. The fix category represents the repair of functional errors, which belongs to the corrective (Corrective) category.

[0018] The revert category is excluded, leaving ten categories that can be explicitly reduced to the three categories of code commit automatic classification tasks. The present application selects the top twenty repositories with the highest number of stars in the top ten programming languages currently in vogue, including C, Python, etc. on GitHub, and manually selects these repositories, excluding some non-software engineering projects such as information sharing and tutorial classes. The selected repositories are downloaded to the local computer, and the development code commit history is selected to select code commit records that meet the conventional commit format. The categories are identified by the above method, and the corresponding classification labels are automatically generated to obtain the data set for pre-training.

[0019] Step 2: Feature extraction.

[0020] In this step, the code commit data set of step 1.1 and step 1.2 is extracted, and the present application extracts two aspects of features from the code commit record, including text features from the code commit change description information, and code features from the code change set. The features extracted in this step will be used in the feature fusion of step 3.

[0021] Step 2.1: Text feature extraction of code commit.

[0022] The present application uses a bidirectional encoding representation (BERT) based on Transformer to extract features from the text of the code commit change description. First, the text of the change description is divided into a series of words using a word segmenter, then the words are mapped into a vector, and the vectors are input into BERT, and finally the vector corresponding to the [cls] word is selected as the feature vector of the entire change description text.

[0023] Step 2.2: Code feature extraction of code commit.

[0024] The files changed in the code commit are divided into three different categories, namely code file category, test file category and document file category. For the code file category and the test file category, the contents of which belong to programming language code, and each line of the programming language code has a different function, the present application divides the code lines of the code file into three categories, namely code lines, comment lines and blank lines. Code lines represent the function of the code itself, comment lines improve the readability of code lines, and blank lines are a special type of comment line that do not provide information but are used to format the code.

[0025] For the code file category and the test file category, the total amount of code line increase and decrease, the total amount of comment line increase and decrease, and the total amount of blank line increase and decrease in all changes belonging to the code file category and the test file category in the code submission record are counted respectively, and the difference value of code line increase and decrease, the difference value of comment line increase and decrease, and the difference value of blank line increase and decrease are calculated respectively, so that the model can easily learn the difference problem between the two versions involved in the code submission. For the document file category, since the document itself is only text, only the total number of line increase, the total number of line decrease, and the difference value of line increase and decrease of all files involved in the code submission are counted.

[0026] Through the above method, 21 numerical values are extracted for the code changes of a code submission, which are used as the code feature vector of the code submission record.

[0027] Step 3: Feature fusion.

[0028] This step fuses the text features obtained in step 2.1 and the code features obtained in step 2.2, so that the code submission classifier can learn the feature information of the code submission description and the code change, capture the feature relationship before the two modalities, and thus improve its classification performance. The present application uses four feature fusion methods to fuse the text feature vector of BERT and the code vector of code change before classification, which are splicing fusion, fusion based on multilayer perception machine method, fusion based on attention mechanism method, and fusion based on multi-modal adaptive gate mechanism.

[0029] Splicing fusion directly splices the text feature vector and the code change feature vector to obtain a new vector for classification. The fusion method based on multilayer perception machine inputs the code change feature vector into a multilayer perception machine, then splices the output of the multilayer perception machine with the text feature vector to obtain a new vector for classification. The method based on attention mechanism calculates the attention score between the text feature and the code feature, then weights to obtain a new fusion vector. The fusion method based on multi-modal adaptive gate uses a gate mechanism function to weight the code change feature vector to the text vector.

[0030] Step 4: Model training.

[0031] Model training includes model pre-training and model fine-tuning. First, the model is pre-trained using the pre-training data set automatically generated in step 1.2, and then fine-tuned using the data set manually annotated in step 1.1, so that the model can learn the feature information of large-scale code submission classification and improve the classification performance for the specified classification task.

[0032] Step 4.1: Model pre-training:

[0033] The training scheme of the present application uses a batch size of 64, a learning rate of 0.00001, a maximum number of training rounds of 10, and a global random number seed of 413, and is trained on an NVIDIA GeForce RTX 3090 graphics card. During training, the present application divides the dataset into a training set and a validation set in a ratio of 8:2, and whenever the model completes a round of training on the training set, it will be validated on the validation set, and the model parameters corresponding to the best effect on the validation set will be saved as the pre-training result. When updating the model parameters, the present application uses the AdamW optimization algorithm to update the gradient of the model parameters, so that the model can be better pre-trained.

[0034] Step 4.2: Model fine-tuning

[0035] The present application uses a training method with a batch size of 24, a learning rate of 0.00005, a maximum number of training rounds of 20, and a global random number seed of 413, and uses an early stopping mechanism with a patience parameter set to 10, i.e. if the validation effect is worse than the historical best effect for more than 10 rounds, the model will end training early. As in step 4.2, the model's fine-tuning also uses the AdamW optimization algorithm to update the model's gradient, and the training is completed on an NVIDIA GeForce RTX 3090 graphics card.

[0036] At this point, after steps 1 to 4, a deep learning-based code submission automatic classification method is completed.

[0037] Advantages

[0038] The deep learning-based code submission automatic classification method proposed by the present application has the following advantages compared to existing classification methods based on code submission description information and features extracted from code changes to represent repairability and refactoring:

[0039] 1. The BERT neural network is fine-tuned during the training of the model, and four different feature fusion methods are used to fuse text features and code features, improving the classification performance of the classifier.

[0040] 2. On the same dataset, the accuracy and macro average F1 score of the present method are better than those of existing methods;

[0041] 3. The fine-grained code line change features used by the present application do not depend on a specific language, so the present application can be used across languages and in complex multilingual environments. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a schematic diagram of the working principle of the present application;

[0043] Figure 2 A schematic diagram of a neural network model of the present application; DETAILED DESCRIPTION

[0044] The present application will be further described and explained with reference to the accompanying drawings and examples.

[0045] Examples

[0046] This embodiment details the method and effect of the code submission automatic classification method based on deep learning of the present application in the implementation of classifying 1793 code submission records collected by Ghadhab et al. The dataset is collected and open sourced by Ghadhab et al. and can be accessed through the link https: / / zenodo.org / record / 4266643# .X6vERuLPxPY .

[0047] Under the hardware environment shown in Table 1, the open source software shown in Table 2 is trained and predicted.

[0048] Table 1: Hardware environment configuration information table

[0049]

[0050]

[0051] Table 2: 1793 code submission basic information table

[0052]

[0053] Step A: Extract the text feature vector from the code submission description information and the fine-grained code line change feature vector from the code change from the 1793 code submissions shown in Table 2. In order to compare with existing methods, and like the method of Ghadhab et al., the 1793 code submissions are divided into training set, validation set and test set according to the ratio of 7:1:2.

[0054] Step B: Feature extraction. Feature extraction is performed on the training set, validation set and test set of step A. BERT is used to extract text features from the code submission description information of the dataset, while fine-grained code line change features are extracted.

[0055] Step C: Initialize the neural network and pre-train. The model is pre-trained using the large-scale dataset automatically generated in the foregoing, so that the model learns the feature information of the large-scale dataset.

[0056] Step D: Input the text feature vector and code feature vector of the training set into the model network as shown in Figure 2 , and fine-tune the model. As shown in Figure 2As shown, the output of the model is a 1*3 vector. The SoftMax value of this output vector is calculated, the index corresponding to the largest position is selected, and the index is converted into the label of the code submission classification. Among them, the index position 0 corresponds to the Corrective category, the index position 1 corresponds to the Perfective category, and the index position 2 corresponds to the Adaptive category.

[0057] Step E: Train the model on the training set according to the method of step D. After training each round, fine-tune it on the validation set in the same way. Save the model parameters that perform best on the validation set during the training process, and use the parameters to test on the test set.

[0058] Table 3: Comparison of the present method with the best existing code submission automatic classification method

[0059] The method The best method currently available Accuracy 81.56％ 79.66％ Macro mean F1 81.48％ 80.00％

[0060] The accuracy in Table 3 = the number of correct classifications / the total number of data;

[0061] The macro average F1 in Table 3 = the sum of the F1 of the three categories / 3

[0062] The results show that:

[0063] 1. The accuracy is 81.56%, which is nearly 2% higher than that of the existing method;

[0064] 2. The macro average F1 is 81.48%, which is nearly 1.5% higher than that of the existing method;

[0065] In addition, the existing method is limited to one language, while the method of the present application can support any programming language.

[0066] The above is only a preferred embodiment of the present application, and the present application should not be limited to the content disclosed in the embodiment and the drawings. Any equivalent or modification made without departing from the spirit of the present application falls within the scope of protection of the present application.

Claims

1. A deep learning-based automatic code submission classification method, characterized by: Step 1: Data Collection; Step 1.1: Manually label and fine-tune the dataset; Step 1.2: Automatically generate pre-trained dataset; Step 2, Feature Extraction; Feature extraction is performed on the dataset submitted by the code in steps 1.1 and 1.

2. The extracted features will be used in the feature fusion in step 3. Step 2.1: Extracting text features from code submissions; The BERT bidirectional encoding representation based on Transformer is used to extract features from the text of the code submission change description. First, the change description text is divided into a series of words using a tokenizer. Then, these words are mapped into a vector. These vectors are input into BERT. Finally, the vector output by BERT to the corresponding [cls] word is selected and used as the feature vector of the entire change description text. Step 2.2: Code feature extraction from code submissions; The files modified in the code commits are divided into code file categories, test file categories, and documentation file categories. For code file categories and test file categories, since the content is programming language code, and each line of programming language code has a different function, the code lines in the code files are divided into three categories: code lines, comment lines, and blank lines. Code lines represent the function of the code itself, comment lines improve the readability of code lines, and blank lines are a special type of comment line that does not provide information itself but is used to format the code. For code file categories and test file categories, we separately count the total number of increases and decreases in code lines, comments, and blank lines among all changes belonging to these categories in the code commit history. We also calculate the differences in the increase / decrease of code lines, comments, and blank lines, allowing the model to easily learn the differences between the two versions involved in the code commit. For document file categories, since documents are simply text, we only count the total number of increases and decreases in the number of lines for all files involving this category in the code commit, as well as the differences in the increase / decrease of the number of lines. Step 3: Feature Fusion The text features obtained in step 2.1 and the code features obtained in step 2.2 are fused together so that the code submission classifier can learn the feature information of code submission description and code change. Step 4: Model Training Step 4.1: Model Pre-training The model is pre-trained on the large-scale dataset automatically generated in step 1.2, and the pre-trained model parameters are saved. Step 4.2: Model fine-tuning; The parameters of the pre-trained model are used to initialize the model, and then fine-tuned on the manually labeled dataset from step 1.

1. Thus, after steps 1 to 4, a method for automatic code submission classification based on deep learning has been completed.

2. The method for automatic classification of code submissions based on deep learning according to claim 1, characterized in that: The feature fusion methods described in step 3 are splicing fusion, fusion based on multilayer perceptron, fusion based on attention mechanism, and fusion based on multimodal adaptive gating mechanism. Concatenation fusion directly concatenates the text feature vector and the code change feature vector to obtain a new vector, which is then used for classification. Multilayer perceptron-based fusion methods input the code change feature vector into a multilayer perceptron, then concatenate the output of the multilayer perceptron with the text feature vector to obtain a new vector for classification. Attention-based methods calculate attention scores between text and code features, then weight them to obtain a new fusion vector. Multimodal adaptive gating-based fusion methods use a gating mechanism function to weight the code change feature vector into the text vector.

Images