A deep learning-based identifier normalization method
By combining the deep learning-based BERT-CRF model with named entity recognition and entity similarity models and programming context information, the problems of splitting and expanding identifiers in identifier normalization are solved, improving the accuracy of identifier normalization and enhancing software development efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
- Filing Date
- 2022-07-14
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies fail to effectively utilize programming context information during identifier normalization, leading to incomplete and excessive splitting in the splitting module, and the extension module cannot accurately handle out-of-vocabulary words and multiple expansions of a single word.
We employ a deep learning-based approach, utilizing the BERT-CRF model for identifier splitting and extending it with named entity recognition and entity similarity models. We leverage programming context information to assist in processing, obtain coarse-grained word roots through a byte pair encoding algorithm, and perform sequence labeling using BMES rules. Finally, we combine the CRF layer and the Seq2Seq task to normalize the identifiers.
It improves the accuracy of identifier normalization, solves problems in splitting and expansion, realizes identifier normalization at mixed granularity, reduces manual workload, and improves software development efficiency.
Smart Images
Figure CN115358218B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of code analysis and understanding in the software development process, and specifically, it designs an identifier normalization method based on deep learning. Background Technology
[0002] Identifiers comprise approximately 70% of the total source code. In large open-source projects, the vast majority of identifiers conform to naming conventions, such as using camelCase and hyphens to separate multiple words. However, non-standard identifier naming conventions still exist in large open-source projects, such as combinations of multiple consecutive lowercase letters or multiple abbreviations. Clearly, such non-standard naming directly affects software developers' understanding of the program, thereby reducing software maintenance efficiency. Existing research has not proposed good solutions for these non-standard identifier combinations, and new identifier standardization methods are urgently needed.
[0003] Identifier normalization is the process of converting source code identifiers into easily understandable natural language words. Specifically, normalization can be divided into two modules: splitting and expansion.
[0004] Identifier splitting (also known as lexical analysis, identifier tokenization, or identifier segmentation) aims to divide program identifiers into intermediate words. Unlike traditional natural language text, words in identifiers are not separated by commonly used delimiters (such as spaces). Java developers typically create identifiers according to camelCase naming conventions, while C programmers typically use underscores as delimiters to connect words. When using explicit delimiters (such as uppercase letters or special delimiters) to separate constituent words, splitting identifiers is simple and straightforward. However, for non-standard identifiers, more efficient splitting methods are required.
[0005] Identifier extension aims to expand abbreviations into natural language words with specific meanings. Common abbreviations can be extended by constructing an abbreviation-extension mapping dictionary. However, in projects across different domains, developers may use abbreviations that are not in the mapping dictionary or that correspond to multiple extensions in the mapping dictionary. This can lead to extensions failing to expand or multiple extensions for a single term, thus increasing the difficulty of abbreviation extension.
[0006] Existing research on identifier normalization has focused primarily on heuristic rules and statistical methods. We will analyze these two aspects below.
[0007] TIDIER overcomes the limitations of existing methods in splitting identifiers containing abbreviations. It specifically identifies transposed words from identifiers by using speech recognition technology to calculate the minimum string edit distance between transposed words in the identifier and words in a predefined thesaurus (including a general English dictionary, a domain-specific program dictionary, and a high-level document context dictionary). TIDIER first splits the identifier into multiple words using word transposition rules and a hill-climbing algorithm. Then, it uses DTW (Dynamic Time Warp) to calculate the Levenshtein distance between words in the identifier and words in the dictionary. Finally, it selects the split with the minimum string edit distance as the correct normalization result. The TIDIER algorithm relies on a dictionary, and its speed slows down as the dictionary size increases because the DTW algorithm calculates the edit distance between words in cubic time.
[0008] Instead of defining transformation rules, LINSEN utilizes the Baeza Yates and Perleberg (BYP) string matching algorithm, which has linear time complexity, to optimize identifier splitting. It first constructs a graph with each character of the identifier as a node and calculates the cost of matching dictionary words through edges. Second, it uses Dijkstra's algorithm to search for paths with minimum cost and returns the matched dictionary words in the path as the correct normalization result. The custom dictionaries it uses include words from comments, source code, common abbreviations, and English dictionaries.
[0009] Normalize expands abbreviations and acronyms using machine translation techniques. It uses a statistical maximum consistency model to calculate the probability of two words appearing simultaneously, applying a five-word window to text extracted from source code, domain-specific documents, and Google datasets published by the Language Data Consortium. It selects the word with the highest probability of co-occurrence as the expansion for the abbreviation. If an abbreviation may have multiple expansions, the method does not expand it.
[0010] Therefore, existing work on identifier normalization focuses primarily on heuristic rules and statistics, neglecting programming context information. Furthermore, the results of identifier normalization are affected by the effectiveness of module splitting and expansion. Regarding splitting, existing methods can lead to incomplete or excessive splitting. Regarding expansion, existing methods cannot effectively handle out-of-vocabulary words and the phenomenon of multiple expansions of a single word.
[0011] As is well known, the formation and construction of English words in natural language follow certain rules, with morphemes at their core. A morpheme generally consists of three parts: the root word, prefixes, and suffixes. The meaning of a word is largely reflected in its root word. Root words can form English words independently or in combination with each other. Prefixes and suffixes are used to change the lexical nature and meaning of a word. For example, in the English word "unanswerable," the root word "unanswerable" indicates an explanation of the question, the prefix "un" indicates that something cannot be done, and the suffix can indicate that the word is an adjective. Through research on the rules of English word formation, we have found that learning by adopting construction rules yields more accurate decomposition results. Therefore, knowledge from the field of natural language processing can be used in the decomposition module for identifier normalization.
[0012] The programming context information of an identifier that is helpful for expansion includes its package name, class name, method name, and relevant comments written by the developer. Existing methods do not fully utilize this information. When encountering situations where a single term has multiple expansions, we can use the above resources as auxiliary information for expansion, allowing the abbreviation to be accurately mapped to its expanded form. Summary of the Invention
[0013] Purpose of the invention: This invention proposes a deep learning-based identifier normalization method to improve the accuracy of identifier normalization, help software developers improve the quality of the code they are writing, especially the quality of the identifiers they create, help software developers quickly understand the semantic functions of the program, reduce manual workload, effectively reduce the communication costs of software developers, and improve the efficiency of collaborative software development.
[0014] Technical solution: This invention provides a deep learning-based identifier normalization method, comprising the following steps:
[0015] (1) Use the byte pair encoding algorithm to obtain coarse-grained word roots for the identifiers, and perform sequence labeling on the word roots according to the BMES rule to obtain the training set required for training the model;
[0016] (2) Input the training set into the BERT-CRF model for training; input the identifier to be split into the trained BERT-CRF model and output the splitting result of the identifier;
[0017] (3) Input the split identifiers into the trained named entity recognition model to identify the abbreviations that need to be expanded, and then input them into the entity similarity model for expansion;
[0018] (4) If the entity similarity model has multiple outputs, add programming context information to assist in the expansion and obtain the normalized identifier.
[0019] Furthermore, the implementation process of step (1) is as follows:
[0020] Identifiers have positional relationships between their characters. In the field of natural language processing, the BMES sequence labeling method can represent these positional relationships, where B represents the beginning of a word, M represents the middle of a word, E represents the end of a word, and S represents a single word; {X1X2...X... n Let} be the set of word roots in a certain identifier X, where each element X represents a word root, and n is 5. X consists of two words, A and B. Word A is composed of word roots X1 and X2, and word B is composed of word roots X3, X4, and X5. Then, the identifier X is sequence-labeled according to the BMES rule as "X1 / B, X2 / E, X3 / B, X4 / M, X5 / E". After sequence labeling, the identifier X is converted into a training set.
[0021] Furthermore, the implementation process of step (2) is as follows:
[0022] BERT includes a multi-layer bidirectional Transformer, which pre-trains the deep bidirectional representation by jointly adjusting the context of each layer. To adapt to the identifier normalization task, the input first needs to be transformed into [CLS] + sentence A + [SEP], where [CLS] represents the special sequence for the classification task and [SEP] represents the delimiter.
[0023] Then, BERT feature extraction is used to output the predicted label probability for each character; finally, the CRF layer constrains the output score and outputs the optimal label sequence; CRF considers the dependency constraints between preceding and subsequent labels and uses the label state transition probability as the score:
[0024]
[0025] The above formula is the score given by the CRF layer for the entire sequence, where x represents the entire sequence and y represents the category of the label. The score for the entire sequence is equal to the sum of the scores at each position. The score consists of two parts: the first part is the emission score output by BERT. Let i be the position index of the word, and y be the position index of the word. i The first part is the index of the category; the second part is determined by the CRF transition matrix A, where Indicates starting from the yth i-1 The tag to the yth i The transfer score of each label;
[0026] Launch Score
[0027] In x1,B, index is 1 and label is B; xindex,label represents the score of the word at index 1 that is labeled as label in the above formula.
[0028] x1,B, x2,M, x3,E, and x4,S are all outputs of BERT; X5,[CLS] and x6,[SEP] are considered as 0; transition scores
[0029] tlabel1->label2 represents the transition probability score from label1 to label2. The transition score comes from the CRF layer, and the optimal path is selected from all paths using the Viterbi algorithm.
[0030] Furthermore, the implementation process of step (3) is as follows:
[0031] Named entity recognition locates and classifies named entities in text into predefined categories, and identifies abbreviations that need to be expanded.
[0032] Modeling entities with contextual relationships based on entity similarity model: Constructing a Seq2Seq task, inputting abbreviations and their corresponding extended words, and attaching programming context information to construct data for text similarity calculation; concatenating the input abbreviations, programming context information and corresponding extended words through SEP, taking dissimilar samples as negative samples, and then calculating the similarity of similar samples through the softmax function;
[0033] An identifier-based extended question answering system based on the BERT deep learning model is constructed using a named entity recognition model and an entity similarity model. The BERT-based automatic question answering is divided into named entity recognition and entity mapping. Entity recognition finds the name of the entity being asked in the question, while entity mapping finds the specific mapping corresponding to the entity.
[0034] Furthermore, the implementation process of step (4) is as follows:
[0035] If the entity similarity model has multiple mapping results when mapping entities, it is necessary to use auxiliary information, namely programming context information, to determine the specific mapping. Collect the class name, declaration type, method name and comments of the identifier from the source code and store them in the database. When multiple expansions of a single word occur, the question answering system will automatically read the information from the database.
[0036] Beneficial Effects: Compared with existing technologies, the beneficial effects of this invention are as follows: This invention considers the specificity of different styles of programming code, takes the lead in utilizing the BERT pre-trained model, solves the problem that existing works do not fully utilize the programming context information of the code, and designs coarse-grained root word annotation, effectively solving the problems of incomplete splitting and over-splitting in the splitting problem; Compared with the original technology, the BERT model can achieve parallel computing, and BERT's long-distance feature capture capability as a feature extractor is significantly stronger than CNN; This invention no longer relies on heuristic rules but expands identifiers by calculating the text similarity between abbreviations and extended words, while effectively solving the problem of multiple expansions of a single word by relying on programming context information; The original technology only implemented the splitting or expansion modules and did not consider the mixed-granularity root words, while this invention realizes the complete process of mixed-granularity identifier normalization. Attached Figure Description
[0037] Figure 1 This is a flowchart of the present invention;
[0038] Figure 2 This is a schematic diagram of the BERT input layer structure. Detailed Implementation
[0039] The present invention will now be described in further detail with reference to the accompanying drawings.
[0040] This invention provides a deep learning-based identifier normalization method, such as... Figure 1 As shown, the entire method is divided into the following two modules: The splitting module: Byte-Pair Encoding (BPE) is used on the original dataset to obtain coarse-grained word roots. Then, the original dataset is preprocessed to obtain a training set, which is then input into the BERT-CRF model for training. Finally, the identifiers to be split are input into the trained BERT-CRF model, and the splitting results are passed to the extension module. The extension module: The identifiers to be extended are fed into a trained Named Entity Recognition (NER) model to identify the abbreviations that need to be extended. Then, they are input into an Entity Similarity (SIM) model for extension. If the SIM model has multiple outputs, programming context information is added to assist in the extension, ultimately resulting in the normalized identifiers. The specific implementation process is as follows:
[0041] Step 1: Use a byte-pair encoding algorithm to obtain coarse-grained word roots for the identifiers, and perform sequence labeling on the word roots according to the BMES rules to obtain the training set required for training the model.
[0042] Identifiers contain positional relationships between characters. In the field of natural language processing, the BMES sequence labeling method can represent these relationships. Here, B represents the beginning of a word, M represents the middle, E represents the end, and S represents a single word. For example, the identifier "toAdding" contains the words "to" and "Adding," and its sequence label is "t / B,o / E,A / B,d / M,d / M,i / M,n / M,g / E." By adding the root "ing" using a byte-pair encoding algorithm, a mixed-granularity labeling is achieved, resulting in the sequence label "t / B,o / E,A / B,d / M,d / M,i / M,ing / E." After sequence labeling, the training data is used as the training set and input into the BERT-CRF model. This preprocessing addresses the issue of previous techniques not considering the root word.
[0043] Step 2: Input the training set into the BERT-CRF model for training; input the identifiers to be split into the trained BERT-CRF model, and output the splitting results of the identifiers.
[0044] The mixed-granularity labels of the identifiers are fed into BERT-CRF for full training. BERT is a pre-trained language representation model whose main architecture is a multi-layer bidirectional Transformer, which pre-trains deep bidirectional representations by jointly tuning the context of each layer. Figure 2 As shown, in order to adapt the BERT model to downstream tasks, the input will be transformed into [CLS] + sentence A + [SEP]. Here, [CLS] represents a special sequence for classification tasks, and [SEP] represents a separator.
[0045] After feature extraction via BERT, the output is the predicted label probability for each character. Then, a CRF layer constrains the output score, resulting in the optimal label sequence. The CRF considers the dependency constraints between preceding and subsequent labels and uses the label state transition probabilities as the overall score.
[0046]
[0047] In the above formula, the CRF layer scores the entire sequence, where x represents the entire sequence and y represents the category of the label. The score of the entire sequence is equal to the sum of the scores at each position. The score consists of two parts:
[0048] 1) The former is the emission fraction output by BERT. Let i be the position index of the word, and y be the position index of the word. i It is an index of the category.
[0049] 2) The latter is determined by the CRF transition matrix A, where Indicates starting from the yth i-1 The tag to the yth iThe transfer score of each label.
[0050] Launch Score
[0051] xindex,label represents the score of the index-th word as label. x1,B,x2,M, etc. are all outputs of BERT. X5,[CLS] and x6,[SEP] can be regarded as 0.
[0052] Transfer score
[0053] tlabel1->label2 represents the transition probability score from label1 to label2. These transition scores come from the CRF layer, and finally the Viterbi algorithm is used to select the optimal path from all possible paths.
[0054] Step 3: Input the split identifiers into the trained Named Entity Recognition (NER) model to identify the abbreviations that need to be expanded, and then input them into the Entity Similarity (SIM) model for expansion.
[0055] Named Entity Recognition (NER) models (also known as entity recognition, entity chunking, and entity extraction) are a subtask of information extraction that aims to locate and classify named entities in text into predefined categories, such as people, organizations, locations, time expressions, quantities, currency values, percentages, etc. In the expansion task, an abbreviation that needs expansion is considered an entity, thus allowing the use of NER models to identify the abbreviation requiring expansion from all identifiers segmented by the chunking modules.
[0056] Building a Seq2Seq task: Training a similarity calculation (SIM) model is a supervised learning task. The training data used is pairs of similar sentences. This task involves inputting text and corresponding similar text. This step requires us to input abbreviations and their corresponding extended words, along with programming context information, to construct data for text similarity calculation.
[0057] Calculating text similarity involves concatenating input abbreviations, programming context information, and corresponding extended words using Semantic Equation Prefix (SEP). For example, given sentence1 and sentence2, the data is transformed into a format like CLS_sentence1_SEP_sentence2_SEP. Then, sentence2 is predicted based on sentence1. The left side represents the semantic similarity part, which essentially treats dissimilar samples as negative samples and then calculates the similarity between similar samples using a softmax function.
[0058] We can build an identifier-based question answering system using the Named Entity Recognition (NER) model and the Entity Similarity (SIM) model. The BERT-based automatic question answering system consists of two main steps: Named Entity Recognition and Entity Mapping. The purpose of Entity Recognition is to find the name of the entity being asked for in the question, while the purpose of Entity Mapping is to find the specific mapping corresponding to that entity.
[0059] Step 4: If the entity similarity (SIM) model has multiple outputs, add programming context information to help expand it and obtain the normalized identifier.
[0060] Entity similarity (SIM) models are a knowledge representation learning method that models entities with contextual relationships. This invention uses the SIMBERT model. If the entity similarity model produces multiple mapping results when mapping entities, auxiliary information, namely programming context information, is needed to determine the specific mapping. For Java, we can collect the class names, declared types, method names, and comments of identifiers from the source code and store them in a database. When multiple expansions of a single term occur, the question-answering system will automatically retrieve the information from the database.
[0061] This invention improves the accuracy of identifier standardization, helps software developers improve the quality of the code they are writing, especially the quality of the identifiers they create, helps software developers quickly understand the semantic function of the program, reduces manual workload, effectively reduces the communication cost of software developers, and improves the efficiency of collaborative software development. It can quickly and accurately standardize identifiers with an estimated accuracy of 70%, which is about 20% higher than the original technology.
Claims
1. A deep learning-based identifier normalization method, characterized in that, Includes the following steps: (1) The identifier is encoded using a byte pair encoding algorithm to obtain coarse-grained word roots, and the word roots are sequence labeled according to the BMES rule to obtain the training set required for training the model; (2) Input the training set into the BERT-CRF model for training; input the identifiers to be split into the trained BERT-CRF model and output the splitting results of the identifiers; (3) Input the split identifier into the trained named entity recognition model to identify the abbreviations that need to be expanded, and then input them into the entity similarity model for expansion; (4) If the entity similarity model has multiple outputs, programming context information is added to assist in the expansion to obtain the normalized identifier; The implementation process of step (3) is as follows: Named entity recognition locates and classifies named entities in text into predefined categories, and identifies abbreviations that need to be expanded. Modeling entities with contextual relationships based on entity similarity model: Constructing a Seq2Seq task, inputting abbreviations and their corresponding extended words, and attaching programming context information to construct data for text similarity calculation; concatenating the input abbreviations, programming context information and corresponding extended words through SEP, taking dissimilar samples as negative samples, and then calculating the similarity of similar samples through the softmax function; An identifier-based extended question answering system based on the BERT deep learning model is constructed using a named entity recognition model and an entity similarity model. The BERT-based automatic question answering is divided into named entity recognition and entity mapping. Entity recognition finds the name of the entity being asked in the question, while entity mapping finds the specific mapping corresponding to the entity.
2. The identifier normalization method based on deep learning according to claim 1, characterized in that, The implementation process of step (1) is as follows: Identifiers have positional relationships between their characters. In the field of natural language processing, the BMES sequence labeling method can represent these positional relationships, where B represents the beginning of a word, M represents the middle of a word, E represents the end of a word, and S represents a single word; {X1X2...X... n Let} be the set of word roots in a given identifier X, where each element X n Let X represent a word root, n is 5, and X is composed of two words A and B. Word A is composed of word roots X1 and X2, and word B is composed of word roots X3, X4 and X5. Then the identifier X is sequence-labeled according to the BMES rule as "X1 / B,X2 / E,X3 / B,X4 / M,X5 / E". After sequence labeling, the identifier X is converted into a training set.
3. The identifier normalization method based on deep learning according to claim 1, characterized in that, The implementation process of step (2) is as follows: BERT includes a multi-layer bidirectional Transformer, which pre-trains the deep bidirectional representation by jointly adjusting the context of each layer. To adapt to the identifier normalization task, the input first needs to be transformed into [CLS] + sentence A + [SEP], where [CLS] represents the special sequence for the classification task and [SEP] represents the delimiter. Then, BERT feature extraction is used to output the predicted label probability for each character; finally, the CRF layer constrains the output score and outputs the optimal label sequence; CRF considers the dependency constraints between preceding and subsequent labels and uses the label state transition probability as the score: The above formula is the score given by the CRF layer for the entire sequence, where x represents the entire sequence and y represents the category of the label. The score for the entire sequence is equal to the sum of the scores at each position. The score consists of two parts: the first part is the emission score output by BERT. Let 'i' be the position index of the word. The first part is the index of the category; the second part consists of the CRF transition matrix. Decision, among which Indicates from the first The first tag to the first The transfer score of each label; Launch Score ; In x1,B, index is 1 and label is B; xindex,label represents the score of the word at index 1 that is labeled as label in the above formula. x1,B、x2,M、 , All are BERT outputs, with X5,[CLS] and x6,[SEP] treated as 0; Transfer score ; tlabel1->label2 represents the transition probability score from label1 to label2. The transition score comes from the CRF layer, and the optimal path is selected from all paths using the Viterbi algorithm.
4. The identifier normalization method based on deep learning according to claim 1, characterized in that, The implementation process of step (4) is as follows: If the entity similarity model has multiple mapping results when mapping entities, it is necessary to use auxiliary information, namely programming context information, to determine the specific mapping. Collect the class name, declaration type, method name and comments of the identifier from the source code and store them in the database. When multiple expansions of a single word occur, the question answering system will automatically read the information from the database.