A code migration method based on deep hashing

Abstract

The present invention discloses a code migration method based on deep hashing, comprising the following steps: (1) selecting an open source data set, extracting all code features, and vectorizing the code features; (2) clustering the code features using the k-means algorithm; (3) constructing a code migration network model based on deep hashing; (4) training the code migration network model using the code feature vectors of the clustered code pairs, so that the hash code similarity matrix learned by the deep hashing module is aligned with the joint code similarity matrix; hashing the codes of the open source code data set using the trained model to construct a code hash database; (5) extracting the code feature vectors for the code to be migrated and inputting them into the trained model, obtaining the corresponding binary hash code through the deep hashing module, and retrieving the matching migrated code from the code hash database. Utilizing the present invention, the accuracy and speed of code migration can be greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of code migration, and in particular relates to a code migration method based on deep hashing. Background Art

[0002] With the localization of operating systems in my country, code migration technology has enormous potential for commercial applications. Domestic operating systems are incompatible with Windows, so migrating Windows-based software to domestic operating systems requires code language conversion. However, current methods for code migration have limited performance.

[0003] In the field of code migration, it can be roughly divided into hard rule-based code migration, information retrieval-based code migration, and deep neural network-based code migration.

[0004] Code migration based on hard rules is to use the rules sorted out during code migration. For example, the Chinese patent with publication number CN115756433A discloses a method for migrating a code platform. Through a visual migration interface, the configuration information of the code platform to be migrated and the configuration information of the target code platform are created, as well as the creation of conversion rules for converting the configuration information of the code platform to be migrated to the configuration information of the target code platform. After obtaining the first domain-specific language file of the code platform to be migrated, the first domain-specific language file is automatically converted into the second domain-specific language file of the target code platform according to the conversion rules. However, this migration is very dependent on the style of the code. Simple modifications will cause the rules to become invalid, and its applicability is low.

[0005] Code migration based on information retrieval often uses text information to perform code migration. For example, the Chinese patent document with publication number CN112181489A discloses a code migration method, including obtaining source files and target conversion framework information; determining the framework information to be converted corresponding to the source file based on the character string to be matched; obtaining a configuration file corresponding to the framework type; determining the target mapping relationship from the multiple mapping relationships based on the framework information to be converted and the target conversion framework information; using a matching method corresponding to the character type to match the character string to obtain a character string to be converted that matches the source framework string; converting the character string to be converted into the target framework string based on the target mapping relationship to obtain the target file of the source file under the target conversion framework. However, the structure of the code is different from that of natural language. Using text information ignores the semantic structure of the code itself, and the retrieval efficiency is relatively low.

[0006] Code migration based on deep neural networks extracts features from code syntax and semantics and uses deep neural networks to learn the relationships between code pairs. However, this approach requires significant computational and storage resources, making it difficult to implement in the industry. Summary of the Invention

[0007] The present invention discloses a code migration method based on deep hashing, which can greatly improve the accuracy and speed of code migration.

[0008] A code migration method based on deep hashing, characterized by comprising the following steps:

[0009] (1) Select an open source dataset and extract all code features in the dataset, including text features and structural features; further vectorize the code features to obtain a code feature vector;

[0010] (2) Using the k-means algorithm, the code features are clustered to obtain multiple cluster groups;

[0011] (3) Constructing a code migration network model based on deep hashing, wherein the code migration network model includes a vector representation code feature distribution module and a deep hashing module;

[0012] The vector representation code feature distribution module is used to normalize the code feature vectors, calculate the code similarity matrix between code pairs, and represent the feature distribution of code pairs through the vectors in the code similarity matrix. It further integrates the information of the code similarity matrix by combining the code similarity matrix to comprehensively represent the distribution in the entire code space.

[0013] The deep hash module is used to convert the code feature vector into a binary hash code vector and calculate the hash code similarity matrix between the code pairs after normalization;

[0014] (4) The code feature vectors of the clustered code pairs are used to train the code transfer network model, so that the hash code similarity matrix learned by the deep hash module is aligned with the joint code similarity matrix, thereby retaining the similarity information between the code pairs in the hash space and the original code space;

[0015] Then, the trained model is used to hash the code of the open source code dataset and build a code hash database;

[0016] (5) For the code to be migrated, the code feature vector is extracted and input into the trained model. The corresponding binary hash code is obtained through the deep hash module, and the matching migrated code is retrieved from the code hash database.

[0017] In step (1), the specific process of extracting code features is as follows:

[0018] (1-1) Use ANTLR to parse the code into a concrete syntax tree CST, unify the words with the same meaning but different names, and simplify them to obtain a simplified concrete syntax tree SCST;

[0019] (1-2) Extract all paths from child nodes to the root node from the simplified concrete syntax tree SCST;

[0020] (1-3) Serialize the entire extracted path.

[0021] In step (2), the number of cluster groups is 10, and different clusters are established by distinguishing based on code paths.

[0022] In step (3), the vector represents the code pair feature distribution module, and the code feature extraction vector is normalized using l2-norm.

[0023] Integrate the code similarity matrix by adding Construct the joint code similarity matrix S, the formula is:

[0024]

[0025]

[0026] Among them, β is the weight parameter, S Co and S Cn is the code feature vector V of the code pair Co 、V Cn The normalized result is then calculated to obtain the code similarity matrix Among them S Co =V Co T V Co 、S Cn =V Cn T V Cn , and finally the joint code similarity matrix S is obtained.

[0027] In the deep hashing module, an improved bag-of-words (NBOW) model is used. Three fully connected FC layers with a specific tanh(·) activation function are used to replace the output layer of the original bag-of-words (BOW) model. The output is controlled within the (-1, 1) interval, thereby converting the original code feature vector into a binary hash code.

[0028] In step (4), a dual encoder architecture is adopted, using independent encoders to encode the code pairs into representation vectors and corresponding binary hash codes respectively; a new triplet metric loss is introduced to ensure that the learned binary hash codes retain the similarity information between the code pairs in the hash space and the original code space.

[0029] The triplet metric loss function is:

[0030]

[0031] In the formula, θ is a hyperparameter, λ1 and λ2 are weight parameters, and B Co 、B Cn is the code feature vector V of the code pair Co 、V Cn The binary hash code vector obtained by binary encoding; using cosine similarity, calculate B Co 、B Cn The gap between them is reduced, thus narrowing the distance between hash code pairs and joint code pairs, i.e., the hash code similarity matrix learned by the deep hashing module is aligned with the joint code similarity matrix.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention parses the code from the perspective of semantic structure, extracts code features by serializing the code path, improves the bag-of-words model and adds a deep hash network, converts code features into binary codes, and improves the efficiency and accuracy of code migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a code migration method based on deep hashing in the present invention;

[0035] Figure 2 Schematic diagram of the model architecture of the method of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0037] like Figure 1 and Figure 2 As shown, a code migration method based on deep hashing includes the following steps:

[0038] Step 1: Extract code features and vectorize them.

[0039] Step 11: The code is parsed into a concrete syntax tree (CST) and a simplified concrete syntax tree (SCST).

[0040] The code was parsed into CST using ANTLR. The simplification rules include the following: (1) unifying words with the same meaning but different names, such as converting while to for; (2) removing extra modifiers such as public and private.

[0041] Step 12: Extract all paths from child nodes to the root node from the parsed Simplified Concrete Syntax Tree (SCST).

[0042] The parsed CST ignores the intermediate nodes, selects the paths between child nodes and leaf nodes, and extracts all paths.

[0043] Step 13: Serialize the entire parsed path.

[0044] Encode the extracted path and encode the path according to the left child right sibling method, and ensure that the child with smaller encoding is on the left of the child with larger encoding.

[0045] Step 2: Use the k-means algorithm to cluster the code structure characteristics. The number of clusters is 10, and different clusters are established by distinguishing based on code paths.

[0046] Step 3: Build a network model for code migration based on deep hashing.

[0047] The code transfer network model includes a vector representation code pair feature distribution module and a deep hashing module.

[0048] In the code pair feature distribution vector representation module, we use l2-norm to normalize the extracted code feature vectors and calculate the similarity matrix. This integrates the information between code pairs and comprehensively represents their distribution in the entire embedding space.

[0049] Extract the feature vector V of the code pair Co (V Co (1) ,...,V Co (n))、V Cn (V Cn (1) ,...,V Cn (n) ) as an example, using l2-norm to convert V Co 、V Cn Normalization, calculation of code similarity matrix Among them, S Co =V Co T V Co 、S Cn =V Cn T V Cn In order to integrate the information between code pairs, they are integrated and added, where β is the weight parameter:

[0050]

[0051] Since the pairwise similar code pair representation vectors still cannot fully reflect their distribution conditions in the entire embedding space, using the matrix To describe the high-order neighbor similarity, the two vectors should also be close to other vectors, and the joint code similarity matrix is ​​constructed:

[0052]

[0053] In the deep hash module, the improved bag-of-words model NBOW is used, and three fully connected FC layers with specific tanh(·) activation functions are used to replace the output layer in the original bag-of-words model BOW, so that the output is controlled in the (-1,1) interval, thereby converting the original code feature vector into a binary hash code. The vector corresponding to the converted binary hash code is B Co (B Co (1) ,...,B Co (n) ) and B Cn (B Cn (1) ,...,B Cn (n) ).

[0054] To ensure that the relative distribution of binary hash codes is similar to the distribution of representation vectors in the original embedding space, we use the following equation as the loss function of the deep hashing module:

[0055]

[0056] Among them, λ1 and λ2 are weight parameters, and cosine similarity is used to calculate B. Co 、B Cn , thereby narrowing the distance between hash code pairs and joint code pairs.

[0057] Step 4: Use the open source training set to train the model, then use the trained model to represent the code hash and build a code hash database.

[0058] We employ a dual-encoder architecture for code pairs, which uses independent encoder-code pairs to encode into representation vectors. We also introduce a novel triplet metric loss to ensure that the learned hash codes preserve the similarity between code pairs in both the hash space and the original code space.

[0059] All experiments were implemented based on Pytorch, and the experimental environment was the Linux operating system. A total of 4 Nvidia GeForce RTX2080ti, 32G, were deployed. During the training process, the present invention adopted the early stopping strategy built into Pytorch to avoid overfitting for all baselines. Time efficiency experiments were conducted on a server using Intel Xeon E5-2698v4 2.2GHz 20-core. The evaluation programming was written in C++, and the program allowed the use of a single thread of the CPU. The learning rate of NBOW was 1e-5, and training was performed using the AdamW algorithm. The hidden layer size was 768, the batch size of each module was 10, and the size of the output binary hash code of all baselines was 128. The hyperparameters β, λ1, and λ2 were 0.6, 0.1, and 0.1, respectively. In the query category prediction module, the cross entropy function was used as the loss function, and the total recall was 100.

[0060] Step 5: Use the query set to search on the trained model to evaluate the retrieval efficiency and accuracy of the model; use the query set to search on the trained model to evaluate the retrieval efficiency and accuracy of the model; classify the codes and select the codes in the corresponding category based on the similarity of the classification.

[0061] To verify the effect of the present invention, the method of the present invention was compared with the current mainstream code migration method.

[0062] Currently, mainstream code migration methods include TRANSCODER, TREE2TREE, and 1PSMT. The network performance was verified on an open-source Java to C# training set. The test results are shown in Table 1. The vertical columns show the migration accuracy and time required for the open-source projects.

[0063] Table 1

[0064]

[0065] It can be seen that although the accuracy of this method is slightly different, it still takes very little time while ensuring high accuracy. This shows the effectiveness and advantages of this method in code migration in industrial applications.

[0066] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A code migration method based on deep hashing, characterized in that: The following steps are involved: (1) Select an open source dataset and extract all code features in the dataset, including text features and structural features; further vectorize the code features to obtain a code feature vector; (2) Using the k-means algorithm, the code features are clustered to obtain multiple cluster groups; (3) Constructing a code migration network model based on deep hashing, wherein the code migration network model includes a vector representation code feature distribution module and a deep hashing module; The vector representation code feature distribution module is used to normalize the code feature vectors, calculate the code similarity matrix between code pairs, and represent the feature distribution of code pairs through the vectors in the code similarity matrix. It further integrates the information of the code similarity matrix by combining the code similarity matrix to comprehensively represent the distribution in the entire code space. Integrate the code similarity matrix by adding Construct the joint code similarity matrix S, the formula is: Among them, β is the weight parameter, S Co and S Cn is the code feature vector V of the code pair Co 、V Cn The normalized result is then calculated to obtain the code similarity matrix Among them S Co =V Co T V Co 、S Cn =V Cn T V Cn , and finally obtain the joint code similarity matrix S; The deep hash module converts code feature vectors into binary hash code vectors and calculates the hash code similarity matrix between code pairs after normalization. In the deep hash module, an improved bag-of-words (NBOW) model is used. Three fully connected FC layers with a specific tanh(·) activation function are used to replace the output layer of the original bag-of-words (BOW) model, keeping the output within the (-1, 1) interval. This allows the original code feature vectors to be converted into binary hash codes. (4) The code feature vectors of the clustered code pairs are used to train the code migration network model, so that the hash coding similarity matrix learned by the deep hashing module is aligned with the joint code similarity matrix, thereby retaining the similarity information between the code pairs in the hash space and the original code space; then the codes of the open source dataset are hashed using the trained model to construct a code hash database; Specifically, a dual-encoder architecture is adopted, using independent encoders to encode code pairs into representation vectors and corresponding binary hash codes respectively. A new triplet metric loss is introduced to ensure that the learned binary hash codes retain the similarity information between code pairs in the hash space and the original code space. The triplet metric loss function is: In the formula, θ is a hyperparameter, λ1 and λ2 are weight parameters, and B Co 、B Cn is the code feature vector V of the code pair Co 、V Cn The binary hash code vector obtained by binary encoding; using cosine similarity, calculate B Co 、B Cn The gap between them is reduced, thereby narrowing the distance between the hash code pairs and the joint code pairs; (5) For the code to be migrated, the code feature vector is extracted and input into the trained model. The corresponding binary hash code is obtained through the deep hash module, and the matching migrated code is retrieved from the code hash database.

2. The code migration method based on deep hashing according to claim 1, characterized in that In step (1), the specific process of extracting code features is as follows: (1-1) Use ANTLR to parse the code into a concrete syntax tree CST, unify the words with the same meaning but different names, and simplify them to obtain a simplified concrete syntax tree SCST; (1-2) Extract all paths from child nodes to the root node from the simplified concrete syntax tree SCST; (1-3) Serialize the entire extracted path.

3. The code migration method based on deep hashing according to claim 1, characterized in that In step (2), the number of cluster groups is 10, and different clusters are established by distinguishing based on code paths.

4. The code migration method based on deep hashing according to claim 1, characterized in that: In step (3), in the vector representation code feature distribution module, the code feature extraction vector is normalized using l2-norm.

Images