Code duplicate removal method and device, equipment and storage medium
By parsing code units and calculating structural overlap, consistency, and error sub-scores, and comprehensively calculating similarity scores, the problem of not being able to identify parameter-differentiated logic in existing technologies is solved, achieving highly accurate and flexible code deduplication.
Patent Information
- Application Number
- CN202511497246.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-20
- Publication Date
- 2026-01-13
AI Technical Summary
Existing code deduplication methods can only identify completely identical code snippets and cannot detect similar logic with different parameters, resulting in coarse identification and a high deduplication error rate.
By parsing the code to obtain individual code units, the structural overlap, consistency sub-score, and error sub-score between code units are calculated. A similarity score is then calculated, and deduplication is performed based on the similarity score, with specific deduplication logic applied to different code types.
It improves the accuracy and flexibility of code deduplication, enabling more precise identification of similar logic with different parameters, and reducing the deduplication error rate.
Smart Images
Figure CN121326344A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of code deduplication, in particular to a code deduplication method, device, equipment and storage medium. BACKGROUND
[0002] Nowadays, code duplication is common in various fields, especially when multiple people collaborate to develop a front-end project. Different developers may independently implement the same functional module, or the same visual effect CSS rules are repeatedly defined, or layout structures and similar components are created multiple times, and so on. Therefore, code duplication affects the efficiency of multiple people collaborating to develop a front-end project. The existing solution is to use a text comparison tool, which is the most basic code similarity detection method. It compares the code text content line by line to identify duplicate fragments.
[0003] However, this solution can only identify identical code fragments and cannot detect similar logic with parameter differentiation. Therefore, the identification is relatively rough, and the error rate of deduplication is high. SUMMARY
[0004] Therefore, the present application provides a code deduplication method, device, equipment and storage medium to solve the problem that the existing solution can only identify identical code fragments and cannot detect similar logic with parameter differentiation. Therefore, the identification is relatively rough, and the error rate of deduplication is high.
[0005] To achieve the above purpose, the present solution is as follows:
[0006] In a first aspect, a code deduplication method comprises:
[0007] parsing the code to be deduplicated to obtain each code unit;
[0008] For each of two code units, calculate the structural coincidence degree between the two code units as a coincidence sub-score, and calculate the consistency sub-score and error sub-score between the two code units;
[0009] Calculate the similarity score according to the coincidence sub-score, consistency sub-score and error sub-score;
[0010] If the similarity score is greater than a preset similarity threshold, the two code units are considered as a group of similar codes;
[0011] Determine the code type of each group of similar codes, and perform deduplication processing on each group of similar codes based on the code type.
[0012] Preferably, the parsing of the code to be deduplicated to obtain each code unit comprises:
[0013] classifying the code to be deduplicated to determine respective code types;
[0014] for code of the code type of JavaScript function, parsing through a syntax tree to extract function features;
[0015] for code of the code type of style rule, converting it into a mathematical vector;
[0016] for code of the code type of Vue component, extracting template features, style features and logic features therein;
[0017] the function features, the mathematical vector, the template features, the style features and the logic features are all code units.
[0018] Preferably, the calculation of the structural coincidence degree between the two code units comprises:
[0019] respectively acquiring tree structures of the two code units;
[0020] one-to-one corresponding tree nodes on the two tree structures and calculating a matching degree between each pair of corresponding tree nodes;
[0021] adding up the respective matching degrees to obtain a matching sum;
[0022] determining a total number of tree nodes contained on the two tree structures;
[0023] dividing the matching sum by the total number to obtain the structural coincidence degree between the two code units.
[0024] Preferably, the calculation of the consistency sub-score between the two code units comprises:
[0025] respectively extracting key control flow node sequences of the two code units;
[0026] determining input data and output data from the key control flow node sequences;
[0027] constructing a mapping relationship from the input data and the output data;
[0028] determining consistency of the mapping relationships corresponding to the two code units respectively to calculate a consistency sub-score.
[0029] Preferably, the calculation process of the error sub-score comprises:
[0030] identifying one or more variable difference points between implementation logics of the two code units to determine a number of the variable difference points;
[0031] Simultaneously identify one or more points of change between the implementation logic of these two code units to determine the number of such points of change;
[0032] Divide the number of variable difference points by the number of change points to obtain the error sub-score.
[0033] Preferably, the step of calculating the similarity score based on the overlap sub-score, consistency sub-score, and error sub-score includes:
[0034] The formula for calculating the similarity score is:
[0035] ;
[0036] in, Represents the similarity score. The sub-fraction represents the degree of overlap. Represents the consistency sub-score. Represents the error sub-fraction. The weighting coefficients for the overlap sub-scores. The weighting coefficients for the consistency sub-scores. These are the weighting coefficients for the error sub-fractions. , , , It is a constant.
[0037] Preferably, the step of performing deduplication on each group of similar codes based on the code type includes:
[0038] For each group of similar codes:
[0039] If the code type of the similar code in the group is a JavaScript function, then the two code units in the similar code in the group will be merged to obtain the merged code;
[0040] Identify the first point of difference between two code units in this group of similar codes;
[0041] The first difference point is used as a parameter to generate a higher-order function corresponding to the merged code, and the group of similar code is deleted;
[0042] If the code type is a style rule, then both code units in the group of similar code are converted into individual reusable style classes;
[0043] The reusable style classes are combined, and each value in the combination is replaced with a preset variable to form a style variable. Similar code in the group is then deleted.
[0044] If the code type is a Vue component, then the layout framework that is the same in two code units in the group of similar code is taken as the base component, and the second difference point between the two code units in the group of similar code is determined.
[0045] The region corresponding to the second difference point is used as a slot, which is reconstructed by the base component and the slot. Then, the interface is configured to form a new component, and the group of similar code is deleted.
[0046] Secondly, a code deduplication device includes:
[0047] The parsing module is used to parse the code to be deduplicated into individual code units.
[0048] The sub-score calculation module is used to calculate the structural overlap between two code units for every two code units, as the overlap sub-score, and to calculate the consistency sub-score and the error sub-score between the two code units.
[0049] The similarity score calculation module is used to calculate the similarity score based on the overlap sub-score, consistency sub-score, and error sub-score.
[0050] The similar code determination module is used to classify the two code units as a group of similar codes if the similarity score is greater than a preset similarity threshold.
[0051] The deduplication module is used to determine the code type of each group of similar code, and perform deduplication processing on each group of similar code based on the code type.
[0052] Thirdly, a code deduplication device, including a memory and a processor;
[0053] The memory is used to store programs;
[0054] The processor is configured to execute the program to implement the steps of the code deduplication method as described in any of the first aspects.
[0055] Fourthly, a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the code deduplication method as described in any of the first aspects.
[0056] As can be seen from the above technical solution, this application obtains each code unit by parsing the code to be deduplicated; for every two code units, the structural overlap between the two code units is calculated as the overlap sub-score, and the consistency sub-score and error sub-score between the two code units are also calculated; a similarity score is calculated based on the overlap sub-score, consistency sub-score and error sub-score; if the similarity score is greater than a preset similarity threshold, the two code units are considered as a group of similar codes; the code type of each group of similar codes is determined, and deduplication processing is performed on each group of similar codes based on the code type. This application first parses the code to be deduplicated, dividing it into various code units. Subsequently, deduplication analysis and processing are performed on multiple code units, improving precision and avoiding the coarseness of existing technologies. For every two code units, three sub-scores are calculated: a structural overlap sub-score, a consistency sub-score, and an error sub-score, taking into account error. A similarity score is then calculated by combining these three sub-scores. This allows for the detection of differences between code units from multiple perspectives, overcoming the limitation of existing technologies that can only identify completely identical code. It accurately identifies whether codes are similar and compares the calculated values with a similarity threshold to determine the similarity of each group of codes. In addition to considering differences when calculating sub-scores, the application also differentiates based on code type during deduplication, implementing different deduplication logics, which greatly improves accuracy, flexibility, and adaptability. Attached Figure Description
[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0058] Figure 1 An optional flowchart of a code deduplication method provided in an embodiment of this application;
[0059] Figure 2 This is a schematic diagram of the structure of a code deduplication device provided in an embodiment of this application;
[0060] Figure 3 This is a schematic diagram of a code deduplication device provided in an embodiment of this application. Detailed Implementation
[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0062] Code duplication is prevalent across various fields, especially in collaborative front-end development projects. This can manifest in various ways, such as different developers independently implementing functional modules with identical logic, repeatedly defining CSS rules for the same visual effects, or creating components with similar layouts and functions multiple times. Therefore, code duplication negatively impacts the efficiency of collaborative front-end development projects. Current solutions rely on text comparison tools, the most basic code similarity detection method, which identifies duplicate segments by comparing line-by-line code text.
[0063] However, this solution can only identify completely identical code snippets and cannot detect similar logic with different parameters. Therefore, the identification is relatively crude and the deduplication error rate is very high.
[0064] To address the shortcomings of the prior art, this invention provides a code deduplication method. This method can be applied to various computer terminals or smart terminals, and its execution entity can be the processor or server of the computer terminal or smart terminal. This invention can be used in numerous general-purpose or special-purpose computing device environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, distributed computing environments including any of the above devices, etc.
[0065] The flowchart of the method is as follows: Figure 1 As shown, it specifically includes:
[0066] S1: Parse the code to be deduplicated to obtain individual code units.
[0067] This step breaks down the large amount of code to be deduplicated into smaller, more manageable independent code units, such as functions, methods, classes, code blocks, etc., laying the foundation for subsequent fine-grained comparison. It avoids the low accuracy and uncertainty caused by existing technologies that can only perform coarse comparisons of code segments in the entire file.
[0068] Furthermore, the parsed code units facilitate subsequent parallel computing and distributed analysis, improving deduplication efficiency.
[0069] S2: For every two code units, calculate the structural overlap between the two code units as the overlap sub-score, and at the same time calculate the consistency sub-score and the error sub-score between the two code units.
[0070] This step calculates sub-scores from three different dimensions:
[0071] First, the calculation of structural overlap focuses on the structural level of code units. It can effectively ignore superficial differences such as variable names, function names, comments, and spaces, and thus directly focus on the logical structure and organization of the code.
[0072] Secondly, the consistency sub-score focuses on the literal consistency between code units at the character, token, or keyword level. It can complement the overlap sub-score to make a preliminary judgment on the literal consistency between code units. The addition of the consistency sub-score calculation can accurately filter out those code units that are obviously similar or completely identical.
[0073] Finally, the error sub-score is calculated to minimize the errors in the overlap and consistency sub-scores, making the calculation results more accurate.
[0074] S3: Calculate the similarity score based on the overlap sub-score, consistency sub-score, and error sub-score.
[0075] The three sub-scores calculated in the above steps are then integrated to achieve a multi-dimensional comprehensive calculation, avoiding the one-sidedness of a single indicator. For example, two code units may have a low consistency sub-score because all variables have been renamed, but because their structural overlap sub-score is high, and error correction is performed through the error sub-score, they can still be accurately determined to be a set of similar code.
[0076] In addition, this multi-dimensional comprehensive calculation method is flexible and adjustable, allowing users to freely set the weights or thresholds involved in the calculation process.
[0077] S4: If the similarity score is greater than the preset similarity threshold, then the two code units are considered as a group of similar codes.
[0078] Different similarity thresholds can be set in different application scenarios. A high similarity threshold can capture only the most similar code units to ensure high accuracy; a low similarity threshold can expand the recognition range, identify more potential duplicates, and prevent omissions. This embodiment does not impose any restrictions on this.
[0079] S5: Determine the code type of each group of similar codes, and perform deduplication processing on each group of similar codes based on the code type.
[0080] In addition to considering differences when calculating sub-fractions, different deduplication logic is also implemented based on differences in code types during specific deduplication. This targeted deduplication process can further ensure the accuracy of deduplication.
[0081] As can be seen from the above technical solution, this application obtains each code unit by parsing the code to be deduplicated; for every two code units, the structural overlap between the two code units is calculated as the overlap sub-score, and the consistency sub-score and error sub-score between the two code units are also calculated; a similarity score is calculated based on the overlap sub-score, consistency sub-score and error sub-score; if the similarity score is greater than a preset similarity threshold, the two code units are considered as a group of similar codes; the code type of each group of similar codes is determined, and deduplication processing is performed on each group of similar codes based on the code type. This application first parses the code to be deduplicated, dividing it into various code units. Subsequently, deduplication analysis and processing are performed on multiple code units, improving precision and avoiding the coarseness of existing technologies. For every two code units, three sub-scores are calculated: a structural overlap sub-score, a consistency sub-score, and an error sub-score, taking into account error. A similarity score is then calculated by combining these three sub-scores. This allows for the detection of differences between code units from multiple perspectives, overcoming the limitation of existing technologies that can only identify completely identical code. It accurately identifies whether codes are similar and compares the calculated values with a similarity threshold to determine the similarity of each group of codes. In addition to considering differences when calculating sub-scores, the application also differentiates based on code type during deduplication, implementing different deduplication logics, which greatly improves accuracy, flexibility, and adaptability.
[0082] The method provided in this embodiment of the invention, which parses the code to be deduplicated to obtain each code unit, is described in detail below:
[0083] The code to be deduplicated is categorized to determine the various code types;
[0084] For JavaScript function code, a syntax tree is used for parsing to extract function characteristics;
[0085] For code of style rule type, convert it into a mathematical vector;
[0086] For code that is a Vue component, extract its template features, style features, and logic features;
[0087] The aforementioned function features, mathematical vectors, template features, style features, and logical features are all treated as code units.
[0088] Specifically, by parsing JavaScript function code using a syntax tree, the extracted function features include function signature features, logical structure features, and context features. Among these, function signature features include the number of parameters and return value type; logical structure features include the depth of nested loops and the number of conditional branches; and context features include the set of external variables referenced.
[0089] For the code of style rules, the mathematical vectors obtained by conversion include size features, visual features, and layout features. Among them, size features include inner margin, outer margin, and size value; visual features include color coding, gradient parameters, and shadow values; and layout features include color code, positioning mode, and flexible layout parameters.
[0090] The template features extracted from the code of Vue components include the root node type and the depth distribution of child nodes; the style features include the selector priority vector; and the logical features include the lifecycle hook call sequence.
[0091] The process of calculating the structural overlap between these two code units in this application is described in detail below.
[0092] Obtain the tree structure of each of these two code units;
[0093] The tree nodes on the two tree structures are matched one-to-one, and the matching degree between each pair of corresponding tree nodes is calculated.
[0094] Add up the matching scores to get the total matching score;
[0095] Determine the total number of tree nodes contained in the two tree structures;
[0096] Divide the sum of the matches by the total number to obtain the structural overlap between the two code units.
[0097] Specifically, compared to simply comparing lines of text, such as string matching, comparing ASTs (Abstract Syntax Trees) can provide a deeper understanding of the logical structure and syntactic composition of code. In one example, comparing tree structures can identify code that has completely different variable names, function names, comments, etc., but identical control flow (such as if-else, for loops) and expression structure.
[0098] In the specific comparison process, tree nodes are matched one by one. That is, if there is a tree node a1 in the tree structure of code unit A, then there is a tree node a2 in the tree structure of code unit B that corresponds to tree node a1. These two tree nodes have similar structures and are in the same position in the tree structure, so these two tree nodes can be matched. Thus, the matching degree between these two corresponding tree nodes can be calculated. The matching degree between each pair of tree nodes is added together to obtain the matching sum between the two tree structures. However, there may be non-corresponding tree nodes. Therefore, the matching sum needs to be divided by the total number to obtain the structural overlap between the two code units.
[0099] The following embodiments provide a detailed explanation of the steps for calculating the consistency sub-score between the two code units in this application.
[0100] Extract the key control flow node sequences from these two code units respectively;
[0101] The input and output data are determined from the sequence of key control flow nodes;
[0102] A mapping relationship is constructed from the input data and the output data;
[0103] Determine the consistency of the mapping relationships corresponding to these two code units to calculate the consistency sub-score.
[0104] Specifically, for the consistency sub-score, it is calculated through the key control flow node sequence. The key control flow node sequence is a simplified sequence extracted from the abstract syntax tree (AST) or control flow graph (CFG) of the code. It retains the key decision points (such as conditional branches and loops) and core operations (such as function calls) of the program execution path, while filtering out the specific implementation details such as variable names, literals, and assignments.
[0105] The extraction process of the key control flow node sequence is as follows: determine the abstract syntax tree of each of the two code units, then traverse the abstract syntax tree in a specific order. During the traversal process, only the node types belonging to the key control nodes are retained, and other details are ignored. Finally, these retained nodes are arranged into a sequence according to the traversal order, which is the key control flow node sequence.
[0106] Therefore, the input and output data of the key control flow node sequence are very important, and there is a mapping relationship between the two. The consistency sub-score can be calculated from this mapping relationship, which can accurately express the degree of consistency between code units.
[0107] Furthermore, the calculation process for the error sub-fraction can include the following steps:
[0108] Identify one or more variable differences between the implementation logic of these two code units to determine the number of such variable differences;
[0109] Simultaneously identify one or more points of change between the implementation logic of these two code units to determine the number of such points of change;
[0110] Divide the number of variable difference points by the number of change points to obtain the error sub-score.
[0111] Specifically, variable differences can include date formats, sorting methods, etc., while things like browser environment dependencies cannot be considered variable differences. Change points refer to all variable or immutable differences. Dividing the number of variable differences by the number of change points reveals the variable differences between two code units; that is, the higher the error sub-score, the greater the similarity between the two code units in a certain aspect.
[0112] Since the variable difference point was not considered when calculating the structural overlap and consistency sub-scores, this application sets an error sub-score to correct the calculation error of the structural overlap and consistency sub-scores, thereby improving the accuracy of the similarity score.
[0113] Optionally, the process of calculating the similarity score based on the overlap sub-score, consistency sub-score, and error sub-score in this application is specifically described below:
[0114] The formula for calculating the similarity score is:
[0115] ;
[0116] in, Represents the similarity score. The sub-fraction represents the degree of overlap. Represents the consistency sub-score. Represents the error sub-fraction. The weighting coefficients for the overlap sub-scores. The weighting coefficients for the consistency sub-scores. These are the weighting coefficients for the error sub-fractions. , , , It is a constant.
[0117] Specifically, this application takes into account the weighting coefficient of the error sub-score when setting the weighting coefficients of the overlap sub-score and the consistency sub-score. Compared with the prior art, this method of setting weights is more accurate and can eliminate calculation errors.
[0118] in, It is a relatively small positive number used to prevent the denominator from being zero, and the weighting coefficient of the error fraction is the largest.
[0119] Furthermore, the process of deduplicating each group of similar code based on the code type may include the following steps:
[0120] For each group of similar codes:
[0121] If the code type of the similar code in the group is a JavaScript function, then the two code units in the similar code in the group will be merged to obtain the merged code;
[0122] Identify the first point of difference between two code units in this group of similar codes;
[0123] The first difference point is used as a parameter to generate a higher-order function corresponding to the merged code, and the group of similar code is deleted;
[0124] If the code type is a style rule, then both code units in the group of similar code are converted into individual reusable style classes;
[0125] The reusable style classes are combined, and each value in the combination is replaced with a preset variable to form a style variable. Similar code in the group is then deleted.
[0126] If the code type is a Vue component, then the layout framework that is the same for two code units in the group of similar code is taken as the base component, and the second difference point between the two code units in the group of similar code is determined.
[0127] The region corresponding to the second difference point is used as a slot, which is reconstructed by the base component and the slot. Then, the interface is configured to form a new component, and the group of similar code is deleted.
[0128] Specifically, this application distinguishes similar code according to code type, namely: JavaScript function type, style rule type, and Vue component type, and performs different deduplication operations for different types.
[0129] For similar code of JavaScript function types, a parameter validation mechanism can be built when generating higher-order functions to ensure type safety. In addition, the original calling interface should be retained after generating higher-order functions to ensure compatibility.
[0130] For similar code of style rule type, after forming style variables, a style variable system can also be established at the same time; for similar code of Vue component type, the configuration interface is to realize the structured abstraction and reuse of basic components, while maintaining the compatibility of the original basic component template structure.
[0131] and Figure 1 Corresponding to the method described above, embodiments of the present invention also provide a code deduplication device for deduplicating code. Figure 1 The specific implementation of the method, the code deduplication device provided in this embodiment of the invention, can be used in computer terminals or various mobile devices, combined with Figure 2 This section introduces code deduplication devices, such as... Figure 2 As shown, the device may include:
[0132] Parsing module 10 is used to parse the code to be deduplicated to obtain individual code units;
[0133] Sub-score calculation module 20 is used to calculate the structural overlap between two code units for every two code units, as the overlap sub-score, and at the same time calculate the consistency sub-score and the error sub-score between the two code units.
[0134] The similarity score calculation module 30 is used to calculate a similarity score based on the overlap sub-score, consistency sub-score, and error sub-score.
[0135] The similar code determination module 40 is used to classify the two code units as a group of similar codes if the similarity score is greater than a preset similarity threshold.
[0136] The deduplication module 50 is used to determine the code type of each group of similar codes, and perform deduplication processing on each group of similar codes based on the code type.
[0137] As can be seen from the above technical solution, this application obtains each code unit by parsing the code to be deduplicated; for every two code units, the structural overlap between the two code units is calculated as the overlap sub-score, and the consistency sub-score and error sub-score between the two code units are also calculated; a similarity score is calculated based on the overlap sub-score, consistency sub-score and error sub-score; if the similarity score is greater than a preset similarity threshold, the two code units are considered as a group of similar codes; the code type of each group of similar codes is determined, and deduplication processing is performed on each group of similar codes based on the code type. This application first parses the code to be deduplicated, dividing it into various code units. Subsequently, deduplication analysis and processing are performed on multiple code units, improving precision and avoiding the coarseness of existing technologies. For every two code units, three sub-scores are calculated: a structural overlap sub-score, a consistency sub-score, and an error sub-score, taking into account error. A similarity score is then calculated by combining these three sub-scores. This allows for the detection of differences between code units from multiple perspectives, overcoming the limitation of existing technologies that can only identify completely identical code. It accurately identifies whether codes are similar and compares the calculated values with a similarity threshold to determine the similarity of each group of codes. In addition to considering differences when calculating sub-scores, the application also differentiates based on code type during deduplication, implementing different deduplication logics, which greatly improves accuracy, flexibility, and adaptability.
[0138] Furthermore, embodiments of this application provide a code deduplication device. Optionally, Figure 3 The hardware structure block diagram of the code deduplication device is shown below. Figure 3 The hardware structure of the code deduplication device may include: at least one processor 01, at least one communication interface 02, at least one memory 03 and at least one communication bus 04.
[0139] In this embodiment, the number of processor 01, communication interface 02, memory 03 and communication bus 04 is at least one, and processor 01, communication interface 02 and memory 03 communicate with each other through communication bus 04.
[0140] Processor 01 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0141] Memory 03 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device.
[0142] The memory stores a program, which the processor can call to execute the deduplication method, including:
[0143] The code to be deduplicated is parsed to obtain individual code units;
[0144] For every two code units, the structural overlap between the two code units is calculated as an overlap sub-score, and the consistency sub-score and error sub-score between the two code units are also calculated.
[0145] The similarity score is calculated based on the overlap sub-score, consistency sub-score, and error sub-score.
[0146] If the similarity score is greater than the preset similarity threshold, then the two code units are considered as a group of similar codes;
[0147] Determine the code type for each group of similar code, and perform deduplication processing on each group of similar code based on the code type.
[0148] Optionally, the refinement and extension functions of the program can be found in the description of the code deduplication method in the method embodiment.
[0149] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor. When the program runs, it controls the device where the storage medium is located to execute the following code deduplication method, including:
[0150] The code to be deduplicated is parsed to obtain individual code units;
[0151] For every two code units, the structural overlap between the two code units is calculated as an overlap sub-score, and the consistency sub-score and error sub-score between the two code units are also calculated.
[0152] The similarity score is calculated based on the overlap sub-score, consistency sub-score, and error sub-score.
[0153] If the similarity score is greater than the preset similarity threshold, then the two code units are considered as a group of similar codes;
[0154] Determine the code type for each group of similar code, and perform deduplication processing on each group of similar code based on the code type.
[0155] Specifically, the storage medium can be a computer-readable storage medium, which can be an electronic storage device such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM.
[0156] Optionally, the refinement and extension functions of the program can be found in the description of the code deduplication method in the method embodiment.
[0157] Furthermore, the functional modules in the various embodiments of this disclosure can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the function is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a live streaming device, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this disclosure.
[0158] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0159] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0160] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A code deduplication method, characterized in that, include: The code to be deduplicated is parsed to obtain individual code units; For every two code units, the structural overlap between the two code units is calculated as an overlap sub-score, and the consistency sub-score and error sub-score between the two code units are also calculated. The similarity score is calculated based on the overlap sub-score, consistency sub-score, and error sub-score. If the similarity score is greater than the preset similarity threshold, then the two code units are considered as a group of similar codes; Determine the code type for each group of similar code, and perform deduplication processing on each group of similar code based on the code type.
2. The method according to claim 1, characterized in that, The parsing of the code to be deduplicated yields individual code units, including: The code to be deduplicated is categorized to determine the various code types; For JavaScript function code, a syntax tree is used for parsing to extract function characteristics; For code of style rule type, convert it into a mathematical vector; For code that is a Vue component, extract its template features, style features, and logic features; The aforementioned function features, mathematical vectors, template features, style features, and logical features are all treated as code units.
3. The method according to claim 1, characterized in that, The calculation of the structural overlap between these two code units includes: Obtain the tree structure of each of these two code units; The tree nodes on the two tree structures are matched one-to-one, and the matching degree between each pair of corresponding tree nodes is calculated. Add up the matching scores to get the total matching score; Determine the total number of tree nodes contained in the two tree structures; Divide the sum of the matches by the total number to obtain the structural overlap between the two code units.
4. The method according to claim 1, characterized in that, The calculation of the consistency sub-score between the two code units includes: Extract the key control flow node sequences from these two code units respectively; The input and output data are determined from the sequence of key control flow nodes; A mapping relationship is constructed from the input data and the output data; Determine the consistency of the mapping relationships corresponding to these two code units to calculate the consistency sub-score.
5. The method according to claim 1, characterized in that, The calculation process of the error sub-fraction includes: Identify one or more variable differences between the implementation logic of these two code units to determine the number of such variable differences; Simultaneously identify one or more points of change between the implementation logic of these two code units to determine the number of such points of change; Divide the number of variable difference points by the number of change points to obtain the error sub-score.
6. The method according to claim 1, characterized in that, The calculation of the similarity score based on the overlap sub-score, consistency sub-score, and error sub-score includes: The formula for calculating the similarity score is: ； in, Represents the similarity score. The sub-fraction represents the degree of overlap. Represents the consistency sub-score. Represents the error sub-fraction. The weighting coefficients for the overlap sub-scores. The weighting coefficients for the consistency sub-scores. These are the weighting coefficients for the error sub-fractions. , , , It is a constant.
7. The method according to any one of claims 1 to 6, characterized in that, The process of deduplicating each group of similar code based on the code type includes: For each group of similar codes: If the code type of the similar code in the group is a JavaScript function, then the two code units in the similar code in the group will be merged to obtain the merged code; Identify the first point of difference between two code units in this group of similar codes; The first difference point is used as a parameter to generate a higher-order function corresponding to the merged code, and the group of similar code is deleted; If the code type is a style rule, then both code units in the group of similar code are converted into individual reusable style classes; The reusable style classes are combined, and each value in the combination is replaced with a preset variable to form a style variable. The similar code in the group is then deleted. If the code type is a Vue component, then the layout framework that is the same for two code units in the group of similar code is taken as the base component, and the second difference point between the two code units in the group of similar code is determined. The region corresponding to the second difference point is used as a slot, which is reconstructed by the base component and the slot. Then, the interface is configured to form a new component, and the group of similar code is deleted.
8. A code deduplication device, characterized in that, include: The parsing module is used to parse the code to be deduplicated into individual code units. The sub-score calculation module is used to calculate the structural overlap between two code units for every two code units, as the overlap sub-score, and to calculate the consistency sub-score and the error sub-score between the two code units. The similarity score calculation module is used to calculate the similarity score based on the overlap sub-score, consistency sub-score, and error sub-score. The similar code determination module is used to classify the two code units as a group of similar codes if the similarity score is greater than a preset similarity threshold. The deduplication module is used to determine the code type of each group of similar code, and perform deduplication processing on each group of similar code based on the code type.
9. A code deduplication device, characterized in that, Including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the code deduplication method as described in any one of claims 1-7.
10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the code deduplication method as described in any one of claims 1-7.