A function level fingerprinting method for JS artifact bundles

Abstract

The application discloses a function-level fingerprint construction method for JS product packages, and comprises the following steps: (1) constructing an open source three-party library product package: generating a product package through a full-automatic construction tool based on JS project source files, and adjusting configuration parameters based on preset rules to improve the construction success rate; (2) performing function-level slicing on the generated product package code based on an AST syntax tree to identify and separate various functions; (3) analyzing the syntax structure of the function-level AST node and extracting feature information of the function code; and (4) calculating the extracted feature information through local sensitive hashing to obtain code fingerprints. The application innovatively realizes the function-level fingerprint in the construction of JS product packages by combining a full-automatic packaging tool and AST syntax analysis, and is helpful to further component identification of JS product packages and solving the safety and reliability problems in the code.

Description

Technical Field

[0001] This invention belongs to the field of code fingerprinting technology, and in particular relates to a method for constructing function-level fingerprints for JS artifact packages. Background Technology

[0002] In today's digital age, JavaScript (JS) has become one of the core languages ​​for web application development. To simplify the development process and improve productivity, developers often introduce external dependencies when developing code. However, with the introduction of artifact packages, issues related to code security and reliability have become more prominent. Therefore, it is necessary to have a comprehensive understanding of the components of code dependencies.

[0003] Traditionally, developers typically use JavaScript package management tools (such as NPM and Yarn) to manage project dependencies, directly identifying dependencies referenced by the project within the source code. Alternatively, they can convert the source code into an Abstract Syntax Tree (AST) and analyze the dependencies introduced by import statements. However, this method struggles to accurately identify the source and dependencies of specific code. Furthermore, as web applications grow in size and complexity, many older or improperly stored JavaScript project source codes are lost, leaving only the artifact packages (deliverable packages generated from the source code build) that function correctly in the web application. Therefore, innovative techniques for JS dependency analysis based on code fingerprinting have become crucial. However, there is currently no feasible solution in the industry for code fingerprinting analysis of JavaScript artifact packages, primarily due to the following unresolved technical issues:

[0004] 1. It is difficult to obtain complete and comprehensive artifact package code for third-party dependencies.

[0005] Building a fingerprint database requires obtaining the artifact package code of third-party dependencies and generating JS fingerprints through calculation. However, most open-source third-party dependency repositories currently only provide source code, not artifact packages. Therefore, we need to first build the source code to generate artifact packages. However, there is currently no universal automated packaging tool. Due to the flexibility and diversity of the JS language ecosystem, many third-party dependencies require manual parameter configuration during the build process; otherwise, they cannot be successfully packaged into executable JS files. Faced with this complex build scenario, a universal automated build solution is needed to complete the data preparation work for the JS fingerprint database.

[0006] 2. It is difficult to achieve accurate matching of JS fingerprint analysis in complex grammatical environments.

[0007] Due to the flexibility of JavaScript code and the high compression of artifact packages, the content of JS artifact packages is difficult to analyze accurately, both semantically and textually. For example, multiple syntax configurations can be used during project building, and a single source code can generate code files under multiple syntax specifications such as ES5 and ES6. Using simple code obfuscation techniques, such as replacing variable names or shuffling statement order, can actually affect the code's distinguishability, thus causing analysis errors.

[0008] 3. How to achieve a balance between efficiency and accuracy.

[0009] To reduce the difficulty of code fingerprint analysis, complex code structures need to be broken down into sums of multiple simpler code snippets. However, this approach has obvious drawbacks: generating and matching fingerprints for multiple simpler code snippets increases analysis time and impacts matching efficiency. Therefore, reducing computation time while ensuring the accuracy of code fingerprint analysis is a challenge. Summary of the Invention

[0010] This invention provides a method for constructing function-level fingerprints for JS artifact packages. It utilizes an automated build tool to create third-party artifact packages. This tool combines preset rule repair and large language model repair to automatically adjust configuration parameters, improving the build success rate. It also parses the artifact package code to generate an AST (Abstract Syntax Tree), performs function-level segmentation and feature extraction based on the AST, and mitigates the impact of code changes caused by building and obfuscation on code analysis. Complex function structures are broken down into the sum of multiple simple functions, ultimately calculating and generating function-level fingerprints within the JS artifact package.

[0011] A method for building function-level fingerprints for JS artifact packages includes the following steps:

[0012] (1) Build open source third-party library artifact package: Generate artifact package from JS project source files using a fully automated build tool. This build tool combines preset rule repair and large language model repair to automatically correct configuration parameters to improve the success rate of the build.

[0013] (2) Based on the AST syntax tree, the generated artifact package code is sliced ​​at the function level to identify and separate each function;

[0014] (3) Analyze the syntax structure of the function-level AST nodes and extract the feature information of the function code;

[0015] (4) The code fingerprint is obtained by extracting the feature information through local sensitive hashing.

[0016] Furthermore, in step (1), the build tool combines preset rule repair and large language model repair to automatically correct configuration parameters to improve the build success rate. The specific implementation method is as follows:

[0017] First, build the project according to the initial configuration and monitor the build process. Then, capture and analyze the error messages of build failures. Next, attempt to match the error messages with pre-defined rules. If a rule is successfully matched, automatically modify the corresponding configuration parameters according to the repair method provided by the rule and execute the build process again. If no known rule is matched, combine the error message and the original configuration information to construct a prompt word and a problem description. Then, input the information into a large conversational language model and apply the corrected configuration information output by the model to execute the build process again. Repeat the above process iteratively until the repair is successful or the specified maximum number of repairs is reached.

[0018] The initial configuration includes the runtime environment, dependencies, and related plugins.

[0019] The preset rules include solutions for missing entry files, missing external dependencies, and compression tool errors.

[0020] Furthermore, in step (2), the artifact package code is sliced ​​at the function level based on the AST syntax tree. Using functions as the basic unit, the complex code is broken down into the sum of multiple simple functions. This simplifies the code structure while maintaining the integrity of individual function content. The specific process is as follows:

[0021] First, a compilation tool is selected to parse the AST syntax tree of the artifact package. Then, the nodes of the AST syntax tree are traversed in post-order using a depth-first search algorithm, and function-level AST nodes are obtained during the traversal. Function nodes with a length greater than a set segmentation threshold are independently segmented for subsequent fingerprinting. Functions with a length less than a specific threshold are ignored because fingerprinting of functions that are too short will be inaccurate. For nested functions, in order to ensure that each function is segmented, the child function nodes are traversed first to generate slices, and the parent function nodes are traversed last.

[0022] The function node position that is split off needs to be replaced with an empty function to maintain the structural integrity of the parent node. The syntax tree is then continuously updated iteratively according to the depth-first search algorithm described above.

[0023] The flexible setting of the splitting threshold is intended to optimize time efficiency while ensuring accuracy. To achieve this, a flexible splitting threshold is used, with larger thresholds set for JS projects with a large overall codebase.

[0024] Furthermore, the specific implementation method of step (3) is as follows:

[0025] First, create an array to store the results; then traverse the specific node type in the AST syntax tree, extract the literals and some variable names and add them to the array, and store the array output in the corresponding structure for subsequent fingerprint calculation.

[0026] The principle for extracting feature information from function code is to retain only literals and variable names that have distinctiveness in the code.

[0027] The specific implementation method of step (4) is as follows:

[0028] First, a locality-sensitive hash function is selected. The selected locality-sensitive hash function (usually a combination of multiple hash functions) is applied to the feature information extracted from the function to obtain a set of hash codes. The hash codes are then combined into a code fingerprint through bit operations or other methods.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. This invention establishes and maintains a JS fingerprint database. It proposes a universal JS automatic packaging tool that is compatible with multiple languages. When packaging errors occur, it modifies the configuration file based on the error message and performs a second packaging, combining two repair methods: preset rule repair and large language model repair. This achieves a universal automated build solution with a success rate of over 95%.

[0031] 2. This invention uses a code segmentation scheme for product packages, taking functions as the basic unit, to break down complex code into the sum of multiple simple functions, simplifying the code structure to be processed, thereby enhancing the accuracy of matching, and flexibly setting the length of the segmented code to optimize time efficiency while ensuring accuracy.

[0032] 3. This invention employs a series of function feature information extraction schemes to selectively retain literals and specific variable names, thereby automatically filtering out ordinary variable names and complex structures to eliminate their features. This solves the problem of large discrepancies between the source code text and the code obfuscation and complex syntax environment, and improves the recall rate of code matching. Attached Figure Description

[0033] Figure 1 This is a flowchart of a function-level fingerprint construction method for JS artifact packages according to the present invention;

[0034] Figure 2 This is a schematic diagram of the process for constructing the product package in this invention;

[0035] Figure 3 This is a schematic diagram of the function segmentation process in this invention;

[0036] Figure 4This is a schematic diagram of the process for extracting function feature information in this invention;

[0037] Figure 5 This is a schematic diagram of the process for calculating code fingerprints in this invention. Detailed Implementation

[0038] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0039] like Figure 1 As shown, a method for building function-level fingerprints for JS artifact packages includes the following steps:

[0040] S10: Prepare data for building the fingerprint database by using an automated packaging tool to generate third-party dependency artifact packages. The specific process is as follows: Figure 2 As shown:

[0041] (1) Obtain a list of currently popular third-party JavaScript repositories. First, download all data from the website https: / / skimdb.npmjs.com / registryg to your local machine using the CouchDB database. This website is an open-source project provided by npm, used to store registration information for all publicly released npm packages, including name, version, author, dependencies, license information, etc. Next, parse the dependencies of all third-party repositories in the registry, construct a dependency graph, and count the number of references for each third-party repository (i.e., node in-degree ranking). Finally, export the list of top-ranked repositories, download the source code of the repositories on the list using npm, and use them as the source of fingerprint database data.

[0042] (2) Build the initial configuration; First, use nodeJS as the project runtime environment, npm as the package management tool, and webpack as the build tool to build the basic build environment. After testing, the various tools can achieve maximum compatibility with various JavaScript syntax versions in the following versions, as shown in Table 1.

[0043] Table 1

[0044] npm 8.19.4 webpack 4.46.0

[0045] Configure webpack and install the corresponding loaders so that webpack can parse CSS, JSX, and SCSS files. See Table 2 for details:

[0046] Table 2

[0047] css-loader Parse CSS files babel-loader Used for parsing JSX files sass-loader and node-sass Used for parsing SCSS files

[0048] (3) Attempt to match the error message with predefined rules to see if there are any predefined repair methods related to the error. The specific predefined rules are shown in Table 3:

[0049] Table 3

[0050]

[0051]

[0052] (4) If a rule is successfully matched, the corresponding configuration parameters are automatically modified according to the repair method provided by the rule, and the build process is executed again;

[0053] (5) If the error message cannot match any predefined rules, combine the error message and the original configuration information to construct a prompt word and a problem description. Then input these information into the conversational large language model. The model will output the corrected configuration parameters based on its trained knowledge and the input problem description. Apply the configuration parameters to execute the construction process again.

[0054] (5) Repeat the above process iteratively until the repair is successful or the specified maximum number of repairs is reached.

[0055] S20: Perform function-level slicing on the artifact package code, breaking down the project's complex code structure into multiple simpler code segments, thus making the generated code fingerprint easier to match. The specific process is as follows: Figure 3 As shown:

[0056] S201. Parse the artifact package code into an abstract syntax tree (AST) using an appropriate compiler or parser; in this embodiment, the espree tool is used to parse the artifact package code.

[0057] S202. Traverse all function nodes using a depth-first search algorithm, including named functions, anonymous functions, arrow functions, etc. In this embodiment, the estraverse tool is used for the facilitation and analysis of the Abstract Syntax Tree (AST). For nested functions, it is necessary to ensure that each function is segmented, so the child function nodes are traversed first to generate slices, and the parent function nodes are traversed last.

[0058] S203. Separate function-level nodes

[0059] If the code length of the current function node exceeds a set threshold, the function is isolated for further function feature extraction, and an empty function is placed in its original location to maintain the structural integrity of the remaining code. If the code length of the current node is less than the set threshold, function feature extraction is not performed, but the node is still deleted from the remaining code and replaced with an empty function. This is because fingerprint calculation for excessively short functions will be inaccurate, and the deletion operation is also to strictly partition the function hierarchy where the code resides, so as to maintain the original code structure. The syntax tree is continuously updated iteratively according to the above algorithm.

[0060] To optimize time efficiency while maintaining accuracy, a flexible approach to setting segmentation thresholds is adopted, with larger thresholds set for JS projects with larger overall code sizes. This is because most short functions have simple structures and single functions, and many even have empty function bodies. These short functions have low distinguishability and perform poorly in source code matching. Testing showed that using a code length of 300 characters as the function segmentation threshold in this embodiment yielded the best source code matching results.

[0061] S204. Results Record: Record the slice code for future reference and team collaboration.

[0062] S30: Analyze the syntactic structure of function-level AST nodes and extract feature information from the function code. The specific process is as follows: Figure 4 As shown:

[0063] The principle for extracting feature information from function code is to retain only literals and unobfuscated variable names, thereby automatically filtering out content in the code that lacks distinguishability due to structural similarity. The extraction process is implemented as follows: First, an array is created to store strings and unobfuscated variable names. Second, the five types of nodes in the AST (Abstract Syntax Tree) are traversed: VariableDeclarator nodes, ObjectExpression nodes, CallExpression nodes, MemberExpression nodes, and Identifier nodes. Literals and some variable names are extracted from these nodes and added to the array. The array output is then stored in the corresponding structure for subsequent fingerprint calculation.

[0064] The more specific steps for extracting feature information from function code are as follows: For variable declaration (VariableDeclarator) nodes, retain the literal value on the right side of the equals sign. For object expression (ObjectExpression) nodes, retain all internal property names, and for property values, retain only simple literals, removing complex data structures. For function call (CallExpression) nodes, remove the parameters whose values ​​are simple literals, retaining the remaining string content and variable names. For property expression (MemberExpression) nodes, retain only the property name and string, removing the object name and complex data structures. Finally, traverse all identifier (Identifier) ​​nodes in the AST, retaining only variable names that are built-in structure names in JavaScript syntax, such as Object, Array, etc.; the remaining identifiers are obfuscated variable names and are not retained. After the AST nodes are modified, the array is converted into a string and output to a generated file.

[0065] S40: Calculate the code fingerprint, the specific process is as follows: Figure 5 As shown:

[0066] First, select an appropriate locality-sensitive hash function, usually a combination of multiple hash functions; apply the selected locality-sensitive hash function to the feature text extracted from the function to obtain a set of hash codes; combine the hash codes into a code fingerprint through bitwise operations or other methods.

[0067] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for constructing function-level fingerprints for JS artifact packages, characterized in that, Includes the following steps: (1) Build open source third-party library artifact package: Generate artifact package from JS project source files using a fully automated build tool. This build tool combines preset rule repair and large language model repair to automatically correct configuration parameters to improve the success rate of the build. (2) Based on the AST syntax tree, the generated artifact package code is sliced ​​at the function level to identify and separate each function; the specific process is as follows: First, a compilation tool is selected to parse the AST syntax tree of the artifact package. Then, the nodes of the AST syntax tree are traversed in post-order using a depth-first algorithm, and function-level AST nodes are obtained during the traversal. Function nodes with a length greater than a set splitting threshold are independently split for subsequent fingerprint calculation. Functions with a length less than a specific threshold are ignored because fingerprint calculations for functions that are too short will be inaccurate. For nested functions, in order to ensure that each function is split, the child function nodes are traversed first to generate slices, and the parent function nodes are traversed last. The function node position that is split off needs to be filled with an empty function to maintain the structural integrity of the parent node. The syntax tree is continuously updated iteratively according to the depth-first search algorithm described above. (3) Analyze the syntax structure of the function-level AST nodes and extract the feature information of the function code; the principle for extracting the feature information of the function code is: only retain the literals and variable names in the code that have distinctiveness; (4) The code fingerprint is obtained by extracting the feature information through local sensitive hashing.

2. The method for constructing function-level fingerprints for JS artifact packages according to claim 1, characterized in that, In step (1), the build tool combines preset rule repair and large language model repair to automatically correct configuration parameters in order to improve the success rate of the build. The specific implementation method is as follows: First, build the project according to the initial configuration and monitor the build process; then capture and analyze the error messages of build failure; further attempt to match the error messages with pre-defined rules. If a rule is successfully matched, automatically modify the corresponding configuration parameters according to the repair method provided by the rule, and execute the build process again. If no known rules are matched, the error message and the original configuration information are combined to construct a prompt word and a problem description. The information is then input into a large conversational language model, and the corrected configuration information output by the model is applied. The build process is executed again. The above process is repeated iteratively until the repair is successful or the specified maximum number of repairs is reached.

3. The method for constructing function-level fingerprints for JS artifact packages according to claim 2, characterized in that, The initial configuration includes the runtime environment, dependencies, and related plugins.

4. The method for constructing function-level fingerprints for JS artifact packages according to claim 2, characterized in that, The preset rules include solutions for missing entry files, missing external dependencies, and compression tool errors.

5. The method for constructing function-level fingerprints for JS artifact packages according to claim 1, characterized in that, To optimize time efficiency while ensuring accuracy, a flexible approach to setting splitting thresholds is adopted, with larger splitting thresholds set for JS projects with larger overall code size.

6. The method for constructing function-level fingerprints for JS artifact packages according to claim 1, characterized in that, The specific implementation method of step (3) is as follows: First, create an array to store the results; then traverse the specific node type in the AST syntax tree, extract the literals and some variable names and add them to the array, and store the array output in the corresponding structure for subsequent fingerprint calculation.

7. The method for constructing function-level fingerprints for JS artifact packages according to claim 1, characterized in that, The specific implementation method of step (4) is as follows: First, a locality-sensitive hash function is selected. The selected locality-sensitive hash function is then applied to the feature information extracted from the function to obtain a set of hash codes. The hash codes are then combined into a code fingerprint through bitwise operations or other methods.

Images