A third-party library detection method for applets

Abstract

The application discloses a third-party library detection method for a small program, which is divided into two sub-methods. The first sub-method is a white list matching method. First, a crawler and a clustering method are used to crawl the third-party libraries that can be used by the small program from platforms such as NPM, CDNJS and the like, and function call graphs of the library files are generated to be added to a white list database. Then, the function call graphs of the JS files in the small program to be detected are generated in sequence, and a graph similarity measurement method is used to measure the similarity of the library files in the white list database. The second sub-method is a heuristic prediction method. A filtering clustering method is designed according to the code calling rules of the small program, the first-party code files are filtered out, the remaining files are clustered, and finally, the unknown libraries used in the small program are output. The application can finally output the third-party library set used in the small program to be detected through the above two sub-methods, and has important significance for small program separation of rights and responsibilities and safety risk analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of code static analysis in small programs, and in particular to a third-party library detection method in a small program. BACKGROUND

[0002] With the development of Internet technology, the third-party platform application represented by WeChat, Alipay and Baidu, namely the small program, has gradually become an indispensable service method in people's life. As of 2021, after 5 years of development, the number of small program DUA has exceeded 450 million, and the number of small program developers has exceeded 3 million. The use of third-party libraries can make the development of small programs more simple, fast and efficient. However, the security of third-party libraries is difficult to guarantee, and most third-party libraries lack security audits, and there are many security vulnerabilities in the code, such as plaintext transmission, which brings great challenges to the security of small programs.

[0003] With the continuous increase in the number of small programs, the use of their corresponding third-party library files is also increasing. Third-party library files are very important for the privacy and security analysis of small programs, and accurately identifying the third-party libraries of small programs is the basis of program analysis and has important application value. However, there are few studies on general third-party library analysis techniques for small programs.

[0004] The third-party library detection of small programs mainly has the following problems:

[0005] 1) The types and quantities of third-party libraries are numerous and the code styles are different, so it is difficult to find a unified separation feature from the third-party library code itself.

[0006] 2) Due to the size limitation of small programs, their third-party library files are usually a js file, and the developer can delete the js file according to the needs, which greatly increases the difficulty of third-party library analysis.

[0007] 3) Most of the current researches on third-party library extraction are based on Android APP, and have not involved small programs. These researches are based on certain prior knowledge or large-scale (usually million-level) APP input, or are targeted at a certain type of third-party library, and there is no heuristic prediction method for general third-party libraries in a single application. SUMMARY

[0008] The application provides a third-party library detection method for small programs to solve the above technical problems. The method includes two sub-methods to detect third-party library files in small programs: (1) a white list matching method, which first uses a crawler and a clustering method to crawl third-party libraries that can be used by small programs from platforms such as NPM, CDNJS, and generates function call graphs of each library file to join the white list database; then the function call graph of the JS file in the to-be-tested small program is generated in turn, and the graph similarity measurement method is used to measure the similarity with the library files in the white list database, if the similarity is greater than a given threshold, the matching is successful; (2) a heuristic prediction method, which is based on the code calling rules in small programs, designs a filtering clustering method to filter the first-party code files in small programs, and clusters the remaining files, and finally outputs the unknown libraries used in the small program. Through the above method, for the to-be-tested small program, first, the known libraries in the small program are identified through feature extraction and white list matching, and the file nodes that fail to match are identified through the heuristic prediction method, and the final output is the set of known libraries and unknown libraries.

[0009] To achieve the above purpose, the application adopts the following technical solutions:

[0010] 1) White list matching method:

[0011] Step S101: Use a crawler method to crawl third-party library files that can be used by small programs from platforms such as NPM, CDNJS, and use a clustering method to extract third-party libraries frequently used by small programs from large-scale small program samples. After obtaining the third-party library files, they need to be preprocessed to generate their function call graphs tcg, index keys k and other information, and finally they are added to the white list dataset D;

[0012] Step S102: De-compile the to-be-tested small program M, and generate the file dependency graph fdg of the small program M, and preprocess each source code file to generate its function call graph scg and index key k and other information to join the to-be-tested small program source code set S;

[0013] Step S103: According to the to-be-tested small program source code set S in step S102, traverse the to-be-tested file in S, use a fast positioning method to find the possible matching library files from the white list dataset D, and perform graph similarity measurement on the to-be-tested file function call graph scg and the library file function call graph tcg. Finally, the file set TS that matches successfully and the file set FS that fails to match are obtained, and TS is the known library set L used by the small program M;

[0014] 2) Heuristic prediction method:

[0015] Step S104: According to the matched file set FS obtained in step S103, the non-library file searching method is used to screen FS, and a non-library file set N and a candidate library set C are obtained;

[0016] Step S105: According to the non-library file set N and the candidate library set C obtained in step S104 and the file dependency graph fdg obtained in step S102, all upper node sets N' of each file node in N in fdg are found, and C-N' is calculated according to the applet file calling rule to obtain a new candidate library set C';

[0017] Step S106: According to the candidate library set C' obtained in step S105 and the file dependency graph fdg obtained in step S102, the file nodes containing dependency relationships in C' are clustered together, and each cluster is an unknown library predicted by the method according to the applet file calling rule. Finally, an unknown library set P is output.

[0018] Step S107: Output the third-party library used by the to-be-tested applet L+P.

[0019] Further, the step S101 needs to use a crawler and a clustering method to obtain the third-party library file. Since the task is relatively simple, a general web crawler can complete the third-party library crawling task. The clustering method mainly depends on the idea that most developers will not modify the third-party library file. When performing the clustering task, third-party library files with the same code characteristics will be clustered together. When the applet sample is large enough, the clustering result of the third-party library file frequently used by the applet will also be large. In this way, the third-party library frequently used by the applet can be found.

[0020] Further, the step S101 needs to preprocess the third-party library file to generate a function call graph, an index and the like. Specifically,

[0021] Step S201: Generate an abstract syntax tree AST of the third-party library file;

[0022] Step S202: Traverse the AST to find CallExpression, FunctionExpression, FunctionDeclaration, and NewExpression nodes n;

[0023] Step S203: Find the nearest p of the n in the same layer and belonging to the above four types;

[0024] Step S204: Add the edge (n, p) to the function call graph edge set E;

[0025] Step S205: Generate other information for the third-party library file, including the index key k and version information. This method uses the size of the function call graph as the index information, i.e., k = len(E).

[0026] Furthermore, step S102 requires generating a file dependency graph (fdg) for the mini-program M under test, traversing the mini-program files, and finding file dependencies based on import (require) information. Specifically, if file A has an import (require(B)), then there is a dependency relationship: A depends on B.

[0027] Furthermore, step S103 specifically includes:

[0028] Step S301: Traverse the set of source code files S after the initialization of the mini-program to be tested. Based on the index information k of the file to be tested, through the first B+ tree, find the set of third-party library file names NS that fluctuate around the same size as the file's code.

[0029] Step S302: Based on the names in the name set, find the storage location of the corresponding third-party library file through the second B+ tree, retrieve the data from the whitelist dataset D according to the storage location, and finally obtain the preprocessed data set D' of the third-party library;

[0030] Step S303: Traverse D' and use the graph kernel method to measure the similarity between the function call graphs tcg of different library files and the function call graph of the file to be tested;

[0031] Step S304: Take the maximum similarity value. If the similarity is greater than the given threshold, the match is successful and the file is added to the successful match set TS. Otherwise, the match fails and the file is added to the failed match set FS.

[0032] Furthermore, step S104 requires using a non-library file search method to find non-library files in the whitelist matching failure set FS. Specifically, this method uses the following three methods to achieve the intended purpose:

[0033] A method for extracting non-library files based on function call graphs: Manual analysis revealed that first-party code functions in a mini-program typically have high coupling within the global function call graph, while third-party library code functions are usually located at the edges of the graph. This method extracts file nodes with high coupling within the global call graph.

[0034] The API-based non-library file extraction method involves extracting the APIs called by the file under test, calculating the file's total weight based on pre-defined API weights, and considering the file as a non-library file if the total weight exceeds a threshold. The extracted APIs primarily include page redirects, routes, and page interactions—types typically not used by third-party libraries.

[0035] The non-library file extraction method based on string features: the method first extracts the string of the small program wxml, i.e., the front-end display, as a global string, and then extracts the string in the file to be tested as a local string, and the global string and the local string are compared by semantics to obtain a matching list, and if the size of the list is greater than a threshold, the file is considered to be a non-library file.

[0036] Further, the step S105 needs to find all parent nodes of the file dependency graph fdg for each non-library file node, specifically:

[0037] Step S401: initialize the non-library file queue Q, and enqueue all non-library files;

[0038] Step S402: determine whether there are still file nodes in Q, if so, dequeue a file node n;

[0039] Step S403: find the father node f of n in fdg, add f to the non-library file set N', and also add f to the queue Q, and return to step S402;

[0040] Step S404: output the non-library file set N'.

[0041] Further, the step S105 needs to calculate according to the small program file calling rule, which is summarized by a large amount of manual analysis, and the specific description is two rules: (1) there is no library file calling non-library file; (2) if two library files have a calling relationship, the two files belong to the same library.

[0042] Further improvement of the application is that: the step S103 uses two B+ trees to quickly locate the possible matching known library, which greatly reduces the time consumption. In addition, the function call graph of the file to be tested is extracted as a code feature, and the white list matching method using graph similarity measurement has good anti-confusion ability.

[0043] Further improvement of the application is that: the steps S104 to S106 propose a heuristic prediction method, summarize the small program file calling rule, and design the filtering clustering basis based on this, so that the application has certain third-party library inference ability.

[0044] Compared with the prior art, the application has the following advantages:

[0045] 1) The method of the application extracts the function call graph as a feature, and uses the graph similarity measurement method for similarity comparison, which solves the problem of poor anti-confusion ability of the white list matching method;

[0046] 2) The method of the application proposes an heuristic prediction method, which uses the regularity of small program file calling to design a filtering clustering method, so that the method of the application has certain inference ability for unknown libraries, and solves the problem that the existing general third-party library detection method for a single APK is generally based on certain prior knowledge; BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The overall flowchart of the third-party library detection method for small programs of the application;

[0048] Figure 2 The flowchart of the source code file preprocessing method of the application;

[0049] Figure 3 The flowchart of the white list matching method of the application;

[0050] Figure 4 The flowchart of the iterative filtering method in the heuristic prediction method of the application; DETAILED DESCRIPTION

[0051] The specific implementation of the third-party library detection method for small programs of the application will be described in detail below with reference to the accompanying drawings.

[0052] Referring to Figure 1 The application discloses a third-party library detection method for small programs, which comprises the following steps:

[0053] Step S101: Use a crawler method to crawl third-party library files that can be used by small programs from platforms such as NPM, CDNJS, etc., and use a clustering method to extract third-party libraries frequently used by small programs from a large number of small program samples. After obtaining the third-party library files, they need to be preprocessed to generate function call graphs tcg, index keys k and other information, and finally added to the white list dataset D;

[0054] Referring to Figure 2 Specifically, it can be divided into the following steps:

[0055] Step S201: Generate an abstract syntax tree AST of the third-party library file;

[0056] Step S202: Traverse the AST to find CallExpression, FunctionExpression, FunctionDeclaration, and NewExpression nodes n;

[0057] Step S203: Find the nearest layer of the parent node p of n and the parent node p belonging to the above four types;

[0058] Step S204: Add the edge edge(n, p) to the set E of function call graph edges;

[0059] Step S205: Generate the third-party library file other information including index key k, version information, etc. The method uses the size of the function call graph as the index information, i.e. k = len(E).

[0060] Step S102: De-compile the to-be-tested small program M, generate the file dependency graph fdg of the small program M, and pre-process each source code file to generate its function call graph scg and index key k and other information, and add them to the to-be-tested small program source code set S;

[0061] Specifically, the file dependency graph fdg is generated by traversing the small program file and finding the file dependency relationship according to the package require information. In particular, if there is a package require (B) in the A file, there is a dependency relationship A depends on B.

[0062] Referring to Figure 3 Specifically, it can be divided into the following steps:

[0063] Step S301: Traverse the to-be-tested small program initialization source code file set S, and find the third-party library file name set NS with approximately the same code size as the file according to the to-be-tested file index information k through the first B+ tree;

[0064] Step S302: According to the name in the name set, find the corresponding third-party library file storage location through the second B+ tree respectively, and get the data from the whitelist data set D according to the storage location, and finally get the third-party library pre-processed data set D';

[0065] Step S303: Traverse D', and use the graph kernel method to measure the similarity between the function call graph tcg of different library files and the function call graph of the to-be-tested file;

[0066] Step S304: Take the maximum value of the similarity, if the similarity is greater than a given threshold, the matching is successful, and the file is added to the matching success set TS, otherwise the matching fails, and the file is added to the matching failure set FS, and TS is the known library set L used by the small program M.

[0067] Step S104: According to the matching failed file set FS obtained in step S103, use the non-library file finding method to screen FS to obtain the non-library file set N and the candidate library set C;

[0068] Specifically, the method uses the following three methods to find non-library files:

[0069] A method for extracting non-library files based on function call graphs: Manual analysis revealed that first-party code functions in a mini-program typically have high coupling within the global function call graph, while third-party library code functions are usually located at the edges of the graph. This method extracts file nodes with high coupling within the global call graph.

[0070] The API-based non-library file extraction method involves extracting the APIs called by the file under test, calculating the file's total weight based on pre-defined API weights, and considering the file as a non-library file if the total weight exceeds a threshold. The extracted APIs primarily include page redirects, routes, and page interactions—types typically not used by third-party libraries.

[0071] A non-library file extraction method based on string features: This method first extracts the string displayed on the front end of the mini-program's wxml file as a global string, and then extracts the string in the file to be tested as a local string. The global string and the local string are semantically compared to obtain a matching list. If the size of the list is greater than a threshold, the file is considered a non-library file.

[0072] Reference Figure 4 As shown, it can be divided into the following steps:

[0073] Step S401: Initialize the non-library file queue Q and enqueue all non-library files;

[0074] Step S402: Determine whether there are still file nodes in Q. If so, dequeue the first file node n.

[0075] Step S403: Find the parent node f of n in fdg, add f to the non-library file set N', and also add f to the queue Q, then return to step S402;

[0076] Step S404: Output the set of non-library files N'.

[0077] Step S106: Based on the candidate library set C' obtained in step S105 and the file dependency graph fdg obtained in step S102, cluster the file nodes with dependencies in C' together. According to the file calling rules of the mini-program, each cluster is an unknown library predicted by this method. Finally, output the unknown library set P.

[0078] Step S107: Output the third-party libraries used by the mini-program under test, combined with L+P.

Claims

1. A method for detecting third-party libraries for small programs, characterized in that, Comprising the following steps: 1) White list matching method: Step S101: Use the crawler method to crawl the third-party library files that can be used by the applet from the NPM, CDNJS platform, use the clustering method to extract the third-party libraries frequently used by the applet from a large number of applet samples, after obtaining the third-party library files, they need to be preprocessed to generate their function call graph tcg, index key k information, and finally they are added to the white list dataset D; The preprocessed third-party library files are as follows: Step S202: Traverse the AST to find CallExpression, FunctionExpression, FunctionDeclaration, NewExpression four types of nodes n; Step S203: Find the nearest layer of n and the parent node p belonging to the above four types; Step S204: Add edge(n, p) to the function call graph edge set E; Step S205: generating the third-party library file other information includes index key k, version information, using function call graph as index information, that is ; Step S102: De-compile the test applet M, generate the file dependency graph fdg of the applet M, and preprocess each source code file to generate its function call graph scg and index key k information to add the test applet source code set S; Step S103: According to the test applet source code set S of step S102, traverse the test file in S, use the fast positioning method to find the library file that may match from the white list dataset D, and measure the graph similarity between the test file function call graph scg and the library file function call graph tcg, finally get the matched file set TS and the matched file set FS, TS is the known library set L used by the applet M; The specific steps are as follows: Step S301: Traverse the test applet initialization source code file set S, according to the test file index information k, pass through the first B+ tree, find the third-party library file name set NS with the same code size floating up and down; Step S302: According to the names in the name set, find the corresponding third-party library file storage location through the second B+ tree respectively, get the data from the whitelist dataset D according to the storage location, and finally get the preprocessed data set of the third-party library ; Step S303: traversing The similarity between the different library file function call graph tcg and the to-be-tested file function call graph is measured using the graph kernel method, respectively. Step S304: Take the maximum similarity value, if the similarity is greater than the given threshold, the matching is successful, and the file is added to the matching success set TS, otherwise the matching fails and the file is added to the matching failure set FS; 2) Heuristic prediction method: Step S104: According to the matched file set FS obtained in step S103, use the non-library file finding method to filter FS to obtain the non-library file set N and the candidate library set C; Step S105: according to the non-library file set N and the candidate library set C obtained in step S104 and the file dependency graph fdg obtained in step S102, find all the upper node sets of each file node in N in fdg , according to the applet file calling rule, calculate C- to obtain a new candidate library set ; Step S106: obtaining the candidate library set according to the candidate library set obtained in step S105 and the file dependency graph fdg obtained in step S102, the file nodes containing the dependency relationship are clustered together, each cluster is an unknown library predicted by the method according to the small program file calling rule, and finally the unknown library set P is output. and the file dependency graph fdg obtained in step S102, the file nodes containing the dependency relationship are clustered together, each cluster is an unknown library predicted by the method according to the small program file calling rule, and finally the unknown library set P is output. Step S107: Output the third-party library used by the test applet L+P.

2. The method of claim 1, wherein, The step S101 uses two methods of crawler and clustering to obtain the third-party library file, the clustering method is based on the idea that most developers will not modify the third-party library file, when performing clustering task, the third-party library files with the same code characteristics will be clustered together, when the applet sample is large enough, the size of the clustering result of the third-party library frequently used by the applet will also be large, so the third-party library frequently used by the applet can be found.

3. The method of claim 1, wherein, The step S102 generates a file dependency graph fdg of the to-be-tested miniprogram M in a manner of traversing the miniprogram file to find file dependency relationship according to the package guide require information. In particular, if the A file exists package guide , there is a dependency relationship A depends on B.

4. The method of claim 1, wherein, The step S104 uses the following three methods to find non-library files in the test applet: 1) Non-library file extraction method based on function call graph: extract file nodes with high coupling degree in global call graph; 2) Non-library file extraction method based on API: extract the API called by the file to be tested, calculate the total weight of the file according to the pre-given API weight, and if the total weight is greater than the threshold, it is considered that the file is a non-library file. The extracted API includes page jump, routing, and page interaction; 3) Non-library file extraction method based on string features: this method first extracts the string of the small program wxml, which is the front-end display, as the global string, and then extracts the string in the file to be tested as the local string. The global string and the local string are compared by semantics to obtain a matching list. If the size of the list is greater than the threshold, it is considered that the file is a non-library file.

5. The method of claim 1, wherein, The step S105 needs to find all the elder nodes of the file dependency graph fdg for each non-library file node, which is specifically: Step S401: Initialize the non-library file queue Q, and enqueue all non-library files; Step S402: Determine whether there are still file nodes in Q, if so, dequeue a file node n; Step S403: find the father node f of n in the fdg, add f to the non-library file set and also add f to the queue Q, return to step S402; Step S404: outputting the non-library file set .

6. The method of claim 1, wherein, The step S105 needs to calculate according to the calling rule of the small program file. This rule is summarized through a large amount of manual analysis and induction, and is specifically described as two rules: (1) there is no case of library file calling non-library file; (2) if there is a calling relationship between two library files, the two files belong to the same library.

Images