APK fingerprint association method and system based on DirHash algorithm

Abstract

This invention discloses an APK fingerprint association method and system based on the DirHash algorithm, belonging to the field of network security technology. The method includes the following steps: Step 1: Receiving and decompressing the APK file to obtain its directory structure; Step 2: Normalizing the directory structure obtained in Step 1: removing all file nodes, the assets directory and its subtrees, and metadata information; Step 3: Performing topological serialization encoding on the normalized directory structure to complete JSON serialization output, obtaining a JSON sequence; Step 4: Calculating the hash value of the JSON sequence obtained in Step 3 using a hash algorithm to obtain the DirHash fingerprint value; Step 5: Using the DirHash fingerprint value obtained in Step 4 to complete fingerprint association and cluster analysis. This invention solves the technical problem that traditional fingerprint methods fail against attacks such as package name randomization, certificate rotation, and resource fine-tuning.

Description

Technical Field

[0001] This invention belongs to the field of network technology, specifically relating to an APK fingerprint association method and system based on the DirHash algorithm. Background Technology

[0002] In mobile application security analysis and network anomaly detection practices, same-origin APK (i.e., Android Package) identification is a key technical requirement. Currently, the industry mainly uses fingerprinting methods such as package name recognition (extracted from AndroidManifest.xml, but easily randomized), certificate fingerprinting, whole-package hashing (calculating APK file SHA-256, but highly sensitive to content), and application name and icon recognition (easily spoofed). However, in actual processing, abnormal APKs exhibit characteristics such as multi-channel distribution, frequent variant iterations, and deep obfuscation and hardening. Attackers use automated tools to implement adversarial measures such as package name randomization, certificate rotation, and resource fine-tuning, causing a single abnormal application family to be scattered into tens of thousands of independent fingerprints in the sample database, rendering traditional clustering methods ineffective. Summary of the Invention

[0003] To address the problems of existing technologies, such as difficulty in resisting attacks like package name randomization, certificate rotation, and resource fine-tuning, as well as the high computational overhead during code decompilation and dynamic execution, this invention proposes an APK fingerprint association method based on the DirHash algorithm, comprising the following steps:

[0004] Step 1: Receive and decompress the APK file to obtain its directory structure;

[0005] Step 2: Standardize the directory structure obtained in Step 1: Remove all file nodes, the assets directory and its subtrees, and metadata information;

[0006] Step 3: Perform topological serialization encoding on the normalized directory structure to complete the JSON serialization output and obtain the JSON sequence;

[0007] Step 4: Calculate the hash value of the JSON sequence obtained in Step 3 using a hash algorithm to obtain the DirHash fingerprint value;

[0008] Step 5: Use the DirHash fingerprint value obtained in Step 4 to complete fingerprint association and cluster analysis.

[0009] Ideally, in step one, the APK file is first checked for integrity to verify that the file extension is .apk and that the file size meets system requirements. The file header magic number is used to confirm that the file is in zip format. Then, the structure of the zip file is extracted, a zip decompression library is called to create a temporary working directory, and the APK file is decompressed as a zip archive to obtain the complete decompressed files and directory list. Finally, a recursive traversal algorithm is used to construct a tree data structure from the decompressed directory structure.

[0010] Ideally, in step two, node type identification is performed first, and the directory tree structure is read before removing information; then, selective filtering rules are executed to remove all file nodes, assets directories and their subtrees, and metadata information in the directory tree; finally, a standardized directory tree structure is generated.

[0011] Preferably, the file nodes include: AndroidManifest.xml, classes.dex, resources.arsc, icon.png, and librustdesk.so; the assets directory format includes assets / and its contained config.json and fonts; the metadata information includes: timestamps, permission attributes (rwx flags), and owner information for each directory.

[0012] Ideally, in step three, a depth-first traversal should be performed first:

[0013] A: Start from the root node;

[0014] B: Visit the first child node;

[0015] C: Access the second child node;

[0016] D: Access the third child node;

[0017] E: Traversal complete;

[0018] Then, the direct child nodes of each directory node are sorted lexicographically, including the child nodes of the res directory and the child nodes of the lib directory. Finally, JSON serialization is performed, the JSON key names are arranged in alphabetical order, and spaces and newlines in the JSON sequence are removed, and then it is converted into a compact string.

[0019] Preferably, in step four, the input string is first converted into a UTF-8 byte sequence during the calculation;

[0020] Then, the SHA-1 algorithm is used to process the byte sequence: padding is added to make the byte sequence length a multiple of 512 bits, then the message is divided into 512-bit blocks, and 80 rounds of iterative operations are performed on each block to generate a 160-bit 20-byte hash value. Finally, the 20-byte binary string is converted into a 40-bit hexadecimal string; the message is a JSON string serialized from the directory structure.

[0021] In step five, the query logic is first executed. Using DirHash nodes as intermediate bridges, all APK nodes connected to the same DirHash node are found, establishing paths that link dirhash_X with apk_hash_A and apk_hash_B, forming a homogeneous cluster. Then, all homogeneous clusters are statistically analyzed and identified. The number of APK nodes connected to each DirHash node is counted, and then they are sorted in descending order of quantity. Large-scale groups are identified, and cluster analysis is completed.

[0022] An APK fingerprint association system based on the DirHash algorithm, characterized by comprising the following modules:

[0023] APK parsing and preprocessing module: It is used to receive the APK file to be analyzed, extract the complete directory tree structure inside the APK using ZIP decompression technology, and then construct a tree data structure of directory nodes;

[0024] DirHash fingerprint generation module: performs structural normalization processing on the directory tree, then performs topological serialization encoding and generates interference-resistant hash fingerprints;

[0025] Fingerprint association and clustering module: Stores and manages the DirHash fingerprint database, then performs same-origin APK matching and association, and finally outputs the group clustering analysis results.

[0026] In a more efficient manner, the DirHash fingerprint generation module performs the following structural normalization process on the directory tree:

[0027] First, recursively traverse the directory tree after the APK is decompressed to obtain the complete directory structure; then filter all file nodes and keep only the directory node information; then delete the assets directory and its child nodes, and remove the metadata information of the directory nodes, and finally build a clean directory topology tree structure.

[0028] The topology serialization encoding process is as follows: The normalized directory topology tree is traversed using a depth-first traversal strategy. Then, the child node names are sorted in lexicographical order at each directory node to serialize the directory tree into a JSON format string. Finally, the JSON key names are arranged in alphabetical order to eliminate the randomness caused by the unordered nature of the dictionary.

[0029] The process of generating an interference-resistant hash fingerprint is as follows: the JSON string generated by the topology serialization encoding process is encoded in UTF-8, then the message digest is calculated using the SHA-1 hash algorithm, and finally the 160-bit binary hash value is converted into a 40-bit hexadecimal string, which is the DirHash fingerprint; the message digest is a 160-bit binary hash value.

[0030] A preferred approach is to use the same APK matching and association method in the fingerprint association and clustering module as follows: After the APK to be detected generates a DirHash fingerprint through the DirHash fingerprint generation module, it first checks whether the same DirHash value exists in the fingerprint database. If it exists, it is determined to be a same APK and classified into the same group; if it does not exist, a new group is created and the DirHash value is added to the database. When outputting the group clustering analysis results, all APKs with the same DirHash value are first classified into one group, and then the sample size, first appearance time, and activity period of each group are counted. Finally, a group association analysis report is output.

[0031] Compared with the prior art, the technical solution of the present invention has the following advantages / benefits:

[0032] 1. This invention utilizes the stable feature of the APK directory topology and adopts a three-stage processing flow of normalization-serialization-hash calculation to generate a DirHash fingerprint that is insensitive to content changes but sensitive to structural changes. This fundamentally solves the problem that traditional fingerprinting methods fail when resisting attacks such as package name randomization, certificate rotation, and resource fine-tuning.

[0033] 2. This invention does not require complex code decompilation or dynamic execution; it only requires decompression and directory traversal, resulting in extremely low computational overhead and making it suitable for rapid processing of large-scale samples. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 This is a flowchart of an APK fingerprint association method based on the DirHash algorithm according to the present invention.

[0036] Figure 2 This is a schematic diagram of an APK fingerprint association system based on the DirHash algorithm according to the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention are described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of this invention, not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention. Therefore, the detailed description of the embodiments of this invention provided below is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the invention.

[0038] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Example 1:

[0039] like Figure 1 As shown, this invention proposes an APK fingerprint association method based on the DirHash algorithm, comprising the following steps:

[0040] Step 1: Receive and decompress the APK file to obtain its directory structure;

[0041] Step 2: Standardize the directory structure obtained in Step 1: Remove all file nodes, the assets directory and its subtrees, and metadata information;

[0042] Step 3: Perform topological serialization encoding on the normalized directory structure to complete the JSON serialization output and obtain the JSON sequence;

[0043] Step 4: Calculate the hash value of the JSON sequence obtained in Step 3 using a hash algorithm to obtain the DirHash fingerprint value;

[0044] Step 5: Use the DirHash fingerprint value obtained in Step 4 to complete fingerprint association and cluster analysis.

[0045] In step one, the integrity of the APK file is first checked. The path to the APK file to be analyzed, ` / data / samples / sample_001.apk`, is entered. The file extension is verified as `.apk`, and the file size is checked to ensure it meets system requirements. The file header magic number is read; in this example, it is 50 4B 03 04. The file is then confirmed to be in zip format. The structure of the zip file that meets the above requirements is extracted, and a temporary working directory, ` / tmp / apk_analysis / sample_001 / `, is created using a zip decompression library. The APK file is then decompressed as a zip archive, and the complete decompressed files and directory list are obtained. Finally, a recursive traversal algorithm is used to construct a tree data structure from the decompressed directory structure.

[0046] The file header magic number is a fixed sequence of bytes located at the beginning of a file, used to identify the file's true type. Operating systems and programs determine the file format by reading these bytes, rather than relying on the file extension (which can be arbitrarily changed).

[0047] The ZIP decompression library used in this embodiment is the zipfile module of Python.

[0048] After decompression, the directory structure is obtained, and a tree data structure is constructed using a recursive traversal algorithm.

[0049] In step two, before removing information, node type identification is performed, the directory tree structure is read, and then selective filtering rules are executed: all file nodes, assets directory and its subtrees, and metadata information in the directory tree structure are removed; finally, a normalized directory tree structure is generated.

[0050] The file nodes include: AndroidManifest.xml, classes.dex, resources.arsc, icon.png, and librustdesk.so; the assets directory format includes assets / and its contained config.json and fonts; the metadata information includes: timestamps for each directory, permission attributes (rwx flags), and owner information.

[0051] Finally, the standardized directory tree structure is obtained.

[0052] It is worth noting that in this directory tree structure, only directory nodes are retained, no file information is available, and the assets directory has been completely removed, with no timestamps or permission information.

[0053] In step three, a depth-first traversal is first performed, and the traversal order is as follows:

[0054] 1. Start from the root node;

[0055] 2. Visit the first child node;

[0056] - If the first child node has no secondary child nodes, backtrack;

[0057] 3. Visit the second child node;

[0058] - Enter the second child node and visit its first-level child nodes;

[0059] - If the first-level child node has no next-level child node, backtrack;

[0060] - Access its second-level child nodes;

[0061] - If the second-level child node has no next-level child node, backtrack;

[0062] - Access its third-level child nodes;

[0063] - If the third-level child node has no next-level child node, backtrack;

[0064] - Access its fourth-level child node;

[0065] - If the fourth-level child node has no next-level child node, backtrack;

[0066] - After traversing all secondary child nodes of the first child node, backtrack;

[0067] 4. Visit the third child node;

[0068] - Enter the third child node and visit its first-level child nodes;

[0069] - If the first-level child node has no next-level child node, backtrack;

[0070] - Access its second-level child nodes;

[0071] - If the second-level child node has no next-level child node, backtrack;

[0072] 5. Traversal complete;

[0073] Then, sort the direct child nodes of each directory node in lexicographical order.

[0074] Finally, perform JSON serialization to output the JSON sequence.

[0075] In step four, the input string is first converted into a UTF-8 byte sequence during the calculation;

[0076] Then, the byte sequence is processed using the SHA-1 algorithm: padding is added to ensure that the byte sequence length is a multiple of 512 bits, and then the message is divided into 512-bit blocks. Here, the message is a JSON string serialized from the directory structure.

[0077] Then, perform 80 rounds of iterative operations on each block to generate a 160-bit, 20-byte hash value. Finally, convert the 20-byte binary string into a 40-bit hexadecimal string to obtain the final DirHash fingerprint value.

[0078] In step five, the query logic is first executed. Using DirHash nodes as intermediate bridges, all APK nodes connected to the same DirHash node are found, establishing paths that link dirhash_X with apk_hash_A and apk_hash_B, forming a homogeneous cluster. Then, all homogeneous clusters are statistically analyzed and identified. The number of APK nodes connected to each DirHash node is counted, and then they are sorted in descending order of quantity. Large-scale groups are identified, and cluster analysis is completed. Example 2:

[0079] like Figure 2 As shown, this invention provides an APK fingerprint association system based on the DirHash algorithm, characterized by comprising the following modules:

[0080] APK parsing and preprocessing module: It is used to receive the APK file to be analyzed, extract the complete directory tree structure inside the APK using ZIP decompression technology, and then construct a tree data structure of directory nodes;

[0081] DirHash fingerprint generation module: performs structural normalization processing on the directory tree, then performs topological serialization encoding and generates interference-resistant hash fingerprints;

[0082] Fingerprint association and clustering module: Stores and manages the DirHash fingerprint database, then performs same-origin APK matching and association, and finally outputs the group clustering analysis results.

[0083] In the DirHash fingerprint generation module, the process of structural normalization of the directory tree is as follows:

[0084] First, recursively traverse the directory tree after the APK is decompressed to obtain the complete directory structure; then filter all file nodes and keep only the directory node information; then delete the assets directory and its child nodes, and remove the metadata information of the directory nodes, and finally build a clean directory topology tree structure.

[0085] The topology serialization encoding process is as follows: The normalized directory topology tree is traversed using a depth-first traversal strategy. Then, the child node names are sorted in lexicographical order at each directory node to serialize the directory tree into a JSON format string. Finally, the JSON key names are arranged in alphabetical order to eliminate the randomness caused by the unordered nature of the dictionary.

[0086] The process of generating an interference-resistant hash fingerprint is as follows: the JSON string generated by the topology serialization encoding process is encoded in UTF-8, then the message digest is calculated using the SHA-1 hash algorithm, and finally the 160-bit binary hash value is converted into a 40-bit hexadecimal string, which is the DirHash fingerprint.

[0087] At this point, the UTF-8 encoded byte sequence is the message, and the 160-bit binary hash value is the message digest.

[0088] In the fingerprint association and clustering module, the method for matching and associating APKs from the same source is as follows: After the APK to be detected generates a DirHash fingerprint through the DirHash fingerprint generation module, it first checks whether there is a DirHash value in the fingerprint database. If it exists, it is determined to be a APK from the same source and classified into the same group; if it does not exist, a new group is created and the DirHash value is added to the database. When outputting the group clustering analysis results, all APKs with the same DirHash value are first classified into one group, and then the sample size, first appearance time, and activity period of each group are counted. Finally, the group association analysis report is output.

[0089] The above are merely preferred embodiments of the present invention. It should be noted that the above preferred embodiments should not be considered as limitations on the present invention, and the scope of protection of the present invention should be determined by the scope defined in the claims. For those skilled in the art, several improvements and modifications can be made without departing from the spirit and scope of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An APK fingerprint association method based on the DirHash algorithm, characterized in that, Includes the following steps: Step 1: Receive and decompress the APK file to obtain its directory structure; Step 2: Standardize the directory structure obtained in Step 1: Remove all file nodes, the assets directory and its subtrees, and metadata information; Step 3: Perform topological serialization encoding on the normalized directory structure to complete the JSON serialization output and obtain the JSON sequence; Step 4: Calculate the hash value of the JSON sequence obtained in Step 3 using a hash algorithm to obtain the DirHash fingerprint value; Step 5: Use the DirHash fingerprint values ​​obtained in Step 4 to complete fingerprint association and clustering analysis; In step three, a depth-first traversal is first performed: A: Start from the root node; B: Visit the first child node; C: Access the second child node; D: Access the third child node; E: Traversal complete; Then sort the direct child nodes of each directory node in lexicographical order: including the child nodes of the res directory and the child nodes of the lib directory; Finally, perform JSON serialization output: sort the JSON keys alphabetically, remove spaces and newlines from the JSON sequence, and then convert it into a compact string; In step four, the input string is first converted into a UTF-8 byte sequence during the calculation. Then, the SHA-1 algorithm is used to process the byte sequence: padding is added to ensure that the byte sequence length is a multiple of 512 bits, then the message is divided into 512-bit blocks, and 80 rounds of iterative operations are performed on each block to generate a 160-bit 20-byte hash value. Finally, the 20-byte binary string is converted into a 40-bit hexadecimal string; the message is a JSON string serialized from a directory structure. In step five, the query logic is first executed. Using the DirHash node as an intermediate bridge, all APK nodes connected to the same DirHash node are found, establishing paths that are mutually associated between dirhash_X and apk_hash_A and apk_hash_B, forming a homogeneous group. Then, all homogeneous groups are statistically analyzed and identified. The number of APK nodes connected to each DirHash node is counted, and then they are sorted in descending order of quantity. Large-scale groups are identified, and cluster analysis is completed.

2. The APK fingerprint association method based on the DirHash algorithm according to claim 1, characterized in that, In step one, the APK file is first checked for integrity to verify that the file extension is .apk and that the file size meets system requirements. The file header magic number is used to confirm that the file is in zip format. Then, the structure of the zip file that meets the requirements is extracted: a temporary working directory is created by calling a zip decompression library, and the APK file is decompressed as a zip archive to obtain the complete decompressed files and directory list. Finally, a recursive traversal algorithm is used to construct a tree data structure for the decompressed directory structure.

3. The APK fingerprint association method based on the DirHash algorithm according to claim 1, characterized in that, In step two, node type identification is performed first, and the directory tree structure is read before removing information; then, selective filtering rules are executed to remove all file nodes, assets directory and its subtrees, and metadata information in the directory tree structure; finally, a standardized directory tree structure is generated.

4. The APK fingerprint association method based on the DirHash algorithm according to claim 3, characterized in that, The file nodes include: AndroidManifest.xml, classes.dex, resources.arsc, icon.png, and librustdesk.so; the assets directory format includes assets / and its contained config.json and fonts; the metadata information includes: timestamps, permission attributes (rwx flags), and owner information for each directory.

5. An APK fingerprint association system based on the DirHash algorithm, characterized in that, Includes the following modules: APK parsing and preprocessing module: It is used to receive the APK file to be analyzed, extract the complete directory tree structure inside the APK using ZIP decompression technology, and then construct a tree data structure of directory nodes; DirHash fingerprint generation module: performs structural normalization processing on the directory tree, then performs topological serialization encoding and generates interference-resistant hash fingerprints; Fingerprint association and clustering module: Stores and manages the DirHash fingerprint database, then performs same-origin APK matching and association, and finally outputs the group clustering analysis results; The DirHash fingerprint generation module performs the following process for structural normalization of the directory tree: First, recursively traverse the directory tree after the APK is decompressed to obtain the complete directory structure; then filter all file nodes and keep only the directory node information; then delete the assets directory and its child nodes and remove the metadata information of the directory nodes, and finally build a clean directory topology tree structure. The topology serialization encoding process is as follows: The normalized directory topology tree is traversed using a depth-first traversal strategy. Then, the child node names are sorted in lexicographical order at each directory node to serialize the directory tree into a JSON format string. Finally, the JSON key names are arranged in alphabetical order to eliminate the randomness caused by the unordered nature of the dictionary. The process of generating an interference-resistant hash fingerprint is as follows: the JSON string generated by the topology serialization encoding process is encoded in UTF-8, then the message digest is calculated using the SHA-1 hash algorithm, and finally the 160-bit binary hash value is converted into a 40-bit hexadecimal string, which is the DirHash fingerprint; the message digest is a 160-bit binary hash value. In the fingerprint association and clustering module, the method for matching and associating APKs from the same source is as follows: After the APK to be detected generates a DirHash fingerprint through the DirHash fingerprint generation module, it first checks whether there is a DirHash value in the fingerprint database. If it exists, it is determined to be a APK from the same source and classified into the same group; if it does not exist, a new group is created and the DirHash value is added to the database. When outputting the group clustering analysis results, all APKs with the same DirHash value are first classified into one group, and then the sample size, first appearance time, and activity period of each group are counted. Finally, a group association analysis report is output.

Images