A file clustering method, apparatus and device
By processing the application interface sequence information of files, feature vectors are generated for clustering, which solves the problem of difficult clustering of files with obfuscated or junk instructions, and improves the accuracy and efficiency of clustering.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TENCENT TECHNOLOGY (SHENZHEN) CO LTD
- Filing Date
- 2019-03-05
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies struggle to effectively handle files with obfuscated or scrambled instructions when clustering files, resulting in inaccurate clustering and high computational complexity, especially with a large number of files, which incurs a huge computational burden.
By obtaining the application interface sequence information of the files to be clustered, combining it into an interface sequence tuple, and using message digest algorithm and locality-sensitive hashing algorithm to generate feature vectors, file clustering is performed.
It improves the accuracy of file clustering, especially for files with obfuscated or junk instructions, while reducing computational complexity and data processing costs.
Smart Images
Figure CN111666404B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of file clustering, in particular to a file clustering method, device and equipment. BACKGROUND
[0002] The current file clustering method is based on the static information of the file, such as collecting various static information of the executable file according to the PE (Portable Executable) structure of the executable file to do weighted calculation, and determining whether two files are similar by comparing the static information of the two files.
[0003] The method of clustering by static information has good universality and does not need to consider the system environment dependent on file execution, but it cannot cluster files using deformation technologies such as shell and flower instructions, although their dynamic behaviors may be similar, and the static characteristics of some files are not obvious, which is easy to cause false positives.
[0004] In order to improve the accuracy of clustering, static clustering usually selects as many features as possible, up to dozens of features. The more features, the higher the computational complexity, and the more complex the index. When the amount of files increases, the clustering calculation becomes extremely large.
[0005] Therefore, it is necessary to propose a file clustering scheme that can easily cope with files using shell and flower instructions. SUMMARY
[0006] The present application provides a file clustering method, device and equipment, and provides a new file clustering scheme that can easily cope with a large amount of files to be clustered. The present application is implemented by the following technical scheme:
[0007] In a first aspect, the present application provides a file clustering method, comprising:
[0008] Obtaining application program interface sequence information called by a plurality of files to be clustered when executed, the application program interface sequence information comprising a plurality of application program interfaces sorted according to the calling time sequence;
[0009] Combining the application program interface sequence information of each file to be clustered into a plurality of interface sequence tuples according to the sorting of the plurality of application program interfaces corresponding to each file to be clustered, the interface sequence tuple containing at least two application program interfaces;
[0010] Determining a plurality of feature vectors of the plurality of interface sequence tuples corresponding to each file to be clustered;
[0011] Determining a feature vector of each file to be clustered based on the plurality of feature vectors corresponding to each file to be clustered;
[0012] cluster the plurality of files to be clustered by using the feature vectors of the plurality of files to be clustered.
[0013] In a second aspect, the present application provides a file clustering device, comprising:
[0014] a obtaining module, configured to obtain application program interface sequence information called by a plurality of files to be clustered when executed, the application program interface sequence information comprising a plurality of application program interfaces sorted according to calling time sequence;
[0015] a combining module, configured to combine the application program interface sequence information of each file to be clustered into a plurality of interface sequence tuples according to the sorting of the plurality of application program interfaces corresponding to the each file to be clustered, the interface sequence tuples comprising at least two application program interfaces;
[0016] a first determining module, configured to determine a plurality of feature vectors of the plurality of interface sequence tuples corresponding to each file to be clustered;
[0017] a second determining module, configured to determine a feature vector of each file to be clustered based on the plurality of feature vectors corresponding to the each file to be clustered;
[0018] a clustering module, configured to cluster the plurality of files to be clustered by using the feature vectors of the plurality of files to be clustered.
[0019] Further, the combining module further comprises:
[0020] a third determining module, configured to determine a first number of application program interfaces contained in an interface sequence tuple;
[0021] a fourth determining module, configured to determine an application program interface extraction window based on the first number of determined application program interfaces;
[0022] a fifth determining module, configured to sequentially extract a first number of adjacent application program interfaces from the application program interface sequence information of each file to be clustered by using the application program interface extraction window, to obtain a plurality of interface sequence tuples, wherein a moving step of the application program interface extraction window is one application program interface when sequentially extracting the first number of adjacent application program interfaces.
[0023] Further, the first determining module further comprises:
[0024] a mapping module, configured to perform mapping processing on the plurality of interface sequence tuples corresponding to each file to be clustered by using a message digest algorithm, to obtain a plurality of feature vectors of the plurality of interface sequence tuples.
[0025] Further, the second determining module further comprises:
[0026] The conversion module is used to convert the multiple feature vectors corresponding to each file to be clustered into feature vectors of each file using the locality-sensitive hashing algorithm.
[0027] Thirdly, the present invention provides a file clustering device, the device comprising a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the file clustering method as described in the first aspect.
[0028] This invention provides a file clustering method, apparatus, and device. It involves splitting the behavioral sequence of the file to be clustered into multiple tuples, mapping these tuples into corresponding vectors, and then transforming them into feature vectors for the file to be clustered. The file is then clustered based on these feature vectors. This solves the problem of difficulty or inaccuracy in file clustering using modification techniques such as obfuscation or junk code in existing technologies, thus improving the accuracy of file clustering. Attached Figure Description
[0029] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0030] Figure 1 This is a schematic diagram of a system provided in an embodiment of the present invention;
[0031] Figure 2 This is a flowchart illustrating a file clustering method provided in an embodiment of the present invention;
[0032] Figure 3 This is a flowchart illustrating a method for determining file feature vectors provided in an embodiment of the present invention;
[0033] Figure 4 This is a flowchart illustrating a file clustering method based on the feature vectors of files to be clustered, provided by an embodiment of the present invention.
[0034] Figure 5 This is a flowchart illustrating another method for file clustering based on the feature vectors of files to be clustered, provided by an embodiment of the present invention.
[0035] Figure 6 This is a flowchart illustrating another method for file clustering based on the feature vectors of files to be clustered, provided by an embodiment of the present invention.
[0036] Figure 7 This is a schematic diagram of the architecture of a file clustering method provided in an embodiment of the present invention;
[0037] Figure 8 This is a flowchart illustrating another file clustering method provided in an embodiment of the present invention;
[0038] Figure 9 This is a schematic diagram of the structure of a document clustering device provided in an embodiment of the present invention;
[0039] Figure 10 This is a schematic diagram of another file clustering device provided in an embodiment of the present invention. Detailed Implementation
[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0041] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.
[0042] Please see Figure 1 , Figure 1 This is a schematic diagram of a system provided in an embodiment of the present invention, such as... Figure 1 As shown, the system may include a sample server 01, a dynamic system server 02, a clustering computing server 03, and a database 04.
[0043] Specifically, the sample server 01 may include a standalone server, a distributed server, or a server cluster consisting of multiple servers. The sample server 01 may include a network communication unit, a processor, and memory, etc. Specifically, the sample server 01 can obtain the files to be clustered to provide sample files for the aforementioned dynamic system server 02.
[0044] Specifically, the dynamic system server 02 may include a standalone server, a distributed server, or a server cluster consisting of multiple servers. The dynamic system server 02 may include a network communication unit, a processor, and memory, etc. Specifically, the dynamic system server 02 can generate sample file behavior data based on samples obtained from the sample server 01, and send the sample file behavior data to the clustering calculation server 03.
[0045] Specifically, the clustering computing server 03 may include a standalone server, a distributed server, or a server cluster consisting of multiple servers. The clustering computing server 03 may include a network communication unit, a processor, and a memory, etc. Specifically, the clustering computing server 03 can cluster sample file behavior data obtained from the dynamic system server 02, send the clustering results to the database 04 for storage, and read file clustering results from the database 04.
[0046] Specifically, the database 04 can be a software or hardware entity. As a hardware entity, it can operate independently or be integrated into other servers. The database 04 can receive and store the clustering results sent by the clustering computing server 03, and can also respond to the read request of the clustering computing server 03 by sending the corresponding clustering results to the clustering computing server 03.
[0047] The following describes the file clustering method of the present invention based on the above-described system. This specification provides the method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual system or server product execution, the method can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment) as shown in the embodiments or drawings.
[0048] Figure 2 This is a flowchart illustrating a file clustering method provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the method specifically includes:
[0049] S201: Obtain the application interface sequence information called during the execution of multiple files to be clustered, wherein the application interface sequence information includes multiple application interfaces ordered according to the calling sequence.
[0050] In a specific example, if the number of files to be clustered is M, where, for example Figure 3As shown, the application interface sequence information called by the file A to be clustered during execution is P1-P2-P3-P4. This interface sequence information includes the calling order information of each interface. If there is an application interface sequence information of P1-P2-P4-P3, then they are different application interface sequence information.
[0051] S203: Based on the sorting of the multiple application interfaces corresponding to each file to be clustered, combine the application interface sequence information of each file to be clustered into multiple interface sequence tuples, wherein each interface sequence tuple contains at least two application interfaces.
[0052] Specifically, the step of combining the application interface sequence information of each file to be clustered into multiple interface sequence tuples according to the sorting of multiple application interfaces corresponding to each file to be clustered includes the following steps:
[0053] (1) Determine the first number of application interfaces contained in the interface sequence tuple;
[0054] The interface sequence tuple can be a binary or a triple, meaning it can contain two or more applications.
[0055] The program interface can also include three application programming interfaces (APIs), and the specific settings can be adjusted according to actual needs.
[0056] This needs to be done. Taking a tuple as an example, then we need to determine the applications contained in the interface sequence tuple.
[0057] The first number of interfaces is 2.
[0058] (2) Extract windows based on the first number of application interface determinations;
[0059] (3) Using the application interface extraction window, extract a first number of adjacent application interfaces from the application interface sequence information of each file to be clustered, and obtain multiple interface sequence tuples, wherein the moving step of the application interface extraction window when extracting the first number of adjacent application interfaces is one application interface.
[0060] Furthermore, taking the aforementioned example, the application's extraction window extracts interface sequence tuples from the M files to be clustered, where, for example... Figure 3 As shown, the application interface extraction window extracts the application interface sequence information P1-P2-P3-P4 of the file to be clustered A by moving one application interface step, sequentially extracting two adjacent application interfaces, resulting in three tuples:
[0061] P1-P2, P2-P3, P3-P4.
[0062] S205: Determine multiple feature vectors for the multiple interface sequence tuples corresponding to each file to be clustered.
[0063] Specifically, determining the multiple feature vectors of the multiple interface sequence tuples corresponding to each file to be clustered includes: using a message digest algorithm to map the multiple interface sequence tuples corresponding to each file to be clustered to obtain multiple feature vectors of the multiple interface sequence tuples.
[0064] The message digest algorithm can specifically be MD5, SHA-1, etc. For example, the three tuples obtained in step S203 are hashed using the MD5 algorithm to obtain three corresponding feature vectors Md5_hash1, Md5_hash2, and Md5_hash3, as shown below. Figure 3 As shown.
[0065] S207: Determine the feature vector of each file to be clustered based on the multiple feature vectors corresponding to each file to be clustered.
[0066] Specifically, determining the feature vector of each file to be clustered based on the multiple feature vectors corresponding to each file to be clustered includes: using a locality-sensitive hashing algorithm to convert the multiple feature vectors corresponding to each file to be clustered into the feature vector of each file to be clustered.
[0067] The locality-sensitive hashing algorithm can be the simhash algorithm. For example, the three feature vectors obtained in step S205 are merged and transformed into a simhash vector, which is equivalent to the fingerprint of the file to be clustered, A. Thus, M files to be clustered can be converted into M feature vectors.
[0068] S209: Cluster the multiple files to be clustered using the feature vectors of the multiple files to be clustered.
[0069] There are various clustering methods depending on different dimensions of files. This specification uses clustering based on the dynamic behavior characteristics of files. Files typically refer to executable files or script files. When a file is executed, it obtains system resources by calling operating system application APIs to complete its program purpose. Therefore, the sequence of API calls (behavioral sequence) has certain characteristics, namely, the behavioral sequences of files of the same type are similar. The solution proposed in this specification is to achieve clustering based on the dynamic behavior of files. Regardless of whether the file is obfuscated or not, it does not affect the clustering effect. The simhash algorithm is selected, and through testing, it achieves an accuracy rate of 95%, while current static clustering only has an accuracy rate of about 80%.
[0070] This specification solves the problem of difficulty or inaccuracy in clustering files using modification techniques such as obfuscation or obfuscation in existing technologies, thereby improving the accuracy of file clustering. It breaks down the behavioral sequences of files to be clustered into multiple tuples, maps these tuples into corresponding vectors, and then transforms them into feature vectors for the files to be clustered. Finally, it clusters the files based on these feature vectors.
[0071] Figure 4 This is a flowchart illustrating a file clustering method based on the feature vectors of files to be clustered, as provided in an embodiment of the present invention. Figure 4 As shown, it specifically includes:
[0072] S401: Calculate the distance between each pair of feature vectors of the file to be clustered.
[0073] Specifically, the method for calculating the pairwise distance between the feature vectors of the file to be clustered can be Euclidean distance, Hamming distance, cosine of the included angle, etc.
[0074] S403: Determine whether the distance between any two feature vectors of the file to be clustered is greater than or equal to a preset distance threshold.
[0075] In the embodiments of this specification, a distance threshold for determining the pairwise distance between the feature vectors of the files to be clustered needs to be preset when clustering the files to be clustered. The specific threshold can be set according to actual needs.
[0076] S405: When the distance between the feature vectors of two files to be clustered is less than or equal to a preset distance threshold, the two files to be clustered are clustered.
[0077] The closer the feature vectors of two files to be clustered, the higher their similarity, and the two files will be grouped into one class.
[0078] This specification takes into account the good comparability of simhash itself, and the ability to compare similarity using Euclidean distance, Hamming distance, cosine similarity, etc. Therefore, file clustering can be achieved based on similarity. Thus, M feature vectors can be used to cluster the M files to be clustered.
[0079] Figure 5 This is a flowchart illustrating another file clustering method based on the feature vectors of files to be clustered, provided by an embodiment of the present invention. Figure 5 As shown, it specifically includes:
[0080] S501: Perform minimum hash calculation on the feature vector of each file to be clustered to obtain multiple minimum hash values.
[0081] The process of calculating the minimum hash is shown in the table below:
[0082]
[0083] Where S1 and S2 represent action sequences, and A, B, C, and D are the binary representations of the action sequences.
[0084] S503: Based on the multiple minimum hash values, the feature vector of each file to be clustered is divided into multiple interface sequence buckets.
[0085] For the multiple minimum hash values obtained in step S501, those with the same value are placed into the same interface sequence bucket. Thus, as... Figure 6 As shown, multiple interface sequence buckets are obtained: bucket 1, bucket 2, ..., bucket N.
[0086] S505: Calculate the pairwise distance between the feature vectors of the files to be clustered in the same interface sequence bucket.
[0087] Specifically, the method for calculating the pairwise distance between feature vectors of files to be clustered in the same interface sequence bucket can be Euclidean distance, Hamming distance, cosine of the included angle, etc.
[0088] S507: Determine whether the distance between any two feature vectors of files to be clustered in the same interface sequence bucket is less than or equal to a preset distance threshold.
[0089] In the embodiments of this specification, a distance threshold needs to be preset to determine the distance between each pair of feature vectors of files to be clustered in the same interface sequence bucket when clustering files to be clustered. The specific threshold can be set according to actual needs.
[0090] S509: When the distance between the feature vectors of two files to be clustered is less than or equal to a preset distance threshold, the two files to be clustered are clustered.
[0091] The closer the feature vectors of two files to be clustered, the higher their similarity, and the two files will be grouped into one class.
[0092] This specification considers that simhash itself has good comparability, and similarity can be compared using Euclidean distance, Hamming distance, and cosine similarity. Therefore, file clustering can be achieved based on similarity. Thus, M feature vectors can be used to cluster the M files to be clustered. Furthermore, file clustering generally involves a comparison process, which is costly for large or massive datasets. Therefore, this specification considers adopting strategies such as bucketing to meet high-performance computing requirements. After bucketing with minhash, pairwise comparisons only need to be performed within the same bucket, thereby reducing the data volume within each bucket. The time complexity of simhash is O(n). Simultaneously, by introducing a bucketing strategy and using the minhash algorithm to build the index, similarity calculation is optimized, easily handling massive samples.
[0093] The present invention also provides an architectural diagram of a file clustering method, as detailed in the embodiments below. Figure 7 The architecture is designed like a sorter in a production line, grouping samples into the same family, with the cluster ID serving as the family identifier. When a sample enters the scheduling queue, its simhash and minhash values are calculated based on the dynamic sequence. The minhash is then used as an index to perform a similarity calculation with the cluster IDs (simhash values) in the database, classifying samples into clusters with similar cluster IDs. The cluster ID is selected from the simhash of the first sample.
[0094] The present invention also provides a flowchart illustrating another file clustering method, see below for details. Figure 8 .
[0095] First, the dynamic sequence of the file is converted into a computable vector. The behavior sequence is split into individual behaviors, and n-grams of behaviors are combined to add temporal information. Then, a hash (using the MD5 algorithm) is generated for the n-grams. Finally, the N generated hashes are merged to generate a simhash. The generated simhash is the computable vector of the file, which is equivalent to the fingerprint of the file.
[0096] Calculating the similarity between two files involves calculating the Hamming distance between the two simhash values. During clustering, if the Hamming distance is less than a set threshold, the two files are considered to be of the same type, i.e., belonging to the same family.
[0097] The final actual clustering results can be seen in the table below:
[0098]
[0099]
[0100] In practical applications, we can first check if the min_hash values are the same. If they are the same, we can then calculate whether the Hamming distance between the simhash values of the two files is less than a threshold (we actually use a threshold of 5.5). If it is less than the threshold, it means that the two files belong to the same class.
[0101] This invention also provides a file clustering device, such as... Figure 8 As shown, the device includes:
[0102] The acquisition module 901 is used to acquire application interface sequence information called during the execution of multiple files to be clustered, wherein the application interface sequence information includes multiple application interfaces ordered according to the calling sequence.
[0103] The combination module 903 is used to combine the application interface sequence information of each file to be clustered into multiple interface sequence tuples according to the order of the multiple application interfaces corresponding to each file to be clustered, wherein the interface sequence tuple contains at least two application interfaces.
[0104] The first determining module 905 is used to determine multiple feature vectors of the multiple interface sequence tuples corresponding to each file to be clustered;
[0105] The second determining module 907 is used to determine the feature vector of each file to be clustered based on the multiple feature vectors corresponding to each file to be clustered.
[0106] Clustering module 909 is used to cluster the multiple files to be clustered using the feature vectors of the multiple files to be clustered.
[0107] Furthermore, the clustering module 909, as... Figure 9 Figure 10 As shown, it includes:
[0108] The first calculation module 1001 is used to perform minimum hash calculation on the feature vector of each file to be clustered to obtain multiple minimum hash values.
[0109] Bucketing module 1003 is used to divide the feature vector of each file to be clustered into multiple interface sequence buckets according to the multiple minimum hash values;
[0110] The second calculation module 1005 is used to calculate the distance between each pair of feature vectors of the files to be clustered in the same interface sequence bucket.
[0111] The first judgment and processing module 1007 is used to determine whether the distance between any two feature vectors of files to be clustered in the same interface sequence bucket is less than or equal to a preset distance threshold; when the distance between the feature vectors of two files to be clustered is less than or equal to the preset distance threshold, the two files to be clustered are clustered.
[0112] Alternatively, further, the clustering module includes:
[0113] The third calculation module is used to calculate the pairwise distances between the feature vectors of the file to be clustered.
[0114] The second judgment and processing module is used to determine whether the distance between any two feature vectors of the files to be clustered is greater than or equal to a preset distance threshold; when the distance between the feature vectors of two files to be clustered is less than or equal to the preset distance threshold, the two files to be clustered are clustered.
[0115] Furthermore, the combined module also includes:
[0116] The third determining module is used to determine the first number of application interfaces contained in the interface sequence tuple;
[0117] The fourth determining module is used to extract windows based on the first number of determined application interface windows;
[0118] The fifth determining module is used to extract a first number of adjacent application interfaces from the application interface sequence information of each file to be clustered using the application interface extraction window, to obtain multiple interface sequence tuples, wherein the moving step of the application interface extraction window when extracting the first number of adjacent application interfaces is one application interface.
[0119] Furthermore, the first determining module also includes:
[0120] The mapping module is used to map the multiple interface sequence tuples corresponding to each file to be clustered using a message digest algorithm, so as to obtain multiple feature vectors of the multiple interface sequence tuples.
[0121] Furthermore, the second determining module also includes:
[0122] The conversion module is used to convert the multiple feature vectors corresponding to each file to be clustered into feature vectors of each file using the locality-sensitive hashing algorithm.
[0123] The apparatus and method embodiments described herein are based on the same inventive concept.
[0124] This invention also provides a file clustering device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set are loaded and executed by the processor to implement the file clustering method described above.
[0125] As can be seen from the embodiments of the file clustering method, apparatus, and device provided by the present invention, the solution of the present invention obtains application interface sequence information called during the execution of multiple files to be clustered, the application interface sequence information including multiple application interfaces ordered according to the call sequence; combines the application interface sequence information of each file to be clustered into multiple interface sequence tuples according to the order of the multiple application interfaces corresponding to each file to be clustered; determines multiple feature vectors of the multiple interface sequence tuples corresponding to each file to be clustered; determines the feature vector of each file to be clustered based on the multiple feature vectors corresponding to each file to be clustered; and clusters the multiple files to be clustered using the feature vectors of the multiple files to be clustered. This solves the problem of difficulty or inaccuracy in file clustering using deformation techniques such as packing and obfuscated instructions in the prior art, and improves the accuracy of file clustering. Furthermore, by using bucketing and intra-bucket clustering methods, the amount of data computation is greatly reduced, and the efficiency of data processing is improved.
[0126] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0127] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and apparatus embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0128] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.
[0129] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A document clustering method, characterized in that, The method includes: Obtain application interface sequence information called during the execution of multiple files to be clustered, wherein the application interface sequence information includes multiple application interfaces ordered according to the calling order; Based on the sorting of multiple application interfaces corresponding to each file to be clustered, the application interface sequence information of each file to be clustered is combined into multiple interface sequence tuples, and each interface sequence tuple contains at least two application interfaces. A message digest algorithm is used to generate hash values for each of the multiple interface sequence tuples corresponding to each file to be clustered, thereby obtaining multiple feature vectors for the multiple interface sequence tuples; The locality-sensitive hashing algorithm is used to fuse multiple feature vectors corresponding to each file to be clustered into a locality-sensitive hash value, thereby obtaining the feature vector of each file to be clustered, which is used as the fingerprint of each file to be clustered. Perform a minimum hash calculation on the feature vector of each file to be clustered to obtain multiple minimum hash values; Feature vectors of files to be clustered with the same minimum hash value are placed into the same interface sequence bucket to obtain multiple interface sequence buckets, and the minimum hash value is used as an index. The index is used to retrieve a list of cluster identifiers from the database after obtaining the local sensitive hash value and the minimum hash value of the task file, using the minimum hash value of the task file as the index, and calculating the similarity between the local sensitive hash value of the task file and the cluster identifiers in the list of cluster identifiers. If the similarity meets the matching condition, the task file is classified into the corresponding cluster identifier; if the similarity does not meet the matching condition, a new cluster is created, and the local sensitive hash value of the task file is used as the cluster identifier of the new cluster. Calculate the pairwise distances between the feature vectors of the files to be clustered in the same interface sequence bucket; Determine whether the distance between any two feature vectors of the files to be clustered in the same interface sequence bucket is less than or equal to a preset distance threshold. When the distance between the feature vectors of two files to be clustered is less than or equal to the preset distance threshold, the two files to be clustered are clustered to obtain a cluster, and the local sensitive hash value of the first file in the cluster is used as the cluster identifier.
2. The method according to claim 1, characterized in that, The step of combining the application interface sequence information of each file to be clustered into multiple interface sequence tuples according to the sorting of multiple application interfaces corresponding to each file to be clustered includes: Determine the first number of application interfaces contained in the interface sequence tuple; Based on the first number, extract windows from the determined application interface; The application interface extraction window is used to extract a first number of adjacent application interfaces from the application interface sequence information of each file to be clustered, resulting in multiple interface sequence tuples. The moving step of the application interface extraction window when extracting the first number of adjacent application interfaces is one application interface.
3. The method according to claim 2, characterized in that, The method further includes: Timing information is added to the interface sequence tuple using the n-gram method.
4. A document clustering device, characterized in that, The device includes: The acquisition module is used to acquire application interface sequence information called during the execution of multiple files to be clustered, wherein the application interface sequence information includes multiple application interfaces ordered according to the calling time sequence; The combination module is used to combine the application interface sequence information of each file to be clustered into multiple interface sequence tuples according to the order of the multiple application interfaces corresponding to each file to be clustered. The interface sequence tuple contains at least two application interfaces. The first determining module is used to generate hash values for the plurality of interface sequence tuples corresponding to each file to be clustered using a message digest algorithm, thereby obtaining a plurality of feature vectors for the plurality of interface sequence tuples. The second determining module is used to fuse multiple feature vectors corresponding to each file to be clustered into a single local sensitive hash value using a local sensitive hash algorithm, thereby obtaining the feature vector of each file to be clustered and using it as the fingerprint of each file to be clustered. The clustering module performs minimum hash calculation on the feature vector of each file to be clustered, obtaining multiple minimum hash values. Feature vectors of files to be clustered with the same minimum hash value are placed into the same interface sequence bucket, resulting in multiple interface sequence buckets. The minimum hash value is used as an index. This index is used to retrieve a list of cluster identifiers from the database after obtaining the local sensitive hash value and minimum hash value of the task file. The similarity between the local sensitive hash value of the task file and the cluster identifiers in the cluster identifier list is calculated. If the similarity meets a matching condition, the task file is clustered... Files are classified under their corresponding cluster identifiers; if the similarity does not meet the matching conditions, a new cluster is created, and the local sensitive hash value of the task file is used as the cluster identifier of the new cluster; the distance between each pair of feature vectors of the files to be clustered in the same interface sequence bucket is calculated; it is determined whether the distance between each pair of feature vectors of the files to be clustered in the same interface sequence bucket is less than or equal to a preset distance threshold; when the distance between the feature vectors of two files to be clustered is less than or equal to the preset distance threshold, the two files to be clustered are clustered to obtain a cluster, and the local sensitive hash value of the first file in the cluster is used as the cluster identifier.
5. The apparatus according to claim 4, characterized in that, The combined module includes: The third determining module is used to determine the first number of application interfaces contained in the interface sequence tuple; The fourth determining module is used to extract windows based on the first number of determined application interface windows; The fifth determining module is used to extract a first number of adjacent application interfaces from the application interface sequence information of each file to be clustered using the application interface extraction window, to obtain multiple interface sequence tuples, wherein the moving step of the application interface extraction window when extracting the first number of adjacent application interfaces is one application interface.
6. The apparatus according to claim 5, characterized in that, The fifth determining module is also used for: Timing information is added to the interface sequence tuple using the n-gram method.
7. A document clustering device, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the file clustering method as described in any one of claims 1 to 3.