Computer security using context-triggered segmentation hashing

Abstract

Generally discussed herein are devices, systems, and methods for clustering based on context triggered piecewise hashing (CTPH). A method can include determining a first index of a first CTPH string of a file. The first index can include contiguous bits of the CTPH string. The first index can be less than the CTPH string, such as a proper subset of the CTPH string. The method can include determining that the first index matches a second index of a cluster of files, and associating the file with the cluster in response to determining that the first index matches the second index of the cluster. The method can include determining, based on the cluster, that the file includes malware.

Description

Background Technology

[0001] Similarity analysis in computer security has been used for many years. Context-triggered segmented hashing (CTPH) is one example of a similarity analysis technique. Other techniques use the Levenshtein distance of the CTPH hashes of the data to determine whether the data is similar. However, this analysis is too expensive in terms of computational resources and time. Furthermore, the determination of the clustering percentage (the threshold distance for determining whether to cluster CTPH strings) is arbitrary and provides unpredictable results. More efficient similarity analysis techniques and more deterministic techniques are desired. Summary of the Invention

[0002] This summary section is provided to introduce various aspects of the embodiments in a simplified form, and the following embodiments are further explained in the detailed description. This summary section is not intended to identify necessary or required features of the claimed subject matter, and the combinations and order of elements listed in this summary section are not intended to impose limitations on the elements of the claimed subject matter.

[0003] Systems, methods, apparatuses, and computer or other machine-readable media may provide improvements over existing CTPH string comparison techniques or malware detection techniques. Improvements may include faster processing times and reduced memory overhead in determining whether a CTPH string matches another CTPH string. Improvements may come at least in part from removing the Levinstein distance calculation from CTPH string comparisons. Instead of determining the distance between CTPH strings, embodiments may compare the indices of the CTPH strings and associate the CTPH strings if the indices match. Another advantage of embodiments may include determinism. Previous distance calculations required the user to define a distance below which a CTPH string would be associated. This requirement made the distance calculation dependent on the defined distance and indeterminate. Further advantages and additional details regarding these advantages are discussed elsewhere herein.

[0004] A system may include processing circuitry and memory. The memory may include instructions that, when executed by the processing circuitry, cause the processing circuitry to perform operations for determining that a file contains malware. Operations may include determining a first index of a first context-triggered segment hash (CTPH) string of the file, the first index comprising consecutive bits of the CTPH string. The first index may be a proper subset of the CTPH string. The CTPH string and the proper subset cannot contain an empty set. Operations may include determining that the first index matches a second index of a cluster of files. Operations may include associating the file with a cluster in response to determining that the first index matches the second index of a cluster. Operations may include determining that the file contains malware based on the cluster. Operations may include marking the file as potentially malware, preventing users from opening or deleting the file, and providing reports indicating the file, associated clusters, etc., to personnel handling malware.

[0005] Determining the first index of the first CTPH string may include splitting the first CTPH string into a plurality of disjoint, non-overlapping parts of consecutive, non-overlapping bits. The first index may be determined as a first number of consecutive (e.g., non-overlapping) bits of a portion of the plurality of parts of the first CTPH string. A cluster may include one of a plurality of corresponding clusters in the file.

[0006] The first index may include the most significant byte of the CTPH string. The operation may also include: determining whether a third index corresponding to the least significant byte of the CTPH string matches the second index. The operation may further include: associating the file with a cluster in a plurality of clusters in response to determining that the third index of the CTPH string matches the second index.

[0007] The third index can be determined (only) in response to the first index not matching any of the corresponding second indexes.

[0008] The operation may further include: for each portion, determining a fourth index that includes the second most significant byte of that portion; and for each portion, determining a fifth index that includes the third most significant byte of that portion. The operation may also include: in response to determining that one of the first, fourth, and fifth indices of a portion of the plurality of portions matches the second index, associating the file with the cluster.

[0009] The third index includes the least significant byte of the section and is determined for each section within that section. The operation may also include: for each section, determining a sixth index that includes the second least significant byte of that section. The operation may also include: in response to determining that either the third or sixth index matches the second index, associating the file with the cluster.

[0010] One approach may include performing operations on the system. A machine-readable medium may include instructions that, when executed by a machine, cause the machine to perform operations on the system. Attached Figure Description

[0011] Figure 1 The flowcharts illustrate, by way of example, an embodiment of a method for detecting malware using CTPH clustering and cluster-based indexing.

[0012] Figure 2 A flowchart illustrating an example of generating a CTPH string based on a computer file is provided.

[0013] Figure 3 A flowchart illustrating an example of splitting a CTPH string into L parts and determining the index of one of the L parts is provided.

[0014] Figure 4 A diagram illustrating an example of the clustering operation of the CTPH clustering method is provided.

[0015] Figure 5 The diagram illustrates the cluster count versus cluster size for the 256-bit Secure Hash Algorithm (SHA256) as an example.

[0016] Figure 6 A graph illustrating the cluster count versus cluster size for the CTPH clustering method of an embodiment is shown as an example.

[0017] Figure 7 A diagram illustrating an embodiment of a malware detection device is provided as an example.

[0018] Figure 8 An example diagram illustrates an embodiment of a method for malware detection using CTPH strings.

[0019] Figure 9 A block diagram illustrating an embodiment of a machine (e.g., a computer system) for implementing one or more embodiments is shown by way of example. Detailed Implementation

[0020] In the following description, reference is made to the accompanying drawings, which form part of the invention and are illustrated by way of illustration, showing specific embodiments that can be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice them. It should be understood that other embodiments may be used, and structural, logical, and / or electrical changes may be made without departing from the scope of the embodiments. Therefore, the following description of the embodiments is not intended to be limiting, and the scope of the embodiments is defined by the appended claims.

[0021] In some embodiments, the operations, functions, or techniques described herein may be implemented in software. Software may include computer-executable instructions stored on a computer or other machine-readable medium or storage device, such as one or more non-transitory memories (e.g., non-transitory machine-readable media) or other types of hardware-based storage devices (local or networked). Furthermore, such functionality may correspond to a subsystem, which may be software, hardware, firmware, or a combination thereof. Multiple functions may be performed as needed in one or more subsystems, and the described embodiments are merely examples. Software may be executed on a digital signal processor, application-specific integrated circuit device (ASIC), microprocessor, central processing unit (CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), or other type of processor that operates on a computer system (such as a personal computer, server, or other computer system) and transforms such a computer system into a specially programmed machine. Functions or algorithms can be implemented using processing circuitry systems, such as electrical and / or electronic components (e.g., one or more transistors, resistors, capacitors, inductors, amplifiers, modulators, demodulators, antennas, radios, regulators, diodes, oscillators, multiplexers, logic gates, buffers, caches, memories, GPUs, CPUs, FPGAs, ASICs, etc.).

[0022] Context-triggered segmented hashing (CTPH) has been used for similarity analysis in the information security industry for over 10 years and has become one of the most popular and standard fuzzy hashing algorithms. CTPH has been used for sample clustering analysis and malware classification. However, using current CTPH clustering techniques, finding groups of samples with a certain degree of similarity is too expensive. To find similar samples in a reduced time, this embodiment provides a simplified CTPH similarity analysis solution to help reduce costs while still maintaining clustering accuracy.

[0023] Fuzzy hashing allows investigators to focus on potential criminal documents that do not appear using other hashing methods. The use of fuzzy hashing is similar to fuzzy logic search. Fuzzy hashing can find similar but not identical documents (sometimes referred to as "same-origin" files).

[0024] Fuzzy hashing involves splitting a file into blocks, hashing each block, and combining all the block hashes into a final hash value. CTPH is a fuzzy hashing technique with variable block sizes. It was originally proposed by Jesse Kornblum in 2006. The output of a CTPH hash is a string (“CTPH string”). CTPH clustering analysis is the task of grouping a set of samples into clusters using CTPH strings, such that samples in the same cluster are more similar to each other than samples in other clusters.

[0025] As discussed earlier in the background section, existing CTPH clustering techniques rely on a similarity metric threshold (Lewinstein distance threshold) to determine whether to associate a CTPH string with a cluster. However, the choice of threshold is arbitrary and provides nondeterministic results. This is illustrated in Table 1:

[0026] 24:hMC5oci9ez+FRbj1NVlPlb8FC3lbmCGi7bCCLLYCy0k04n3… 24:hMC5oci9ez+FRbj1NVlPlb8FC3lbmCGi7bCCLLYCy0k04n3… 12:GXwnacE61wW7N8auUa7moij1muD8Ztl75olbzUOlXBDeZs… 12: GXwnacE629W7N8aC96Al8iJ1muD8X+tl75olbzUOlXBDeZs… 12: GXwnacE671W7N8aG2oAkiJ1muD8X+tl75olbzUOlXBDeZs… 12:GXwnacE6AW7N8aIl6238af7wiJ1muD8X+tl75olbzUOlXBD… 12:GXwnacE6gW7N8aHf0oQOiiJ1muD8X+tl75olbzUOlXBDeZ… 12:GXwnacE6MW7N8an0d7oiJ1muD8X+tl75olbzUOlXBDeZs…

[0027] Table 1: Example CTPH strings

[0028] Now consider an example where the similarity threshold is set to 90%. The strings in the first two rows of Table 1 will be associated with the first cluster, while the remaining six CTPH strings will remain unclustered. Next, consider an example where the similarity threshold is set to 80%. In this example, the strings in the first two rows of Table 1 will be associated with the first cluster, the CTPH strings in the third, fifth, and seventh rows will be associated with the second cluster, and the remaining rows will be associated with the third cluster. Finally, consider another example where the similarity threshold is set to 50%. In this example, the strings in the first two rows of Table 1 will be associated with the first cluster, and the remaining rows of Table 1 will be associated with another cluster.

[0029] The nondeterministic nature of this clustering technique makes its results problematic at best and only occasionally useful. Furthermore, determining the Levenstein distance between strings is computationally expensive. Consider n as the total number of CTPH strings, p as the average number of CTPH strings per cluster, and m as the number of clusters. The computational complexity using the Levenstein distance technique is O((n*m+n) / 2). Moreover, the time required for clustering using this technique is not linear but grows exponentially with the total number of CTPH strings. A deterministic clustering technique that saves computational time and resources is desired.

[0030] Figure 1 A flowchart illustrating an embodiment of a method 100 for detecting malware using CTPH clustering and cluster-based indexing is provided as an example. The illustrated method 100 includes generating a CTPH string for a file at operation 110. Operation 110 is optional. In some cases, an entity may have access to CTPH strings for at least some files in the file for processing using method 100. In other cases, an entity may generate corresponding CTPH strings for one or more files in its file for processing by method 100. Further details regarding operation 110 are provided in [link to documentation]. Figure 2 It is provided in the description and its corresponding description.

[0031] At operation 120, the CTPH string is split into L parts, where L is a positive integer greater than one (1). These parts may have the same or similar size. In some embodiments, each of the L parts includes a minimum number of bytes. The minimum number of bytes may be a positive integer greater than one (1). In some embodiments, the minimum number of bytes may be seventeen (17). More details about operation 120 are available in... Figure 3 It is provided in the description and its corresponding description.

[0032] At operation 130, an index can be determined for a portion of the L portions generated at operation 120. The index is an integer value corresponding to a subset of the corresponding portion among the L portions. For example, the index may correspond to a value represented by a consecutive number of bits of that portion. In some embodiments, the index may be seven consecutive bytes, or more or fewer bytes. Further details regarding operation 130 are provided in... Figure 3 It is provided in the description and its corresponding description.

[0033] At operation 140, the generated CTPH string can be clustered based on the index determined at operation 130. The determined index of each of the L parts can be compared with one or more indices of each cluster. In response to a determined index matching any index in the cluster's indices, the CTPH string can be added to that cluster. In response to a determined index not matching any index in the cluster's indices, a new cluster can be generated using the CTPH string. More details about operation 140 are available in... Figure 4 It is provided in the description and its corresponding description.

[0034] At operation 150, the cluster(s) to which the CTPH string is mapped at operation 140 can be used to determine whether a file from which the CTPH string is generated contains malware. Operation 150 may include classifying the cluster(s) using artificial intelligence (AI), machine learning (ML), heuristics, or other techniques. Prior knowledge of which files contain malware may be used to label each cluster as containing malware. In some embodiments, the label may include an indication of the percentage of files in the cluster known to contain malware. Heuristics may include indicating that a file contains malware in response to determining that the cluster to which a file is mapped contains more than a threshold percentage of files known to contain malware mapped thereto.

[0035] Figure 2 A flowchart illustrating an example of generating a CTPH string based on computer file 202 is provided. Figure 2 Provided information about Figure 1More details on operation 110. Many different types of computer files exist. Files can be designed to store images, written information, video, audio, text, vector graphics, computer programs, or various other types of data. Some types of files can store multiple types of information simultaneously. Using computer programs, users can open, read, change, save, and close computer files. Computer files can be reopened, modified, and copied. File 202 may include binary data or data that can be converted to binary form. In some embodiments, file 202 may include a block hash of the original file. The block hash can accept variable-size input and produce a fixed-length string (hash).

[0036] File 202 can be split into blocks 204A, 204B, 204C, etc. Each block in blocks 204A-204C may include a disjoint subset of file 202 or a portion thereof, such as metadata. Blocks 204A, 204B, and 204C may have the same or different sizes. Blocks 204A-204C can be scaled such that file 202 is split into Q blocks, where Q is a positive integer greater than one. Therefore, the size of each block can be Z / Q rounded up to the nearest integer, where Z is the file size in bytes, bits, etc.

[0037] At operations 206A, 206B, and 206C, checksums can be performed on each of blocks 204A-204C. Operations 206A-206C use checksum functions to determine the integer values ​​for blocks 204A-204C. Checksums are commonly used to verify data integrity. For small changes to the input, some checksum functions output significantly different values. This is especially true for cryptographic hash functions, which can be used to detect data corruption errors and verify overall data integrity. If the calculated checksum of the current data input matches the stored value of a previously calculated checksum, the probability that the data has not been accidentally altered or corrupted is very high. Parity bits and check bits are special cases of checksums, suitable for small data blocks. Some error-correcting codes are based on special checksums that not only detect common errors but also allow recovery of the original data in certain cases.

[0038] At operations 208A, 208B, and 208C, the results of operations 206A-206C can be encoded respectively. Encoding in CTPH typically uses Base64 character encoding. Base64 is a group of binary-to-text encoding schemes that represent binary data in ASCII string format by converting binary data into a radix-64 representation. Each Base64 number represents 6 bits of data. Therefore, three 8-bit bytes (24 bits in total) can be represented by four 6-bit Base64 numbers. Base64 is just one example, and other bases such as Base32, Base128, Base256, or others are possible.

[0039] To generate CTPH string 220, all encoded checksum blocks 210A, 210B, and 210C from operations 208A-208C can be concatenated to form CTPH string 220. The order of the encoded checksum blocks 210A-210C in CTPH string 220 can be relative to the positions of their corresponding blocks 204A-204C.

[0040] Figure 3 A flowchart illustrating an embodiment of splitting a CTPH string into L parts 330A, 330B, 330C, 330D, and 330E, and determining indices 350A, 350B, 350C, and 350D for each of the L parts 330A-330E, is provided as an example. The L parts can be subsets of bytes (e.g., non-overlapping subsets, consecutive subsets, overlapping subsets, or combinations thereof). Index 350 can be a specified number of (e.g., consecutive) bytes from a given portion of the L parts. Figure 3 Provided information about Figure 1 More details on operating 120 and 130.

[0041] exist Figure 3 In the text, the CTPH string 220 is split into L parts 330A-330E, each N bytes in size. L and N are both positive integers greater than one (1). Figure 3 In the example, L is 5. N can have a minimum value, such as 7 bytes, 11 bytes, 17 bytes, 21 bytes, a larger or smaller number of bytes, or a number in between. The minimum value of N can affect L, the number of parts into which the CTPH string is divided. For example, for a CTPH string 220 with only fifty (50) bytes and N with seventeen (17) bytes, L can be three. In some cases, parts 330A-330E can be padded (with zeros) so that each part in parts 330A-30E has the same size.

[0042] At operation 130, byte subsets 340A, 340B, 340C, and 340D (e.g., consecutive byte subsets) of each part of portions 330A-330E can be determined. The number of byte subsets 340A-340D can be determined based on the size of N and the number of bytes in each part of portions 330A-330E. Subset 340A is the most significant byte of portion 330A (in big-endian notation), and subset 340D is the least significant byte of portion 330A. In the illustrated embodiment, seven (7) consecutive subsets 340A, 340B, 340C, and 340D are used. Subsets 340A-340D can be consecutive (as shown), overlapping, non-overlapping, or a combination thereof. However, subsets of more or fewer bytes can be used. The number of consecutive subsets 340A-340D can be referred to as M, where M is a positive integer greater than zero. In some embodiments, M may be a subset of five (5), six (6), seven (7) or more or fewer 340A-340D.

[0043] Consider the portion including bytes [1, 35]. The most significant byte includes bytes [1, 7], the second most significant byte includes bytes [2, 8], the third most significant byte includes bytes [3, 9], and so on. The least significant byte includes bytes [29, 35], the second least significant byte includes bytes [28, 34], the third most significant byte includes bytes [27, 33], and so on.

[0044] Each subset of 340A-340D can be used as an index of 350A, 350B, 350C, 350D for operation 140. In the example where the contiguous subsets are seven (7) bytes long, the probability of a collision (one index equals another) is 1 / 64. 7 ~ = 1 / 4398 billion. This is extremely small, meaning that if a collision occurs, the corresponding file associated with the index is likely to be a related binary sequence or contain the same binary sequence. Therefore, indices 350A-350D can be grouped equally to cluster related files 202. If the block hashes match, the output Base64 characters also match. Therefore, if several consecutive block hashes match, the corresponding CTPH substrings match. The probability of finding a block hash match increases with the length of the matching CTPH substring, and the similarity between files increases with the length of the matching CTPH substring. Therefore, CTPH clustering can be used to find similar files.

[0045] exist Figure 3In the example, contiguous subsets 340A and 340B come from the beginning of portion 330A, while contiguous subset 340B comes from the end of portion 330A. In some embodiments, contiguous subsets 340A and 340B from the beginning of portion 330A can be used to determine indices 350A and 350B. Then, indices 350A and 350B can be used in operation 140 before contiguous subsets 340C and 340D from the end of portion 330A are used to determine indices 350C and 350D. In response to no matching clusters 440A, 440B, 440C, and 440D being found, see [link to relevant documentation]. Figure 4 The contiguous subsets 340C and 340D from the end of portion 330A can be used to determine indices 350C and 350D, and compared with clusters 440A-440D. In some embodiments, the contiguous subsets from the beginning of each of portions 330A-330E can be used to determine an index for each portion, and all determined indices can be used in operation 140 before the contiguous subsets from the end of portions 330A-330E are used to determine the index. Then, in response to no matching clusters 440A-440D being found, the contiguous subsets from the end of portions 330A-330E can be used to determine an index, and the index can be compared with clusters 440A-440D.

[0046] Figure 4 A diagram illustrating an example of a clustering operation is provided. Figure 4 Provided information about Figure 1 More details on Operation 140. Operation 140 involves comparing indices 350A-350D with the indices of clusters 440A, 440B, 440C, and 440D. The indices of clusters 440A-440D can be defined by the following: the first file placed in clusters 440A-440D, a unique index of the first file placed in the cluster (an index that is not currently in any other cluster), a single index of a file in the cluster that started the cluster, or all indexes of all files placed in clusters 440A-440D, etc.

[0047] Figure 4 The diagram illustrates P clusters, where P is a positive integer greater than one. However, more or fewer clusters can be used. Further discussion on the number of clusters used is available on [link to diagram]. Figure 5 and Figure 6 Provided. Operation 140 may include: comparing each index 350A-350D with at least one index from each cluster 440A-440D. Figure 4In the example, each cluster has L*M indexes because there are M indexes and L parts for each file. However, as mentioned earlier, more or fewer indexes may be associated with each cluster 440A-440D. In response to the determined index appearing in cluster 440A-440D, file 202 may be associated with that cluster 440A-440D. In response to the determination that file 202 includes index 350A-350D common to multiple clusters 440A-440D, file 202 may be associated with each of the multiple clusters 440A-440D. Figure 4 In the example, indices 350A, 350C, and 350D are mapped to clusters 440A, 440B, and 440C, while index 350B is not mapped to any cluster.

[0048] In some embodiments, indices 350A-350D that are not currently in clusters 440A-440D may be added to clusters 440A-440D for future cluster testing. In some other embodiments, indices 350A-350D that are not currently in clusters 440A-440D are not included in future cluster testing. In some embodiments, in response to determining that an index such as index 350B is not mapped to clusters 440A-440D, index 350B may form the basis of a new cluster 440A-440D.

[0049] In response to the determination that indices 350A-350D of CTPH string 220 do not match any of the clusters 440A-440D, CTPH string 220 can be used as the basis for a new cluster (if more clusters are allowed). In some embodiments, a comparison between indices 350C and 350D and indices 350A and 350B can be performed to help identify binary duplicates in file 202.

[0050] The computational complexity of clustering operation 140 is only O(n), providing a significant improvement over existing Leavenstein distance clustering techniques. Furthermore, in this embodiment, whether to cluster CTPH strings is deterministic (the CTPH string (or file) is associated with a cluster in response to determining that the index of the CTPH string matches the index of the cluster). Additionally, the CTPH string clustering in this embodiment can detect similarities in files that cannot be detected by block hash clustering techniques (e.g., SHA256 hashes).

[0051] Consider the following SHA256 representation of the file:

[0052] 1: 8bfb9c2c9d11c7080a72db22d8fe43c5c9508899da1f16ac55b0e9af276ab60c

[0053] 2:09bdb7514e4cce9760f23c6302623895def5d78e7265e105d5526ce1f0d75f3f

[0054] 3: 49808e3beb9124702e22b333e3a11c6bde95bce659bf003d15fb733019373b6e

[0055] 4: 82c3e16c825ce84981696b8615ee71b16c701c77ae63b825769ef25120c77585

[0056] 5: abb011d36eccd9493fb37724bdd939496de5a2ac7593ab01346fb2b9b2d3d7cc

[0057] 6:24d16f95b80f247b6aa390da0d027343bcd24e7b119aa82847b1e67bbad85a0c

[0058] 7:1e75ff5e4d62db3b2e41ac5f0336b0cldccae3c141645bdce7bcb79785dacca8

[0059] These SHA256 representations can have the following CTPH string representations:

[0060] 1:e7OEqtoBMbJsv6tWKFdu9C5OL34qtEUhmkkYVE

[0061] 2:e7OEqtoBMbJsv6tWKFdu9C5OL3aQPxZoJqkJ5

[0062] 3:e7OEqtoBMbJsv6tWKFdu9C5OL3EqGy5UDM

[0063] 4:e7OEqtoBMbJsv6tWKFdu9C5OL3fZZnpzjYj6

[0064] 5:e7OEqtoBMbJsv6tWKFdu9C5OL3jQPxZoJqkJ5

[0065] 6:e7OEqtoBMbJsv6tWKFdu9C5OL3UQPxZoJqkJ5

[0066] 7:e7OEqtoBMbJsv6tWKFdu9C5OL3Yc+ZiHh

[0067] For each CTPH string, the string length (e.g., the number of bits or bytes) can be determined. The index length can be predetermined or determined based on the number of indices to be compared. The offset for determining the next CTPH string can be determined. The offset can be greater than, equal to, or less than the length of the CTPH string. Assuming, for this example, the index length is 7 bytes. For the CTPH string of the first file "1", the index number (rounded down) could be 5, as follows:

[0068] Index 1: e7OEqto

[0069] Index 2: BMbJsv6

[0070] Index 3: tWKFdu9

[0071] Index 4: C5OL34q,

[0072] Index 5: tEUhmkk

[0073] The same process can be used to determine the index for each CTPH string. For the CTPH string in file 7, the offset could be 6, and the index length could be 7. The configured index could be:

[0074] Index 1: e7Oeqto,

[0075] Index 2: oBMbJsv

[0076] Index 3: v6tWKFd

[0077] Index 4: du9C5OL

[0078] Index 5: L3Yc+Zi

[0079] After determining the index at the initial position, strings that include the equality index 1 can be clustered, and index 1 can be the representative index of the cluster. Similarly, strings that include the equality index 2 can be clustered, and index 2 can be the representative index of the cluster, and so on, until the last index.

[0080] Based on the clustering at that initial location, all files 1-7 can be clustered together. This may not be the case in distance calculations using Levenstein distance. For example, files 1 and 7 have 77% similarity (distance), which is unlikely to be clustered because most techniques require a similarity greater than 80%.

[0081] For files that were not clustered in the first round of clustering, a next round of clustering can be performed. Consider the following CTPH string:

[0082] 8:Pf7OEqtoBMbJsv6tWKFdu9C5OL34qtEUhmkkYVE

[0083] 9:xf7OEqtoBMbJsv6tWKFdu9C5OL3Yc+ZiHh

[0084] The first round of clustering will not cluster these files together, even if they are very similar (the second through 27th bytes of the CTPH string are equal). In the second round of clustering (or another subsequent round of clustering), the first byte of the CTPH string can be removed. The indices of the resulting shorter CTPH strings can then be determined, compared, and clustered (as discussed in the first round).

[0085] If the file is still not clustered after a second round of clustering (or another subsequent round), the index from the second round (or another subsequent round) can be compared with the index from one or more previous rounds. Consider the following CTPH string:

[0086] 10:e7OEqtoBMbJsv6tWKFdu9C5OL34qtEUhmkkYVE

[0087] 11:xe7OEqtoBMbJsv6tWKFdu9C5OL3Yc+ZiHh

[0088] These two files, 10 and 11, contain the same CTPH string of 26 bytes, but will not be clustered based on the previous rounds described. In the third round described, these two files can be clustered. Note that in the third or subsequent rounds, index 1 from round 1 can be compared to index 1 from round 2, index 2 from round 1 can be compared to index 2 from round 2, and so on.

[0089] Further rounds of bit removal, removing more than two bits from the beginning of the CTPH string, determine the index, and then compare it with the indexes from the same or different rounds, performing clustering accordingly. Note that so far, this clustering involves comparisons based on the most significant byte. The same process can be performed on the least significant byte.

[0090] For files 1-7, index 5, which is determined to be the least significant byte, can be as follows:

[0091] File 1: hmkkYVE

[0092] File 2: ZoJqkJ5

[0093] File 3: qGy5UDM

[0094] File 4: npzjYj6

[0095] File 5: ZoJqkJ5

[0096] Document 6: ZoJqkJ5

[0097] File 7: Yc+ZiHh

[0098] These indexes can then be compared and clustered (if any indexes are equal). If the file is still not clustered, the least significant byte can be removed and clustering can be performed again (least significant byte alignment). Similar to the most significant byte alignment round, indexes from the least significant byte removal round can be compared with indexes from the same or previous rounds, and if the indexes are equal, they can be clustered.

[0099] Typically, the computational complexity of distance editing (an exponential function) is compared to that of the indexed comparison technique (a linear function), which significantly reduces CPU costs. Many rounds can be optimized; the basic solution is to split the string and compare substrings over all samples, rather than calculating the distance for each pair.

[0100] Figure 5 The diagram illustrates the cluster count versus cluster size for the 256-bit Secure Hash Algorithm (SHA256) as an example. As can be seen, the cluster size (in terms of the number of CTPH strings 220 (or associated files 202)) decreases as the cluster count increases.

[0101] Figure 6 A graph illustrating cluster count versus cluster size for method 100 is provided as an example. As can be seen, the cluster size decreases as the cluster count increases. However, compared to the SHA256 clustering technique, method 100 results in a more gradual increase in cluster size. Although not illustrated, it corresponds to... Figure 5 and Figure 6 The percentage of unclustered data remained roughly the same in both techniques of plotting (approximately 14%).

[0102] Figure 7A diagram illustrating an embodiment of a malware detection device 772 is provided by way of example. The malware detection device 772 may receive cluster identifiers 770 as input and generate a classification 776 using classification circuitry 774. Classification 776 indicates whether a file 202 corresponding to the cluster identifiers 770 corresponds to malware (the probability of it corresponding to malware). The cluster identifiers 770 indicate the clusters 440A-440D to which the CTPH string 220 of file 202 is mapped (e.g., at operation 140). The cluster identifiers 770 may be associated with an indication regarding whether a cluster is known to contain files with malware. In some embodiments, the percentage of known malware-containing files in clusters 440A-440D may be included in the cluster identifiers 770.

[0103] The classification circuitry 774 includes one or more electrical or electronic components configured to determine a classification 776 based on cluster identifiers 770. The classification circuitry 774 may implement heuristic-based techniques or be trained to implement ML- or AI-based techniques for determining classifications. Heuristic-based techniques define rules for providing classification 776 based on cluster identifiers 770. For example, in clusters corresponding to cluster identifiers 770, in response to a mathematical combination determining that a percentage of files identified as containing malware is greater than or equal to a threshold, classification 776 may indicate that a file contains malware or otherwise warrants further consideration. The threshold may be 25%, 30%, 50%, 75%, some higher or lower percentage, or some percentage in between.

[0104] The ML or AI-based technique for determining category 776 may include using cluster identifier(s) 770, and corresponding labels (malware, percentage of files containing malware, not malware, etc.) may be used to train classification circuit device 774 to determine category 776. Then, classification circuit device 774 may acquire a new set of cluster identifier(s) 770 and determine category 776 based on the training.

[0105] The electrical or electronic components of the circuit device 774 or other circuit devices may include one or more transistors, resistors, capacitors, inductors, amplifiers, modulators, demodulators, antennas, radios, regulators, diodes, oscillators, multiplexers, logic gates (e.g., AND, OR, XOR, No, etc.), buffers, caches, memories, GPUs, CPUs, FPGAs, ASICs, etc.

[0106] Figure 8A flowchart illustrating a method 800 for identifying malicious files using an index of a file's CTPH string is provided as an example. The illustrated method 800 includes: at operation 810, determining (e.g., index 350A) a first index of a first CTPH string 220 of file 202 (e.g., index 350A) via a processing circuitry of a computing device (e.g., processing unit 902 of device 900); at operation 820, determining that the first index matches a second index of a cluster of files (e.g., clusters 440A-440D); at operation 830, associating the file with a cluster in response to determining that the first index matches the second index of the cluster; and at operation 840, determining, based on the cluster, that the file includes malware (e.g., using a classification circuitry 774 of a malware detection device 772 that is the same as or different from the device including the processing circuitry). The first index may include consecutive bits of the CTPH string. The first index may be smaller than the CTPH string, such as being a proper subset of the CTPH string. A proper subset of set A is a subset of A that is not equal to A. In other words, if B is a proper subset of A, then all elements of B are in A, but A contains at least one element that is not in B.

[0107] Operation 810 may include splitting the first CTPH string into multiple disjoint portions of consecutive, non-overlapping bits (e.g., portions 330A-330E). A first index may be determined as a first number of consecutive, non-overlapping bits of one of the portions of the first CTPH string. The cluster may be one of multiple corresponding clusters in the file.

[0108] The first index may include the most significant byte of the CTPH string. Method 800 may further include: determining whether a third index corresponding to the least significant byte of the CTPH string matches the second index. Method 800 may further include: associating the file with a cluster in a plurality of clusters in response to determining that the third index of the CTPH string matches the second index. The third index may be determined in response to the first index not matching any of the corresponding second indices.

[0109] The first index may include the most significant byte of the portion. The first index may be determined for each portion in the portion. Method 800 may further include: for each portion in the portion, determining a fourth index that includes the second most significant byte of that portion. Method 800 may further include: for each portion in the portion, determining a fifth index that includes the third most significant byte of that portion. Method 800 may further include: in response to determining that one of the first, fourth, and fifth indices of a portion of the plurality of portions matches a second index, associating the file with the cluster.

[0110] The third index may include the least significant byte of the portion. The third index may be determined for each portion within the portion. Method 800 may further include: for each portion within the portion, determining a sixth index that includes the second least significant byte of that portion. Method 800 may further include: in response to determining that one of the third or sixth indices matches the second index, associating the file with the cluster.

[0111] Figure 9 A block diagram illustrating an embodiment of a machine 900 (e.g., a computer system) for implementing one or more embodiments is shown by way of example. Machine 900 may implement techniques for CTPH string clustering or malware detection, such as method 100 or 800. Malware detection device 772 or classification circuit device 774 may include one or more components of machine 900. An example machine 900 (in the form of a computer) may include a processing unit 902, memory 903, removable storage device 910, and non-removable storage device 912. Although the example computing device is illustrated and described as machine 900, the computing device may take different forms in different embodiments. For example, instead, the computing device may be a smartphone, tablet computer, smartwatch, or include features related to… Figure 9 Other computing devices with the same or similar elements as those illustrated and described. Devices such as smartphones, tablets, and smartwatches are generally referred to collectively as mobile devices. Furthermore, although various data storage elements are illustrated as part of machine 900, storage devices may also or alternatively include cloud-based storage devices accessible via a network (such as the Internet).

[0112] Memory 903 may include volatile memory 914 and non-volatile memory 908. Machine 900 may include or access a computing environment that includes various computer-readable media, such as volatile memory 914 and non-volatile memory 908, removable storage device 910, and non-removable storage device 912. Computer storage devices include random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices, or other magnetic storage devices capable of storing computer-readable instructions for execution to perform the functions described herein.

[0113] Machine 900 may include or access a computing environment including input 906, output 904, and communication connection 916. Output 904 may include a display device, such as a touchscreen, which can also be used as an input device. Input 906 may include one or more of the following: a touchscreen, touchpad, mouse, keyboard, camera, one or more device-specific buttons, one or more sensors integrated into machine 900 or coupled to machine 900 via a wired or wireless data connection, and other input devices. The computer may use the communication connection to operate in a networked environment to connect to one or more remote computers, such as database servers, including cloud-based servers and storage devices. Remote computers may include personal computers (PCs), servers, routers, network PCs, peer-to-peer devices, or other public network nodes. Communication connection may include a local area network (LAN), a wide area network (WAN), a cellular network, IEEE 802.11 (Wi-Fi), Bluetooth, or other networks.

[0114] Computer-readable instructions stored on a computer-readable storage device are executable by a processing unit 902 (sometimes referred to as a processing circuit device) of machine 900. Hard disk drives, CD-ROMs, and RAM are some examples of articles that include non-transitory computer-readable media such as storage devices. For example, a computer program 918 can be used to cause the processing unit 902 to perform one or more methods or algorithms described herein.

[0115] Additional notes and examples:

[0116] Example 1 may include a system comprising processing circuitry and memory including instructions stored thereon that, when executed by the processing circuitry, cause the processing circuitry to perform operations for determining that a file contains malware, the operations including: determining a first index of a first context-triggered segment hash (CTPH) string of the file, the first index comprising consecutive bits of the CTPH string, the first index being less than the CTPH string; determining that the first index matches a second index of a cluster of files; in response to determining that the first index matches the second index of the cluster, associating the file with the cluster; and determining that the file contains malware based on the cluster.

[0117] In Example 2, Example 1 may further include: wherein determining the first index of the first CTPH string includes splitting the first CTPH string into a plurality of disjoint, non-overlapping parts.

[0118] In Example 3, Example 2 may further include: wherein the first index is determined as a first number of consecutive bits of a portion of a plurality of portions of the first CTPH string.

[0119] In Example 4, Example 3 may also include: where the cluster is one of a plurality of clusters of the file.

[0120] In Example 5, Example 4 may further include: wherein the first index includes the most significant byte of the CTPH string, and the operation further includes: determining that a third index of the CTPH string corresponding to the least significant byte of the CTPH string matches the second index; and in response to determining that the third index of the CTPH string matches the second index, associating the file with a cluster in a plurality of clusters.

[0121] In Example 6, Example 5 may also include: wherein the third index is determined in response to the first index not matching any of the second indices in the corresponding second index.

[0122] In Example 7, at least one of Examples 4-6 may further include: wherein the first index includes the most significant byte of the portion, and is determined for each portion in the portion, and the operation further includes: for each portion in the portion, determining a fourth index including the second most significant byte of the portion; for each portion in the portion, determining a fifth index including the third most significant byte of the portion; and in response to determining that one of the first, fourth, and fifth indices of the portion in the plurality of portions matches the second index, associating the file with the cluster.

[0123] In Example 8, at least one of Examples 6-7 may further include: wherein the third index includes the least significant byte of the portion and is determined for each portion in the portion, and the operation further includes: for each portion in the portion, determining a sixth index that includes the second least significant byte of the portion; and in response to determining that one of the third index or the sixth index matches the second index, associating the file with the cluster.

[0124] Example 9 includes a method for determining that a file contains malware using context-triggered segmented hash (CTPH) clustering, the method comprising: determining, by a processing circuitry of a computing device, a first index of a first CTPH string of the file, the first index comprising consecutive bits of the CTPH string, the first index being less than the CTPH string; determining that the first index matches a second index of a cluster of files; associating the file with the cluster in response to determining that the first index matches the second index of the cluster; and determining that the file contains malware based on the cluster.

[0125] In Example 10, Example 9 may further include: wherein determining the first index of the first CTPH string includes splitting the first CTPH string into a plurality of disjoint parts of consecutive, non-overlapping positions.

[0126] In Example 11, Example 10 may further include: wherein the first index is determined as a first number of consecutive bits of a portion of a plurality of portions of the first CTPH string.

[0127] In Example 12, at least one of Examples 9-11 may also include: wherein the cluster is one of a plurality of clusters of the file.

[0128] In Example 13, at least one of Examples 9-12 may further include: wherein the first index includes the most significant byte of the CTPH string, and the method further includes: determining that a third index of the CTPH string corresponding to the least significant byte of the CTPH string matches the second index; and in response to determining that the third index of the CTPH string matches the second index, associating the file with a cluster in a plurality of clusters.

[0129] In Example 14, Example 13 may also include: wherein the third index is determined in response to the first index not matching any of the second indices in the corresponding second index.

[0130] In Example 15, at least one of Examples 12-14 may further include: wherein the first index includes the most significant byte of the portion and is determined for each portion, and the method further includes: for each portion, determining a fourth index including a second most significant byte of the portion; for each portion, determining a fifth index including a third most significant byte of the portion; and in response to determining that one of the first, fourth, and fifth indices of the portions matches the second index, associating the file with the cluster.

[0131] In Example 16, at least one of Examples 13-15 may further include: wherein the third index includes the least significant byte of the portion and is determined for each portion in the portion, and the method further includes: for each portion in the portion, determining a sixth index that includes the second least significant byte of the portion; and in response to determining that one of the third index or the sixth index matches the second index, associating the file with the cluster.

[0132] Example 17 includes a non-transitory machine-readable medium comprising instructions that, when executed by a machine, cause the machine to perform operations for determining that a file contains malware using a context-triggered segmented hash (CTPH), the operations including: determining, by processing circuitry of a computing device, a first index of a first CTPH string of the file, the first index comprising consecutive bits of the CTPH string, the first index being less than the CTPH string; determining that the first index matches a second index of a cluster of the file; associating the file with the cluster in response to determining that the first index matches the second index of the cluster; and determining, based on the cluster, that the file contains malware.

[0133] In Example 18, Example 17 may further include: wherein determining the first index of the first CTPH string includes splitting the first CTPH string into a plurality of disjoint, non-overlapping parts.

[0134] In Example 19, at least one of Examples 17-18 may further include: wherein the first index is determined as a first number of consecutive bits of a portion of a plurality of portions of the first CTPH string.

[0135] In Example 20, at least one of Examples 17-19 may also include: wherein the cluster is one of a plurality of corresponding clusters of the file.

[0136] In Example 21, Example 20 may further include: wherein the first index includes the most significant byte of the CTPH string, and the operation further includes: determining that a third index of the CTPH string corresponding to the least significant byte of the CTPH string matches the second index; and in response to determining that the third index of the CTPH string matches the second index, associating the file with a cluster in a plurality of clusters.

[0137] In Example 22, Example 21 may also include: wherein the third index is determined in response to the first index not matching any of the second indices in the corresponding second index.

[0138] In Example 23, at least one of Examples 21-22 may further include: wherein the first index includes the most significant byte of the portion and is determined for each portion, and the operation further includes: for each portion, determining a fourth index including a second most significant byte of the portion; for each portion, determining a fifth index including a third most significant byte of the portion; and in response to determining that one of the first, fourth, and fifth indices of the portions matches the second index, associating the file with the cluster.

[0139] In Example 24, at least one of Examples 21-23 may further include: wherein the third index includes the least significant byte of the portion and is determined for each portion in the portion, and the operation further includes: for each portion in the portion, determining a sixth index that includes the second least significant byte of the portion; and in response to determining that one of the third index or the sixth index matches the second index, associating the file with the cluster.

[0140] Some embodiments may be further understood with reference to computer software code configured as part of implementing the methods in discussion (e.g., methods 100, 800, or combinations thereof). For ease of reference, such computer software code is provided herein:

[0141]

[0142]

[0143]

[0144]

[0145]

[0146]

[0147]

[0148]

[0149]

[0150]

[0151]

[0152]

[0153]

[0154]

[0155]

[0156]

[0157]

[0158]

[0159]

[0160]

[0161]

[0162]

[0163]

[0164]

[0165]

[0166]

[0167]

[0168]

[0169]

[0170]

[0171]

[0172]

[0173]

[0174]

[0175]

[0176]

[0177]

[0178]

[0179]

[0180]

[0181]

[0182]

[0183]

[0184]

[0185]

[0186]

[0187]

[0188]

[0189]

[0190]

[0191]

[0192]

[0193]

[0194]

[0195] Although several embodiments have been described in detail above, other modifications are possible. For example, the logical flow depicted in the figures does not require the shown order or sequential order to achieve the desired result. Other steps may be provided or excluded from the described flow, and other components may be added to or removed from the described system. Other embodiments are within the scope of the following claims.

Claims

1. A system comprising: Processing circuit device; A memory, including instructions stored thereon, which, when executed by the processing circuitry, cause the processing circuitry to perform operations for determining that a file contains malware, the operations including: Determine the first index of the first context-triggered segmented hash CTPH string of the file, wherein the first index is a proper subset of the CTPH string, wherein determining the first index of the first CTPH string includes: splitting the first CTPH string into a plurality of disjoint parts of consecutive bits, and the first index is determined as a first number of consecutive bits of the plurality of parts of the first CTPH string. Compare the first index with the second index of the file cluster; In response to determining that the first index matches the second index of the cluster, the file is associated with the cluster; and Based on the cluster, it was determined that the file contained malware.

2. The system according to claim 1, wherein the cluster is one of a plurality of clusters of files.

3. The system of claim 2, wherein the first index includes the most significant byte of the CTPH string, and the operation further includes: Determine that the third index corresponding to the least significant byte of the CTPH string matches the second index; as well as In response to determining that the third index of the CTPH string matches the second index, the file is associated with one of the plurality of clusters.

4. The system of claim 3, wherein the third index is determined in response to a mismatch between the first index and the second index.

5. The system according to claim 2, further comprising: For each of the plurality of portions, a fourth index is determined that includes the second most significant byte of the portion; as well as In response to determining that the fourth index matches the second index, the file is associated with the cluster.

6. The system of claim 4, wherein the operation further comprises: For each of the plurality of parts, a sixth index is determined that includes the second least significant byte of the part; as well as In response to determining that the sixth index matches the second index, the file is associated with the cluster.

7. A method for determining that a file contains malware using a cluster based on a context-triggered segmented hash (CTPH), the method comprising: The processing circuitry of the computing device determines a first index of a first CTPH string of the file, the first index being a proper subset of the CTPH string, wherein determining the first index of the first CTPH string includes splitting the first CTPH string into a plurality of disjoint parts of consecutive positions, and the first index is determined as a first number of consecutive positions of the part of the plurality of parts of the first CTPH string. Compare the first index with the second index of the file cluster; In response to determining that the first index matches the second index of the cluster, the file is associated with the cluster; and Based on the cluster, it was determined that the file contained malware.

8. The method of claim 7, wherein the cluster is one of a plurality of clusters of files.

9. The method of claim 8, wherein the first index comprises the most significant byte of the CTPH string, and the method further comprises: Determine that the third index corresponding to the least significant byte of the CTPH string matches the second index; as well as In response to determining that the third index of the CTPH string matches the second index, the file is associated with one of the plurality of clusters.

10. The method of claim 9, wherein the third index is determined in response to a mismatch between the first index and the second index.

11. The method of claim 8, wherein the first index comprises the most significant byte of the portion, and the method further comprises: Determine the fourth index that includes the second most significant byte of the aforementioned portion; as well as In response to determining that the fourth index matches the second index, the file is associated with the cluster.

12. The method of claim 10, wherein the third index comprises the least significant byte of the portion, and the method further comprises: Determine the sixth index that includes the second least significant byte of the aforementioned portion; as well as In response to determining that the sixth index matches the second index, the file is associated with the cluster.

13. A machine-readable medium comprising instructions that, when executed by a machine, cause the machine to perform operations for determining, using a context-triggered segmented hash (CTPH) cluster, that a file contains malware, the operations comprising: The processing circuitry of the computing device determines a first index of a first CTPH string of the file, the first index being a proper subset of the CTPH string, wherein determining the first index of the first CTPH string includes splitting the first CTPH string into a plurality of disjoint parts of consecutive positions, and the first index is determined as a first number of consecutive positions of the part of the plurality of parts of the first CTPH string. Compare the first index with the second index of the file cluster; In response to determining that the first index matches the second index of the cluster, the file is associated with the cluster; and Based on the cluster, it was determined that the file contained malware.

14. The machine-readable medium of claim 13, wherein the cluster is one of a plurality of clusters of a file.

15. The machine-readable medium of claim 14, wherein the first index comprises the most significant byte of the CTPH string, and the operation further comprises: Determine that the third index corresponding to the least significant byte of the CTPH string matches the second index; as well as In response to determining that the third index of the CTPH string matches the second index, the file is associated with one of the plurality of clusters.

16. The machine-readable medium of claim 15, wherein the third index is determined in response to a mismatch between the first index and the second index.

17. The machine-readable medium of claim 15, wherein the first index comprises the most significant byte of the portion, and the machine-readable medium further comprises: Determine the fourth index that includes the second most significant byte of the aforementioned portion; as well as In response to determining that the fourth index matches the second index, the file is associated with the cluster.

18. The machine-readable medium of claim 17, wherein the third index includes the least significant byte of the portion, and the operation further includes: Determine the sixth index that includes the second least significant byte of the aforementioned portion; as well as In response to determining that the sixth index matches the second index, the file is associated with the cluster.

Images