Technique of pattern matching for executable binary files

Abstract

A technique of pattern matching for executable binary files comprises the following steps: a) preparing one or more sequences data digests; b) computing locality-sensitive hash; c) providing a process of reduction of the plausible matchings; wherein said step of reduction of the plausible matchings is carried out by using filters based on a threshold of Hamming distance between hashes.

Description

[0001] TECHNIQUE OF PATTERN MATCHING FOR EXECUTABLE BINARY FILES

[0002] Description

[0003] Technical Field

[0004] The present invention find application in the technical field of electric digital data processing and, in particular, it has as object a technique of pattern matching for executable binary files.

[0005] State of the art

[0006] As known, regular expressions are commonly used to search for patterns in bodies of text for rapid comparison, for example for using within search engines and data operations,

[0007] In this filed, a pattern matching technique is a technique allowing to check a given sequence of tokens for the presence of the constituents of some pattern.

[0008] Properly, in the field of data analysis, pattern matching is the action of checking the presence of a certain pattern inside a composite object.

[0009] Specifically, the expression may refer to pattern recognition within a character string or within abstract data structures such as lists or trees, a syntactic construct of functional programming languages used to condition recursion.

[0010] Uses of pattern matching include outputting the locations of a pattern within a token sequence, to output some component of the matched pattern, and to substitute the matching pattern with some other token sequence (i.e., search and replace).

[0011] The common solutions designed to detect and classify data sequences provide the use of an exact binary patterns data set.

[0012] Some extensions of this method allow fuzzy matching with specified similarity threshold using e.g. Hamming distance.

[0013] A fuzzy multi-pattern matching is a problem with high computational complexity wherein, in many cases, the pattern is detected and categorized not only by values of a sequence but also by its surroundings, thus adding further context to the specific data sequence.

[0014] The main drawback of the known techniques is that they are memory and computationally expensive when applied on the medium size pattern over a large alphabet. Moreover, the generalization of the matching sequence, like accepting changes to its shape or slight differences of values, makes this approach a complex problem when the patterns data set is large.

[0015] Hash-based solutions are valuable for exact matching, but due to the avalanche effect are useless for approximate matching.

[0016] Furtherly, with additional matching criteria like context, an exact pattern matching is not valuable for many similar patterns distinctive only by their surroundings.

[0017] Therefore, without significant reduction of potential candidates of matching patterns, these methods are not valuable for cases when the number of data patterns is large.

[0018] Li Weng - Perceptual Multimedia Hashing discloses a perceptual hashing (PH) of multimedia files, such as image, sound, video.

[0019] However, methods used on multimedia are not valuable on executable binary files. For example, there is no option to scale, blur, reduce color information, convert to greyscale or use similar techniques to reduce the input file size or use as a method of data pre-processing on executable binary files.

[0020] As matter of fact, compression of multimedia files typically has much better ratios than on executable binary files, which makes the problem of applying LSH much more complex.

[0021] Moreover, the PH methods are not resistant to pre- and post-padding of input files with noise or constant values.

[0022] Scope of the invention

[0023] The object of the present invention is to provide a technique of pattern matching for executable binary files characterized by high efficiency e relative cost-effectiveness.

[0024] A proper object that the present invention intends to solve is to increase the speed and / or quality of sequence matching by reducing the number of potential pattern candidates.

[0025] A further object is to provide a technique of pattern matching for executable binary files providing a fast method of reducing the number of potential patterns which are matching the analyzed sequence, allowing processing more data at the same time.

[0026] These objects, as well others those will become more apparent hereinafter, are fulfilled by a technique of pattern matching for executable binary files according to claim 1.

[0027] The solution to the above drawbacks is an augmentation of the pattern-matching process with efficient context matching, which is achievable with the proposed technique using novel LSH hash prepared especially for semi-monotonic data sequences intrinsic to executable binary files.

[0028] The main value of the technique is the ability to appliance in a fast pattern-matching process, with medium and large-size context surroundings, along with a potential significant mismatch of the data pattern area.

[0029] A classical approach based on specific, predefined, provided by expert data markers, or using exact pattern matching, is slow and requires a much larger data set.

[0030] Other techniques, e.g. fuzzy pattern search, is memory and computationally expensive. Advantageous embodiments of the invention are obtained in accordance with the dependent claims.

[0031] Brief disclosure of the drawings

[0032] Further features and advantages of the invention will become more apparent in the light of the detailed description of preferred but not exclusive embodiments of the technique according to the invention, shown by way of non-limiting example with the aid of the attached drawing tables wherein:

[0033] FIG. 1 is a diagram showing a first series of steps of the technique;

[0034] FIG. 2 is a diagram showing a second series of steps of the technique;

[0035] FIG. 3 is a diagram showing a third series of steps of the technique.

[0036] FIG. 4 is a flow chart showing a possible application of the technique in automotive sector on ECU executable binary files for classification of ECU binary file features;

[0037] FIG. 5 is a flow chart showing a possible application of the technique in automotive sector on ECU executable binary files for classification with pruning for wide context analysis of data sequences.

[0038] Best mode of carrying out the invention

[0039] The present invention refers to a new technique for reducing the matching patterns data set and in particular to a new method of fuzzy, context-aware subsequence matching, based on a new locality-sensitive hash for organized and partially monotonic data patterns.

[0040] In particular, the technique is a fuzzy pattern matching technique designed for searching for patterns inside the large binary data block having executable binary file as input data. Basically, the technique comprise a first series of steps designed to prepare one or more sequences data digests, a second series of steps for computing locality-sensitive hash and a final series of steps of reduction of the plausible matchings.

[0041] Fig- 1 shows the first series of steps suitable to prepare the sequences data digest. According to a preferred but not exclusive embodiment for the present invention, the technique provides that, after insertion of input and training data, the data is analysed for containing semi-organized, partially monotonic sequences.

[0042] A preprocessing step is also provided for filtering monotonic sequences, during which step potential sequences are discarded or accepted based on several statistical criteria. Each accepted sequence is then encoded with metadata and stored as part of the data sequences digest so obtained.

[0043] Said digest contains information describing the spatial organization of sequences, their monotonicity properties and other features valuable for approximate matching.

[0044] Fig- 2 shows a possible operative diagram for computing the locality-sensitive hash, wherein locality-sensitive hashes are computed for each pattern in the data set and the analyzed positions in the input data.

[0045] The proprietary locality-sensitive hash is calculated based on a method of data compression.

[0046] In a preferred but non-exclusive way, the proprietary locality-sensitive hash is calculated based on the wavelets analysis.

[0047] The most significant parameters of data compression are then encoded with single bits. Such an approach reduces data digest to a 64-bit hash which is convenient for fast Hamming distance calculation.

[0048] In this way, similar sequences with the same surroundings have hashes that differ only on a small number of bits.

[0049] Moreover, 64-bit is a native machine word size of modern CPUs, allowing efficient computation of Hamming distance as most of the architectures have dedicated CPU instructions for such purpose. By contrast, typical PH hashes are longer.

[0050] Reduction of plausible matchings is achieved using filters based on a threshold of Hamming distance between hashes, in such a way that the matching candidate are accepted if distance is below a set threshold.

[0051] Furtherly, the meta-data digest may be used for hash collision resolving of the same LSH hash values, as edit-distance metrics are applicable on the meta-data digest to measure surroundings similarity ratio.

[0052] Differently form PH methods, the present technique, due to intrinsic features of the meta-data digest construction technique, is resistant to pre- and post-padding of input files with noise or constant values.

[0053] A possible application of the technique according to the invention is into the automotive sector on ECU executable binary files for classification of ECU binary file features.

[0054] The technique will allow to solve problems related to the analysis of ECU binary executable files typical of automotive sector, in particular:

[0055] • Detection of ECU brand, ECU family, vehicle brand;

[0056] • Detection of software version

[0057] • Detection of engine power variant

[0058] • Detection of irregularities in the ECU control process

[0059] • Detection of DTC areas

[0060] • Detection and classification of control map areas.

[0061] As an improvement part of the following AI / ML tasks, the technique allows:

[0062] • Finding patterns in both code and data segments with medium to large context size;

[0063] • Reducing the number of potential match candidates (pruning) for data classification;

[0064] • Finding areas of interest based on surroundings with a loose threshold for the content match (fuzzy pattern matching using intrinsic syntax relations);

[0065] • Providing an efficient method for data clustering and reducing the training dataset size.

[0066] As a way to solve specific problems related to binary executable files, including but not limited to the automotive sector, the technique also allows:

[0067] • Detection of used linker, compiler;

[0068] • Detection of linked libraries;

[0069] • Detection of code type and data modification;

[0070] The possible appliances are: code analysis, digital forensics, and antivirus software.

Claims

Claims1. A technique of pattern matching for executable files comprising the following steps: a) preparing one or more sequences data digests; b) computing locality-sensitive hash LSH; c) providing a process of reduction of the plausible matchings; wherein said executable files are executable binary files; wherein the LSH size is 64-bit; wherein said step of reduction of the plausible matchings is carried out by using filters based on a threshold of Hamming distance between hashes and wherein each accepted sequence is encoded with metadata and stored as part of the data sequences digest.

2. Technique as claimed in claim 1, wherein said step a) of preparing the digest comprises a step of data input and a subsequent step of training said data.

3. Technique as claimed in claim 2, comprising a pre-processing step of said input data.

4. Technique as claimed in claim 3, wherein said pre-processing step provides the analysis of said data for containing semi-organized, partially monotonic sequences.

5. Technique as claimed in claim 3 or 4, wherein in said pre-processing step potential sequences are discarded or accepted based on statistical criteria.

6. Technique as claimed in any preceding claim, wherein in said step of computing locality-sensitive hash, locality-sensitive hash is calculated based on a method of data compression.

7. Technique as claimed in claim 6, wherein said method of data compression is wavelets analysis.

8. Technique as claimed in claim 6 or 7, wherein the most significant parameters of data compression are encoded with single bits.

9. Technique as claimed in claim 8, wherein said step of reduction of the plausible matchings provides that the matching candidate are accepted if distance is below a set threshold.

10. Technique as claimed in any preceding claim, characterized by being resistant to pre- and post-padding of input files with noise or constant values.

11. Technique as claimed in any preceding claim, characterized by being a fuzzy pattern matching technique designed for searching for patterns inside the large binary data block having executable binary file as input data.

12. Technique as claimed in any preceding claim, characterized by the fact that said executable binary files are ECU executable binary files, a classification step of said ECU binary file features being provided for detecting ECU brand, ECU family, vehicle brand, software version, engine power variant, irregularities in the ECU control process, DTC areas, control map areas.

Images