A storage method for a large number of MD5 feature codes

Abstract

The storage method for a large number of MD5 feature codes proposed by the present invention includes the following steps: Read the MD5 strings in the MD5 data file, classify the MD5 strings according to the content of the first two bytes, and generate 256 sub-files; Establish a linked list array, using the file names of the sub-files as the data in the linked list array; Process the sub-files separately, convert each MD5 string in the sub-file into an integer data, and establish an array for each MD5 string to store the converted integer data; Add the arrays to the linked list array according to the file names of the sub-files where they are located; Sort the arrays; Store the sorted data. This method greatly improves the efficiency of storing and matching a large number of MD5 feature codes.

Description

Technical Field

[0001] The present invention relates to the technical field of password management, and particularly relates to a method for storing a large number of MD5 feature codes. Background Art

[0002] In the process of searching and matching computer virus files, it is inevitable to use the MD5 features of files for matching calculations to discover the traces of virus files. In the existing systems, the storage and matching of the MD5 of files adopt the general string matching method. This method becomes very cumbersome when dealing with hundreds of millions of MD5 feature codes:

[0003] 1) During the storage process of hundreds of millions of feature codes, a large amount of system memory will be used, which brings many restrictions to the popularization and use of this technology;

[0004] 2) Adopting the calculation method of string matching takes a long time in the computer and is not conducive to quickly matching a large amount of data.

[0005] Therefore, when there are hundreds of millions of MD5 feature codes, efficient storage and matching are necessary for the search and matching of computer virus files. Summary of the Invention

[0006] The present invention provides a method for storing a large number of MD5 feature codes, which is used to solve the technical problem of efficient storage and matching of a large number of MD5 feature codes.

[0007] The embodiments of the present invention are as follows:

[0008] Read the MD5 strings in the MD5 data file, classify the MD5 strings according to the content of the first two bytes, and generate 256 sub-files; establish a linked list array, and use the file names of the sub-files as the data in the linked list array; process the sub-files separately, convert each MD5 string in the sub-file into an integer data, and establish an integer array for storing the converted integer data; add the array to the linked list array according to the file name of the sub-file where it is located; sort the array; store the sorted data.

[0009] Further, before reading the MD5 strings, preprocess the MD5 data file, retrieve the content of the data file to ensure that each piece of data is a compliant MD5 string (0-10a-f), with a length of 32 bytes.

[0010] Further, the 256 sub-files are named according to the content of the first two bytes of the MD5 string, that is, the file names are 00, 01, 02....A0, B0,....., up to FF. At this time, the MD5 entries in the 256 split files are all MD5 data starting with the corresponding file name prefix.

[0011] Further, 256 original data files are processed separately, and the MD5 string (32 bytes) is converted into integer data of 2 * int64 (16) bytes and stored in an array in int64 format.

[0012] Further, the sorting method used is the quicksort method, and other common sorting methods can also be used, such as the bubble sort method, the merge sort method, etc.

[0013] Further, in order to prevent file corruption or malicious tampering, when storing the sorted data, a signature and verification information are added to the file header.

[0014] Further, when a data matching request is received, the position of the array is located according to the content of the first two bytes of the MD5 signature to be matched, and the MD5 signature to be matched is queried in the array by the binary search method.

[0015] The storage method of a large number of MD5 signatures proposed by the present invention partitions a large number of MD5 data files according to the content of the first two bytes of the MD5 signature, and stores the MD5 signature in double Int64 integers, which greatly improves the storage and matching efficiency of a large number of MD5 signatures. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 Flowchart of the storage method of a large number of MD5 signatures;

[0018] Figure 2 MD5 signature matching flowchart;

[0019] Figure 3 Schematic diagram of the storage format of sorted data of a large number of MD5 signatures in memory. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0021] According to an embodiment of the present invention, a method for storing a large number of MD5 feature codes is provided. Figure 1 As shown in the flowchart of the storage method, the following steps are included: reading the MD5 strings in the MD5 data file, classifying the MD5 strings according to the content of the first two bytes, and generating 256 sub-files; establishing a linked list array, using the file names of the sub-files as the data in the linked list array; processing the sub-files respectively, converting each MD5 string in the sub-file into an integer data, and establishing an integer array for storing the converted integer data; adding the array to the linked list array according to the file name of the sub-file where it is located; sorting the array; in order to avoid sorting the data file again when loading next time, storing the sorted data of each partition in the original data format file, and adding signature and verification information to the file header. In order to prevent file damage or malicious tampering, relevant signature and verification information is designed in the file header:

[0022]

[0023]

[0024] This data structure occupies 1024 bytes of the export file header, and relevant verification is performed during loading to prevent the file from being maliciously tampered with by a third party.

[0025] In a preferred embodiment of the present invention, before reading the MD5 strings, preprocessing is performed on the MD5 data file, and the content of the data file is retrieved to ensure that each piece of data is a compliant MD5 string (0-10a-f), with a length of 32 bytes.

[0026] In a preferred embodiment of the present invention, the 256 sub-files are named according to the content of the first two bytes of the MD5 string, that is, the file names are 00, 01, 02....A0, B0,..... until FF. At this time, the MD5 entries in the 256 split files are all MD5 data starting with the corresponding file name prefix.

[0027] In a preferred embodiment of the present invention, the 256 original data files are processed respectively, converting the MD5 string (32 bytes) into an integer data of 2*int64 (16) bytes, and storing it in an array in int64 format. For example, for example: converting the string "6b5e4c956ccfab36b9e314e13cf35a5c" (32-byte length) into two integers in memory (stored in an int64 array):

[0028] a[0] = 0x6b5e4c956ccfab36;

[0029] a[1] = 0xb9e314e13cf35a5c。

[0030] Add the a variable (int64[2]) to the linked list array numbered 6b (the first two bytes), process the next record until all files are processed. Finally, the partitioned MD5 sorted data as shown in Figure 3 will be formed in memory. Calculated according to 200 million MD5 data volumes, the overall memory occupancy is:

[0031] 200000000 / (1024*1024*1024)*16 = 2.98G Bytes

[0032] For general server configurations (memory greater than or equal to 8G), it is sufficient to bear.

[0033] In a preferred embodiment of the present invention, the sorting method used is the quicksort method. Of course, other common sorting methods can also be used, such as the bubble sort method, the merge sort method, etc., as long as it can ensure fast and correct sorting, it can be used.

[0034] When receiving a data matching request, as shown in Figure 2 , receive the matching access request, check the format of the request. If it is not successful, return an error; locate the internal sorting array position according to the first byte of the Md5 of the request content; perform a binary search in the located array; return the query matching result. After testing with relevant virus samples collected from the Internet, the average partition queue length for storing 200 million MD5s is about 78000. Testing on an ordinary server takes about 3.1us, meeting the system requirements.

[0035] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A storage method for a large amount of MD5 feature codes, characterized in that, The steps include: Preprocess the MD5 data file, retrieve the content of the MD5 data file to ensure that each piece of data is a compliant MD5 string; Read the MD5 strings in the MD5 data file, classify the MD5 strings according to the content of the first two bytes, and generate 256 chunk files; Establish a linked list array with the file names of the chunk files as the data in the linked list array; Process the chunk files respectively, convert each MD5 string in the chunk file into two 16-byte data in int64 format, and establish an int64 format integer array for storing the converted 16-byte data in int64 format; Add the array to the linked list array according to the file name of the chunk file where it is located; Sort the array and store the sorted data.

2. The storage method of a large number of MD5 feature codes according to claim 1, characterized in that, The 256 chunk files are named according to the content of the first two bytes of the MD5 string.

3. The storage method of a large number of MD5 feature codes according to claim 1, characterized in that, The sorting method used is the quicksort method.

4. The storage method of a large number of MD5 feature codes according to claim 1, characterized in that, When storing the sorted data, add signature and verification information at the file header.

5. The storage method of a large number of MD5 feature codes according to claim 1, characterized in that, When receiving a data matching request, locate the position of the array according to the content of the first two bytes of the MD5 feature code to be matched, and query the MD5 feature code to be matched in the array using the binary search method.

Images