Efficient malware signature matching method and storage medium

Abstract

The application relates to the technical field of malware detection, and discloses an efficient malware feature code matching method and a storage medium, wherein a feature library is constructed according to target data strings; feature values in the feature library are initialized to the memory; to-be-detected data is received, and the effective length is calculated; matching is performed in the feature library according to the length, matching of a first feature segment of the to-be-detected data is performed when the matching is successful, matching of a second feature segment of the to-be-detected data is performed when the matching of the first feature segment is successful, and the to-be-detected data is determined as malware when the matching of the second feature segment is successful; the key data segment and the effective data segment of the malware are extracted, hash values are respectively calculated, a malware feature library is constructed, hash value taking of the first feature segment of the to-be-detected data is performed to make initial filtering, hash value taking of the second feature segment is performed to make judgment, and hash value comparison is performed in the feature library; the matching efficiency is high, real-time scanning is suitable, the segmented crc32 hash algorithm is used for value taking, the calculation speed of the hash value itself is improved, and the overall efficiency and the accuracy are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of malware detection, and in particular to an efficient malware signature matching method and a storage medium. BACKGROUND

[0002] With the rapid development of information technology, malware (including botnets, Trojans, worms, etc., referred to as "bot Trojans") has become one of the main threats in the field of network security. These malicious software can cause serious harm to personal privacy, business operations, and even national network security through methods such as stealing sensitive information, destroying system functions, and launching network attacks. According to industry reports, the number of malware worldwide is growing at an exponential rate, with its variants updating at a shorter and shorter cycle and becoming increasingly complex, posing a huge challenge to detection and defense work.

[0003] In the field of malware detection technology, signature matching algorithms are a widely used core technology. This technology extracts unique signatures (such as specific byte sequences, instruction fragments, etc.) of malware and compares them with data in the file to be detected, thereby quickly identifying known malware. However, as the number and complexity of malware increase, the shortcomings of existing pattern matching algorithms in efficiency are becoming increasingly apparent.

[0004] Currently, some mainstream pattern matching algorithms use a "one-to-one" comparison method, which directly calculates the hash of the full text data during detection and compares the results one by one with the preset signature hash value. Although this method is simple to implement, it has significant efficiency problems: on the one hand, full-text hash calculation requires traversing all the data in the file, which consumes a large amount of computing resources and time when the size of the file to be detected is large or the size of the detection sample library is large; on the other hand, the "one-to-one" comparison mode lacks targeted optimization of signatures, making it difficult to adapt to the rapid iteration of malware variants, leading to missed or false detections, and also failing to meet the needs of real-time detection.

[0005] In addition, in practical applications, real-time monitoring of network traffic, rapid scanning of terminal devices, and other scenarios require higher efficiency from the matching algorithm. The low efficiency of traditional algorithms not only leads to detection delays and affects user experience, but also may leave a time window for the spread and destruction of malware. Therefore, developing an algorithm that can significantly improve the efficiency of signature matching while ensuring detection accuracy has become a key problem in the field of malware detection that needs to be solved. SUMMARY

[0006] In view of the deficiencies of the prior art, the purpose of the present application is to provide an efficient malware signature matching method.

[0007] In order to achieve the above object, the present application provides the following technical solutions:

[0008] The high-efficiency malicious software feature code matching method comprises the following steps:

[0009] According to the target data string, a feature library is constructed, and the feature value corresponding to each data string is stored in the feature library; the feature values in the feature library are initialized to the memory; a to-be-detected data is received, and the effective length of the to-be-detected data is calculated; the to-be-detected data is matched in the feature library according to the length; when the matching is successful, the first feature segment of the to-be-detected data is matched; otherwise, the scanning process is ended; when the first feature segment is matched successfully, the second feature segment of the to-be-detected data is matched; when the second feature segment is matched successfully, the to-be-detected data is determined as malicious software.

[0010] In the present application, preferably, the construction of the matching feature library from the target data string comprises the following steps:

[0011] The first feature segment of the target matching data string is extracted, and the crc32 hash value of the first feature segment is calculated to obtain a first hash value; the byte length of the first feature segment and the first hash value are combined to form a first feature value; the second feature segment of the target matching data string is extracted, and the crc32 hash value of the second feature segment is calculated to obtain a second hash value; the byte length of the second feature segment and the second hash value are combined to form a second feature value; and the first feature value and the second feature value are taken as the feature value of the target matching data.

[0012] In the present application, preferably, the initialization of the feature values in the feature library to the memory comprises the following steps:

[0013] The low 16 bits of the feature values in the feature library are taken as a first feature index array and stored in an index table, and the high 16 bits are taken as a second feature index array and stored in a linked list array; and the first feature index array and the second feature index array are associated.

[0014] In the present application, preferably, when the low 16 bits are taken as the first feature index array, the element data type of the first feature index array is a bool value, and the second feature index array is associated with each bool value of the first feature index array.

[0015] In the present application, preferably, before the effective length of the to-be-detected data is calculated, the tail invalid bytes of the to-be-detected data are removed.

[0016] In the present application, preferably, the matching of the first feature segment of the to-be-detected data comprises the following steps:

[0017] The crc32 hash value of the first feature section is calculated, the low 16 bits of the crc32 hash value are directly indexed in the index table of the corresponding first feature section, when indexed, the crc32 hash value of the high 16 bits is searched in the corresponding linked list, when successfully searched, the first feature section is matched successfully, and the next step is performed, otherwise the scanning is ended.

[0018] In the application, preferably, the matching of the second feature section of the to-be-detected data specifically comprises: calculating the crc32 hash value of the second feature section, directly indexing the low 16 bits of the crc32 hash value in the index table of the corresponding second feature section, when indexed, searching the crc32 hash value of the high 16 bits in the corresponding linked list, when successfully searched, the second feature section is matched successfully, and the to-be-detected data is malicious software, otherwise the scanning is ended.

[0019] In the application, preferably, 4096 bytes are taken as the first feature section from the place with a data offset of 0.

[0020] In the application, preferably, the starting offset position of the second feature section is the same as that of the first feature section, and the length is the total number of valid bytes of the data.

[0021] Compared with the prior art, the application has the beneficial effects that:

[0022] The method of the application first extracts the key data section and the effective data section of the malicious software to calculate hash values respectively, constructs the feature library corresponding to the malicious software, then performs hash value taking on the first feature section of the to-be-detected data as an initial filtering condition, and performs hash value taking on the second feature section for judgment, and uses an optimized index method to perform hash value comparison in the feature library, the matching efficiency is high, is suitable for real-time scanning, the segmented crc32 hash algorithm is used for value taking, the calculation speed of the hash value itself can be greatly improved, so that the overall efficiency is improved, and the segmented matching improves the detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The flowchart of the efficient malicious software feature code matching method is shown.

[0024] Figure 2 The structure diagram of the first feature section in the memory after initialization in the embodiment of the application is shown.

[0025] Figure 3 The structure diagram of the second feature section in the memory after initialization in the embodiment of the application is shown. DETAILED DESCRIPTION

[0026] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0028] Please refer to Figure 1 A preferred embodiment of the present application provides a high-efficiency malicious software signature matching method, which extracts features by segmenting a target malicious program file and using a CRC32 hash algorithm, and constructs a feature library according to the features and reduces the storage volume by optimizing the data structure. Meanwhile, the CRC32 value of a first feature segment is split into a low 16-bit and a high 16-bit, the low 16-bit is used as an index array and stored in an index table, and the high 16-bit is stored in a linked list associated with the index group. When data matching is performed, the CRC32 value of the first feature segment of the to-be-detected file is searched in the low 16-bit index and the high 16-bit associated linked list. If no match is found, the process is ended. If a match is found, the CRC32 value of a second feature segment is calculated. If a match is found, the process is successful. Otherwise, the process is ended. The CRC32 hash algorithm is used to replace the traditional MD5, SHA1 and other algorithms, the time consumption is short, the fast filtering is realized by segment feature calculation, and non-malicious software can be excluded without calculating the complete hash value. The CRC32 value is split into high and low 16-bit and indexed, the search range is reduced, and the matching efficiency is improved.

[0029] The matching method specifically includes the following steps:

[0030] A feature library is constructed according to a target data string, and the feature library has a feature value corresponding to each data string;

[0031] The feature values in the feature library are initialized to the memory;

[0032] A to-be-detected data is received, and the effective length thereof is calculated;

[0033] The length is matched in the feature library. When a match is found, the first feature segment of the to-be-detected data is matched. Otherwise, the scanning process is ended.

[0034] When the first feature segment is matched successfully, the second feature segment of the to-be-detected data is matched again, and when the matching is successful, it is determined that the to-be-detected data is malicious software.

[0035] In the embodiment, constructing the matching feature library for the target data string comprises:

[0036] The first feature segment is extracted from the target matching data string, and a crc32 hash value of the first feature segment is calculated to obtain a first hash value; the first feature value is combined by the byte length of the first feature segment and the first hash value; the second feature segment is extracted from the target matching data string, and a crc32 hash value of the second feature segment is calculated to obtain a second hash value; the second feature value is combined by the byte length of the second feature segment and the second hash value; and the first feature value and the second feature value are taken as the feature value of the target matching data.

[0037] Specifically, the target matching data string is generally a malicious program file, and is segmented and extracted, each segment comprising a segment size and a crc32 value calculated for the segment data. The first feature segment has a fixed segment size, which is 4096 in this embodiment, and a data width of 32 bits, i.e. 4 bytes. Since the segment hash uses crc32, the data width is also 32 bits, i.e. 4 bytes. The segment size of the first feature segment can also be 2048, and the feature value can not record the segment size but only record the crc32 hash value of the first feature segment. For example, for a 16k (16384 byte) file, the first feature segment is extracted first, and the crc32 hash value of the first 4096 bytes is calculated from the file offset of 0 (the first byte), and the calculation result is 0xd4978a8a, and the length is 4096, so the first segment feature value is 0x00001000 concatenated with 0xd4978a8a. The start of the second feature segment can overlap with the start of the first feature segment, and to improve efficiency, the second feature segment uses the same start offset position as the first feature segment, i.e. the offset is 0 (i.e. the first byte), and the crc32 hash value of the effective byte segment of the target matching data string is calculated, and the crc32 hash values of the first feature segment and the second feature segment are stored in the hash table together with the corresponding byte length.

[0038] In the embodiment, when the to-be-detected data is scanned, the feature values in the feature library need to be initialized into the memory first: the low 16 bits of the feature values in the feature library are stored in the index table as a first feature index array, and the high 16 bits are stored in the linked list array as a second feature index array; and the first feature index array and the second feature index array are associated.

[0039] Specifically, since the size of the first feature segment in this example is a fixed value 4096, the crc32 feature value of the first feature segment is directly calculated, the crc32 value is 32 bits, and a first segment feature index array with a length of 65536 elements is established by taking the low 16 bits in the 32 bits. The element data type of the array is a bool value. For example, the value of the low 16 bits of the first segment feature value is 0x8a8a (decimal 35466), so the 35467th (35466+1) bool value in the first segment feature index array is set to true. The high 16 bits of the crc32 hash value are stored in a linked list array, and this linked list array is associated with each bool value of the first segment feature index array, which is 0x8a8a+1 in this example, i.e., the 35467th. In this way, when scanning, the calculated crc32 low 16-bit value and the 35467th bool value of the feature index array match successfully, and the value of the high 16 bits can be searched in the associated linked list array. The scanning program can reduce the range of maximum search and comparison of feature values during scanning. For example, in this example, the low 16 bits are used as index values, so the search range is 32 bits-16 bits=16 bits, i.e., the length of the search space is the remaining 16 bits, i.e., 2^16, i.e., 65536. That is, the remaining 16 bits are matched, and the maximum search and comparison times are 65536 times. If the feature value index table is not used, the maximum range of one search and comparison is 2^32, i.e., 4294967296 times, and 4294967296 / 65536=65536. Therefore, after using the index, the maximum search range is reduced to 1 / 65536 of the original, thereby greatly improving the search and matching efficiency. This strategy is a "space for time" strategy, which uses some large storage space to reduce the running time of the algorithm. In this example, the size of a single index table is 65536, i.e., 64k element units, and each element unit is associated with a linked list array. According to different platforms (e.g., 32-bit or 64-bit), the actual occupied memory space is not more than 1MB.

[0040] In a specific embodiment, it is assumed that there are 4 features in the feature library, and the following is a pseudo code description of the 4 features in the feature library:

[0041] / / Define the data structure of the feature entry

[0042] struct SigRec

[0043] {

[0044] struct SigCRC32WithFixedSize4096 stSigCRC32_1;

[0045] struct SigCRC32 stSigCRC32_2;

[0046] }

[0047] struct SigRec record0, record1, record2, record3

[0048] / / First feature

[0049] record0.stSigCRC32_1.uiCRC32 = 0x43210000

[0050] record0.stSigCRC32_2.uiLength = 80823 (0x13bb7)

[0051] record0.stSigCRC32_2.uiCRC32 = 0xa71ed633

[0052] / / Second feature

[0053] record0.stSigCRC32_1.uiCRC32 = 0xd4978a8a

[0054] record0.stSigCRC32_2.uiLength = 4096 (0x1000)

[0055] record0.stSigCRC32_2.uiCRC32 = 0xd4978a8a

[0056] / / Third feature

[0057] record0.stSigCRC32_1.uiCRC32 = 0x1234ffff

[0058] record0.stSigCRC32_2.uiLength = 20250 (0x4f1a)

[0059] record0.stSigCRC32_2.uiCRC32 = 0x463b53e8

[0060] / / Fourth feature

[0061] record0.stSigCRC32_1.uiCRC32 = 0x8c24ffff

[0062] record0.stSigCRC32_2.uiLength = 146359 (0x23bb7)

[0063] record0.stSigCRC32_2.uiCRC32 = 0x86241990

[0064] Figure 2 The data structure after the first feature segment of the four features in the feature library in this example is initialized to the memory.

[0065] In this embodiment, when the low 16 bits of the feature value are used as the first feature index array, the element data type of the first feature index array is a bool value, the second feature index array is associated with each bool value of the first feature index array, and the association is achieved by a pointer.

[0066] In this embodiment, before calculating the effective length of the to-be-detected data, the tail invalid bytes of the to-be-detected data need to be removed.

[0067] In this embodiment, the matching of the first feature segment of the to-be-detected data specifically includes:

[0068] The crc32 hash value of the first feature segment is calculated, the low 16 bits of the crc32 hash value are directly indexed in the index table corresponding to the first feature segment, when the index is found, the crc32 hash value of the high 16 bits is searched in the corresponding linked list, when the crc32 hash value of the high 16 bits is successfully searched, the first feature segment is matched successfully, and the next step is performed, otherwise the scanning is ended.

[0069] In this embodiment, the matching of the second feature segment of the to-be-detected data specifically includes:

[0070] The crc32 hash value of the second feature segment is calculated, the low 16 bits of the crc32 hash value are directly indexed in the index table corresponding to the second feature segment, when the index is found, the crc32 hash value of the high 16 bits is searched in the corresponding linked list, when the crc32 hash value of the high 16 bits is successfully searched, the second feature segment is matched successfully, and the to-be-detected data is malicious software, otherwise the scanning is ended.

[0071] Specifically, after the feature library initialization is completed, the comparison process is started, and first, the size in the second feature section is compared with the size of the object file to be scanned. The size is the size after removing the tail 0x00, 0x0a, 0x0d, and the like, so the target comparison data string, i.e., the file, is first processed to remove the tail invalid bytes. As described above, the size in the second feature section records the size of the entire file after removing the tail 0x00, 0x0a, 0x0d, and the like. If there is no second feature section with such a size in the feature library, it is indicated that there is no feature in the feature library that can match the file, and the scanning process is ended. The size of the entire file can be used as the first filtering condition to optimize the algorithm efficiency.

[0072] When the file size is successfully matched in the above step, the matching of the first feature section is started, and the structure of the first feature section in the memory after the initialization is shown in FIG. 2. Figure 2 The first feature section can also be referred to as the first filtering section, because its existence can quickly exclude the files that cannot match the features in the feature library. If the matching fails, the scanning process of the file is immediately ended, and there is no need to wait for the calculation of the hash value of the entire file before the matching, so that the algorithm efficiency is improved. The specific matching process is that the low 16 bits of the crc32 calculated by using the first 4096 bytes of the scanned object are directly indexed in the index table of the corresponding first feature section. When the indexing is successful, the high 16 bits are searched in the corresponding linked list, otherwise the matching fails, and the scanning is ended.

[0073] If the above steps are all successfully matched, the last step, i.e., the matching of the crc32 value in the second feature section, i.e., the matching of the full-text crc32 value, is performed. In this example, the value offset start of the second data section is the same as that of the first data section, i.e., 0x0, i.e., the value is taken from the first byte, and the length of the value is the length after removing the tail 0x00, 0x0a, 0x0d, and the like. For example, the entire file to be matched is 16k (16384 bytes), and the file content is as follows:

[0074] The output result of hexdump -C binary.test is as follows, and the middle part is omitted:

[0075] 00000000 ba a6 7b c6 56 a6 e5 1c c9 a3 3f aa c4 de 7d a0 |..{.V...?...}.|

[0076] 00000010 26 71 0f 43 5e 3e 32 94 e0 4c d1 9d 04 6f 86 d8 |&q.C^>2..L...o..|

[0077] 00000020 ee 79 36 0e 13 d1 ee 2586 33 dd b7 b0 33 c6 f3|.y6....%.3...3..

[0078] 0000003060 7a 2f 55 d3 64 e0 f364 ab 54 6b 86 bf 3b 5d|`z / U.d..d.Tk..;]

[0079] ……………………………………………………………………

[0080] 00003fc0ef b7 3a 70 b7 a6 ce 810b a7 7f 16 0c fb 37 77|..:p..........7w|

[0081] 00003fd0f7 5f d7 b2 84 ca 31 0ddc 7d 77 a8 5b d1 02 76|._....1..}w.[..v|

[0082] 00003fe073 6c 7c c4 24 5b 5e f481 3d 9e e7 f3 70 04 e2|sl|.$[^..=...p..|

[0083] 00003ff005 6b b5 7e ee e9 14 34 6d 2b 44 00 f0 0d 00 0a|.k.~...4m+D.....|

[0084] Note that the end of the file is (0d 00 0a) which needs to be removed before the crc32 calculation as described above, so the actual length of the second data string is the overall length of the file minus the length of the three bytes (0d 00 0a) at the end, i.e. 16384-3=16381. Finally, the crc32 of the 16381 bytes of data after processing is calculated, and then searched in the optimization table of the second feature section. If the matching scan is successful, the scan ends, otherwise the scan ends. The specific matching process is not described again, and the matching method of the first feature section is the same, i.e. first index the low 16 bits, then search the high 16 bit chain table. If there is no match, the scan ends directly, and if the high 16 bit chain table is matched, the data to be detected is malicious software.

[0085] In some other preferred embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to make the processor execute the steps of the method described in the above embodiments.

[0086] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0087] The above description is a detailed description of the preferred embodiments of the present application, but the embodiments are not intended to limit the scope of the patent application of the present application. Any equivalent changes or modifications made under the technical spirit of the present application should be included in the scope of the patent.

Claims

1. A method of efficient malware signature matching, characterized in that, The method comprises the following steps: constructing a feature library according to target data strings, wherein each data string corresponds to a feature value in the feature library; initializing the feature values in the feature library to the memory; receiving data to be detected, and calculating the effective length thereof; matching according to the length in the feature library, and when the matching is successful, matching the first feature segment of the data to be detected, otherwise ending the scanning process; when the first feature segment is matched successfully, matching the second feature segment of the data to be detected, and when the matching is successful, determining that the data to be detected is malicious software; initializing the feature values in the feature library to the memory specifically comprises: storing the low 16 bits of the feature values in the feature library as a first feature index array in an index table, and storing the high 16 bits as a second feature index array in a linked list array; associating the first feature index array with the second feature index array; matching the first feature segment of the data to be detected specifically comprises: calculating the crc32 hash value of the first feature segment, directly indexing the low 16 bits of the crc32 hash value in the index table corresponding to the first feature segment, when the indexing is successful, searching the high 16 bits of the crc32 hash value in the corresponding linked list, when the searching is successful, the first feature segment is matched successfully, and the next step is performed, otherwise the scanning is ended; matching the second feature segment of the data to be detected specifically comprises: calculating the crc32 hash value of the second feature segment, directly indexing the low 16 bits of the crc32 hash value in the index table corresponding to the second feature segment, when the indexing is successful, searching the high 16 bits of the crc32 hash value in the corresponding linked list, when the searching is successful, the second feature segment is matched successfully, and the data to be detected is malicious software, otherwise the scanning is ended; the starting offset position of the second feature segment is the same as that of the first feature segment, and the length is the effective byte number of the data.

2. The method of claim 1, wherein, constructing a matching feature library for target data strings comprises: extracting a first feature segment from the target matching data string, and calculating the crc32 hash value of the first feature segment to obtain a first hash value, combining the byte length of the first feature segment and the first hash value to form a first feature value; extracting a second feature segment from the target matching data string, and calculating the crc32 hash value of the second feature segment to obtain a second hash value, combining the byte length of the second feature segment and the second hash value to form a second feature value; combining the first feature value and the second feature value as the feature value of the target matching data.

3. The method of claim 2, wherein, When the low 16 bits are used as the first feature index array, the element data type of the first feature index array is a bool value, and each bool value of the first feature index array is associated with the second feature index array.

4. The method of claim 1, wherein, Before calculating the effective length of the data to be detected, the tail invalid bytes of the data to be detected need to be removed.

5. The method of claim 2, wherein, 4096 bytes are taken from the place with a data offset of 0 as the first feature segment.

6. A storage medium, characterized by A computer program is stored, and when the computer program is executed by a processor, the processor executes the steps of the efficient malicious software feature code matching method according to any one of claims 1-5.

Images