Fine-grained flow lossless compression method combined with multiple threads

Abstract

The invention relates to the field of traffic storage optimization and traffic data compression, in particular to a multithreading-combined lossless compression method for fine-grained traffic, which comprises the following steps of: acquiring a traffic data file, extracting a header triple based on the traffic data file, generating an identifier according to the header triple, and transmitting the identifier to a server; aggregating the traffic data files with the same identifier to obtain data streams, and generating a sorted data stream size table according to the data volume of the data streams; creating threads according to system computing power, distributing the data flow to the thread with the minimum thread load according to the data flow size table and the thread load condition, and outputting a data flow distribution scheme; and performing fine-grained characterization and serialization on each data stream to obtain an integer sequence and a byte sequence, selecting a compression processing method according to redundancy characteristics of the integer sequence and the byte sequence, obtaining a redundancy-eliminated sequence, and writing the redundancy-eliminated sequence into a compressed file. The invention aims to provide high compression efficiency to reduce the storage cost while ensuring the data precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of flow storage optimization and flow data compression, in particular, especially relates to a fine-grained flow lossless compression method combined with multi-thread. BACKGROUND

[0002] With the rapid development of Internet technology, the scale of the network is expanding, and the flow data in the Internet is increasing dramatically. These data contain valuable information that can be used for network measurement, resource optimization, and network security analysis tasks, so the data needs to be stored for subsequent business use. However, the large amount of data brings huge storage pressure to the flow storage system. In order to effectively reduce the storage cost, a suitable compression strategy can be used to identify and eliminate data redundancy, reduce the storage space requirement of data, and reduce the operating cost. In addition, compression processing can also speed up data transmission, optimize the utilization of network resources, and thus improve the overall performance of the system.

[0003] Currently, many network storage systems use general compression algorithms to process flow data, such as GZIP, LZMA, BZ2, etc. These algorithms are not limited to the content and format of the data. When processing data, the data is treated as a byte stream, a window is maintained, and the repeated content in the window is scanned and identified. The redundancy is represented by a shorter symbol or code, thereby reducing the storage space usage. However, as can be seen from the above redundancy processing flow, this type of algorithm has some defects. First, the general algorithm can only handle a single type of redundancy, and can only eliminate repeated byte segments in the byte stream. If the data has multiple and complex types of redundancy, the general algorithm cannot effectively identify and eliminate these redundancies, resulting in low compression efficiency. Second, if the redundancy of the data is not evenly distributed and the repeated data is not in the same window, the redundancy cannot be identified and eliminated. In addition, general compression algorithms usually compress the entire data, and each time the data is accessed, the entire compressed block must be decompressed to extract the required part, which not only increases the decompression overhead, but also reduces the system response speed.

[0004] Based on the above problems, the present application provides a fine-grained flow lossless compression method combined with multi-thread. SUMMARY

[0005] In view of the problems of low compression efficiency of general compression algorithms and the inability to support selective decompression, thereby making it difficult to effectively reduce the storage cost of massive flow data, a fine-grained flow lossless compression method combined with multi-thread is provided.

[0006] The technical means adopted by the present application are as follows:

[0007] A fine-grained flow lossless compression method combined with multi-thread, comprising the following steps:

[0008] S1. Collecting traffic data files, extracting header triplets based on the traffic data files, generating identifiers according to the header triplets, aggregating the traffic data files with the same identifier, obtaining data streams, and generating a sorted data stream size table according to the data volume of the data streams;

[0009] S2. Creating threads according to the system computing power, assigning data streams to threads with the smallest load according to the data stream size table and thread load conditions, outputting a data stream allocation scheme, and realizing multi-thread parallel compression;

[0010] S3. Fine-grained characterization and serialization of each data stream to obtain integer sequences and byte sequences, selection of compression processing methods for the redundant characteristics of the integer sequences and byte sequences, and writing of the sequences after eliminating redundancy into a compressed file.

[0011] The modified inferable sequence is processed using run-length encoding.

[0012] Further, the S1 step specifically includes:

[0013] S11. Collecting traffic data files, extracting header triplets based on the traffic data files, and the header triplets including source address, destination address and protocol type field value;

[0014] S12. Generating identifiers using the triplets splicing method;

[0015] S13. Aggregating data packets with the same identifier to form data streams;

[0016] S14. Statistics of the data volume of each data stream, and generation of a data stream size table sorted in descending order of data volume.

[0017] The modified inferable sequence is processed using run-length encoding.

[0018] Further, the S2 step specifically includes:

[0019] S21. Creating threads according to the system computing power;

[0020] S22. Managing the threads using a thread pool;

[0021] S23. Assigning data streams to threads with the smallest load according to the data stream size table and thread load conditions, and outputting a load-balanced data stream allocation scheme.

[0022] The modified inferable sequence is processed using run-length encoding.

[0023] Further, the S3 step specifically includes:

[0024] S31. Fine-grained characterization is performed on each data stream to represent each data packet in the data stream in the form of fields and payloads;

[0025] S32. Serialization is performed on each data stream to merge the same fields and payloads in the data stream into an integer sequence and a byte sequence;

[0026] S33. A compression processing method is selected according to the redundancy characteristics of the integer sequence and the byte sequence, the redundancy characteristics including a continuous repeated field sequence, a fixed value field sequence, an incremental field sequence, a timestamp field sequence, an inferable field sequence, and a remaining field sequence;

[0027] S34. The sequence after the redundancy is eliminated is obtained and written into a compressed file.

[0028] Run-length encoding processing is performed on the modified inferable sequence.

[0029] Further, the compression processing method for the repeated field sequence is:

[0030] The repeated field sequence is processed by using a mask combined with run-length encoding;

[0031] In a conventional run-length encoding manner, the processed repeated field sequence is divided into a plurality of repeated equal data segments, if the storage space required for storing the data segments in the format of run value-run length is smaller than the original storage space required by the data segments, the run value-run length is used for representation, and the corresponding mask position is 1, otherwise, the original format is used for storage, and the corresponding mask position is 0, and the encoding rule is:

[0032] S={s1,s2,…,s n}→R′={bitmask,r′1,r′2,…,r′ m}

[0033] bitmask={b1b2…b m}

[0034] b k =0 or 1,k∈[1,m]

[0035] r′ k ={(v k ,l k )}or{v k},k∈[1,m],v k ∈S,l k ∈N +

[0036] Wherein, S is the repeated field sequence, n is the sequence length, s iR' is the result of run-length encoding, m is the number of data segments, r' is the i-th data in the sequence of repeated fields k v is the encoding form of data segment k l is the value of the k-th run k bitmask is the mask, b is the length of the k-th run k b is 0 or 1, indicating the encoding form of the data segment, 0 means that the data segment is stored in the original value, and 1 means that the data segment is stored in the format of run value-run length, N + N is a positive integer.

[0037] The modified inferable sequence is processed by run-length encoding.

[0038] Further, the compression processing method for the fixed value field sequence is:

[0039] The fixed value sequence is stored in a file name, and the file name format is source IP-destination IP-protocol.bin.

[0040] The modified inferable sequence is processed by run-length encoding.

[0041] Further, the compression processing method for the incremental field sequence is:

[0042] The incremental field sequence is processed by using the difference calculation combined with the dictionary compression algorithm, and the encoding rule is:

[0043] S={s1,s2,…,s n}→S'={s1,s2-s1,…,s n -s n-1}

[0044] Z=LZMA(S')

[0045] Where S is the incremental field sequence, n is the sequence length, s i is the i-th data in the sequence of repeated fields, S' is the incremental field sequence after difference, LZMA(·) is a dictionary-based compression algorithm, and Z is the byte stream after compression processing.

[0046] The modified inferable sequence is processed by run-length encoding.

[0047] Further, the compression processing method for the timestamp field sequence is:

[0048] The timestamp field sequence is processed by difference;

[0049] The differential processed timestamp field sequence is expressed in the form of control bit-data bit, the control bit records the data encoding mode, and the data bit records the actual data, and the encoding rule is as follows:

[0050] If the actual data is 0, the control bit is expressed in one-bit binary;

[0051] If the actual data is greater than or equal to 1 and less than or equal to 256, the control bit is expressed in two-bit binary, and the data bit is expressed in eight-bit binary;

[0052] If the actual data is greater than or equal to 257 and less than or equal to 1280, the control bit is expressed in three-bit binary, and the data bit is expressed in ten-bit binary;

[0053] If the actual data is greater than or equal to 1281 and less than or equal to 5376, the control bit is expressed in four-bit binary, and the data bit is expressed in twelve-bit binary;

[0054] When the actual data is greater than 5376, the encoding method of the data bit includes a first method, a second method, a third method, and a fourth method, the method with the smallest storage space is selected by comparing the storage spaces of the first method, the second method, the third method, and the fourth method, and different control bits are used to distinguish the corresponding methods;

[0055] The first method is to use an eight-bit binary mask to represent whether each byte of data is 0, and to append non-zero byte data after the mask;

[0056] The second method is to use a six-bit binary mask to record the number of consecutive zeros starting from the first bit of the actual data, and to append non-zero bit data after the mask;

[0057] The third method is to use a three-bit binary mask to record the number of consecutive zero bytes starting from the first bit of the actual data, and to use a three-bit binary mask to record the number of consecutive zero bytes starting from the last bit of the actual data, and to append non-zero byte data of the actual data after the mask;

[0058] The fourth method is to use a six-bit binary mask to record the number of consecutive zeros starting from the last bit of the actual data, and to append non-zero bit data before the mask.

[0059] The modified inferable sequence is processed by run-length encoding.

[0060] Further, the compression processing method of the inferable field sequence is as follows:

[0061] The inferable field sequence is modified to zero by the method of in-packet verification;

[0062] The modified inferable sequence is subjected to run-length encoding processing.

[0063] Further, the compression processing method for the remaining field sequence is:

[0064] The remaining field sequence is converted into a byte stream.

[0065] The byte stream is processed by using a general dictionary compression algorithm to output a compressed byte stream.

[0066] Compared with the prior art, the present application has the following advantages:

[0067] 1. The present application provides a fine-grained flow lossless compression method combined with multi-threading, which aggregates redundant data and isomorphic data through flow classification, facilitates the identification and elimination of redundancy by the compression algorithm, and the subsequent compression task is performed for each data flow, so that the data flow has an independent compressed file, and when decompressing, the corresponding data flow can be decompressed according to the needs, avoiding decompressing the entire compressed file and speeding up the system response.

[0068] 2. The present application performs fine-grained characterization and serialization on data flow with multi-modal redundancy, converts it into multiple sequences with single redundancy, and designs processing methods according to the redundancy characteristics to fully eliminate various redundancies within the flow and improve the utilization rate of storage space.

[0069] 3. The present application introduces flow classification, serialization, and multiple redundancy processing methods, which may result in slower compression speed. To this end, the present application uses a multi-threaded approach to parallel processing of data flow compression and optimizes the allocation of data flow and threads through dynamic programming algorithm to achieve load balancing and speed up the overall compression speed.

[0070] 4. Compared with general compression algorithms, the compression rate of the present application is significantly better than that of general compression algorithms, which can effectively reduce storage cost consumption, and the compression speed and decompression speed are close to the fastest general compression algorithm, and the present application supports selective decompression, and the time required to restore part of the data flow is significantly better than other general algorithms, which can effectively improve the response speed of the system.

[0071] Based on the above reasons, the present application can be widely popularized in the fields of flow storage optimization and flow data compression, and can be used in scenarios such as massive flow data storage, monitoring, and analysis, effectively reducing system storage costs. BRIEF DESCRIPTION OF DRAWINGS

[0072] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the following embodiment or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0073] Figure 1 Figure 1 is a schematic diagram of the overall architecture of the present application.

[0074] Figure 2 Figure 2 is a graph comparing the compression efficiency of the present application with a general compression algorithm.

[0075] Figure 3 Figure 3 is a graph comparing the compression time of the present application with a general compression algorithm.

[0076] Figure 4 Figure 4 is a subgraph comparing the compression time of the present application with a general compression algorithm.

[0077] Figure 5 Figure 5 is a graph comparing the decompression time of the present application with a general compression algorithm.

[0078] Figure 6 Figure 6 is a subgraph comparing the decompression time of the present application with a general compression algorithm.

[0079] Figure 7 Figure 7 is a graph comparing the decompression data consumption time of the present application with a general compression algorithm.

[0080] Figure 8 Figure 8 is a subgraph comparing the decompression data consumption time of the present application with a general compression algorithm. DETAILED DESCRIPTION

[0081] In order to make the technical personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0082] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0083] As Figure 1As shown, the present application provides a fine-grained flow lossless compression method combined with multi-threading, and the steps are specifically as follows:

[0084] S1. Collect flow data files, extract header triplets based on the flow data files, generate identifiers according to the header triplets, aggregate flow data files according to the same identifiers, obtain data streams, and generate a sorted data stream size table according to the data amount of the data streams.

[0085] Specifically, the S1 step is:

[0086] S11. Collect flow data files, extract header triplets based on the flow data files, and the header triplets include Source IP (source address), Destination IP (destination address), and Protocol (protocol type) field values.

[0087] S12. Generate FlowID (identifier) by splicing the header triplets. Classify the data streams according to the identifiers, so that redundant data and isomorphic data are aggregated together, which facilitates the identification and elimination of redundancy by the compression algorithm. The subsequent compression task is performed for each data stream, and the data stream has an independent compression file, which provides support for selective compression.

[0088] S13. Aggregate data packets with the same identifier to form data streams.

[0089] S14. Count the data amount of each data stream and generate a Payload_Table (data stream size table) arranged in descending order of data amount, which facilitates the subsequent allocation of threads and data streams.

[0090] S2. Create threads according to the system computing power, allocate data streams to the thread with the smallest load according to the data stream size table and thread load, output the data stream allocation scheme, achieve load balancing, and parallelly compress the data streams to speed up the overall speed.

[0091] Specifically, the S2 step is:

[0092] S21. Create threads according to system computing power. The number of threads is determined according to the system computing power and the type of task. The task is a CPU-intensive task. If the number of CPU cores is n, the number of threads should be set in the interval [n+1, n+2]. Ideally, the number of threads is equal to the number of cores, which is the best effect. It can make full use of each core and avoid frequent thread context switching. However, if there is a block, the operating system will reclaim the CPU resources of the thread, and the corresponding CPU core will be idle. Here, "+1 / +2" is to allow the remaining threads to continue to use CPU resources when a thread is blocked, preventing CPU idling. And this candidate value cannot be too large, otherwise it will cause frequent thread context switching due to too many threads.

[0093] S22. Manage threads using thread pool. The threads before compression are reused when the traffic files captured in different collection periods are compressed, avoiding the overhead caused by frequent thread creation / destruction.

[0094] S23. According to the data flow size table and the thread load, the data flow is allocated to the thread with the smallest load, and the load-balanced data flow allocation scheme is output. Specifically, the threads and data flows are pre-allocated to reduce the overhead of task scheduling. According to Payload_Table in S1, the data flow is allocated to the thread with the smallest load in turn, ensuring that the total amount of data flow allocated to each thread is basically consistent, achieving load balancing, avoiding thread idling or overloading, reducing task execution time fluctuation, and thus improving the overall compression speed.

[0095] Further, the above process is specifically illustrated by example. Assume that there are 10 data flows, and their sizes after sorting are 10MB, 10MB, 9MB, 8MB, 7MB, 7MB, 6MB, 6MB, 5MB, and 1MB, respectively. There are three threads, thread1, thread2, and thread3. The allocation process is as follows:

[0096] For the first 10MB data flow, in the initial state, the loads of thread1, thread2, and thread3 are all 0. At this time, any thread can be selected for allocation. Here, it is allocated to thread1. At this time, the load of thread1 becomes 10MB, i.e. thread1:{10MB}, thread2:{0MB}, and thread3:{0MB}.

[0097] Next, a second 10MB data stream is allocated, and since threadl already has a load of 10MB and thread2 and thread3 have a load of 0, the data stream is allocated to thread2, and the load of thread2 becomes 10MB, i.e., threadl : {10MB}, thread2 : {10MB}, thread3 : {0MB}.

[0098] For a 9MB data stream, the load of threadl and thread2 is 10MB and the load of thread3 is 0, so the data stream is allocated to thread3, which has the smallest load, and the load of thread3 becomes 9MB, i.e., threadl : {10MB}, thread2 : {10MB}, thread3 : {9MB}.

[0099] For an 8MB data stream, the load of threadl and thread2 is 10MB and the load of thread3 is 9MB, so the data stream is allocated to thread3, which has the smallest load, and the load of thread3 becomes 17MB, i.e., threadl : {10MB}, thread2 : {10MB}, thread3 : {9MB, 8MB}.

[0100] For a first 7MB data stream, the loads of the three threads are compared, and the load of thread3 is the largest and the loads of threadl and thread2 are the same and relatively small, so one of them, e.g., threadl, is selected, and the load of threadl becomes 17MB, i.e., threadl : {10MB, 7MB}, thread2 : {10MB}, thread3 : {9MB, 8MB}.

[0101] For a second 7MB data stream, the loads of threadl and thread3 are large, so the data stream is allocated to thread2, and the load of thread2 becomes 17MB, i.e., threadl : {10MB, 7MB}, thread2 : {10MB, 7MB}, thread3 : {9MB, 8MB}.

[0102] Next, a first 6MB data stream is allocated, and the loads of the three threads are the same, so one of them, e.g., threadl, is selected, and the load of threadl becomes 23MB, i.e., threadl : {10MB, 7MB, 6MB}, thread2 : {10MB, 7MB}, thread3 : {9MB, 8MB}.

[0103] For the second data stream of 6MB, the thread with the minimum load (thread2) is selected again, and the load of thread2 becomes 23MB after allocation, i.e., thread1:{10MB, 7MB, 6MB}, thread2:{10MB, 7MB, 6MB}, thread3:{9MB, 8MB}.

[0104] For the data stream of 5MB, thread3 is selected after comparison, and the load of thread3 becomes 22MB after allocation, i.e., thread1:{10MB, 7MB, 6MB}, thread2:{10MB, 7MB, 6MB}, thread3:{9MB, 8MB, 5MB}.

[0105] Finally, the data stream of 1MB is allocated, and the thread with the minimum load (thread3) is selected again, and the load of thread3 becomes 23MB after allocation, i.e., thread1:{10MB, 7MB, 6MB}, thread2:{10MB, 7MB, 6MB}, thread3:{9MB, 8MB, 5MB, 1MB}.

[0106] S3. Fine-grained characterization and serialization are performed on each data stream to obtain integer sequences and byte sequences, and a compression processing method is selected according to the redundancy characteristics of the integer sequences and byte sequences to obtain the sequences after eliminating redundancy and write them into a compressed file.

[0107] S31. Fine-grained characterization is performed on each data stream to represent each data packet in the data stream in the form of fields and loads.

[0108] S32. Serialization is performed on each data stream to merge the same fields and loads in the data stream into integer sequences and byte sequences, the redundancy within each sequence is single, and the redundancy between sequences is different.

[0109] S33. A compression processing method is selected according to the redundancy characteristics of the integer sequences and byte sequences, and the redundancy characteristics include continuous repeated field sequences, fixed value field sequences, incremental field sequences, timestamp field sequences, inferable field sequences, and remaining field sequences.

[0110] In the same data stream, the Destination / Source MAC Address, Type, Version, Internet Header Length, Type of Service, Flags, Fragment Offset, Time to Live, Data Offset, Urgent Pointer and other field values are basically unchanged, and the sequence composed of the same values will appear continuously. The compression processing method for the repeated field sequence is as follows:

[0111] The repeated field sequence is processed by using a mask combined with run-length encoding. Run-length encoding regards continuous same data as a run, and represents it as a run value-run length form. The entire sequence will be encoded as several run value-run length forms, which greatly reduces the consumption of storage space. The processing rules of run-length encoding are as follows:

[0112] S={s1,s2,…,s n}→R={r1,r2,…,r m}

[0113] r k ={(v k ,l k )},v k ∈S,l k ∈N +

[0114] Wherein, S is the repeated field sequence, n is the sequence length, s i is the i-th data in the repeated field sequence, R is the result of run-length encoding, m is the number of data segments, r k is the encoding form of the data segment, v k is the value of the k-th run (continuous same data value), and l k is the length of the k-th run (continuous repetition times).

[0115] Suppose there is an original repeated field sequence:

[0116] S={1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,3,3,3,3,3,3,3,3,3}

[0117] Wherein each data occupies 1 Byte, and the sequence occupies 24 Bytes. After run-length encoding, the result is:

[0118] R={(1,9),(2,6),(3,9)}

[0119] Wherein the run length adopts 1 byte, the run value adopts 1 byte, then the sequence after run length encoding occupies 6 bytes, greatly reduces the storage space, but the traditional run length encoding has a problem, when the run length is short, the data segment is encoded as the form of run value-run length, which may increase the storage occupation, for example, there is a sequence:

[0120] S={1,1,1,2,3,3,1,2,2,3,2,3,3,3}

[0121] The sequence occupies 14 bytes run length encoding result is:

[0122] R={(1,3),(2,1),(3,2),(1,1),(2,2),(3,1),(2,1),(3,3)}

[0123] The encoded sequence occupies 16 bytes, which increases the storage space, because the storage space required by such (1,1), (2,1), (3,1) encoding is greater than the storage space of the original data, in order to solve this problem, the run length encoding is improved, a mask is added in the encoding header, according to the traditional run length encoding mode, the sequence is divided into several equal repeated data segments, if the storage space required by the data segment according to the run value-run length (repetition value-repetition length) format is less than the original storage space required by the data segment, then the run value-run length is used to represent, and the corresponding mask position is 1, otherwise the original format is used to store, and the corresponding mask position is 0, the overall encoding rule is as follows:

[0124] S={s1,s2,…,s n}→R′={bitmask,r′1,r′2,…,r′ m}

[0125] bitmask={b1b2…b m}

[0126] b k =0 or 1,k∈[1,m]

[0127] r′ k ={(v k ,l k )}or{v k},k∈[1,m],v k ∈S,l k ∈N +

[0128] Wherein, S is a repeated field sequence, n is the sequence length, s iR' is the result of run-length encoding, m is the number of data segments, r' is the i-th data in the repeated field sequence k v is the encoding form of the data segment k l is the value of the k-th run k bitmask is the mask, b is the length of the k-th run k b is 0 or 1, indicating the data segment encoding mode, 0 represents that the data segment is stored in the original value, and 1 represents that the data segment is stored in the run value-run length format, N + N is a positive integer (run length is greater than or equal to 1).

[0129] For example, the sequence is:

[0130] S={1,1,1,2,3,3,1,2,2,3,2,3,3,3}

[0131] The sequence occupies 14 bytes, and the improved run-length encoding result is:

[0132] R'={169,(1,3),2,(3,2),1,(2,2),3,2,(3,3)}

[0133] The encoded sequence occupies 13 bytes, including 1 byte of mask, the mask value is 169, the binary representation is 10101001, 1 indicates that the data segment is represented by run value-run length, a total of 4, occupying 8 bytes, 0 indicates that the data segment is represented by the original value, a total of 4, occupying 4 bytes, reducing the storage space occupation, solving the problem of traditional run-length encoding.

[0134] For the sequence generated by the Source / Destination Address, Protocol and other fields, these fields are part of the identifier in flow classification, and the values of these fields in the same data stream are the same. The compression processing method for the fixed value field sequence is:

[0135] The corresponding sequence presents a fixed and unchanged redundancy characteristic, and the application does not store in the compressed file, but stores the fixed value sequence in the file name. The file name format is Source Address-Destination Address-Protocol.bin. Through the name of the compressed file, the values of the redundant fields can be directly restored when decompressing.

[0136] In the same data stream, the values of Identification, Sequence Number, Acknowledgment Number and other fields generally show a regular increasing trend, and direct use of dictionary compression algorithm cannot identify and eliminate such regular change redundancy.

[0137] The compression processing method for the increasing field sequence is as follows: for the sequence generated by these fields, the difference calculation combined with the dictionary compression algorithm is used to process the field sequence in the application, and the encoding rule is as follows:

[0138] S={s1,s2,…,s n}→S′={s1,s2-s1,…,s n -s n-1}

[0139] Z=LZMA(S′)

[0140] Wherein, S is the increasing field sequence, n is the sequence length, s i is the i-th data in the increasing field sequence, S' is the difference processed increasing field sequence, the first-order difference calculation can simplify the data mode, convert the data value into relative change (i.e. the difference between adjacent data), thereby significantly reducing the diversity of data, facilitating compression processing, LZMA(·) is a dictionary-based compression algorithm, which can identify and eliminate the redundancy in the sequence after difference processing, so as to improve the storage space utilization, and Z is the byte stream after compression processing.

[0141] The redundancy of the Timestamp (time stamp) field generated sequence is relatively special, the number of bytes occupied by the field itself is relatively large (8 bytes), but the difference between the front and rear fields is much smaller than the range that can be represented by the number of bytes of the field, the application adopts the way of difference calculation combined with adaptive variable length coding to process the timestamp sequence, and the processing method is as follows:

[0142] First, the timestamp field sequence is processed by difference.

[0143] Next, the difference processed timestamp field sequence is represented in the form of control bit-data bit, the control bit records the data encoding mode, and the data bit records the actual data, and the encoding rule is as follows:

[0144] If the actual data is 0, a one-bit binary (1b'0) is used to represent the control bit.

[0145] If the actual data is in [1, 256], a two-bit binary (2b'10) is used to represent the control bit, and an eight-bit binary is used to represent the data bit.

[0146] If the actual data ∈ [257, 1280], the control bit is represented by three binary digits (3b'110), and the data bit is represented by ten binary digits.

[0147] If the actual data ∈ [1281, 5376], the control bit is represented by four binary digits (4b'1110), and the data bit is represented by twelve binary digits.

[0148] When the actual data is greater than 5376, the encoding method of the data bit includes a first method, a second method, a third method, and a fourth method. By comparing the storage spaces of the first method, the second method, the third method, and the fourth method, the method with the smallest storage space is selected to represent the data bit, and the control bit is used to distinguish the corresponding method. The control bit of the first method is (5b'11110), the control bit of the second method is (6b'111110), the control bit of the third method is (7b'1111110), and the control bit of the fourth method is (7b'1111111).

[0149] The first method is to use an eight-bit binary mask to represent whether each byte of data is 0, and to append non-zero byte data after the mask. Specifically, for an 8-byte data, an 8-bit binary mask is used to represent whether each byte of the data is all 0, and if it is 0, the mask position is 0, and if it is not 0, the mask is set to 1. Then the non-zero byte data (0x11 and 0x12) is appended after the binary mask. The result of the data encoding is 0b000001010001000100010010, which converts the original 8-byte (64-bit) data into 3-byte (24-bit) data, reducing the space occupation.

[0150] The second method is to use a six-bit binary mask to record the number of consecutive zeros from the first bit of the actual data, and to append the non-zero data after the mask. Specifically, for an 8-byte data, a 6-bit binary mask is used to record the number of consecutive zeros from the first bit of the data, and the non-zero data is appended after the mask. For example, the data 0x0000000000001112 (0x11 in binary representation 00010001) has 51 consecutive zeros from the first bit, which is represented in binary as (6b'110011). The non-zero data is 0b1000100010010. The result of the data encoding is 0b1100111000100010010, which converts the original 8-byte (64-bit) data into 19-bit data, reducing the space occupation.

[0151] The third method is: using a three-bit binary mask to record the number of continuous zero bytes from the first bit of the actual data, using a three-bit binary mask to record the number of continuous zero bytes from the last bit of the actual data, and appending the non-zero byte data of the actual data after the mask. Specifically, for an 8-byte data, a three-bit binary mask is used to record the number of continuous zero bytes from the first bit of the data, a three-bit binary mask is used to record the number of continuous zero bytes from the last bit of the data, and the non-zero bit of the data is appended after the mask. For example, the data is 0x0000000101000000, the number of continuous zero bytes from the first bit of the data is 3, which is represented in binary as (3b'011), the number of continuous zero bytes from the last bit of the data is 3, which is represented in binary as (3b'011), and the non-zero bit of the data is 0b0000000100000001. The encoding result of the data is 0b0110110000000100000001, which converts the original 8-byte (64-bit) data into 22-bit data, reducing the space occupation.

[0152] The fourth method is: using a six-bit binary mask to record the number of continuous zero bits from the last bit of the actual data, and appending the non-zero bit data of the actual data before the mask. Specifically, for an 8-byte data, a six-bit binary mask is used to record the number of continuous zero bits from the last bit of the data, and the non-zero bit data before the mask is appended after the mask. For example, the data is 0x1100000000000000 (0x11 in binary representation 00010001), the number of continuous zero bits from the last bit of the data is 56, which is represented in binary as (6b'111000), and the non-zero bit before the mask is 0b00010001. The encoding result of the data is 0b11100000010001, which converts the original 8-byte (64-bit) data into 14-bit data, reducing the space occupation.

[0153] For the fields of Original Length, Total Length, Header Checksum, Length, Checksum, etc., most of these fields can be calculated from other fields of the data packet, payload, and the calculation rules are as follows:

[0154] Header Checksum needs to split the IP header into multiple parts according to 16-bit words, and all 16-bit words are added together. If the result overflows, the high bit is added back to the low bit, and finally the complement code of the sum result is taken.

[0155] Checksum needs to construct a pseudo header temporarily, which is used to ensure end-to-end data integrity, including source IP, destination IP, reserved field, protocol number, TCP length, and the pseudo header and TCP data are regarded as a whole, which is split according to 16-bit words, all 16-bit words are added, if the result overflows, the high bit carry is added back to the low bit, and finally the sign of the sum is taken.

[0156] The Total Length and Length fields can be inferred from the Original Length field, the value of the Original Length field minus 14 (the length of the link layer header) is the value of the Total Length field, and the value of the Length field is equal to the value of the Original Length field minus 14 minus the IP header length, and the Original Length is equal to the Captured Length in most cases.

[0157] The compression processing method for the above inferable field sequence is:

[0158] First, the intra-packet check is performed, if the field value can be inferred from the packet information, the value is modified to '0', and the modified sequence as a whole shows the characteristics of slow change (most of them are '0').

[0159] Then, the run-length encoding processing is performed on the modified sequence, and when decompressing, only the sequence is restored, and then the values of '0' in the decompressed sequence are changed to the original values according to the calculation of the packet data.

[0160] The remaining sequence does not show obvious redundancy types, and the general compression algorithm is used to process this kind of sequence to eliminate the possible redundancy as much as possible, and the compression processing method for the remaining field sequence is:

[0161] First, the remaining field sequence is converted into a byte stream. The integer value in the integer sequence is first converted into the corresponding byte, and then spliced into a byte stream, and the byte sequence is directly spliced to obtain a byte stream.

[0162] Then, the general dictionary compression algorithm is used to process the byte stream to output the compressed byte stream. The repeated byte segment is mapped to a shorter index or identifier, thereby eliminating data redundancy and improving storage space utilization.

[0163] S34. The sequence after eliminating redundancy is obtained and written into a compressed file. The file name is like the fixed field value sequence processing method, different streams are written into different files, and when decompressing later, only the corresponding data stream can be decompressed according to the needs, instead of decompressing all the data, and then finding out the required data stream according to the needs.

[0164] S1 is the data stream classification, which provides support for subsequent compression processing and selective decompression. S3 is the specific compression processing method for the data stream. The combination of S1 and S3 can solve the above problems, while S2 is designed to optimize the overall compression process and achieve fast compression.

[0165] In the embodiment of the present invention, multiple indicators are used for comparative analysis, including compression ratio, compression time, decompression time, and time required to recover part of the data. The compression ratio refers to the ratio of the initial size of the data to the compressed size. The higher the compression ratio, the better the compression performance and the lower the storage cost. The compression time and decompression time refer to the time required to complete the compression and decompression operations. The shorter the time required, the higher the data processing efficiency and the faster the overall data compression and recovery. The time required to recover part of the data refers to the time required to recover the required data (not all data) according to business needs, and is an important indicator of the rapid response of the system.

[0166] The comparison algorithm selects the currently commonly used general compression algorithms, including GZIP, LZMA, and BZIP2. These three algorithms are widely used in traffic data storage systems. The algorithm introduction is as follows:

[0167] GZIP: Based on a dictionary compression algorithm, it can compress files and generate compressed packages in the .gz format. Due to its fast compression speed and moderate compression ratio, it is widely used in scenarios such as file compression, web page transmission, and log archiving.

[0168] LZMA: It is a high-compression algorithm that combines a dictionary compression algorithm and a Markov chain model. It can generate compressed packages in .xz or .7z formats. It has a high compression ratio but a slow speed. It is often used in scenarios such as software distribution, backup, and big data storage.

[0169] BZIP2 (BZ2): A compression tool based on the Burrows-Wheeler transform (BWT), it is typically used to compress single files. It produces compressed files in the .BZ2 format. Its compression ratio is generally higher than GZIP's, while its compression speed is moderate. It's suitable for scenarios that require a high compression ratio but are not speed-sensitive, such as file backup and archiving.

[0170] The traffic data was captured randomly by the laboratory server at different time periods. Wireshark was used to capture the traffic link frames (transmission form of traffic data) sent / received by the host network card within a certain time interval. A total of 8 groups of traffic files were captured and saved. The data (number and size of data packets) are shown in Table 1:

[0171] Table 1

[0172]

[0173]

[0174] Figure 2 The compression rates of different algorithms on different data sets are shown, and it can be seen that the compression of the application performs better than the general compression algorithm and leads other algorithms on multiple data sets.

[0175] Figure 3 The compression time required by different compression algorithms on different data sets is shown, and the difference in algorithm compression time is larger for the first two data sets compared with the latter data sets due to the smaller data volume, Figure 3 The results may not be very intuitive, therefore, the subgraph ( Figure 4 ) is used to more clearly show the comparison of compression time of different compression algorithms on the first two data sets. Figure 4 For the compression time required by different compression algorithms on the first two data sets, combined with Figure 3 It can be seen that the application has a faster compression speed, close to GZIP, performs superiorly, and significantly leads LZMA and BZIP2, and can realize fast compression.

[0176] Figure 5 The decompression time required by different compression algorithms on different data sets is shown, and the difference in algorithm decompression time is larger for the first two data sets compared with the latter data sets due to the smaller data volume, Figure 5 The results may not be very intuitive, therefore, the subgraph ( Figure 6 ) is used to more clearly show the comparison of compression time of different compression algorithms on the first two data sets. Figure 6 For the decompression time required by different compression algorithms on the first two data sets, combined with Figure 5 It can be seen that the application has a faster decompression speed, close to GZIP, performs superiorly, and significantly leads LZMA and BZIP2, and can realize fast decompression.

[0177] Figure 7 The time required for recovering part of data by different compression algorithms on different data sets is shown, and the difference in time required for recovering part of data is larger for the first two data sets compared with the latter data sets due to the smaller data volume, Figure 7 The results may not be very intuitive, therefore, the subgraph ( Figure 8 ) is used to more clearly show the comparison of compression time of different compression algorithms on the first two data sets. Figure 8 For the time required for recovering part of data by different compression algorithms on the first two data sets, combined with Figure 7 It can be seen that the application supports selective decompression, and the time consumption for recovering part of data is far less than GZIP, LZMA and BZIP2, which can provide support for fast response of the system.

[0178] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A fine-grained traffic lossless compression method combined with multithreading, characterized in that: The following steps are involved: S1. Collect traffic data files, extract the header triples based on the traffic data files, generate identifiers based on the header triples, aggregate the traffic data files with the same identifiers to obtain data streams, and generate a sorted data stream size table based on the amount of data streams; S2. Create threads based on system computing power, assign data streams to threads with the smallest thread loads based on the data stream size table and thread load, output a data stream allocation plan, and implement multi-threaded parallel compression; S3. Perform fine-grained characterization and serialization on each data stream to obtain integer sequences and byte sequences. Select a compression processing method based on the redundant characteristics of the integer and byte sequences to obtain a sequence after eliminating redundancy and write it into a compressed file.

2. The fine-grained traffic lossless compression method combined with multithreading according to claim 1 is characterized in that: The S1 step specifically includes: S11. Collect traffic data files and extract the header triplet based on the traffic data files. The header triplet includes the source address, destination address, and protocol type field values. S12. Using triple concatenation, generate an identifier for the head triple; S13 aggregates packets with the same identifier to form a data stream; S14. Count the data volume of each data flow and generate a data flow size table sorted in descending order of data volume.

3. The fine-grained traffic lossless compression method combined with multithreading according to claim 1 is characterized in that: The S2 step specifically includes: S21. Create threads based on system computing power; S22. Using a thread pool to manage the threads; S23. According to the data flow size table and the thread load, the data flow is allocated to the thread with the smallest thread load, and a load-balanced data flow allocation solution is output.

4. The fine-grained traffic lossless compression method combined with multithreading according to claim 1 is characterized in that: The S3 step specifically includes: S31. Perform fine-grained characterization on each data flow, representing each data packet in the data flow in the form of fields and payloads; S32. Serialize each data stream, merge the same fields and payloads in the data stream, and convert them into integer sequences and byte sequences; S33. Select a compression processing method for the redundant characteristics of integer sequences and byte sequences, wherein the redundant characteristics include a continuously repeated field sequence, a fixed value field sequence, an incrementing field sequence, a timestamp field sequence, an inferable field sequence, and other field sequences; S34. Obtain the sequence after eliminating redundancy and write it into a compressed file.

5. The fine-grained traffic lossless compression method combined with multithreading according to claim 4 is characterized in that: The compression processing method for the repeated field sequence is: Using mask combined with run-length encoding to process the repeated field sequence; According to the traditional run-length encoding method, the processed repeated field sequence is divided into several repeated equal data segments. If the storage space required to store the data segment in the format of run value-run length is less than the original storage space required by the data segment, the run value-run length representation is used and the corresponding mask position is set to 1. Otherwise, the original format is used for storage and the corresponding mask position is set to 0. The encoding rules are as follows: S={s1,s2,…,s n }→R′={bitmask,r′1,r′2,…,r′ m } bitmask={b1b2…b m } b k =0 or 1,k∈[1,m] r′ k ={(v k ,l k )}or{v k },k∈[1,m],v k ∈S,l k ∈N + Among them, S is the repeated field sequence, n is the sequence length, s i is the i-th data in the repeated field sequence, R′ is the result after run length encoding, m is the number of data segments, r′ k is the encoding form of the data segment, v k is the value of the k-th run, l k is the length of the k-th run, bitmask is the mask, b k It is 0 or 1, indicating the data segment encoding mode. If it is 0, it means that the data segment is stored in the original value format. If it is 1, it means that the data segment is stored in the format of run value-run length. + Is a positive integer.

6. The fine-grained traffic lossless compression method combined with multithreading according to claim 4 is characterized in that: The compression processing method for the fixed value field sequence is: The fixed value sequence is stored in a file name, and the file name format is source IP-destination IP-protocol.bin.

7. The fine-grained traffic lossless compression method combined with multithreading according to claim 4 is characterized in that: The compression processing method for the incremental field sequence is: The base increment field sequence is processed by using differential calculation combined with a dictionary compression algorithm. The encoding rule is: S={s1,s2,…,s n }→S′={s1,s2-s1,…,s n -s n-1 } Z=LZMA(S′) Among them, S is the increasing field sequence, n is the sequence length, s i is the i-th data in the increasing field sequence, S′ is the increasing field sequence after difference, LZMA(·) is the dictionary-based compression algorithm, and Z is the byte stream after compression.

8. The fine-grained traffic lossless compression method combined with multithreading according to claim 4 is characterized in that: The compression processing method for the timestamp field sequence is: Performing differential processing on the timestamp field sequence; The timestamp field sequence after the differential processing is represented in the form of control bit-data bit, where the control bit records the data encoding method and the data bit records the actual data. The encoding rules are as follows: If the actual data is 0, use one binary bit to represent the control bit; If the actual data is greater than or equal to 1 and less than or equal to 256, two bits of binary are used to represent the control bit and eight bits of binary are used to represent the data bit; If the actual data is greater than or equal to 257 and less than or equal to 1280, three binary digits are used to represent the control bit and ten binary digits are used to represent the data bit; If the actual data is greater than or equal to 1281 and less than or equal to 5376, four binary bits are used to represent the control bit and twelve binary bits are used to represent the data bit; When the actual data is greater than 5376, the data bit encoding methods include the first method, the second method, the third method, and the fourth method. By comparing the storage space of the first method, the second method, the third method, and the fourth method, the method with the smallest storage space is selected to represent the data bit, and different control bits are used to distinguish the corresponding methods; The first method is: using an eight-bit binary mask to indicate whether each byte data is 0, and appending non-zero byte data after the mask; The second method is: using a six-bit binary mask to record the number of consecutive zero bits starting from the first bit of the actual data, and appending the non-zero bits after the actual data to the mask; The third method is: using a three-bit binary mask to record the number of bytes that are consecutively zero starting from the first bit of the actual data, and using a three-bit binary mask to record the number of bytes that are consecutively zero starting from the last bit of the actual data, and appending the non-zero bytes of the actual data after the mask; The fourth method is: using a six-bit binary mask to record the number of bits that are consecutively zero starting from the last bit of the actual data, and appending the non-zero bits in front of the actual data to the back of the mask.

9. The fine-grained traffic lossless compression method combined with multithreading according to claim 4 is characterized in that: The compression processing method for the inferable field sequence is: By using the in-packet check method, the inferred field sequence is modified to zero; Run-length encoding is applied to the modified inferable sequence.

10. The fine-grained traffic lossless compression method combined with multithreading according to claim 4, characterized in that: The compression processing method for the remaining field sequences is: Converting the remaining field sequence into a byte stream; Use a general dictionary compression algorithm to process the byte stream and output a compressed byte stream.

Images