A real-time flow retention and batch extraction method

Abstract

The application relates to a real-time flow retention and batch extraction method, which adopts a brand-new design logic and is designed in a unified logic scheduling relationship for the retention and extraction of network data flow. For the retention, a cache relay and a file system writing are introduced to obtain the storage of each to-be-analyzed network data flow in each file in the file system, and each aggregation key corresponding to each time mark information corresponding to each to-be-analyzed network data flow divided according to a preset network data flow aggregation rule is combined to form each to-be-analyzed key, and each to-be-analyzed value formed by the storage address of each to-be-analyzed network data flow in the file system corresponding to each to-be-analyzed key. Based on this, for a target extraction request, the target aggregation key and the target time period are compared with each to-be-analyzed key and the corresponding to-be-analyzed value, so that the target extraction request is responded, the storage speed of real-time flow is improved, and the efficiency of batch extraction during subsequent use is improved. The application has important significance for the rapid extraction and analysis of retained flow.

Description

Technical Field

[0001] This invention relates to a method for real-time traffic retention and batch extraction, belonging to the field of network traffic storage and extraction technology. Background Technology

[0002] With the rapid development of network technologies such as 5G mobile internet, big data, artificial intelligence, the Internet of Things, and blockchain, network information security faces new technological challenges. Analyzing real-time network traffic, including traffic identification, filtering, and parsing, to obtain network security situation analysis is a crucial means of maintaining network security. For real-time traffic, retaining network conditions over a past period in the form of traffic summaries (mainly pcap files) allows for the reconstruction of the situation when new security incidents occur. In particular, retaining massive access logs such as DNS, Netflow, and HTTP logs can maximize the reproduction of past network conditions. The main issues to consider for real-time network traffic retention are the storage and extraction efficiency of massive traffic data. Currently, there is limited research and methodology in the industry regarding real-time traffic retention and batch extraction technologies.

[0003] Current industry-standard traffic retention technologies primarily store traffic data according to type (e.g., video, audio, images, text, binary files, etc.) in message queues or distributed file systems. Then, specific types of traffic data can be extracted based on type or folder. However, existing technologies have several drawbacks: when traffic data is small, increased disk seek times during storage slow down the process. Secondly, message queues or distributed file systems can only support extraction based on a single data type. If source IP, time period, or other dimensions are specified during extraction, all retained traffic data must be compared line by line to determine which data to extract, which is clearly inefficient. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for real-time traffic retention and batch extraction, which can improve the storage speed of real-time traffic and the efficiency of batch extraction during subsequent use, and is of great significance for the rapid extraction and analysis of retained traffic.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: The present invention designs a real-time traffic retention and batch extraction method, which is based on the storage of each network data stream to be analyzed in each file in the file system obtained by real-time acquisition. According to the network data stream to be analyzed, each aggregation key is divided according to the preset network data stream aggregation rules, and each analysis key is formed by combining each aggregation key with its corresponding time stamp information. The analysis value is formed by the storage address of each network data stream to be analyzed in the file system corresponding to each analysis key. The method responds to the target extraction request containing the target aggregation key and the target time period according to the following steps A to D.

[0006] Step A. Determine if any of the keys to be analyzed contain an aggregate key that is identical to the target aggregate key, and if any of the timestamp information corresponding to that aggregate key contains a key to be analyzed within the target time period. If so, obtain the aggregate key that is identical to the target aggregate key and the timestamp information corresponding to that aggregate key within the target time period, and obtain the analysis value corresponding to each key to be analyzed as a candidate value, then proceed to Step B; otherwise, the response to the target extraction request will be empty.

[0007] Step B. Filter out the candidate values ​​that are different from each other, and use them as non-repeating candidate values, and proceed to step C;

[0008] Step C. Sort the storage addresses corresponding to each non-repeating candidate value in ascending or descending order of storage address, and merge each group of adjacent and consecutive storage addresses in the sorted order to form each target storage address, and then proceed to step D.

[0009] Step D. Read all network data streams to be analyzed from each target storage address in the file system as each candidate network data stream, and obtain the collection time corresponding to each candidate network data stream. Then obtain each candidate network data stream whose collection time is within the target time period as the response to the target extraction request.

[0010] As a preferred embodiment of the present invention, before performing step A, the following operation is also included:

[0011] The operation uses the result of the target time period start time minus the target time period start time * 5% as the new target time period start time, and the result of the target time period end time minus the target time period end time * 5% as the new target time period end time. The new target time period is then formed by the new target time period start time and the new target time period end time.

[0012] Then, in step A, it is determined whether there exists an aggregate key that is the same as the target aggregate key among the keys to be analyzed, and whether there are keys to be analyzed in the time stamp information corresponding to the aggregate key that are located in the new target time period. If so, the aggregate key that is the same as the target aggregate key and the time stamp information corresponding to the aggregate key that are located in the new target time period are obtained, and the analysis value corresponding to each key to be analyzed is obtained as each candidate value, and then step B is entered; otherwise, the response to the target extraction request is empty.

[0013] As a preferred technical solution of the present invention: In step A, based on the ascending order of the timestamp information of each key to be analyzed, each key to be analyzed is traversed sequentially. First, it is determined whether there is a key to be analyzed whose timestamp information is greater than or equal to the start time of the new target time period. If so, the first key to be analyzed whose timestamp information is greater than or equal to the start time of the new target time period is obtained, and then each key to be analyzed whose timestamp information is less than or equal to the end time of the new target time period is obtained. Then, the analysis value corresponding to each obtained key to be analyzed is obtained as each candidate value, and then the process proceeds to step B; otherwise, the response to the target extraction request is empty.

[0014] As a preferred technical solution of the present invention: for each acquired network data stream to be analyzed in real time, according to the following steps i to vii, the acquired network data stream to be analyzed is stored in each file in the file system in real time, and each analysis key is constructed by combining each aggregation key with its corresponding time stamp information under the preset network data stream aggregation rules, and each analysis value is constructed by the storage address of each network data stream to be analyzed in the file system corresponding to each analysis key.

[0015] Step i. Divide the network data stream according to the preset network data stream aggregation rules, obtain the aggregation key corresponding to the network data stream to be analyzed, and determine whether the data volume of the network data stream to be analyzed is less than the preset traffic data volume threshold. If yes, proceed to step ii; otherwise, proceed to step vii.

[0016] Step ii. Determine if the cache corresponding to the aggregation key exists in the cache area. If it does, proceed directly to step iii; otherwise, create the cache corresponding to the aggregation key and then proceed to step iii.

[0017] Step iii. Write the network data stream to be analyzed into the cache corresponding to the aggregation key in the cache area, and update the minimum and maximum collection times of the network data stream to be analyzed in the cache corresponding to the aggregation key according to the collection time of the network data stream to be analyzed, and then proceed to step iv.

[0018] Step iv. Determine whether the amount of data in the cache corresponding to the aggregation key is greater than or equal to the preset cache data amount threshold. If yes, divide the network data stream to be analyzed in the cache corresponding to the aggregation key into segments from the minimum to the maximum collection time according to the preset time span value, obtain each time interval of the network data stream to be analyzed in the cache corresponding to the aggregation key, and use the start time of each time interval as each time mark information. Combine the aggregation key with each time mark information to form each key to be analyzed, and then proceed to step v; otherwise, the processing of the network data stream to be analyzed ends.

[0019] Step v. Obtain the storage address of the file marked as writable in the file system, write each network data stream to be analyzed in the cache corresponding to the aggregation key into the file at the storage address, and use the storage address to form the analysis value corresponding to each analysis key obtained in the adjacent steps iii. Then delete the aggregation key and its corresponding cache in the cache area, and proceed to step vi.

[0020] Step vi. Determine whether the amount of data written to the file in the adjacent steps of the execution sequence is greater than or equal to the preset file data writing threshold. If yes, update the file's mark to be unwritable, then create a new file and define its mark as writable, and the processing of the network data stream to be analyzed ends; otherwise, keep the file's mark as writable, and the processing of the network data stream to be analyzed ends.

[0021] Step vii. Use the acquisition time of the network data stream to be analyzed as the timestamp information. Combine the aggregation key with the timestamp information to form the analysis key. Obtain the storage address of the file marked as writable in the file system. Write the network data stream to be analyzed into the file at the storage address. The storage address constitutes the analysis value corresponding to the analysis key. Then return to step vi.

[0022] As a preferred technical solution of the present invention: in step vi, a new file is created at the next storage address of the maximum storage address of an existing file in the file system, according to the preset maximum storage data amount of the file, and its mark is defined as writable.

[0023] As a preferred technical solution of the present invention: the storage address constituting the value to be analyzed includes a file sequence number composed of the file's starting storage address, a storage address offset, and a preset maximum storage data volume for the file.

[0024] The real-time traffic retention and batch extraction method described in this invention, compared with the prior art, has the following technical advantages:

[0025] This invention presents a real-time traffic retention and batch extraction method. Employing a novel design logic, it establishes a unified logical scheduling relationship for both the retention and extraction stages of network data streams. For retention, it introduces two stages: cache relay and file system writing. This obtains the storage of each network data stream to be analyzed within its respective file system file. Furthermore, it constructs analysis keys for each network data stream, based on preset aggregation rules, where each aggregation key is combined with its corresponding timestamp information. It also constructs analysis values ​​for each analysis key, corresponding to the storage addresses of the network data streams in the file system. Based on this, for target extraction requests, it compares the target aggregation key and target time period with each analysis key and its corresponding analysis value to achieve a response to the target extraction request. This improves the storage speed of real-time traffic and the efficiency of subsequent batch extraction, which is significant for the rapid extraction and analysis of retained traffic. Attached Figure Description

[0026] Figure 1 This is a flowchart of the real-time network data stream retention process in this invention.

[0027] Figure 2 This is a schematic diagram of the key and value storage format in the design of this invention;

[0028] Figure 3 This is a flowchart of the batch extraction process of network data streams in the design of this invention. Detailed Implementation

[0029] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0030] This invention designs a method for real-time traffic retention and batch extraction. In practical applications, it first performs real-time extraction on each acquired network data stream to be analyzed, according to... Figure 1 As shown, perform steps i to vii to achieve real-time storage of the acquired network data streams to be analyzed in each file of the file system, and construct each analysis key composed of each aggregation key and its corresponding timestamp information under the preset network data stream aggregation rules, and each analysis value composed of the storage address of each network data stream to be analyzed in the file system corresponding to each analysis key.

[0031] Step i. Divide the network data stream according to the preset network data stream aggregation rules, obtain the aggregation key corresponding to the network data stream to be analyzed, and determine whether the data volume of the network data stream to be analyzed is less than the preset traffic data volume threshold, such as 1MB. If yes, proceed to step ii; otherwise, proceed to step vii.

[0032] This section pre-defines network data stream aggregation rules, such as by website address. For example, if the aggregation key is "taobao", compared to traditional categorized storage, storing by aggregation key allows for faster extraction after the traffic data is stored.

[0033] Step ii. Determine if a cache corresponding to the aggregate key exists in the map-based cache. If yes, proceed directly to step iii; otherwise, create the cache corresponding to the aggregate key and then proceed to step iii.

[0034] For example, if the current cache contains caches with the three aggregate keys taobao, tecent, and baidu, then searching for the cache corresponding to the taobao aggregate key in the cache will be successful. Conversely, if there is no cache corresponding to taobao in the cache, then 1.2MB of memory will be allocated, and the taobao aggregate key and its corresponding memory space will be added to the cache, that is, inserted into the cache in the form of a map.

[0035] Step iii. Write the network data stream to be analyzed into the cache corresponding to the aggregation key in the cache area, and update the minimum and maximum collection times of the network data stream to be analyzed in the cache corresponding to the aggregation key according to the collection time of the network data stream to be analyzed, and then proceed to step iv.

[0036] For example, if the cache corresponding to the taobao aggregation key contains 300KB of data, and 850KB of network data stream to be analyzed is appended, the size of the cache corresponding to the taobao aggregation key will reach 1150KB. Furthermore, if the time period of the cache corresponding to the taobao aggregation key is [1664037004, 1664037010], and the collection time of the network data stream to be analyzed is 1664037012, then after the update, the time period of the minimum collection time and the maximum collection time of the network data stream to be analyzed in the cache corresponding to this aggregation key will be [1664037004, 1664037012].

[0037] Step iv. Determine whether the amount of data in the cache corresponding to the aggregation key is greater than or equal to the preset cache data amount threshold. If yes, divide the time span of the network data stream to be analyzed in the cache corresponding to the aggregation key into the range from the minimum to the maximum acquisition time according to the preset time interval value, such as 5 seconds, to obtain each time interval of the network data stream to be analyzed in the cache corresponding to the aggregation key. Use the start time of each time interval as each time mark information. Combine the aggregation key with each time mark information to form each key to be analyzed, and then proceed to step v; otherwise, the processing of the network data stream to be analyzed ends.

[0038] For example, if the preset cache data size threshold is 1MB, and the current cache size corresponding to the taobao aggregation key is 1150KB, which exceeds the preset cache data size threshold, then it is necessary to divide the cache corresponding to the taobao aggregation key into time intervals according to step iv above, and then proceed to step v.

[0039] Step v. Obtain the storage address of the file marked as writable in the file system, write each network data stream to be analyzed in the cache corresponding to the aggregation key into the file at the storage address, and use the storage address to form the analysis value corresponding to each analysis key obtained in the adjacent steps iii. Then delete the aggregation key and its corresponding cache in the cache area, and proceed to step vi.

[0040] Here, as Figure 2 As shown, the storage address that constitutes the value to be analyzed includes the file sequence number composed of the file's starting storage address, the storage address offset, and the preset maximum storage data size of the file.

[0041] Step vi. Determine whether the amount of data written to the file in the adjacent steps of the execution sequence is greater than or equal to the preset file data writing threshold. If yes, update the file's mark to be unwritable, then create a new file and define its mark to be writable, and the processing of the network data stream to be analyzed ends; otherwise, keep the file's mark as writable, and the processing of the network data stream to be analyzed ends.

[0042] The preset file write data volume threshold is designed to prevent a single file from becoming too large. For example, if the preset file write data volume threshold is 1GB, and the current size of 0019.dat is 1.01GB, exceeding the preset file write data volume threshold, then a new file needs to be created before the cache is written.

[0043] In practical applications, step vi here is specifically designed to create a new file based on the next storage address of the maximum storage address of an existing file in the file system, according to the preset maximum storage data volume of the file, and define its mark as writable.

[0044] Step vii. Use the acquisition time of the network data stream to be analyzed as the timestamp information. Combine the aggregation key with the timestamp information to form the analysis key. Obtain the storage address of the file marked as writable in the file system. Write the network data stream to be analyzed into the file at the storage address. The storage address constitutes the analysis value corresponding to the analysis key. Then return to step vi.

[0045] Following steps i to vii above, each key to be analyzed and its corresponding value are obtained and stored in a key-value database engine, RocksDB. In this embodiment, the cached time period corresponding to the taobao aggregation key is [1664037004, 1664037012], which can be divided into three time intervals in 5-second increments: [1664037000, 1664037005), [1664037005, 1664037010), and [1664037010, 1664037015]. The storage address of the corresponding network data stream to be analyzed is 0020-0-1177600, where 0020 represents the file sequence number, 0 represents the offset within the file, and 1177600 represents the data size (1177600B = 1150KB). The key-value pairs stored in the RocksDB key-value database engine are as follows:

[0046] (1) Key: taobao-1664037000 Value: 0020-0-1177600

[0047] (2) Key: taobao-1664037005 Value: 0020-0-1177600

[0048] (3) Key: taobao-1664037010 Value: 0020-0-1177600

[0049] Then, based on a key-value database engine like RocksDB, press Figure 3 As shown, perform the following operations, and steps A to D, to respond to a target extraction request containing the target aggregation key and the target time period. For example, in the design embodiment, the target aggregation key is the taobao aggregation key, and the target time period is [1664037006, 1664037011].

[0050] Operation: The result of subtracting 5% from the start time of the target time period is taken as the start time of the new target time period, and the result of subtracting 5% from the end time of the target time period is taken as the end time of the new target time period. The new target time period is then formed by the start time and the end time of the new target time period. In this embodiment, the calculation is performed for the target time period [1664037006, 1664037011) to obtain the start time of the new target time period = 1664037005 and the end time of the new target time period = 1664037010.

[0051] Step A. Determine if any of the keys to be analyzed contain an aggregate key that is identical to the target aggregate key, and if any of the timestamp information corresponding to that aggregate key contains a key to be analyzed within the new target time period. If so, obtain the aggregate key that is identical to the target aggregate key and the timestamp information corresponding to that aggregate key within the new target time period, and obtain the analysis value corresponding to each key to be analyzed as a candidate value, and then proceed to Step B; otherwise, the response to the target extraction request will be empty.

[0052] In practical applications, during the implementation of step A above, based on the ascending order of the timestamp information of each key to be analyzed, each key to be analyzed is traversed sequentially. First, it is determined whether there is a key to be analyzed whose timestamp information is greater than or equal to the start time of the new target time period. If so, the first key to be analyzed whose timestamp information is greater than or equal to the start time of the new target time period is obtained, and then each key to be analyzed whose timestamp information is less than or equal to the end time of the new target time period is obtained. Then, the analysis value corresponding to each obtained key to be analyzed is obtained as each candidate value, and then the process proceeds to step B; otherwise, the response to the target extraction request is empty.

[0053] Step B. Filter out the candidate values ​​that are different from each other, and use them as non-repeating candidate values, and proceed to step C; In this embodiment, for example, the candidate values ​​obtained are 0020-0-1177600, 0020-0-1177600, 0020-2358798-1060054, 0020-1177600-1049600. After deduplication and sorting, the non-repeating candidate values ​​are obtained as follows: 0020-0-1177600, 0020-1177600-1049600, 0020-4358798-1060054.

[0054] Step C. Sort the storage addresses corresponding to each non-repeating candidate value in ascending or descending order of storage address, and merge each group of adjacent and consecutive storage addresses in the sorted order to form each target storage address, and then proceed to step D; In this embodiment, the obtained target storage addresses include 0020-0-2227200 and 0020-4358798-1060054.

[0055] Step D. Read all network data streams to be analyzed from each target storage address in the file system as each candidate network data stream, and obtain the collection time corresponding to each candidate network data stream. Then obtain each candidate network data stream whose collection time is within the target time period as the response to the target extraction request.

[0056] In this embodiment, the new target time period is traversed using 1664037005 as the start time and 161664037010 as the end time. The location information corresponding to taobao-1664037005 actually stores data in the range [1664037005, 161664037010), and the location information corresponding to taobao-1664037010 actually stores data in the range [1664037010, 161664037015). Therefore, in step D, the actual time range for reading all network data streams to be analyzed from each target storage address in the file system is [1664037005, 161664037015]. The target time period in the target extraction request is [1664037006, 1664037011). Therefore, data traffic exceeding the target time period is filtered out, and the remaining candidate network data streams are the final batch extraction results for the target extraction request.

[0057] The real-time traffic retention and batch extraction method designed in the above technical solution adopts a completely new design logic. It implements a unified logical scheduling relationship for both the retention and extraction stages of network data streams. For retention, it introduces two stages: cache relay and file system writing. This obtains the storage of each network data stream to be analyzed within its respective file system file. It also constructs analysis keys for each network data stream, based on preset aggregation rules, where each aggregation key is combined with its corresponding timestamp information. Furthermore, it generates analysis values ​​for each analysis key, corresponding to the storage addresses of the network data streams in the file system. Based on this, it further responds to target extraction requests by comparing the target aggregation key and target time period with each analysis key and its corresponding analysis value. This improves the storage speed of real-time traffic and the efficiency of subsequent batch extraction, which is of great significance for the rapid extraction and analysis of retained traffic.

[0058] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for real-time traffic retention and batch extraction, characterized in that: Based on the storage of each network data stream to be analyzed in each file in the file system obtained in real time, according to the preset network data stream aggregation rules, each aggregation key is combined with its corresponding timestamp information to form each analysis key, and each analysis value is formed by the storage address of each network data stream to be analyzed in the file system corresponding to each analysis key. Then, respond to the target extraction request containing the target aggregation key and the target time period according to the following steps A to D. Step A. Determine if any of the keys to be analyzed contain an aggregate key that is identical to the target aggregate key, and if any of the timestamp information corresponding to that aggregate key contains keys to be analyzed within the target time period. If so, obtain the aggregate key that is identical to the target aggregate key and the timestamp information corresponding to that aggregate key within the target time period, and obtain the analysis value corresponding to each key to be analyzed as each candidate value, and then proceed to Step B; otherwise, the response to the target extraction request will be empty. Step B. Filter out the candidate values ​​that are different from each other, and use them as non-repeating candidate values, then proceed to step C; Step C. Sort the storage addresses corresponding to each non-repeating candidate value in ascending or descending order of storage address, and merge each group of adjacent and consecutive storage addresses in the sorted order to form each target storage address, and then proceed to step D. Step D. Read all network data streams to be analyzed from each target storage address in the file system as each candidate network data stream, and obtain the collection time corresponding to each candidate network data stream. Then obtain each candidate network data stream whose collection time is within the target time period as the response to the target extraction request.

2. The method for real-time traffic retention and batch extraction according to claim 1, characterized in that, Before performing step A, the following operations are also included: Operation, based on the start time of the target time period - the start time of the target time period. The 5% result will be used as the start time of the new target time period, calculated as the difference between the end time of the target time period and the end time of the target time period. The result of 5% is used as the end time of the new target time period. The new target time period is then composed of the start time and the end time of the new target time period. Then, in step A, it is determined whether there exists an aggregate key that is the same as the target aggregate key among the keys to be analyzed, and whether there are keys to be analyzed in the time stamp information corresponding to the aggregate key that are located in the new target time period. If so, the aggregate key that is the same as the target aggregate key and the time stamp information corresponding to the aggregate key that are located in the new target time period are obtained, and the analysis value corresponding to each key to be analyzed is obtained as each candidate value, and then step B is entered; otherwise, the response to the target extraction request is empty.

3. The method for real-time traffic retention and batch extraction according to claim 2, characterized in that: In step A, based on the ascending order of the timestamp information of each key to be analyzed, each key to be analyzed is traversed sequentially. First, it is determined whether there is a key to be analyzed whose timestamp information is greater than or equal to the start time of the new target time period. If so, the first key to be analyzed whose timestamp information is greater than or equal to the start time of the new target time period is obtained, and then each key to be analyzed whose timestamp information is less than or equal to the end time of the new target time period is obtained. Then, the analysis value corresponding to each obtained key to be analyzed is obtained as each candidate value, and then the process proceeds to step B; otherwise, the response to the target extraction request is empty.

4. The method for real-time traffic retention and batch extraction according to claim 1, characterized in that: For each acquired network data stream to be analyzed, according to steps i to vii, the acquired network data stream to be analyzed is stored in each file in the file system in real time. Each network data stream to be analyzed is divided according to the preset network data stream aggregation rules. Each aggregation key is combined with its corresponding timestamp information to form each analysis key. Each analysis key is also composed of the storage address of each network data stream to be analyzed in the file system. Step i. Divide the network data stream according to the preset network data stream aggregation rules, obtain the aggregation key corresponding to the network data stream to be analyzed, and determine whether the data volume of the network data stream to be analyzed is less than the preset traffic data volume threshold. If yes, proceed to step ii; otherwise, proceed to step vii. Step ii. Determine if the cache corresponding to the aggregation key exists in the cache area. If it does, proceed directly to step iii; otherwise, create the cache corresponding to the aggregation key and then proceed to step iii. Step iii. Write the network data stream to be analyzed into the cache corresponding to the aggregation key in the cache area, and update the minimum and maximum collection times of the network data stream to be analyzed in the cache corresponding to the aggregation key according to the collection time of the network data stream to be analyzed, and then proceed to step iv. Step iv. Determine whether the amount of data in the cache corresponding to the aggregation key is greater than or equal to the preset cache data amount threshold. If so, divide the network data stream to be analyzed in the cache corresponding to the aggregation key into segments according to the preset time span value, from the minimum collection time to the maximum collection time, to obtain each time interval of the network data stream to be analyzed in the cache corresponding to the aggregation key. Use the start time of each time interval as each time stamp information. Combine the aggregation key with each time stamp information to form each key to be analyzed, and then proceed to step v; otherwise, the processing of the network data stream to be analyzed ends. Step v. Obtain the storage address of the file marked as writable in the file system, write each network data stream to be analyzed in the cache corresponding to the aggregation key into the file at the storage address, and use the storage address to form the analysis value corresponding to each analysis key obtained in the adjacent steps iii. Then delete the aggregation key and its corresponding cache in the cache area, and proceed to step vi. Step vi. Determine whether the amount of data written to the file in the adjacent steps of the execution sequence is greater than or equal to the preset file data writing threshold. If so, update the file's mark to be unwritable, then create a new file and define its mark to be writable. The processing of the network data stream to be analyzed ends here. Otherwise, keep the file marked as writable, and the processing of the network data stream to be analyzed ends; Step vii. Use the acquisition time of the network data stream to be analyzed as the timestamp information, combine the aggregation key with the timestamp information to form the analysis key, obtain the storage address of the file marked as writable in the file system, write the network data stream to be analyzed into the file at the storage address, and use the storage address to form the analysis value corresponding to the analysis key, and then return to step vi.

5. The method for real-time traffic retention and batch extraction according to claim 4, characterized in that: In step vi, a new file is created at the next storage address of the maximum storage address of an existing file in the file system, according to the preset maximum storage data volume of the file, and its mark is defined as writable.

6. The method for real-time traffic retention and batch extraction according to claim 4 or 5, characterized in that: The storage address of the key to be analyzed includes a file sequence number composed of the file's starting storage address, a storage address offset, and a preset maximum storage data size for the file.

Images