A Caching Method and System for Structured Streaming Data

Abstract

This invention discloses a caching method and system for structured streaming data. The method includes: setting up a cache space using different pre-allocation methods based on the type and length of the structured streaming data to be received; wherein, when the data is fixed-length, a fixed-length cache space is pre-allocated; when the data is variable-length, a variable-length cache space or a composite cache space is pre-allocated; receiving structured streaming data, when the data is fixed-length, storing the structured streaming data in the pre-allocated fixed-length cache space; when the data is variable-length, dynamically expanding the pre-allocated cache space according to the actual length of the received structured streaming data until the structured streaming data is completely stored. This invention fundamentally solves the overflow problem that fixed-length cache spaces may face, while the application of the variable-length caching mechanism also greatly saves storage space, balancing the needs of memory resource optimization and caching performance.

Description

Technical Field

[0001] This invention belongs to the field of information technology, and in particular relates to a caching method and system for structured streaming data. Background Technology

[0002] In network auditing, network traffic often reaches speeds exceeding 10 Gbps, resulting in massive amounts of monitoring and auditing data. Processing terabytes of network log data is not uncommon. In this context, the efficiency of organizing and managing the monitoring and auditing data becomes a bottleneck for the performance of network data analysis systems. In the application scenario described in this invention, the network auditing application collects network traffic data, analyzes, statistically analyzes, and monitors various information contained within the network traffic, outputting analysis results for the parts of interest. These analysis results are output in data format according to a specific structure. As the network traffic processed by the program continuously increases, the application's output is constantly generated. How to efficiently cache and manage this output data is the application scenario that this invention addresses.

[0003] In this application scenario, the core element is the result data generated by the network auditing application. Based on specific scenarios and practical operations, we can summarize and explain the characteristics of this type of data. First, there is a certain correspondence between the input and output of the auditing program. At the time level, because the network traffic data input to the application has streaming characteristics, the corresponding output data also has streaming characteristics: real-time, sequential, large-scale, fast, and continuous arrival. Second, network traffic content has inherent connections. When the application parses network traffic, it divides network packets into different network flows according to the five-tuple attributes of network traffic: source / destination address, source / destination port, and protocol type. Network packets belonging to the same network flow have highly correlated audit data parsed from them. Simultaneously, for specific types of network flows, the data parsed by the auditing application has a definite category. For example, for HTTP protocol traffic, the auditing program often parses the corresponding MAC address, IP address, port number, HTTP protocol version, server / client version, client method, URL, Host, cookie, and response content, etc., which are output in the form of fields. For a specific type of network stream, the parsed output data has defined categories, and there are definite relationships between these categories. The categories of output data can be defined by the application, therefore the output data in this application scenario simultaneously possesses structured characteristics. This type of data, output by a network auditing application, has both streaming and structured characteristics, hence it is called structured streaming data.

[0004] Structured streaming data is characterized by its real-time, high-volume, and continuous arrival, requiring organization, caching, and batch processing for storage in disk files or databases. How to cache structured streaming data is the problem this invention addresses. Based on practical experience, structured streaming data can be divided into two categories: one with a fixed length, such as the MAC field (48 bits, 6 bytes) in various data obtained from parsing HTTP traffic. Caching this type of data is relatively simple, requiring only sufficient cache space. The other category, with variable lengths, can be further divided into two types: one where the length is completely unpredictable, such as database protocol response fields; and another where theoretically there is no fixed length, but in practice there is often a typical threshold. For example, even from HTTP traffic, URL fields can vary in length. Although clients or servers limit the length of URL fields in practice, the HTTP protocol itself does not impose a limit on URL length. Various problems arise when caching this type of data with uncertain lengths. Structured streaming data of uncertain length has streaming characteristics, and its length is unpredictable. Using traditional fixed-length cache space may lead to overflow, while truncating the data will cause information loss. Simply using fixed-length cache cannot solve the caching problem of variable-length structured streaming data. The only way to reduce the probability of overflow is to increase the fixed-length cache space, but on the other hand, this will waste memory resources.

[0005] For the problem of caching variable-length content, existing patented technologies offer the following solutions:

[0006] Reference [1] (application number 201310631997.6, patent "A method and apparatus for storing variable-length data") proposes a method for storing variable-length TLV structures using contiguous cache space. This method first applies for a sufficiently large contiguous cache space, and then uses this space to cache variable-length data. When new variable-length data arrives, it will determine whether the existing remaining space can meet the needs of the new data. If the space is insufficient, it will apply for a new cache space and mark the termination in the original cache space and point to the new cache space. This patent application uses a large cache space to avoid space overflow, but this method does not distinguish the categories of cached content and may generate memory fragments, which is a waste of memory resources.

[0007] Reference [2] (application number 201810023778.2, patent "A method for writing and reading variable-length data segments based on network coding") utilizes the characteristics of a distributed file system to divide variable-length data into multiple fixed-length data blocks for multi-location encoding and storage. This method also does not perform data differentiation processing, and the data truncation method cannot meet the performance requirements of data caching.

[0008] For the problem of streaming data caching, existing patented technologies propose the following solutions:

[0009] Reference [3] (application number 201911302664.2, "A method and apparatus for caching streaming data") proposes a sampling prediction method for the length of streaming data, but does not explain the actual caching method, which is different from the focus of this invention.

[0010] Reference [4] (application number 201310741116.6, "A Method for Writing Large-Scale Network Streaming Data Cache") proposes to adopt different caching strategies according to the data source, and write data that meets the triggering conditions of the strategy into files and databases and periodically merge small data. The focus of this patent is on the cache output strategy, which is different from the focus of this invention. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the prior art and to propose a caching method and system for structured streaming data.

[0012] To achieve the above objectives, this invention proposes a caching method for structured streaming data, the method comprising:

[0013] Depending on the type and length of the structured streaming data to be received, different pre-allocation methods are used to set the buffer space; specifically, when the data is of fixed length, a fixed-length buffer space is pre-allocated; when the data is of variable length, a variable-length buffer space or a composite buffer space is pre-allocated.

[0014] When receiving structured streaming data, if the data is of fixed length, the structured streaming data is stored in a pre-allocated fixed-length buffer space; if the data is of variable length, the pre-allocated buffer space is dynamically expanded according to the actual length of the received structured streaming data until the structured streaming data is completely stored.

[0015] As an improvement to the above method, when the data is fixed-length data, a fixed-length cache space is pre-allocated; specifically, a contiguous memory space is set according to the length of the fixed-length data, the size of the space is equal to the length of the fixed-length data, and it cannot be expanded.

[0016] As an improvement to the above method, the types of variable-length data include: a first type of variable-length data and a second type of variable-length data; wherein,

[0017] The length of the first type of variable-length data is completely unpredictable;

[0018] The length of the second type of variable-length data is 90% likely to be below a certain length threshold.

[0019] As an improvement to the above method, the pre-allocation of a variable-length cache space or a composite cache space when the data is of variable length is specifically:

[0020] When the data is of the first type of variable length data, a variable-length cache space is pre-allocated for storage;

[0021] When the data is the second type of variable-length data, a fixed-length cache space is pre-allocated for storage, and the size of the space is equal to the length threshold.

[0022] As an improvement to the above method, when the data is of variable length, the pre-allocated buffer space is dynamically expanded according to the actual length of the received data until the structured streaming data is completely stored; specifically:

[0023] For the first type of variable-length data, when the sum of the actual received data length and the data length already stored in the existing storage space exceeds the existing storage space size, the variable-length cache space is dynamically expanded.

[0024] For the second type of variable-length data, when the sum of the actual received data length and the data length already stored in the existing storage space exceeds the length threshold, a dynamically allocated variable-length cache space is used to replace the original fixed-length cache space, and the data stored in the original cache space is copied to the newly allocated variable-length cache space, thereby realizing the dynamic expansion of the cache space and the switching of the cache space.

[0025] A caching system for structured streaming data, the system comprising: a storage space pre-allocation module and a data receiving and storage module; wherein,

[0026] The storage space pre-allocation module is used to set the cache space according to the type and length of the structured streaming data to be received, using different pre-allocation methods; wherein, when the data is fixed-length data, a fixed-length cache space is pre-allocated; when the data is variable-length data, a variable-length cache space or a composite cache space is pre-allocated.

[0027] The data receiving and storage module is used to receive structured streaming data. When the data is fixed-length data, the structured streaming data is stored in a pre-allocated fixed-length cache space. When the data is variable-length data, the pre-allocated cache space is dynamically expanded according to the actual length of the received structured streaming data until the structured streaming data is completely stored.

[0028] As an improvement to the above system, the storage space pre-allocation module includes: a fixed-length data storage space pre-allocation sub-module and a variable-length data storage space pre-allocation sub-module; wherein,

[0029] The fixed-length data storage space pre-allocation submodule is used to set up a continuous memory space according to the length of the fixed-length data. The size of the space is equal to the length of the fixed-length data and cannot be expanded.

[0030] The variable-length data storage space pre-allocation submodule is used to pre-allocate variable-length cache space or composite cache space for variable-length data.

[0031] As an improvement to the above system, the types of variable-length data include: a first type of variable-length data and a second type of variable-length data; wherein,

[0032] The length of the first type of variable-length data is completely unpredictable;

[0033] The length of the second type of variable-length data is 90% likely to be below a certain length threshold.

[0034] As an improvement to the above system, the specific implementation process of the variable-length data storage space pre-allocation submodule is as follows:

[0035] When the data is of the first type of variable length data, a variable-length cache space is pre-allocated for storage;

[0036] When the data is the second type of variable-length data, a fixed-length cache space is pre-allocated for storage, and the size of the space is equal to the length threshold.

[0037] As an improvement to the above system, the specific implementation process of the data receiving and storage module is as follows:

[0038] Receive structured streaming data and determine the data type;

[0039] For fixed-length data, the received data is stored in the cache space set by the pre-allocated submodule of the fixed-length data storage space until the data storage is completed;

[0040] For the first type of variable-length data, when the sum of the actual received data length and the data length already stored in the cache space set by the variable-length data storage space pre-allocation submodule exceeds the set storage space size, the cache space is dynamically expanded until the data is completely stored.

[0041] For the second type of variable-length data, when the sum of the actual received data length and the data length already stored in the cache space set by the variable-length data storage space pre-allocation submodule exceeds the length threshold, a dynamically allocated variable-length cache space is used to replace the original fixed-length cache space, and the data stored in the original cache space is copied to the newly applied variable-length cache space, thereby realizing the dynamic expansion of the cache space and the switching of the cache space until the data is completely stored.

[0042] Compared with the prior art, the advantages of the present invention are:

[0043] When managing structured streaming data of variable length in a cache, the method of this invention fundamentally solves the overflow problem that fixed-length cache space may face. At the same time, the application of the variable-length cache mechanism also greatly saves storage space, taking into account both the needs of memory resource optimization and cache performance. Attached Figure Description

[0044] Figure 1 These are schematic diagrams of the three cache structures of this invention;

[0045] Figure 2 This is a schematic diagram of the cache switching of the composite cache structure of the present invention;

[0046] Figure 3 This is a flowchart of the caching method for structured streaming data according to the present invention. Detailed Implementation

[0047] This invention discloses a cache management method for structured streaming data. The method employs different caching methods based on the different attributes of the input structured streaming data: for streaming data of fixed length categories, a fixed-length cache structure is used. For streaming data of variable length categories, when the length is completely unpredictable, a variable-length cache structure is used; when the length has a certain range, a length threshold is set. If the length is below the threshold, a fixed-length cache structure is used; if the length is above the threshold, the caching method switches from a fixed-length cache structure to a variable-length cache structure.

[0048] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0049] Example 1

[0050] like Figure 1 As shown, Embodiment 1 of the present invention proposes a caching method for structured streaming data. This method employs different caching methods depending on the attributes of the input structured streaming data: for streaming data of fixed length categories, a fixed-length caching structure is used. For streaming data of variable length categories, when the data length is completely unpredictable, a variable-length caching structure is used directly; when the data length has a certain range, a length threshold is set. When the length is below the set threshold, a fixed-length caching structure is used; when the length is above the set threshold, the caching method switches from using a fixed-length caching structure to using a variable-length caching structure. See the figure for a schematic diagram of the caching switching of the composite caching structure.

[0051] Figure 3 This is a flowchart illustrating the caching method for structured streaming data according to the present invention.

[0052] The following provides an example of applying this method to a log management system. This method addresses the problem of caching streaming data during network activity monitoring and auditing, and provides a complete solution for managing structured streaming data, generating custom-formatted logs, and outputting logs. It should be noted that this method can be applied to various streaming data caching systems, not just the system in this example.

[0053] Based on practical application scenarios, a system for generating and outputting logs using this method was designed. This system receives parsed structured streaming data, caches the received data by field into the cache structure corresponding to the cache management method, generates a custom-formatted log after collecting all the streaming data, and outputs the logs in batches to files for backend database analysis and presentation.

[0054] In this system, custom-formatted logs are generated at the granularity of sessions or network behaviors. The network flow corresponding to a network session is defined by a five-tuple: source IP address, destination IP address, source port number, destination port number, and protocol type. Based on the log content format, logs are divided into two categories: session logs that record data flow information and operation logs that record various behaviors within the flow. Only one session log is generated for each data flow, but multiple operation logs can be generated.

[0055] The key data caching method in this data structure is the data caching management method proposed in this invention. In this invention, the structured streaming data as input is cached in a log cache structure by field. Fields can be divided into fixed-length fields and variable-length fields based on whether their length is fixed. Variable-length fields are further divided into two categories based on the predictability of their length: a first category that is completely unpredictable and a second category where a length threshold can be set. According to the method proposed in this invention, a fixed-length cache space is allocated to cache fixed-length fields; for the first category of variable-length fields, a variable-length cache structure is directly requested to cache the fields; for the second category of variable-length fields, a fixed-length cache space equivalent to the length threshold is first allocated to cache the data; when the data length exceeds the threshold, a variable-length cache space is requested and allocated, and the data cached in the fixed-length space is copied to the variable-length cache space, realizing dynamic switching of the caching method.

[0056] In this system, log generation begins after the input of structured streaming data from a network stream has finished and all data has been cached. At this point, the data structure mentioned above uses the data caching management method proposed in this invention to cache all previously received data in the form of fields. These fields are then concatenated into a complete log according to a custom format, thus completing the log generation process.

[0057] In this system, logs are immediately sent to the output queue after generation, awaiting output. This invention employs batch processing for log output. When logs arrive in the output queue, it is determined whether the size of the log content currently waiting in the queue reaches the set batch processing threshold. If the condition is met, the logs in the queue are output in batches. Simultaneously, to ensure the timeliness of logs, this invention introduces a timeout mechanism, recording the arrival time of each log entry. When the log waiting time exceeds the set maximum waiting time, the timeout mechanism is triggered, and all logs in the queue are written to the log file for output.

[0058] The method proposed in this invention is used in the log module of a network traffic parsing system to achieve data storage. The input data to be stored is the streaming data result obtained by the system, and the final output of the log module is a log file in a system-defined format. The following uses the processing of an HTTP stream as an example to illustrate the workflow of the log module using this method.

[0059] The logging module's caching system first performs space pre-allocation. For HTTP streams, the fields to be stored include system-defined general logging fields (start time, end time, identifier ID, etc.), upper-level protocol fields (MAC address, transport layer protocol, source IP address, destination IP address, source port, destination port, etc.), and HTTP protocol-specific fields (URL, HTTP action, HTTP response content). Different cache spaces are pre-allocated based on the different attributes of these fields. For fields such as MAC address, transport layer protocol, source IP address, destination IP address, source port, and destination port, which have fixed lengths, the logging module allocates fixed-length cache space. Fields such as HTTP response content not only have variable lengths but also unpredictable lengths; therefore, the logging module allocates variable-length cache space for these fields. For URL fields, although their length is not fixed, it is highly likely that its length will not exceed a certain threshold. For URL fields, their length generally does not exceed the 2083 characters that Internet Explorer can process; therefore, the logging module pre-allocates a composite cache space with a length threshold of 2083 for URLs.

[0060] After the pre-allocation process is complete, the parent system sends the HTTP stream's start time, source and destination addresses, port, protocol type, and other data, along with the identification ID, to the log module. The log module uses the identification ID to locate the corresponding cache structure for that HTTP stream and stores the data—start time, source and destination addresses, port, protocol type, etc.—in the space corresponding to that cache structure. This data is fixed-length and is cached in the pre-allocated fixed-length cache.

[0061] The upper-level system detects the operations performed by the client in the HTTP stream, such as GET operations. The system then sends the information parsed from the GET operation, such as the client method, URL, HOST, and response content, along with the identification ID, to the log module via an interface function. The received data is then stored in the corresponding cache space. For the response content field, a variable-length cache space is pre-allocated, and the input data is stored in this variable-length space. For the URL field, a composite cache space is allocated. When the URL length does not exceed the threshold set for the URL field (i.e., less than 2083 characters), the URL is directly stored in the fixed-length space of the composite cache structure. When the URL length exceeds the threshold, the existing data is first moved to the variable-length space of the composite cache structure, and then the data exceeding the threshold continues to be stored in the variable-length space.

[0062] The parent system detects the end of the HTTP stream and calls the log termination interface. After the interface call, the logging module begins generating logs. The module combines the data in the log cache structure according to a defined format to generate a single log entry.

[0063] The generated logs are not directly output to a file, but are instead added to an output buffer. The logging module uses a batch output method. Only when the size of the log content in the output buffer reaches a set threshold will it be output to a file. Furthermore, to ensure the timeliness of the output logs, a timeout mechanism is incorporated into the logging module's output management. If a log entry waits too long in the output buffer, triggering the timeout mechanism, the logging module will also output the log to a file.

[0064] The above describes the workflow of the log module implemented in the network traffic analysis system using the method proposed in this invention.

[0065] Example 2

[0066] Based on the above method, Embodiment 2 of the present invention proposes a caching system for structured streaming data. The system includes: a storage space pre-allocation module and a data receiving and storage module; wherein,

[0067] The storage space pre-allocation module is used to set the cache space according to the type and length of the structured streaming data to be received, using different pre-allocation methods; specifically, when the data is fixed-length data, a fixed-length cache space is pre-allocated; when the data is variable-length data, a variable-length cache space or a composite cache space is pre-allocated.

[0068] The storage space pre-allocation module includes a fixed-length data storage space pre-allocation submodule and a variable-length data storage space pre-allocation submodule. The fixed-length data storage space pre-allocation submodule is used to allocate contiguous memory space based on the length of the fixed-length data. The space size is equal to the length of the fixed-length data and cannot be expanded. The variable-length data storage space pre-allocation submodule is used to pre-allocate variable-length cache space or composite cache space for variable-length data. When the data is the first type of variable-length data, the variable-length cache space is pre-allocated for storage; when the data is the second type of variable-length data, a fixed-length cache space is pre-allocated for storage, and the size of this space is equal to the length threshold.

[0069] The data receiving and storage module is used to receive structured streaming data. When the data is of fixed length, it is stored in a pre-allocated fixed-length buffer space. When the data is of variable length, the pre-allocated buffer space is dynamically expanded according to the actual length of the received data until the structured streaming data is completely stored.

[0070] The specific implementation process is as follows:

[0071] Receive structured streaming data and determine the data type;

[0072] For fixed-length data, the received data is stored in the cache space set by the pre-allocated submodule of the fixed-length data storage space until the data storage is completed;

[0073] For the first type of variable-length data, when the sum of the actual received data length and the data length already stored in the cache space set by the variable-length data storage space pre-allocation submodule exceeds the set storage space size, the cache space is dynamically expanded until the data is completely stored.

[0074] For the second type of variable-length data, when the sum of the actual received data length and the data length already stored in the cache space set by the variable-length data storage space pre-allocation submodule exceeds the length threshold, a dynamically allocated variable-length cache space is used to replace the original fixed-length cache space, and the data stored in the original cache space is copied to the newly applied variable-length cache space, thereby realizing the dynamic expansion of the cache space and the switching of the cache space until the data is completely stored.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A caching method for structured streaming data, the method comprising: Depending on the type and length of the structured streaming data to be received, different pre-allocation methods are used to set the buffer space; specifically, when the data is of fixed length, a fixed-length buffer space is pre-allocated; when the data is of variable length, a variable-length buffer space or a composite buffer space is pre-allocated. When receiving structured streaming data, if the data is of fixed length, the structured streaming data is stored in a pre-allocated fixed-length buffer space; if the data is of variable length, the pre-allocated buffer space is dynamically expanded according to the actual length of the received structured streaming data until the structured streaming data is completely stored. The types of variable-length data include: first type of variable-length data and second type of variable-length data; wherein... The length of the first type of variable-length data is completely unpredictable; The length of the second type of variable-length data has a 90% probability of being lower than a certain length threshold; The provision of pre-allocating variable-length or composite cache space when the data is of variable length; specifically: When the data is of the first type of variable length data, a variable-length cache space is pre-allocated for storage; When the data is the second type of variable-length data, a fixed-length cache space is pre-allocated for storage, and the size of the space is equal to the length threshold. When the data is of variable length, the pre-allocated buffer space is dynamically expanded according to the actual length of the received data until the structured streaming data is completely stored; specifically: For the first type of variable-length data, when the sum of the actual received data length and the data length already stored in the existing storage space exceeds the existing storage space size, the variable-length cache space is dynamically expanded. For the second type of variable-length data, when the sum of the actual received data length and the data length already stored in the existing storage space exceeds the length threshold, a dynamically allocated variable-length cache space is used to replace the original fixed-length cache space, and the data stored in the original cache space is copied to the newly allocated variable-length cache space, thereby realizing the dynamic expansion of the cache space and the switching of the cache space.

2. The caching method for structured streaming data according to claim 1, characterized in that, The method of pre-allocating a fixed-length cache space when the data is fixed-length data refers to setting a contiguous memory space based on the length of the fixed-length data. The size of the space is equal to the length of the fixed-length data and cannot be expanded.

3. A caching system for structured streaming data, characterized in that, The system includes: a storage space pre-allocation module and a data receiving and storage module; wherein... The storage space pre-allocation module is used to set the cache space according to the type and length of the structured streaming data to be received, using different pre-allocation methods; wherein, when the data is fixed-length data, a fixed-length cache space is pre-allocated; when the data is variable-length data, a variable-length cache space or a composite cache space is pre-allocated. The data receiving and storage module is used to receive structured streaming data. When the data is fixed-length data, the structured streaming data is stored in a pre-allocated fixed-length cache space. When the data is variable-length data, the pre-allocated cache space is dynamically expanded according to the actual length of the received structured streaming data until the structured streaming data is completely stored. The storage space pre-allocation module includes: a variable-length data storage space pre-allocation sub-module; used to pre-allocate variable-length cache space or composite cache space for variable-length data; The types of variable-length data include: first type of variable-length data and second type of variable-length data; wherein... The length of the first type of variable-length data is completely unpredictable; The length of the second type of variable-length data has a 90% probability of being lower than a certain length threshold; The specific implementation process of the variable-length data storage space pre-allocation submodule is as follows: When the data is of the first type of variable length data, a variable-length cache space is pre-allocated for storage; When the data is the second type of variable-length data, a fixed-length cache space is pre-allocated for storage, and the size of the space is equal to the length threshold. For the first type of variable-length data, when the sum of the actual received data length and the data length already stored in the cache space set by the variable-length data storage space pre-allocation submodule exceeds the set storage space size, the cache space is dynamically expanded until the data is completely stored. For the second type of variable-length data, when the sum of the actual received data length and the data length already stored in the cache space set by the variable-length data storage space pre-allocation submodule exceeds the length threshold, a dynamically allocated variable-length cache space is used to replace the original fixed-length cache space, and the data stored in the original cache space is copied to the newly applied variable-length cache space, thereby realizing the dynamic expansion of the cache space and the switching of the cache space until the data is completely stored.

4. The caching system for structured streaming data according to claim 3, characterized in that, The storage space pre-allocation module includes: a fixed-length data storage space pre-allocation sub-module; It is used to set up a contiguous memory space based on the length of fixed-length data. The size of the space is equal to the length of the fixed-length data and cannot be expanded.

Images