Data storage method and device and computer equipment
By using smaller data blocks and compression technology in the data storage method in the field of network security, the problem of raw data packet storage resources is solved, and the memory and hard disk resources are saved and query efficiency is improved.
Patent Information
- Application Number
- CN202411223967.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2025-01-03
AI Technical Summary
In the field of network security, the storage of raw data packets requires a large amount of memory and hard disk resources, resulting in performance degradation.
By storing traffic data into a smaller first data block, when the preset capacity threshold is reached, it is compressed into a larger second data block and stored on the hard disk, using the smaller memory A data block stores traffic data, saves memory resources, and improves query efficiency through data collection areas and hash processing.
It effectively reduces the use of memory and hard disk resources, and improves the performance of data storage and query.
Smart Images
Figure CN119248185A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of information security technology, and in particular to a data storage method, device and computer equipment. Background Art
[0002] In the field of network security, retaining and querying original data packets is an important technical means for network analysts to discover network attacks. However, the number of original data packets is large, and storing this data requires a lot of storage resources, which also leads to poor performance when querying original data packets.
[0003] In the traditional technology, when storing the original data packets, there is a problem of occupying too much memory resources and hard disk resources. Summary of the invention
[0004] Based on this, it is necessary to provide a data storage method, device and computer equipment to address the above technical problems, which can reduce the usage of memory resources and hard disk resources when storing original data packets.
[0005] In a first aspect, the present application provides a data storage method, the method comprising:
[0006] Get traffic data for different sessions;
[0007] storing each of the flow data in a first data block corresponding to each session in the memory; wherein the flow data stored in each first data block belongs to the same session;
[0008] After the accumulated traffic data of each of the first data blocks in the memory reaches a preset capacity threshold, the traffic data stored in each of the first data blocks is compressed into a second data block in the memory; wherein the data capacity of the second data block is greater than the data capacity of the first data block;
[0009] The second data block is stored in the hard disk.
[0010] In an embodiment of the present application, the data capacity of the second data block is greater than the data capacity of the first data block; the first data block with a smaller memory is used to store traffic data, thereby saving the loss of memory resources; when the compressed second data block is stored in the hard disk, the occupancy of the hard disk resources is also saved, thereby reducing the occupancy of memory resources and hard disk resources.
[0011] In one embodiment, after the accumulated traffic data of each of the first data blocks in the memory reaches a preset capacity threshold, compressing the traffic data stored in each of the first data blocks into a second data block in the memory includes:
[0012] For each first data block, after the data in the first data block is stored, the data stored in the first data block is migrated to a data collection area of the corresponding session;
[0013] After the accumulated flow data in each of the data collection areas reaches the preset capacity threshold, the accumulated flow data in each of the data collection areas is compressed into the second data block in the memory.
[0014] In the embodiment of the present application, by compressing the accumulated flow data in each data collection area into the second data block in the memory, the occupation of memory resources can be saved.
[0015] In one embodiment, the step of migrating the data stored in the first data block to a data collection area of the corresponding session includes:
[0016] determining, according to the session identification data corresponding to the first data block, a data collection area of the session to which the first data block belongs;
[0017] The traffic data stored in the first data block is migrated to the corresponding data collection area.
[0018] In the embodiment of the present application, a unique data collection area is determined by session identification data, so that the traffic data of the same session can be stored in the same data collection area, which is convenient for subsequent query of the traffic data of the same session.
[0019] In one of the embodiments, determining the data collection area of the session to which the first data block belongs according to the session identification data corresponding to the first data block includes:
[0020] The session identification data corresponding to the first data block is hashed to obtain an area identification of a data collection area of the session to which the first data block belongs.
[0021] In an embodiment of the present application, by performing hash processing on the session identification data to determine a unique data collection area, the traffic data of the same session can be stored in the same data collection area, facilitating subsequent queries on the traffic data of the same session.
[0022] In one embodiment, after compressing the accumulated flow data in each of the data collection areas into the second data block in the memory, the method further includes:
[0023] Determining storage location information of traffic data of the same session in the second data block;
[0024] The storage location information is associated with the second data block and stored.
[0025] In the embodiment of the present application, the length of the traffic data corresponding to each session and the position information in the second data block are associated and stored with the second data block, so as to facilitate querying the traffic data corresponding to the session identifier according to the session identifier.
[0026] In one embodiment, storing each of the traffic data in a first data block corresponding to each session in the memory includes:
[0027] The flow data is stored in the first data block corresponding to each session in the memory through the main thread, and the data block identifier of each first data block that has been stored is stored in the first queue;
[0028] Correspondingly, after the accumulated flow data of each of the first data blocks in the memory reaches a preset capacity threshold, the flow data stored in each of the first data blocks is compressed into a second data block in the memory, including:
[0029] Determine, by merging threads, whether the accumulated flow data of the first data blocks corresponding to each data block identifier in the first queue reaches a preset capacity threshold;
[0030] If so, the accumulated traffic data of each of the first data blocks in the first queue is compressed into the second data block in the memory through the merging thread.
[0031] In an embodiment of the present application, the operation of storing the traffic data of the session is performed by the main thread, and the operation of compressing the traffic data of the session is performed by the merging thread, which can improve the efficiency and real-time performance of storing the traffic data of the session.
[0032] In one embodiment, storing the second data block in a hard disk includes:
[0033] storing the second data block in a second queue;
[0034] The second data blocks in the second queue are stored in the hard disk in sequence through the hard disk compression thread.
[0035] In the embodiment of the present application, the second data block is stored in the hard disk through the hard disk compression thread to relieve the pressure of the main thread.
[0036] In one embodiment, if there is a merge thread, storing the second data block in the second queue includes:
[0037] The second data block is stored in a second queue through the merging thread.
[0038] In the embodiment of the present application, the second data blocks in the second queue are sequentially stored in the hard disk through the hard disk compression thread, which can relieve the pressure of the main thread to ensure the efficiency and real-time performance of the main thread in storing the flow data of the session.
[0039] In a second aspect, the present application further provides a data storage device, the device comprising:
[0040] The acquisition module is used to obtain the traffic data of different sessions;
[0041] A first storage module, used for storing each of the flow data in a first data block corresponding to each session in the memory; wherein the flow data stored in each first data block belongs to the same session;
[0042] A compression module, configured to compress the traffic data stored in each of the first data blocks into a second data block in the memory after the accumulated traffic data of each of the first data blocks in the memory reaches a preset capacity threshold; wherein the data capacity of the second data block is greater than the data capacity of the first data block;
[0043] The second storage module is used to store the second data block in a hard disk.
[0044] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:
[0045] Get traffic data for different sessions;
[0046] storing each of the flow data in a first data block corresponding to each session in the memory; wherein the flow data stored in each first data block belongs to the same session;
[0047] After the accumulated traffic data of each of the first data blocks in the memory reaches a preset capacity threshold, the traffic data stored in each of the first data blocks is compressed into a second data block in the memory; wherein the data capacity of the second data block is greater than the data capacity of the first data block;
[0048] The second data block is stored in the hard disk.
[0049] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0050] Get traffic data for different sessions;
[0051] storing each of the flow data in a first data block corresponding to each session in the memory; wherein the flow data stored in each first data block belongs to the same session;
[0052] After the accumulated traffic data of each of the first data blocks in the memory reaches a preset capacity threshold, the traffic data stored in each of the first data blocks is compressed into a second data block in the memory; wherein the data capacity of the second data block is greater than the data capacity of the first data block;
[0053] The second data block is stored in the hard disk.
[0054] In a fifth aspect, the present application further provides a computer program product, the computer program product comprising a computer program, which implements the following steps when executed by a processor:
[0055] Get traffic data for different sessions;
[0056] storing each of the flow data in a first data block corresponding to each session in the memory; wherein the flow data stored in each first data block belongs to the same session;
[0057] After the accumulated traffic data of each of the first data blocks in the memory reaches a preset capacity threshold, the traffic data stored in each of the first data blocks is compressed into a second data block in the memory; wherein the data capacity of the second data block is greater than the data capacity of the first data block;
[0058] The second data block is stored in the hard disk.
[0059] The above data storage method, device and computer equipment obtain the flow data of different sessions; then store each flow data in the first data block corresponding to each session in the memory, so that the flow data stored in each first data block belongs to the same session; and after the accumulated flow data of each first data block in the memory reaches the preset capacity threshold, the flow data stored in each first data block is compressed into the second data block in the memory; and then the second data block is stored in the hard disk. In the above scheme, the data capacity of the second data block is greater than the data capacity of the first data block; using the first data block with a smaller memory to store flow data saves the loss of memory resources; when the compressed second data block is stored in the hard disk, the hard disk resource occupation is also saved, thereby reducing the occupation of memory resources and hard disk resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 A schematic diagram of a flow chart of a data storage method in one embodiment;
[0061] Figure 2is a schematic diagram of a flow chart of compressing flow data into a second data block in one embodiment;
[0062] Figure 3 A schematic diagram of a process of migrating traffic data to a data collection area in one embodiment;
[0063] Figure 4 is a schematic diagram of a process of storing a second data block to a hard disk in one embodiment;
[0064] Figure 5 A schematic diagram of a process of storing traffic data in a main thread in one embodiment;
[0065] Figure 6 A schematic diagram of a process of merging threads to compress traffic data in one embodiment;
[0066] Figure 7 A schematic diagram of a process of storing a second data block by a hard disk compression thread in one embodiment;
[0067] Figure 8 A schematic diagram of a flow chart of another data storage method in one embodiment;
[0068] Fig. 9 is a structural block diagram of a data storage device in one embodiment;
[0069] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0070] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0071] The data storage method provided in the embodiment of the present application can be applied to the application scenario of storing the flow data of the session. The method can be executed by a server or by a terminal with a certain computing power. Among them, the server can be implemented as an independent server or a server cluster composed of multiple servers. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc.
[0072] In one embodiment, Figure 1 As shown, a data storage method is provided, which is described by taking the method applied to a server as an example, and includes the following steps:
[0073] S101, obtaining traffic data of different sessions.
[0074] A session can be understood as a group of data packets transmitted in the network, which constitute a complete communication process from start to finish. The flow data of a session can be understood as the data generated during the session, including the number, size, transmission direction, source address and destination address of the data packets. For example, the flow data of different sessions can be captured through the flow monitoring function of the server.
[0075] Specifically, the main thread can be set to obtain traffic data. When traffic data of different sessions are obtained, a cumulative thread packet number can be set for the traffic data of the same session according to the order in which the main thread obtains the traffic data. The thread packet number can be used to distinguish the order of each data packet in a session, which is convenient for sorting when extracting traffic data in subsequent queries.
[0076] S102: Store each flow data in a first data block corresponding to each session in the memory.
[0077] The first data block can be understood as a data block with a relatively small storage capacity in the memory. For example, the memory capacity of the first data block can be preset to 16KB. Each flow data can be stored in the first data block in the memory, and the flow data stored in each first data block belongs to the same session. For example, there may be multiple flow data in the same session, and the flow data of the same session can be stored in one or more first data blocks, and one first data block can only store flow data of one session, so as to establish a corresponding relationship between the first data block and the session.
[0078] For example, for each flow data, before writing into the first data block, the thread receiving packet number, timestamp and the byte size of the flow data and other header information can be written into the first data block in sequence, and then the flow data can be written into the first data block. In this way, the receiving order and data volume of each flow data can be determined.
[0079] S103, after the accumulated flow data of each first data block in the memory reaches a preset capacity threshold, compressing the flow data stored in each first data block into a second data block in the memory.
[0080] The second data block can be understood as a data block with a larger storage capacity in the memory. For example, the storage capacity of the second data block can be 1GB, wherein the data capacity of the second data block is greater than the data capacity of the first data block. The accumulated traffic data of each first data block can be understood as the total amount of traffic data stored in each first data block. The preset capacity threshold is the same as the capacity of the second data block; thus, after the accumulated traffic data of each first data block in the memory reaches the preset capacity threshold, a second data block can be applied for, and the traffic data stored in each first data block can be classified according to the session, and the traffic data corresponding to each session can be arranged in order, and then the traffic data corresponding to each session can be spliced, and the spliced traffic data can be compressed into the second data block using a compression algorithm.
[0081] S104, storing the second data block in the hard disk.
[0082] Furthermore, the second data block can be stored in a folder in the hard disk, and the second data block can be associated with the folder. For example, a file_id can be assigned to each second data block, and the file_id is used to identify the folder where the second data block is located; log information is generated based on the file_id and the folder storage path to facilitate querying the second data block.
[0083] The above data storage method obtains the flow data of different sessions; then stores each flow data in the first data block corresponding to each session in the memory, so that the flow data stored in each first data block belongs to the same session; and after the accumulated flow data of each first data block in the memory reaches the preset capacity threshold, the flow data stored in each first data block is compressed into the second data block in the memory; and then the second data block is stored in the hard disk. In the above scheme, the data capacity of the second data block is greater than the data capacity of the first data block; the first data block with a smaller memory is used to store the flow data, which saves the loss of memory resources; when the compressed second data block is stored in the hard disk, the hard disk resource occupation is also saved, thereby reducing the occupation of memory resources and hard disk resources.
[0084] In some optional implementations, in order to further save memory resources, when the accumulated traffic data of the first data block reaches the capacity of a second data block, the traffic data in each first data block can be compressed and stored in the second data block to further save memory resources.
[0085] See also Figure 2 , Figure 2 A schematic diagram of a process for compressing flow data into a second data block in a memory is provided, which specifically includes the following steps:
[0086] S201 , for each first data block, after the data in the first data block is completely stored, the data stored in the first data block is migrated to a data collection area of the corresponding session.
[0087] The completion of storage of the data in the first data block can be understood as the data stored in the first data block reaching the data capacity of the first data block; or the session corresponding to the stored traffic data ends, even if the traffic data stored in the first data block does not reach the data capacity of the first data block, the data is considered to be stored.
[0088] It should be noted that one data collection area corresponds to one session, that is, the traffic data of the same session will be stored in one data collection area. After the data in the first data block is stored, the data stored in the first data block can be migrated to the data collection area of the session to which it belongs. After the data stored in the first data block is migrated to the data collection area of the session to which it belongs, these first data blocks can be released so that these data blocks can continue to store other traffic data, so as to improve the utilization rate of memory resources.
[0089] Exemplarily, the data stored in the first data block may be migrated to the data collection area of the corresponding session in sequence according to the corresponding session and the order of receipt.
[0090] S202, after the accumulated flow data in each data collection area reaches a preset capacity threshold, compress the accumulated flow data in each data collection area into a second data block in the memory.
[0091] After the accumulated flow data in each data collection area reaches a preset capacity threshold, the accumulated flow data in each data collection area may be compressed into a second data block in the memory.
[0092] In the embodiment of the present application, by compressing the accumulated flow data in each data collection area into the second data block in the memory, the occupation of memory resources can be saved.
[0093] Furthermore, a data collection area stores the flow data of a session. After the accumulated flow data in each data collection area is compressed and stored in the second data block, the storage location information of the flow data of the same session in the second data block can be determined, for example, the length of each data collection area and the starting position in the second data block can be determined; and then the storage location information can be stored in association with the second data block, for example, the length of each data collection area and the starting position in the second data block can be stored with the second data block identifier. In this way, by associating the storage location information of the flow data of the same session in the second data block with the second data block, it is convenient to query the flow data corresponding to the session identifier according to the session identifier information.
[0094] In some optional implementations, the present application embodiment provides an implementation of how to determine the data aggregation area corresponding to each session. Figure 3 , Figure 3 A flowchart for migrating traffic data to a data collection area is provided, which specifically includes the following steps:
[0095] S301: Determine, according to session identification data corresponding to a first data block, a data collection area of a session to which the first data block belongs.
[0096] Exemplarily, the session identification data may be at least two of a source Internet Protocol (IP) address, a source port number, a destination IP address, a destination port number, and a transport layer protocol type; each first data block stores traffic data of a session, and the session identification data corresponding to the first data block is the session identification data corresponding to the traffic data stored in the first data block.
[0097] The data collection area can be an area for temporarily storing traffic data in the first data block, for example, it can be a linked list in a hash table; thereby, based on the session identification data corresponding to the first data block, the data collection area of the session to which the first data block belongs is determined; the session identification data of the same session corresponds to the same data collection area.
[0098] S302, migrate the traffic data stored in the first data block to the corresponding data collection area.
[0099] The flow data stored in the first data block can then be migrated to the corresponding data collection area, so that the flow data of the same session can be migrated to the same data collection area, which is convenient for subsequent query of the flow data of the same session.
[0100] In the embodiment of the present application, a unique data collection area is determined by session identification data, so that the traffic data of the same session can be stored in the same data collection area, which is convenient for subsequent query of the traffic data of the same session.
[0101] Furthermore, in the above S301, when determining the data collection area of the session to which the first data block belongs, the session identification data corresponding to the first data block can be hashed to obtain the area identification of the data collection area of the session to which the first data block belongs, i.e., the hash value key; and then the linked list in the hash table, i.e., the data collection area, can be determined according to the hash value key. It should be noted that the hash value key determined by the same session identification is the same, and the first data block with the same key needs to establish a conflict linked list, so that the traffic data stored in the first data block corresponding to the same session can be stored on the same linked list.
[0102] On this basis, when compressing the accumulated traffic data into the second data block, after the accumulated traffic data in each data collection area reaches the preset capacity threshold, that is, after the size of the traffic data accumulated on the linked list of the hash table reaches the preset capacity threshold, a second data block is applied. Then, the linked lists in the hash table are traversed, and all the traffic data of the same session are merged together according to the thread packet receiving number sequence in the header information, and then the traffic data corresponding to each merged session is compressed and stored in the second data block; and the length of the traffic data corresponding to each merged session and the position information in the second data block are determined, and the length of the traffic data corresponding to each session and the position information in the second data block are associated with the second data block for storage, so as to facilitate the query of the traffic data corresponding to the session identifier according to the session identifier.
[0103] In some optional implementations, in order to improve the efficiency and real-time performance of storing session traffic data, different threads may be set to perform traffic data storage and traffic data compression operations. The specific implementation process is as follows:
[0104] Each flow data is stored in the first data block corresponding to each session in the memory through the main thread, and the data block identifier of each stored first data block is stored in the first queue.
[0105] Exemplarily, the session identifier corresponding to each first data block can be stored in the session structure, and the data block identifier of each first data block that has been stored can be stored in the session structure, wherein the data block identifier is used to identify the first data block and the storage address of the first data block in the memory. The main thread stores each flow data in the first data block corresponding to each session in the memory, and after the first data block is stored, the session structure containing the data block identifier of each first data block that has been stored is stored in the first queue Q1. This is to facilitate data compression operations.
[0106] Furthermore, the data compression operation can be completed by the merge thread, which does not affect the flow data of the main thread storing the session, thereby improving the efficiency and real-time performance of the flow data of the storage session.
[0107] Specifically, by merging threads, the data collection area corresponding to the session identifier can be determined according to the session identifier in each session structure in the first queue Q1; for example, a hash value key can be determined according to the session identifier, and a linked list in the hash table can be determined according to the hash value key; and then the data in the first data block can be migrated to the data collection area according to the data block identifier in the session structure.
[0108] Then, it is determined whether the cumulative flow data of the first data blocks corresponding to each data block identifier in the first queue Q1 reaches the preset capacity threshold; that is, it can be understood as determining whether the cumulative flow data in each data collection area reaches the preset capacity threshold; if the preset capacity threshold is reached, the cumulative flow data of each first data block in the first queue can be compressed into the second data block in the memory through the merge thread; that is, the cumulative flow data in each data collection area is compressed into the second data block in the memory.
[0109] In this way, the operation of storing the traffic data of the session is performed by the main thread, and the operation of compressing the traffic data of the session is performed by the merging thread, which can improve the efficiency and real-time performance of storing the traffic data of the session.
[0110] In some optional implementations, a hard disk compression thread may be used to store the second data block in the hard disk to relieve the pressure of the main thread.
[0111] See also Figure 4 , Figure 4 A schematic diagram of a process of storing a second data block to a hard disk is provided, which specifically includes the following steps:
[0112] S401: Store the second data block in the second queue.
[0113] Exemplarily, the main thread may be used to store the second data block in the second queue Q2, so that the hard disk compression thread stores the second data block in the second queue Q2 to the hard disk at one time.
[0114] Specifically, if there is a merge thread, the second data block can be stored in the second queue through the merge thread, so that the hard disk compression thread can store each second data block in the second queue in the hard disk in sequence. Among them, using the merge thread to store the second data block in the second queue can share the pressure of the main thread, so as to ensure the efficiency and real-time performance of the main thread in storing the flow data of the session.
[0115] S402, storing the second data blocks in the second queue in sequence into the hard disk through the hard disk compression thread.
[0116] Specifically, the second data blocks in the second queue can be stored in the hard disk in sequence through the hard disk compression thread, so as to relieve the pressure of the main thread and ensure the efficiency and real-time performance of the main thread in storing the flow data of the session.
[0117] Exemplarily, when there are a main thread, a merge thread, and a hard disk compression thread, the steps executed by each thread are described below.
[0118] See also Figure 5 , Figure 5A schematic diagram of the process of storing traffic data in the main thread is provided, which specifically includes the following steps:
[0119] First, obtain the traffic data of different sessions.
[0120] A session is constructed according to the session identifier in the flow data, and an association relationship is established between the first data block storing the flow data and the session.
[0121] The flow data is stored in the first data block. Specifically, the data stored in the first data block includes header information and flow data, wherein the header information at least includes a thread packet receiving number, a timestamp, and a byte size of the flow data.
[0122] It is determined whether the data in the first data block has been stored. If yes, the data block identifier and session identifier of the first data block are stored in the first queue Q1. If no, the operation of storing the traffic data in the first data block is continued.
[0123] See also Figure 6 , Figure 6 A schematic diagram of a process for merging thread compression flow data is provided, which specifically includes the following steps:
[0124] The data block identifier and the session identifier are obtained from the first queue Q1.
[0125] Determine whether the node corresponding to the session identifier already exists in the hash table according to the session identifier. If so, migrate the traffic data stored in the first data block corresponding to the data block identifier to the conflict linked list corresponding to the node. If not, determine a hash head node according to the session identifier, and migrate the traffic data stored in the first data block corresponding to the data block identifier to the hash head node.
[0126] After the traffic data accumulated and stored in the hash table reaches the preset capacity threshold, a second data block is requested.
[0127] The hash table and conflict chain are traversed to sort the traffic data corresponding to each session according to the thread packet receiving number in the header data, and the sorted traffic data of each session are concatenated.
[0128] The spliced traffic data is compressed into a large data block using a compression algorithm; and a log is generated based on the traffic data length of each session, the position information in the second data block, and the session identifier, and the log is stored so that the traffic data of the corresponding session can be obtained based on the log.
[0129] After the traffic data stored in the hash table and the conflict linked list are compressed and stored in the second data block, the second data block is pushed to the second queue Q2.
[0130] See also Figure 7 , Figure 7 A schematic diagram of a process of storing a second data block in a hard disk compression thread is provided, which specifically includes the following steps:
[0131] The second data block is obtained from the second queue Q2.
[0132] A file identifier file_id is allocated to each second data block.
[0133] The second data block is stored in the hard disk, and a log of the second data block is generated, wherein the log includes a file identifier for storing the second data block and a storage path of the file.
[0134] When a user wants to query the traffic data of a session, he can query the relevant logs according to the session identifier and timestamp information sent by the user to confirm the file path, starting position and length of the session traffic data. After reading the traffic data of the same session from multiple files, sort them according to the timestamp in the header information of each traffic data, and then write them into the packet capture (pcap) file in sequence, and finally get all the traffic data of the session you want to query.
[0135] See also Figure 8 , Figure 8 A flowchart of another data storage method is provided, which specifically includes the following steps:
[0136] S801, obtaining traffic data of different sessions through the main thread.
[0137] S802, storing each flow data in a first data block corresponding to each session in the memory through the main thread, and storing the data block identifier of each stored first data block in the first queue.
[0138] S803, by merging threads, for each first data block corresponding to the data identifier in the first queue, hash processing is performed on the session identifier data corresponding to the first data block to obtain the area identifier of the data collection area of the session to which the first data block belongs.
[0139] S804, migrate the traffic data stored in the first data block to the corresponding data collection area through the merging thread.
[0140] S805, by merging threads, determine whether the accumulated flow data of the first data block corresponding to each data block identifier in the first queue reaches a preset capacity threshold, if so, execute S806, if not, continue to execute S805.
[0141] S806, compressing the accumulated traffic data in each data collection area into a second data block in the memory by merging threads.
[0142] S807: Store the second data block into the second queue by merging threads.
[0143] S808, storing the second data blocks in the second queue in sequence in the hard disk through the hard disk compression thread.
[0144] The data storage method provided in the embodiment of the present application uses two data blocks of different sizes to store traffic data, which can better adapt to the real-time storage of raw data traffic in complex traffic conditions, while saving memory loss; multiple first data blocks use a combination of hash tables and linked lists to make the merging performance of the same session higher, and merging the same session for processing in the merging thread will not hinder other operations of the session in the main thread, reducing the pressure on the main thread. In addition, after merging the traffic data in multiple first data blocks, a compression algorithm is used to compress it from memory to memory, which not only reduces the disk occupancy consumption after the data is written to the disk, but also improves the compression performance.
[0145] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0146] Based on the same inventive concept, the embodiment of the present application also provides a data storage device for implementing the data storage method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data storage device embodiments provided below can refer to the limitations on the data storage method above, and will not be repeated here.
[0147] In one embodiment, Fig. 9 As shown, a data storage device is provided, comprising:
[0148] An acquisition module 10, used to acquire traffic data of different sessions;
[0149] The first storage module 20 is used to store each flow data in a first data block corresponding to each session in the memory; wherein the flow data stored in each first data block belongs to the same session;
[0150] The compression module 30 is used to compress the traffic data stored in each first data block into a second data block in the memory after the accumulated traffic data of each first data block in the memory reaches a preset capacity threshold; wherein the data capacity of the second data block is greater than the data capacity of the first data block;
[0151] The second storage module 40 is used to store the second data block in the hard disk.
[0152] The above data storage device obtains the flow data of different sessions; then stores each flow data in the first data block in the memory, so that the flow data stored in each first data block belongs to the same session; and after the accumulated flow data of each first data block in the memory reaches the preset capacity threshold, the flow data stored in each first data block is compressed into the second data block in the memory; and then the second data block is stored in the hard disk. In the above scheme, the data capacity of the second data block is greater than the data capacity of the first data block; the first data block with a smaller memory is used to store the flow data, which saves the loss of memory resources; when the compressed second data block is stored in the hard disk, the hard disk resource occupation is also saved, thereby reducing the occupation of memory resources and hard disk resources.
[0153] In one embodiment, the compression module 30 specifically includes:
[0154] A migration unit, configured to migrate the data stored in each first data block to a data collection area of the corresponding session after the data in the first data block is stored;
[0155] The compression unit is used to compress the accumulated flow data in each data collection area into a second data block in the memory after the accumulated flow data in each data collection area reaches a preset capacity threshold.
[0156] In one embodiment, the migration unit specifically includes:
[0157] A determination subunit, configured to determine a data collection area of a session to which the first data block belongs according to the session identification data corresponding to the first data block;
[0158] The migration subunit is used to migrate the traffic data stored in the first data block to the corresponding data collection area.
[0159] In one embodiment, the determining subunit is specifically configured to:
[0160] The session identification data corresponding to the first data block is hashed to obtain an area identification of a data collection area of the session to which the first data block belongs.
[0161] In one embodiment, the device further comprises an associated storage module, which is used to: after compressing the accumulated flow data in each data collection area into the second data block in the memory, the associated storage module is used to:
[0162] Determine storage location information of the traffic data of the same session in the second data block; and store the storage location information in association with the second data block.
[0163] In one embodiment, the first storage module 20 is specifically used for:
[0164] The main thread is used to store each flow data in a first data block corresponding to each session in the memory, and the data block identifier of each first data block that has been stored is stored in the first queue;
[0165] Accordingly, the compression module 30 is specifically used for:
[0166] Through the merging thread, determine whether the cumulative flow data of the first data blocks corresponding to each data block identifier in the first queue reaches the preset capacity threshold; if so, compress the cumulative flow data of each first data block in the first queue into the second data block in the memory through the merging thread.
[0167] In one embodiment, the second storage module 40 specifically includes:
[0168] A first storage unit, used for storing the second data block in the second queue;
[0169] The second storage unit stores the second data blocks in the second queue in sequence in the hard disk through the hard disk compression thread.
[0170] In one embodiment, the first storage unit is specifically used for:
[0171] By merging the threads, the second data block is stored in the second queue.
[0172] Each module in the above data storage device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0173] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the flow data of the session. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data storage method is implemented.
[0174] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0175] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0176] Get traffic data for different sessions;
[0177] storing each flow data in a first data block corresponding to each session in the memory; wherein the flow data stored in each first data block belongs to the same session;
[0178] After the accumulated flow data of each first data block in the memory reaches a preset capacity threshold, the flow data stored in each first data block is compressed into a second data block in the memory; wherein the data capacity of the second data block is greater than the data capacity of the first data block;
[0179] The second data block is stored in the hard disk.
[0180] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0181] For each first data block, after the data in the first data block is stored, the data stored in the first data block is migrated to the data collection area of the session to which it belongs; after the accumulated traffic data in each data collection area reaches a preset capacity threshold, the accumulated traffic data in each data collection area is compressed into the second data block in the memory.
[0182] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0183] According to the session identification data corresponding to the first data block, the data collection area of the session to which the first data block belongs is determined; and the flow data stored in the first data block is migrated to the corresponding data collection area.
[0184] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0185] The session identification data corresponding to the first data block is hashed to obtain an area identification of a data collection area of the session to which the first data block belongs.
[0186] In one embodiment, after compressing the accumulated flow data in each data collection area into the second data block in the memory, the processor further implements the following steps when executing the computer program:
[0187] Determine storage location information of the traffic data of the same session in the second data block; and store the storage location information in association with the second data block.
[0188] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0189] The main thread stores each flow data in the first data block corresponding to each session in the memory, and stores the data block identifier of each stored first data block in the first queue; through the merge thread, determine whether the cumulative flow data of the first data blocks corresponding to each data block identifier in the first queue reaches the preset capacity threshold; if so, the merge thread compresses the cumulative flow data of each first data block in the first queue into the second data block in the memory.
[0190] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0191] The second data block is stored in the second queue; and each second data block in the second queue is stored in the hard disk in sequence through the hard disk compression thread.
[0192] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0193] By merging the threads, the second data block is stored in the second queue.
[0194] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data storage method described in any of the above embodiments is implemented.
[0195] In one embodiment, a computer program product is provided, including a computer program, which implements the data storage method described in any of the above embodiments when executed by a processor.
[0196] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.
[0197] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0198] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0199] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A data storage method, characterized in that: The method comprises: Get traffic data for different sessions; storing each of the flow data in a first data block corresponding to each session in the memory; wherein the flow data stored in each first data block belongs to the same session; After the accumulated traffic data of each of the first data blocks in the memory reaches a preset capacity threshold, the traffic data stored in each of the first data blocks is compressed into a second data block in the memory; wherein the data capacity of the second data block is greater than the data capacity of the first data block; The second data block is stored in the hard disk.
2. The method according to claim 1, characterized in that After the accumulated flow data of each of the first data blocks in the memory reaches a preset capacity threshold, compressing the flow data stored in each of the first data blocks into a second data block in the memory includes: For each first data block, after the data in the first data block is stored, the data stored in the first data block is migrated to a data collection area of the corresponding session; After the accumulated flow data in each of the data collection areas reaches the preset capacity threshold, the accumulated flow data in each of the data collection areas is compressed into a second data block in the memory.
3. The method according to claim 2, characterized in that The step of migrating the data stored in the first data block to a data collection area of the corresponding session includes: determining, according to the session identification data corresponding to the first data block, a data collection area of the session to which the first data block belongs; The traffic data stored in the first data block is migrated to the corresponding data collection area.
4. The method according to claim 3, characterized in that The step of determining, according to the session identification data corresponding to the first data block, a data collection area of the session to which the first data block belongs includes: The session identification data corresponding to the first data block is hashed to obtain an area identification of a data collection area of the session to which the first data block belongs.
5. The method according to claim 2, characterized in that: After compressing the accumulated flow data in each of the data collection areas into the second data block in the memory, the method further includes: Determining storage location information of traffic data of the same session in the second data block; The storage location information is associated with the second data block and stored.
6. The method according to claim 1, characterized in that The storing of each of the traffic data into the first data block corresponding to each session in the memory includes: The flow data is stored in the first data block corresponding to each session in the memory through the main thread, and the data block identifier of each first data block that has been stored is stored in the first queue; Correspondingly, after the accumulated traffic data of each of the first data blocks in the memory reaches a preset capacity threshold, the traffic data stored in each of the first data blocks is compressed into a second data block in the memory, including: Determine, by merging threads, whether the accumulated flow data of the first data blocks corresponding to each data block identifier in the first queue reaches a preset capacity threshold; If so, the accumulated traffic data of each of the first data blocks in the first queue is compressed into the second data block in the memory through the merging thread.
7. The method according to any one of claims 1 to 6, characterized in that The storing the second data block in the hard disk includes: storing the second data block in a second queue; The second data blocks in the second queue are stored in the hard disk in sequence through the hard disk compression thread.
8. The method according to claim 7, characterized in that If there is a merge thread, storing the second data block in the second queue includes: The second data block is stored in a second queue through the merging thread.
9. A data storage device, characterized in that: The device comprises: The acquisition module is used to obtain the traffic data of different sessions; A first storage module, used for storing each of the flow data in a first data block corresponding to each session in the memory; wherein the flow data stored in each first data block belongs to the same session; A compression module, configured to compress the traffic data stored in each of the first data blocks into a second data block in the memory after the accumulated traffic data of each of the first data blocks in the memory reaches a preset capacity threshold; wherein the data capacity of the second data block is greater than the data capacity of the first data block; The second storage module is used to store the second data block in a hard disk.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.