Cluster-oriented network communication log storage method, cluster server and cluster

Abstract

The application discloses a cluster-oriented network communication log storage method, a cluster server and a cluster, and belongs to the field of cluster communication log processing.The method comprises the following steps: acquiring network communication log records from a message queue in a server, extracting five-tuple information as a communication connection record, searching for the communication connection record in a log bucket, updating the last use time of the communication connection record in the log bucket and a graph database if the search is successful, otherwise, generating an ID for the communication connection record and inserting the communication connection record into the log bucket and the graph database, wherein the log bucket is a six-layer multi-way tree in memory, the second to fifth layers of nodes of the six-layer multi-way tree are source IP, source port, destination IP and destination port in sequence, and leaf nodes store the ID, the last use time and a protocol of the communication connection record, and the graph database is a graph structure in a persistent storage device, nodes of the graph structure are host IPs, and edges exist between host nodes that have network communication. The application can effectively improve the storage efficiency of massive network communication logs in a cluster.

Description

Technical Field

[0001] This invention belongs to the field of cluster communication log processing, and more specifically, relates to a cluster-oriented network communication log storage method, cluster server, and cluster. Background Technology

[0002] In a host cluster, to ensure the communication and security of the entire cluster, it is necessary to monitor the existing communication connections. By recording network communication logs of these connections, it is possible to better identify network faults, intrusions, and call relationships within the cluster. Typically, each host collects its own communication log information and sends it to a designated server for aggregation. The server then aggregates, stores, and displays the network communication logs of the current cluster.

[0003] Typically, clusters contain a large number of hosts, resulting in massive amounts of data being sent to servers, placing significant storage pressure on them. Furthermore, communication logs are often time-dependent, requiring queries based on time ranges. Within a given timeframe, two hosts may initiate multiple communications, leading to numerous redundant 5-tuples in the collected communication traffic logs, reducing storage and query efficiency. These factors present numerous challenges to the collection, storage, and visualization of cluster network communication logs. Existing methods often employ simple relational databases to store network communication logs, failing to effectively address the storage and redundancy issues associated with communication traffic logs.

[0004] Furthermore, network communication logs exhibit characteristics of hot and cold data; specifically, data from more recent times is accessed less frequently, while more recent data is accessed more frequently. This characteristic can be used to improve the query efficiency of network communication logs. However, existing methods for storing network communication logs using relational databases cannot fully utilize this hot and cold data characteristic, thus query efficiency needs further improvement. Summary of the Invention

[0005] In response to the shortcomings and improvement needs of existing technologies, this invention provides a network communication log storage method, a cluster server, and a cluster for clusters, with the aim of effectively improving the storage efficiency of massive network communication logs in a cluster.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for storing network communication logs for clusters is provided, comprising: simultaneously, while each host in the cluster sends the acquired network communication log records to the message queue of the server, the following steps are performed on the server:

[0007] Retrieve network communication log records from the message queue, extract the 5-tuple information as a communication connection record, search for the communication connection record in the log bucket. If the search is successful, update the last usage time of the communication connection record in the log bucket and the graph database to the current time. If the search is unsuccessful, generate an ID for the communication connection record, insert the communication connection record into the log bucket and the graph database, and set its last usage time to the establishment time of the communication connection.

[0008] The communication log records include 5-tuple information and the establishment time of the communication connection; the log bucket is a six-level multi-way tree created in memory, where the nodes of levels 2 to 5 are the source IP, source port, destination IP, and destination port, respectively, and each connection record corresponds to a path from the root node to a leaf node in the multi-way tree. The leaf node stores information including the ID of the communication connection record, the last time it was used, and the protocol; the graph database is a graph structure created in a persistent storage device, where the nodes are host IPs, and there are edges between the nodes corresponding to the hosts that have conducted network communication, and the edge information includes the ID of the communication connection record, the last time it was used, the protocol, the source port, and the destination port.

[0009] Furthermore, when inserting communication connection records that do not exist in the log bucket into the log bucket and the database, the process also includes: using the establishment time of the communication connection as a timestamp, and inserting it into the hot database along with the ID of the communication connection record; the hot database is a time-series database used to store communication connection records within the most recent N time windows.

[0010] Furthermore, cluster-oriented network communication log storage methods also include:

[0011] At fixed times within each time window, data from the earliest time window in the hot storage is migrated to the cold storage; the cold storage is a time-series database used to store communication connection records within the most recent N+1 to M time windows;

[0012] Where M and N are both positive integers, and M > 2N.

[0013] Furthermore, N time windows correspond to one week, and M time windows correspond to one month.

[0014] Furthermore, the process of migrating data from the hot storage to the cold storage also includes deleting data from the earliest time window in the cold storage.

[0015] Furthermore, in addition to deleting the data from the earliest time window in the cold storage, it also includes:

[0016] Determine whether each ID currently being deleted was used in other time windows. If not, delete the communication connection record corresponding to that ID in the log bucket and graph database.

[0017] Furthermore, the ID of the communication connection record is an integer number assigned starting from 0 and incrementing sequentially. The method for generating the ID for the communication connection record is as follows:

[0018] Retrieve the list of IDs corresponding to the communication connection records stored in the current graph database, and find the smallest unused value from the list of IDs to generate the ID.

[0019] Furthermore, the cluster-oriented network communication log storage method provided by the present invention further includes: performing the following steps after the server receives a query request:

[0020] (S1) Determine whether the communication connection record within the requested time range R is currently stored based on the time range of the data stored in the hot storage and cold storage. If yes, proceed to step (S2); otherwise, determine that the query result does not exist and proceed to step (S4).

[0021] (S2) Query the hot storage or cold storage according to the time range R to obtain the ID of the corresponding communication connection record and remove duplicates to obtain the ID list;

[0022] (S3) Query the graph database according to the ID list to obtain the five-tuple information of the communication connection record corresponding to each ID, complete the recovery of the network communication log, and return the recovered network communication log.

[0023] (S4) Query ends.

[0024] Furthermore, after retrieving network communication log records from the message queue and extracting the quintuple information, the process also includes: converting the four numbers separated by "." in the IP address of the quintuple information into 2-digit hexadecimal strings, thereby converting the IP address into an 8-digit hexadecimal string; and converting the port number in the quintuple information into a 2-digit hexadecimal string.

[0025] Furthermore, the communication connection record is the converted quintuple information.

[0026] Furthermore, the network communication log storage method for clusters provided by the present invention also includes a dynamic adjustment step of the transmission frequency executed on the host side;

[0027] The dynamic adjustment steps for the sending frequency include: real-time detection of network latency; if the latency exceeds a preset threshold, the frequency of sending network communication log records to the server's message queue is reduced; and when the network latency does not exceed the threshold, the frequency of sending network communication log records to the server's message queue is restored to the initial setting value.

[0028] According to another aspect of the present invention, a cluster server is provided, comprising:

[0029] A first computer-readable storage medium for storing computer programs;

[0030] And a first processor, configured to read a computer program stored in a first computer-readable storage medium and execute the cluster-oriented network communication log storage method provided by the present invention.

[0031] According to another aspect of the present invention, a cluster is provided, including the cluster server provided by the present invention; and in the cluster, each host is equipped with a dynamic frequency adjustment module.

[0032] The dynamic frequency adjustment module includes:

[0033] A second computer-readable storage medium for storing computer programs;

[0034] And a second processor, used to read the computer program stored in the second computer-readable storage medium and execute the dynamic adjustment step of the transmission frequency in the cluster-oriented network communication log storage method provided by the present invention.

[0035] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0036] (1) This invention utilizes memory-based log buckets on the server side to store network communication logs. Information from a network communication log is only inserted into the log bucket when a 5-tuple in the log log does not have a corresponding record in the log bucket, thus achieving deduplication and effectively improving the storage efficiency of massive network communication logs. Simultaneously, since the log bucket is a multi-way tree structure, and the 5-tuples in the network communication log are stored on nodes at different levels within this multi-way tree structure, overlapping 5-tuples in the first half will share the same nodes, further compressing the amount of stored data. Furthermore, it allows for quick determination of whether a network communication log record has been stored by querying the log bucket, further improving storage efficiency. In addition, a graph database is created in the persistent storage device, and its content is synchronized with the log bucket, ensuring the security and availability of log bucket information. In summary, this invention employs a storage scheme that combines log buckets and a graph database, effectively improving the storage efficiency of massive network communication logs in a cluster.

[0037] (2) In a preferred embodiment of the present invention, a hot database and a cold database are further maintained in the server based on a time-series database. These databases are used to store recently established communication connection records and communication connection records established over a longer period of time, respectively. This achieves separate storage of hot and cold data. According to the access characteristics of network communication logs in the cluster, recently generated network communication logs are more likely to be queried and tend to account for a smaller proportion. Network communication logs generated over a longer period of time are less likely to be queried and account for a larger proportion. Therefore, the data requested by the network communication log query request will mainly be concentrated in the hot database with a smaller data volume, and the query efficiency can be effectively improved.

[0038] (3) In the preferred embodiment of the present invention, while ensuring the data quality in the hot storage by periodically migrating the earliest established data in the hot storage to the cold storage, the earliest established data in the cold storage is further deleted periodically. According to the access frequency of the network communication log, the probability of these data being queried later is extremely small. Deleting these data in a timely manner can avoid the data volume in the cold storage being too large, thus saving storage space while ensuring the query efficiency of the data in the cold storage.

[0039] (4) In a preferred embodiment of the present invention, the ID of the communication connection record is set to an integer number that is incremented starting from 0. While the data is stored in the cold storage, for communication connection record IDs that are not used in other time windows, the corresponding communication connection records in the log bucket and graph database will be deleted at the same time, and the corresponding IDs will be deleted. Each time a new ID is generated, the minimum unused value is selected, thereby effectively avoiding ID depletion and avoiding the storage of invalid data in the log bucket and graph database.

[0040] (5) In a preferred embodiment of the present invention, before storing the quintuple information in the network communication log, the IP and port are compressed into hexadecimal strings, thereby improving the comparison efficiency and reducing the storage space.

[0041] (6) In a preferred embodiment of the present invention, each host will also monitor network latency in real time and reduce the frequency of sending network communication log records to the message queue of the server when the network latency is large, thereby avoiding the impact on the normal business of the cluster. Attached Figure Description

[0042] Figure 1 A flowchart of a cluster-oriented network communication log storage method provided in an embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the log bucket structure provided in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of cold and hot separation storage of a time-series database provided in an embodiment of the present invention;

[0045] Figure 4 This is a flowchart for searching log information by time range, provided as an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0047] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0048] To address the low storage efficiency of existing relational database-based methods for storing network communication logs, which suffer from low efficiency when storing massive amounts of logs, this invention provides a cluster-oriented network communication log storage method, a cluster server, and a cluster. The overall approach involves improving the data structure for storing network communication logs on the server and then refining the storage process based on this improved data structure. This allows for deduplication and compression of network communication logs during storage, effectively improving storage efficiency. Furthermore, a time-series database is used to separate hot and cold data in the network communication logs, thereby leveraging the hot and cold characteristics of the logs to improve query efficiency.

[0049] The cluster contains multiple hosts, one or more of which are selected as servers. Each host collects its own communication log information in real time and sends it to the server for storage, which can then be queried according to time range.

[0050] The host typically obtains network connection log information by listening to network connection events in the host kernel. Specifically, it uses the Netfilter conntrack module to set the reporting time interval, and within each time interval, it retrieves all flow objects in the current system and iterates through all network flow connections.

[0051] The process of obtaining network data streams through the Netfilter conntrack module is as follows:

[0052] (1) For traffic packets received in the system, try to obtain their corresponding conntrack_info and connection records;

[0053] (2) If the corresponding conntrack_info and connection record do not exist, determine whether the packet needs to be tracked. If the connection record exists or the packet does not need to be tracked, ignore the packet and do not perform any operation.

[0054] (3) Extract information from the L4 header of the traffic packet, initialize the protocol-related connection variables, define the callback method corresponding to the connection, and check the integrity of the packet, etc.

[0055] (4) Call resolve_normal_ct() to start connection tracing, and at the same time create a new conntrackentry for the package or update the corresponding conntrack entry record;

[0056] (5) In each time interval, traverse all surviving entry records, retrieve the corresponding communication information of both parties and return it.

[0057] After obtaining network connection flow information through the Netfilter conntrack module, it retrieves the five-tuple (connection initiator, source port, protocol, connection receiver, destination port) for all network connections, performs preprocessing operations such as serialization to binary, compresses the data, appends the current time, and then sends the compressed data to the message queue on the server. The server will then retrieve and store the network communication log information from this message queue.

[0058] The following is an example.

[0059] Example 1:

[0060] A cluster-oriented network communication log storage method, such as Figure 1 As shown, this includes: while each host in the cluster sends the acquired network communication log records to the server's message queue, the server performs the following steps:

[0061] Retrieve network communication log records from the message queue, extract the 5-tuple information as a communication connection record, search for the communication connection record in the log bucket. If the search is successful, update the last usage time of the communication connection record in the log bucket and the graph database to the current time. If the search is unsuccessful, generate an ID for the communication connection record, insert the communication connection record into the log bucket and the graph database, and set its last usage time to the establishment time of the communication connection.

[0062] The communication log records include quintuple information and the establishment time of the communication connection.

[0063] In this embodiment, the log bucket is a six-level multi-way tree created in memory, and its structure is as follows: Figure 2As shown, the nodes at levels 2 to 5 are, in order, the source IP, source port, destination IP, and destination port. Each connection record corresponds to a path from the root node to a leaf node in this multi-way tree. The leaf node stores information including the communication connection record's ID, last usage time, and protocol. Based on this log bucket structure, when storing quintuple data, overlapping quintuples in the first half will share the same nodes. For example, quintuples of communication connections initiated by two different hosts in the same sending direction will share the source IP node; quintuples of communication connections initiated by the same sender to two different hosts via the same port will share the source IP node and source port node, and so on. This storage method effectively compresses the amount of stored data and allows for quick determination of whether a network communication log record has been stored by querying the log bucket. It should be noted that in this embodiment, the hierarchical arrangement of the information in the quintuple within the log corresponds to the unidirectional nature of network communication connections, which improves the efficiency of querying the existence of a record in the log bucket.

[0064] In this embodiment, the graph database is a graph structure created in a persistent storage device. The nodes are host IPs, and edges exist between nodes corresponding to hosts that have engaged in network communication. Edge information includes the connection record ID, last access time, protocol, source port, and destination port. It's easy to understand that since the number of hosts in a cluster is often large, and the server cannot accurately know all the nodes in the cluster initially, when inserting a communication connection record into the graph database, it's first necessary to determine if the host nodes corresponding to the source and destination IPs already exist in the database. If they do, the corresponding edges are created and their information is recorded; otherwise, the corresponding host nodes need to be created in the graph database first. In this embodiment, the graph database created in the persistent storage device maintains consistency between record insertion and updates and the log buckets in memory, thereby achieving log bucket persistence. This ensures high query efficiency while guaranteeing the security and availability of log bucket information.

[0065] In this embodiment, when storing network communication log information using log buckets and graph databases, the information of the network communication log is only inserted into the log bucket and graph database when the five-tuple in the network communication log does not have a corresponding record in the log bucket. This achieves deduplication and effectively improves the storage efficiency of massive network communication logs.

[0066] It's easy to understand that since the host compresses the 5-tuple information when sending network communication log information, it needs to be decompressed after retrieving relevant information from the message queue. Because querying the log bucket to determine if a record is stored there requires comparing the information in the 5-tuple, to improve comparison efficiency, as a preferred implementation, this embodiment first converts the source IP, source port, destination IP, and destination port into hexadecimal strings after obtaining the 5-tuple information. Specifically, the four numbers separated by "." in the IP address of the 5-tuple information are converted into 2-digit hexadecimal strings, thus converting the IP address into an 8-digit hexadecimal string; the port address in the 5-tuple information is also converted into a 2-digit hexadecimal string; for example, the IP address "255.255.255.255" is mapped to "EEEEEEEE", and the port "65535" is mapped to "EE".

[0067] Furthermore, the communication connection record is the converted quintuple information.

[0068] By using the above conversion methods, while improving comparison efficiency, the data length is shortened after conversion, thus achieving a certain compression effect and further reducing storage space overhead.

[0069] Considering the characteristic of "hot" and "cold" data in network communication logs—that is, data from more recent times is accessed less frequently, while more recent data is accessed more frequently—this embodiment maintains a "hot" and a "cold" database based on a time-series database to fully utilize this characteristic and further improve the query efficiency of network communication logs by time range. The hot database stores recently generated network communication log data, while the cold database stores network communication log data generated relatively earlier. This ensures that the data requested by query requests is concentrated in the hot database. Since the hot database only stores recently generated data, the data volume is smaller, resulting in higher access efficiency. Compared to directly using a single time-series database to store all network communication log records, this effectively improves the overall query efficiency.

[0070] This embodiment further analyzes the distribution of hot and cold data in the cluster network communication logs and finds that network communication logs generated within the past week are accessed more frequently, while those generated more than a month ago are almost never accessed. Based on this analysis, as a preferred implementation, in this embodiment, the hot storage is specifically used to store network communication log data generated within the past week, while the cold storage is specifically used to store network communication log data generated more than a week ago and within the past month. Further analysis in this embodiment reveals that, typically, network communication activity within the cluster is least active at 12:00 midnight. Therefore, this embodiment uses a daily time window for data storage and migration, and performs data migration from the hot storage to the cold storage at 12:00 midnight every night to ensure the data's availability in the hot storage and prevent data migration from affecting the normal storage of network communication logs.

[0071] In order to maintain the aforementioned hot and cold storage facilities, such as Figure 3 As shown in this embodiment, when inserting communication connection records that do not exist in the log bucket into the log bucket and the database, the method further includes: using the establishment time of the communication connection as a timestamp, and inserting it into the hot database along with the ID of the communication connection record; the hot database is a time-series database used to store communication connection records within the most recent week.

[0072] Furthermore, cluster-oriented network communication log storage methods also include:

[0073] At 12:00 every night, the earliest data from the hot storage is migrated to the cold storage; the cold storage is a time-series database used to store communication connection records generated within the past week and the past month.

[0074] Migrating data from the hot storage to the cold storage also includes deleting data from the earliest time window in the cold storage.

[0075] For IDs deleted from the cold storage, if they are not used on other dates, the network communication log data corresponding to those IDs will no longer be accessed and will be considered invalid data. Furthermore, these IDs can be used by other communication connection records. Based on this consideration, this embodiment, while deleting data from the earliest time window in the cold storage, also includes:

[0076] Determine whether each ID currently being deleted was used in other time windows. If not, delete the communication connection record corresponding to that ID in the log bucket and graph database.

[0077] Furthermore, the ID of the communication connection record is an integer number assigned starting from 0 and incrementing sequentially. The method for generating the ID for the communication connection record is as follows:

[0078] Retrieve the list of IDs corresponding to the communication connection records stored in the current graph database, and find the smallest unused value from the list of IDs to generate the ID.

[0079] This embodiment, by combining the aforementioned cold storage data deletion mechanism with the ID generation method, can avoid ID depletion and improve ID utilization. At the same time, it can reduce the storage of invalid data in the log bucket and graph database, further improving storage utilization.

[0080] It should be noted that in practical applications, the time range of data storage in hot and cold storage, the length of the time window, and the specific timing of data migration can all be set according to the current behavior characteristics of the cluster. The above is only a preferred setting and should not be construed as the only limitation of the present invention.

[0081] like Figure 4 As shown, in this embodiment, the server will perform the following steps after receiving a query request:

[0082] (S1) Determine whether the communication connection record within the requested time range R is currently stored based on the time range of the data stored in the hot storage and cold storage. If yes, proceed to step (S2); otherwise, determine that the query result does not exist and proceed to step (S4).

[0083] (S2) Query the hot storage or cold storage according to the time range R to obtain the ID of the corresponding communication connection record and remove duplicates to obtain the ID list;

[0084] It is easy to understand that the time range R may be entirely located in cold storage, entirely in hot storage, or partially in cold storage and partially in hot storage. The corresponding ID can be obtained by querying the hot storage or cold storage according to the specific time.

[0085] (S3) Query the graph database according to the ID list to obtain the five-tuple information of the communication connection record corresponding to each ID, complete the recovery of the network communication log, and return the recovered network communication log.

[0086] It is easy to understand that, since each ID in the graph database records the source port, destination port, and protocol information on the corresponding edge, and the two nodes connected by the edge record the source IP and destination IP, the complete 5-tuple information can be obtained by accessing the graph database based on the ID.

[0087] Since the information stored on the server is preprocessed by serialization and other processes performed by the host, and the quintuple information stored in the graph database is a converted hexadecimal string, after obtaining the queried quintuple information, it is necessary to first convert it into the original format and perform preprocessing such as deserialization before returning it to the user.

[0088] (S4) Query ends.

[0089] To prevent the host from affecting the normal operation of the cluster when sending network communication log information to the message queue in the server, as a preferred implementation, this embodiment also includes a step of dynamically adjusting the sending frequency executed on the host side;

[0090] The dynamic adjustment step of the transmission frequency includes: real-time detection of network latency; if the latency exceeds a preset threshold, the frequency of sending network communication log records to the server's message queue is reduced; and when the network latency does not exceed the threshold, the frequency of sending network communication log records to the server's message queue is restored to the initial setting value. Specifically, in this embodiment, each time the network latency exceeds the threshold, it indicates that other services require more network resources, so the transmission frequency of network communication log data is reduced to half of the current transmission frequency to reduce the network resources occupied by sending network communication log data; once the network latency does not exceed the threshold, it indicates that there is idle network resources, so the transmission frequency is restored to the initial transmission frequency to send the collected network communication log information to the server as soon as possible.

[0091] It should also be noted that the specific threshold for judging network latency, the initial value of the sending frequency, and the magnitude of each reduction in the sending frequency should all be set according to the characteristics of the cluster application. This is only an illustrative example and should not be construed as the sole limitation of the present invention.

[0092] In summary, this embodiment achieves deduplication of network communication log data by creating log buckets and a graph database, effectively improving storage efficiency. Maintaining cold and hot databases based on a time-series database enables separate storage of cold and hot data in the network communication logs, thereby fully utilizing the cold and hot characteristics of the network communication logs and improving query efficiency. Compressing IP and port information in the network communication logs into hexadecimal improves string comparison efficiency and further reduces storage space. Dynamically adjusting the frequency of host network communication log transmission based on the current network latency improves the transmission efficiency of network communication logs without affecting the normal operation of the cluster.

[0093] Example 2:

[0094] A cluster server includes:

[0095] A first computer-readable storage medium for storing computer programs;

[0096] And a first processor, used to read the computer program stored in the first computer-readable storage medium and execute the cluster-oriented network communication log storage method provided in Embodiment 1 above.

[0097] Example 3;

[0098] A cluster includes the cluster server provided in Embodiment 2 above; and in the cluster, each host is equipped with a dynamic frequency adjustment module.

[0099] The dynamic frequency adjustment module includes:

[0100] A second computer-readable storage medium for storing computer programs;

[0101] And a second processor, used to read the computer program stored in the second computer-readable storage medium and execute the dynamic adjustment step of the transmission frequency in the cluster-oriented network communication log storage method provided in Embodiment 1 above.

[0102] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for storing network communication logs for clusters, characterized in that, include: While each host in the cluster sends the acquired network communication log records to the server's message queue, the following steps are executed on the server: Retrieve network communication log records from the message queue, extract the 5-tuple information as a communication connection record, search for the communication connection record in the log bucket, if the search is successful, update the last usage time of the communication connection record in the log bucket and the graph database to the current time; if the search is unsuccessful, generate an ID for the communication connection record, insert the communication connection record into the log bucket and the graph database, and set its last usage time to the establishment time of the communication connection; The communication log records include 5-tuple information and the establishment time of the communication connection; the log bucket is a six-level multi-way tree created in memory, where the nodes of levels 2 to 5 are, in order, the source IP, source port, destination IP, and destination port, and each connection record corresponds to a path from the root node to a leaf node in the multi-way tree. The information stored in the leaf node includes the ID of the communication connection record, the last time it was used, and the protocol; the graph database is a graph structure created in a persistent storage device, where the nodes are host IPs, and there are edges between the nodes corresponding to the hosts that have conducted network communication, and the edge information includes the ID of the communication connection record, the last time it was used, the protocol, the source port, and the destination port; When inserting communication connection records that are not present in the log bucket into the log bucket and the database, the process further includes: using the establishment time of the communication connection as a timestamp, and inserting it along with the ID of the communication connection record into the hot database; the hot database is a time-series database used to store the most recent... N Communication connection records within a time window; Furthermore, the cluster-oriented network communication log storage method also includes: At fixed times within each time window, data from the earliest time window in the hot storage is migrated to the cold storage; the cold storage is a time-series database used to store the most recent... N+ 1~ M Communication connection records within a time window; in, M and N All are positive integers, and M >2 N ; Migrating data from the hot storage to the cold storage also includes deleting data from the earliest time window in the cold storage. In addition to deleting the data from the earliest time window in the cold storage, the process also includes: Determine whether each ID currently being deleted was used in other time windows. If not, delete the log bucket and the communication connection record corresponding to that ID in the graph database. Furthermore, the ID of the communication connection record is an integer number assigned starting from 0 and incrementing sequentially. The method for generating the ID for the communication connection record is as follows: Obtain the list of IDs corresponding to the communication connection records stored in the graph database, and find the smallest unused value from the list of IDs to generate the ID.

2. The cluster-oriented network communication log storage method as described in claim 1, characterized in that, N Each time window corresponds to one week. M Each time window corresponds to one month.

3. The cluster-oriented network communication log storage method as described in claim 1, characterized in that, Also includes: After the server receives the query request, it performs the following steps: (S1) Determine the requested time range based on the time range of the data stored in the hot storage and the cold storage. R Check if the communication connection record is currently stored. If yes, proceed to step (S2); otherwise, determine that the query result does not exist and proceed to step (S4). (S2) Based on the time range R Query the hot or cold storage to obtain the IDs of the corresponding communication connection records and remove duplicates to get a list of IDs; (S3) Query the graph database according to the ID list to obtain the five-tuple information of the communication connection record corresponding to each ID, complete the recovery of the network communication log, and return the recovered network communication log; (S4) Query ends.

4. The cluster-oriented network communication log storage method as described in any one of claims 1 to 3, characterized in that, After retrieving network communication log records from the message queue and extracting the 5-tuple information therein, the method further includes: converting the four numbers separated by "." in the IP address of the 5-tuple information into 2-digit hexadecimal strings, thereby converting the IP address into an 8-digit hexadecimal string; and converting the port number in the 5-tuple information into a 2-digit hexadecimal string. Furthermore, the communication connection record is the converted quintuple information.

5. The cluster-oriented network communication log storage method as described in any one of claims 1 to 3, characterized in that, It also includes a dynamic adjustment step for the transmission frequency performed on the host side; The dynamic adjustment step of the transmission frequency includes: real-time detection of network latency; if the latency is greater than a preset threshold, reducing the frequency of sending network communication log records to the server's message queue; and restoring the frequency of sending network communication log records to the server's message queue to the initial set value when the network latency does not exceed the threshold.

6. A cluster server, characterized in that, include: A first computer-readable storage medium for storing computer programs; And a first processor, configured to read the computer program stored in the first computer-readable storage medium and execute the cluster-oriented network communication log storage method according to any one of claims 1 to 5.

7. A cluster, characterized in that, The cluster server includes the one described in claim 6; and in the cluster, each host is equipped with a dynamic frequency adjustment module. The dynamic frequency adjustment module includes: A second computer-readable storage medium for storing computer programs; And a second processor, configured to read the computer program stored in the second computer-readable storage medium and execute the dynamic adjustment step of the transmission frequency in the cluster-oriented network communication log storage method of claim 5.

Images