A log storage method, a computer readable storage medium and a program product

Abstract

The one or more embodiments of the specification provide a log storage method, a computer readable storage medium and a program product, the method is applied to a master cluster, the master cluster is configured with a first message queue; the method comprises: obtaining log data of a content distribution network (CDN); storing the log data to the first message queue; sending the log data in the first message queue to a slave cluster, so that the slave cluster stores the received log data to a second message queue configured by the slave cluster.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of computer technology, and particularly relate to a log storage method, a computer readable storage medium and a program product. BACKGROUND

[0002] As an intelligent virtual network built on the existing network infrastructure, a content delivery network (CDN) generates hundreds of millions of log data every day, which comes from both self-built global edge nodes and backhaul logs from multiple CDN vendors. To ensure the high stability, security and reliability of processing such a large amount of data, the industry usually deploys a primary and backup dual-cluster mode to achieve redundancy protection. This architecture can maintain the continuity of log processing and the consistency of final data when an exception occurs in a cluster, thereby meeting the strict requirements of enterprises for service high availability and data reliability.

[0003] In the related art, there are two main ways to implement the primary and backup cluster CDN log storage: one is to deploy a collection program on the edge node, which sends the log in parallel to the receiving interfaces of the primary and backup clusters through the HyperText Transfer Protocol (HTTP), and this method is commonly used to process the logs of self-built edge nodes; the other is to build an Nginx service through the gateway layer, which uses the Mirror function to copy and forward the request to the primary and backup clusters, and this method is commonly used to process the backhaul logs from multiple CDN vendors. However, the former will occupy additional bandwidth and computing resources of the edge node, increasing the system load; the latter not only increases the load of the receiving gateway, but also is easily affected by network fluctuations, and there is a risk of single-point failure of the gateway. In summary, no matter which way is used, the data received by the primary and backup clusters has a high inconsistency probability. SUMMARY

[0004] Therefore, one or more embodiments of the present specification provide technical solutions as follows:

[0005] According to a first aspect of one or more embodiments of the present specification, a log storage method is provided, applied to a primary cluster, wherein the primary cluster is configured with a first message queue; the method comprises:

[0006] obtaining log data of a content delivery network (CDN);

[0007] storing the log data to the first message queue;

[0008] sending the log data in the first message queue to a secondary cluster, so that the secondary cluster stores the received log data to a second message queue configured by the secondary cluster.

[0009] According to a second aspect of one or more embodiments of the present specification, a computer readable storage medium is provided, having stored thereon computer instructions that, when executed by a processor, implement the steps of the method according to the first aspect.

[0010] According to a third aspect of one or more embodiments of the present specification, a computer program product is provided, comprising computer program / instructions that, when executed by a processor, implement the steps of the method according to the first aspect.

[0011] As can be seen from the above embodiments, the scheme of the present specification effectively avoids the extra bandwidth and performance overhead caused by the double sending of the prior art at the edge node, and eliminates the load pressure, network sensitivity and single point failure risk introduced by relying on the gateway layer to copy the request, thereby significantly reducing the probability of data inconsistency between the primary and backup clusters while ensuring the continuity of log transmission and system reliability. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 FIG. 1 is a schematic diagram of an architecture of a log storage system according to an example embodiment;

[0013] Figure 2 FIG. 1 is a schematic diagram of an architecture of a log storage system according to an example embodiment;

[0014] Figure 3 FIG. 1 is a schematic diagram of an architecture of a log storage system according to an example embodiment;

[0015] Figure 4 FIG. 1 is a schematic diagram of an architecture of a log storage system according to an example embodiment;

[0016] Figure 5 FIG. 1 is a schematic diagram of an architecture of a log storage system according to an example embodiment;

[0017] Figure 6 FIG. 1 is a schematic diagram of an architecture of a log storage system according to an example embodiment; DETAILED DESCRIPTION

[0018] The example embodiments will be described in detail herein with reference to the attached drawings. The following description is only exemplary and is not intended to represent all embodiments consistent with the present disclosure. Rather, the following description is intended only to represent some of the embodiments consistent with the present disclosure.

[0019] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0020] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0021] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0022] Figure 1 This is a schematic diagram of the architecture of a traffic testing system provided in an exemplary embodiment. For example... Figure 1 As shown, the system may include a primary cluster 12 and a secondary cluster 14.

[0023] The primary cluster 12 is the core execution entity of the log storage method, and it is configured with a first message queue. The primary cluster 12 is responsible for directly acquiring raw log data from the CDN. This data source can cover self-built edge nodes or logs from one or more CDN providers. After acquiring the CDN logs, the primary cluster 12 does not need to immediately perform persistent storage or forwarding. Instead, it can first write the corresponding log data into its internal first message queue for buffering, thereby decoupling high-throughput data ingestion from subsequent processing, effectively handling data flow fluctuations, and avoiding system overload. Subsequently, the primary cluster 12 can independently and asynchronously send the log data in the first message queue to the secondary cluster 14. By having the primary cluster uniformly handle log reception and distribution, the performance overhead of edge nodes autonomously sending logs is eliminated, and the additional load and single point of failure risk caused by traffic replication at the receiving gateway layer are avoided.

[0024] The secondary cluster 14 is configured with a second message queue as a redundant backup node. The secondary cluster 14 does not directly interface with the CDN log source, but receives log data forwarded from the primary cluster 12. Its core function is to reliably store the data replica sent by the primary cluster. Once the log data is received, the secondary cluster 14 stores it persistently in its own second message queue. This ensures that the secondary cluster 14 has a data replica that is nearly consistent with the primary cluster 12, but the entire process is completely transparent to the log data source, greatly simplifying the logic of the data collection end. Through this asynchronous replication architecture actively pushed by the primary cluster, the secondary cluster 14 can complete data backup in a decoupled and reliable manner, together with the primary cluster 12 to form a highly available, highly reliable, and maximally guaranteed log storage system.

[0025] It should be noted that whether it is the primary cluster 12 or the secondary cluster 14, its essence is a distributed computing system composed of multiple physical servers or virtual machines. Among them, the concept of cluster is to organize multiple devices into a unified whole to jointly bear data processing and storage tasks, thereby providing processing power, availability, and scalability much higher than a single machine. Specifically, the multiple devices in the primary cluster 12 can work together to jointly complete the tasks of log acquisition, buffering, and forwarding, and share the pressure internally through load balancing and other technologies; similarly, the secondary cluster 14 is also composed of multiple devices, and its second message queue is distributed on these devices to jointly provide high-throughput, high-availability data persistent storage capabilities to ensure the safety and reliability of the data replica. This multi-device architecture enables the entire system to easily handle the processing challenge of billions of log data, and effectively avoids single-point failures through redundant design to ensure service continuity.

[0026] Based on this, there are multiple optional architecture modes on the deployment relationship of the first message queue and the devices in the primary cluster 12:

[0027] One is a centralized queue, that is, the first message queue exists as a middleware service independent of the devices in the primary cluster, such as an independent Kafka message queue (Kafka) or Rocket message queue (RocketMQ) cluster, etc. The devices in the above primary cluster all act as producers and write log data uniformly to the shared queue.

[0028] The second is a local queue, that is, each device in the primary cluster maintains a local message queue, such as a local disk file or an embedded database. This architecture is simple and has no central node bottleneck, but coordination is needed when forwarding data.

[0029] The third is a hybrid mode, that is, the devices in the primary cluster are grouped, and each group of devices shares a message queue instance, thereby balancing performance and complexity.

[0030] For the above various architecture patterns, the present specification can be adjusted according to the actual scene. The relationship between the second message queue and the devices in the secondary cluster 14 is the same, and a suitable queue deployment strategy can also be selected according to the actual scene to realize reliable and efficient data persistence and inter-cluster synchronization.

[0031] Figure 2 is a flowchart of a log storage method disclosed in an exemplary embodiment of the present specification, which is applied to a primary cluster configured with a first message queue. The method can specifically include the following steps:

[0032] Step S202, obtaining log data of a content distribution network (CDN).

[0033] The above primary cluster serves as a unified log access layer and is responsible for receiving raw log data from the content distribution network. The data sources cover a wide range of distributed nodes, including self-built global edge nodes and logs returned by multiple integrated CDN vendors. The primary cluster receives high-availability, high-parallelism interfaces to continuously collect massive log data streams reaching hundreds of millions per day, providing stable and reliable data input for subsequent processing.

[0034] According to the data source of the log, it can be divided into different types, for example: the first type of log data directly generated by the CDN self-built edge node, and the second type of log data generated and returned by the service node of the third-party CDN service vendor. In actual application scenarios, the primary cluster can receive one or more types of log data in combination, and the present specification does not limit the specific source and combination form.

[0035] In the architecture implementation of data acquisition, the primary cluster in the present specification can also support multiple optional log receiving modes, including but not limited to:

[0036] 1. Load balancing reception: The data stream corresponding to the log data can first be distributed to multiple devices in the primary cluster through a load balancer, thereby realizing traffic distribution and parallel processing.

[0037] 2. Special gateway reception: A special device such as a gateway or a log ingestor is set up inside the primary cluster to be responsible for uniformly receiving all log data, and the remaining devices focus on subsequent processing tasks to realize decoupling of functions.

[0038] 3. Single entry reception: In the case of small data size or specific debugging scenarios, a device in the primary cluster can independently undertake the responsibility of log reception.

[0039] Step S204, storing the log data to the first message queue.

[0040] After obtaining the original log data, the master cluster can store the log data into the configured first message queue. By using the buffering characteristics of the message queue, the traditional high-throughput data ingestion process can be effectively decoupled from the subsequent storage and forwarding operations. In other words, regardless of whether the first message queue adopts a centralized, local or hybrid deployment mode, the temporary storage mechanism based on the message queue here can effectively cope with the instantaneous fluctuations of the data flow, improve the throughput and stability of the system, and at the same time avoid the system overload that may be caused by directly processing the downstream pressure.

[0041] Further, after successfully obtaining the log data of the CDN, the process or device as the producer in the master cluster can call the client interface of the first message queue to initiate a write request in the form of a "message" to write the log data into the first message queue. The request is sent to the cluster where the first message queue service is located through the network. After receiving the message, the first message queue service does not only temporarily store it in the memory, but can immediately start the persistence process. At this time, the server corresponding to the first message queue service can verify and classify the message, and then append it as a new record to the log file of the non-volatile storage medium such as a hard disk drive (HDD), a solid state drive (SSD) in a sequential writing manner. Only after confirming that the message has been successfully written to the disk, the first message queue service returns a successful confirmation to the producer in the master cluster, indicating that the log data has been reliably stored, and also lays a solid foundation for subsequent stable synchronization to the secondary cluster.

[0042] Step S206, sending the log data in the first message queue to the secondary cluster, so that the secondary cluster stores the received log data into the second message queue configured by the secondary cluster.

[0043] The master cluster can send the log data in the first message queue to the secondary cluster. This step can be initiated by the master cluster actively, and the secondary cluster does not need to interface with the data source, but only needs to receive and store into its second message queue. Through this design, decoupled log synchronization between the master and secondary clusters is realized, which not only guarantees the eventual consistency of the data replica, but also completely avoids various performance and reliability problems caused by repeated sending at the edge node or gateway layer.

[0044] In an embodiment, the log data can be consumed from the first message queue to obtain to-be-sent data, and the to-be-sent data is sent to the secondary cluster based on a preset transmission strategy. Specifically, a specific service or consumer group in the primary cluster can first consume the log data from the first message queue, and after successfully pulling and confirming the log data, the log data is converted into to-be-sent data to be sent to the secondary cluster. Then, the primary cluster can send the to-be-sent data to the secondary cluster based on the preset transmission strategy. The transmission strategy can be flexibly configured to adapt to network fluctuations, maximize transmission throughput, and ensure the final reliability of data transmission.

[0045] Based on the transmission strategy, a sending mechanism with periodic and batch characteristics can be implemented. The mechanism effectively adapts to the synchronization requirements in different network environments and business scenarios through parameterized configuration, thereby balancing the real-time performance and throughput efficiency of data transmission.

[0046] In an embodiment, the transmission strategy includes two core parameters: a minimum sending data amount and a sending duration. After each interval of the sending duration, a target data amount of the to-be-sent data can be obtained, and the to-be-sent data can be batch sent to the secondary cluster if the target data amount is not less than the minimum sending data amount. If the minimum sending data amount threshold is not reached, the next sending period can be waited for, thereby avoiding network overhead and performance loss caused by frequent sending of a large number of small data packets. This embodiment uses a dual-judgment method based on data amount and sending duration, which can ensure batch transmission of data at a high throughput during a data generation peak period or when the network condition is good, thereby maximizing the use of bandwidth and reducing the system burden. On the other hand, during a data sparse period, a reasonable period can be waited for to aggregate an appropriate amount of data, thereby avoiding ineffective or inefficient transmission caused by a small data amount, and thereby ensuring the efficiency and economy of data transmission as a whole.

[0047] The transmission strategy of the present specification can also include a perfect fault tolerance and cleaning mechanism to ensure the robustness of the data transmission process. Specifically, the mechanism can control the failed retry behavior by defining a maximum number of retransmissions.

[0048] In an embodiment, in the case that the sending of the to-be-sent data fails and the number of failures is less than the maximum number of retransmissions, the to-be-sent data can be sent at a preset retransmission interval. The above can serve as a retry process triggered when the master cluster fails to send the to-be-sent data to the secondary cluster. If the condition that the number of cumulative failures of the to-be-sent data is less than the configured maximum number of retransmissions is met, the sending task corresponding to the to-be-sent data will not be immediately abandoned, but will wait for a preset retransmission interval, and then the same batch of data will be sent again. This process is executed in a loop until success or the maximum number of retries is reached. The retransmission interval can be dynamically calculated using an exponential backoff algorithm, or manually input to achieve customization. This specification does not limit this.

[0049] In another embodiment, in the case that the sending of the to-be-sent data fails and the number of failures is not less than the maximum number of retransmissions, the to-be-sent data can be marked as an abnormal log, and the local persistent copy corresponding to the abnormal log can be queried and deleted. If the number of sending failures has reached or exceeded the maximum number of retransmissions, it indicates that the data cannot be successfully delivered due to reasons such as continuous network interruption, unavailability of the destination, or abnormal data format in the current environment. At this time, the master cluster marks the batch of to-be-sent data as an abnormal log. To further release storage resources and avoid the accumulation of invalid data, the following cleaning operation can be performed, that is, the local copy corresponding to the batch of abnormal logs, which is persisted in the first message queue or the storage associated therewith, is queried and deleted. This embodiment ensures automatic recovery in the face of temporary failures, while performing clear failure isolation and resource recovery for abnormal data confirmed to be unable to process, preventing it from occupying system resources indefinitely, and facilitating subsequent centralized auditing and problem troubleshooting.

[0050] In the cleaning operation in the previous embodiment, in addition to the already persisted local copy, the data in the first message queue also needs to be removed. Different queue removal methods are introduced in this specification for different consumption models involved in the message queue to ensure the accuracy of message queue consumption management and the reliability of data processing, and to avoid problems such as data loss or repeated processing due to mismatch between the consumption model and the data removal operation.

[0051] Specifically, the consumption model of the first message queue can adopt an automatic acknowledgment mode or a manual acknowledgment mode, and different modes have different processing conditions.

[0052] In the first case, if the consumption model adopts an automatic confirmation mode, the system can store the obtained to-be-sent data in a preset cache for temporary storage after consuming the log data from the queue. Thereafter, the system removes the corresponding to-be-sent data from the cache in response to a removal instruction. The cache can be, for example, a Redis (Remote Dictionary Server) database or a local memory database, and the removal instruction indicates that the data has been successfully sent to the secondary cluster or has been marked as abnormal log. In this mode, the server corresponding to the message queue immediately marks the message as consumed due to the automatic confirmation mechanism, and therefore relies on an external cache to ensure the traceability and reliability of the data during processing.

[0053] In the second case, if the consumption model adopts a manual confirmation mode, there is no need to introduce an external cache compared to the first case. The reason is that after the system consumes the data, the to-be-sent data remains in the message queue, and its state is invisible to other consumers. Only when the system responds to the removal instruction, does it send an acknowledgement (Acknowledge, ACK) to the message queue server to officially remove the message from the first message queue. This mode completely entrusts the data persistence and life cycle management to the message queue itself, thereby simplifying the architecture.

[0054] In summary, through the different mechanisms adopted in the different cases, the system corresponding to the primary cluster can flexibly adapt to different message queue middleware and their confirmation models, while achieving abnormal data cleaning and successful data removal, fundamentally ensuring the accuracy of data processing semantics and avoiding data loss or duplication.

[0055] It is worth mentioning that the present specification also introduces serialization and compression steps in the sending process, thereby ensuring the efficiency and compatibility of data transmission and performing necessary optimization processing on the to-be-sent log data before transmission. This processing can significantly reduce network transmission bandwidth occupation and improve the ability of data cross-platform interaction.

[0056] In an embodiment, after the primary cluster obtains the to-be-sent data, the to-be-sent data can be subjected to serialization encoding processing to generate a serialized data stream conforming to a preset standardized format before the to-be-sent data is officially sent. At the same time, the serialized data stream is subjected to data compression processing based on a preset compression algorithm, thereby generating compressed to-be-sent data.

[0057] In particular, the serialization encoding process described above converts the log data, which can originally exist in the form of memory objects or specific data structures, into a serialized data stream conforming to a preset standardized format, such as a byte stream, where the so-called standardized format can include formats such as JavaScript Object Notation (JSON), eXtensible Markup Language (XML), Protocol Buffers, or Avro Data Serialization System (Avro), etc. In summary, the serialization process not only eliminates the dependence of data on the running environment, enabling correct parsing between different systems, but also prepares a structured binary data basis with high redundancy for the subsequent compression step.

[0058] Subsequently, the primary cluster can perform data compression processing on the serialized data stream based on a preset compression algorithm, such as GNU Zip (GZIP), LZ4 Compression Algorithm (LZ4), Snappy Compression Algorithm (Snappy), or Zstandard Compression Algorithm (Zstandard), etc. The above compression algorithms greatly reduce the volume of data to be transmitted by eliminating redundant information in the data, thereby generating compressed data to be sent. This directly reduces the bandwidth consumption and transmission time of network transmission, especially in the scenario of processing hundreds of millions of log data per day, which can effectively save costs and improve synchronization efficiency.

[0059] Through the above two-step processing of serialization and compression, the data to be sent is minimized in size while maintaining semantic integrity, laying a solid foundation for subsequent efficient and economical batch transmission to the secondary cluster.

[0060] For the secondary cluster, its responsibility is to reliably receive and store log data from the primary cluster to ultimately achieve data consistency with the primary cluster. When it receives the data packet sent by the primary cluster from the network, it can first perform decompression and deserialization processing to restore the structured log data content. Subsequently, the secondary cluster can persistently store the parsed original log data as a message in its own configured second message queue. The second message queue also has the same high availability and persistence characteristics as the first message queue, ensuring that data on the secondary cluster side will not be lost due to system restarts or other abnormalities.

[0061] After successfully storing the log data in the second message queue, the secondary cluster completes the data persistence between the primary and secondary clusters. This step signifies that a log data entry has successfully passed through the complete pipeline from collection, temporary storage, and transmission in the primary cluster to reception and storage in the secondary cluster, achieving redundant data backup. Subsequently, other downstream processing programs in the secondary cluster, such as real-time computing engines or offline analysis systems, can consume data from the second message queue for further statistical analysis. Since all the above operations are based on the same reliable data copy—the log data from the primary cluster mentioned earlier—the eventual consistency of data processing results between the primary and secondary clusters is fundamentally guaranteed.

[0062] The following is combined Figure 3 The complete process and system architecture of the above log storage method will be further explained. Figure 3 As shown, the system architecture clearly demonstrates the data flow and collaboration relationships between the main cluster, the secondary cluster, and related nodes.

[0063] First, during the data collection and storage phase, CDN log data from two main sources are uniformly received: firstly, access logs generated by edge nodes of the self-built CDN are reported to the receiving interface of the main cluster through the collection program; secondly, CDN logs returned by the converged CDN vendor are also aggregated to the main cluster. After receiving this data, the devices corresponding to the main cluster store it as messages in the configured first message queue for buffering. Taking a Kafka cluster as the first message queue as an example, this message queue provides a unified, high-throughput data source for all subsequent tasks such as real-time statistics, offline computing, and permanent disk storage. Subsequently, during the data synchronization and backup phase, dedicated service components deployed in the main cluster can continuously consume log data from the first message queue, Kafka. This component can use high-performance log synchronization tools such as Rsyslog to reliably send log data to the receiving gateway of the secondary cluster. This sending process is configured with a failure retry mechanism to ensure that it can continue to try even when network fluctuations occur until the data is successfully delivered to the secondary cluster, thereby strongly guaranteeing the eventual consistency of data between the main and backup clusters at the transport layer. After receiving the data, the secondary cluster stores it in its own second message queue, which is also based on Kafka.

[0064] Through the above process, all aggregated CDN access logs, after being stored in the primary cluster's Kafka queue, can be fully, orderly, and reliably synchronized to the secondary cluster and stored in the backup cluster's Kafka queue by this service component. This aims to provide the secondary cluster with source data that is completely consistent with the primary cluster for all subsequent statistical tasks, permanent disk storage, offline log generation, and secondary log distribution, fundamentally ensuring strong consistency of CDN logs between the primary and backup clusters, as well as consistency of statistical results between the two.

[0065] In addition, it is worth noting that, Figure 3 The service component described in the foregoing can be designed as a distributed service, that is, support for scale-out and distributed multi-node deployment, thereby avoiding single point of failure, guaranteeing high availability of the data synchronization process, and being able to linearly improve processing capacity as the data volume grows. In other words, the scheme efficiently solves the problem of inconsistent data between the primary and standby clusters, and the problem of excessive consumption of machine and performance resources in the synchronization process in the traditional scheme.

[0066] The following will be described in combination with Figure 4 The method performed by the service component in the foregoing will be introduced, as shown in FIG. 4, the method comprises the following steps: Figure 3 Figure 4 Step S402, the service component is initialized and HTTP transmission configuration is loaded.

[0067] In an embodiment, the service component can first perform an initialization process after starting, and the HTTP parameter configuration module built therein loads necessary HTTP transmission parameters from a local configuration file or a remote configuration center. These parameters can include but are not limited to: URL address of the receiving gateway of the secondary cluster, authentication token, connection timeout, content type (Content-Type), and related parameters for subsequent batch sending, such as the minimum sending data volume, sending duration, and maximum retransmission number as described above. This step ensures that the service component has all the necessary information to initiate an HTTP request to the correct target, and prepares for subsequent data sending.

[0068] Step S404, the service component consumes log data from the first message queue.

[0069] In an embodiment, the consumption Kafka module in the service component can start one or more consumers to continuously pull CDN log data from the first message queue Kafka of the primary cluster. The module is responsible for maintaining the connection with the Kafka cluster, managing the consumption offset, and delivering the obtained raw log data stream to the subsequent processing unit, thereby providing a stable and ordered data source for the entire synchronization process.

[0070] Step S406, the service component performs compression processing on the data obtained by consumption.

[0071] In an embodiment, the data compression module in the service component can perform compression processing on the raw log data consumed from Kafka to optimize transmission efficiency. The module can use a preset compression algorithm to convert batch log data into compressed binary data packets, significantly reducing data volume, thereby reducing network bandwidth occupation and improving the efficiency of subsequent HTTP transmission.

[0072] In an embodiment, the data compression module in the service component can perform compression processing on the raw log data consumed from Kafka to optimize transmission efficiency. The module can use a preset compression algorithm to convert batch log data into compressed binary data packets, significantly reducing data volume, thereby reducing network bandwidth occupation and improving the efficiency of subsequent HTTP transmission.​

[0073] Step S408, the service component stores the compressed data into the sending queue cache.

[0074] In an embodiment, the compressed data is not immediately sent, but is temporarily stored based on the sending queue cache module of the service component. The cache can be implemented based on a memory queue, used to buffer the speed difference between production and consumption, absorb transient traffic peaks, prevent overload of the downstream sending link, and support subsequent batch assembly and sending strategies.

[0075] As understood by those skilled in the art, the queue cache in this step is not the same as the preset cache mentioned above. The former is a queue working buffer, used to temporarily store data that has been consumed from Kafka, compressed, and ready for batch sending; the latter is a persistent state storage, only used to solve data loss problem in the automatic confirmation mode. To avoid subsequent ambiguity, it is assumed that Figure 4 the consumption mode of the queue is manual confirmation.

[0076] Step S410, the service component assembles batch data from the cache and performs HTTP sending.

[0077] In an embodiment, the HTTP batch sending module of the service component obtains multiple compressed data from the sending queue cache according to the preset strategy, assembles them into a batch message body, and then uses the HTTP configuration loaded in step S402 to initiate an HTTP POST request to the receiving gateway address of the replica cluster, sending the batch data at once. This batch processing mechanism greatly improves network throughput and resource utilization efficiency.

[0078] Step S412, the service component processes the sending result and performs timeout retransmission.

[0079] In an embodiment, the HTTP timeout retransmission module of the service component is responsible for processing the response of the HTTP request. If the sending is successful, for example, the message "HTTP 200 OK" is received, it is confirmed that the batch data synchronization is completed. If network timeout or server error is encountered, for example, the message HTTP 500 is received, the retransmission mechanism is triggered. The module will wait for a period of time according to the preset retry strategy, then take out the original data from the sending queue cache, and send it again by the HTTP batch sending module until success or the maximum number of retries is reached, so as to ensure the final reliability of data transmission.

[0080] Figure 5 is a schematic structural diagram of a device provided by an exemplary embodiment. Please refer to Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and can also include other hardware required by functions. One or more embodiments of the present specification can be implemented in a software manner, such as reading a corresponding computer program from the non-volatile memory 510 into the memory 508 by the processor 502 and then running. Of course, in addition to the software implementation, one or more embodiments of the present specification do not exclude other implementation manners, such as a logic device or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0081] Corresponding to the foregoing embodiments of the log storage method, the present specification also provides an embodiment of a log storage device. Please refer to Figure 6 , the device is applied to a master cluster, and the master cluster is configured with a first message queue; the device includes:

[0082] A log data acquisition unit 602 is configured to acquire log data of a content distribution network (CDN).

[0083] A message queue storage unit 604 is configured to store the log data to the first message queue.

[0084] A log data sending unit 606 is configured to send the log data in the first message queue to a secondary cluster, so that the secondary cluster stores the received log data to a second message queue configured by the secondary cluster.

[0085] Optionally, the log data acquisition unit 602 is specifically configured to:

[0086] acquire first type log data generated by an edge node of the CDN; and / or,

[0087] acquire second type log data generated by a third party service node connected to the CDN.

[0088] Optionally, the message queue storage unit 604 is specifically configured to:

[0089] write the log data into the first message queue and persist to a non-volatile storage medium.

[0090] Optionally, the log data sending unit 606 is specifically configured to:

[0091] consume the log data from the first message queue to acquire to-be-sent data;

[0092] send the to-be-sent data to the secondary cluster based on a preset transmission strategy.

[0093] Optionally, the transmission strategy includes a minimum sending data amount and a sending duration; and the log data sending unit 606 is specifically configured to:

[0094] acquire a target data amount of the to-be-sent data every interval of the sending duration;

[0095] in a case where the target data amount is not less than the minimum sending data amount, batch sending the to-be-sent data to the secondary cluster.

[0096] Optionally, the transmission strategy includes a maximum retransmission number; and the log data sending unit 606 is specifically configured to:

[0097] in a case where the to-be-sent data fails to be sent and a failure number is less than the maximum retransmission number, sending the to-be-sent data every interval of a preset retransmission duration;

[0098] in a case where the to-be-sent data fails to be sent and the failure number is not less than the maximum retransmission number, marking the to-be-sent data as an abnormal log; and querying and deleting a local persistent copy corresponding to the abnormal log.

[0099] Optionally, the apparatus further includes:

[0100] a data removing unit configured to, in a case where a consumption model of the first message queue adopts an automatic confirmation mode, store the to-be-sent data into a preset cache; and in response to a removing instruction for the to-be-sent data, remove the to-be-sent data from the cache, the removing instruction being used to represent that the to-be-sent data is an abnormal log or the to-be-sent data is sent successfully;

[0101] in a case where the consumption model of the first message queue adopts a manual confirmation mode, in response to a removing instruction for the to-be-sent data, remove the to-be-sent data from the first message queue, the removing instruction being used to represent that the to-be-sent data is an abnormal log or the to-be-sent data is sent successfully.

[0102] Optionally, the apparatus further includes:

[0103] a data compression unit configured to perform serialization coding processing on the to-be-sent data to generate a serialized data stream conforming to a preset standardized format;

[0104] perform data compression processing on the serialized data stream based on a preset compression algorithm to generate compressed to-be-sent data.

[0105] Based on the same idea as the above method, the specification also provides a computer readable storage medium, which stores computer instructions, and the instructions are executed by a processor to implement the steps of the method according to any one of the above embodiments.

[0106] Based on the same idea as the above method, the specification also provides a computer program product, which includes computer program / instructions, and the instructions are executed by a processor to implement the steps of the method according to any one of the above embodiments.

[0107] Although the present application contains many specific embodiments, these should not be construed as limiting the scope or spirit of any application in any way. Rather, they are provided to illustrate particular embodiments of the application. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in subcombination or in a different combination in a single embodiment. Furthermore, features can be implemented in a single application even though they are described in the context of separate applications.

[0108] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring or implying that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0109] Accordingly, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0110] The above descriptions are only the preferred embodiments of the present application, and are not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A log storage method characterized by comprising: The method is applied to a master cluster, and the master cluster is configured with a first message queue; the method comprises: obtaining log data of a content distribution network (CDN); storing the log data into the first message queue; sending the log data in the first message queue to a slave cluster, so that the slave cluster stores the received log data into a second message queue configured by the slave cluster; the sending of the log data in the first message queue to the slave cluster comprises: consuming the log data from the first message queue to obtain to-be-sent data; sending the to-be-sent data to the slave cluster based on a preset transmission strategy, the transmission strategy comprising a minimum sending data amount and a sending time length; the sending of the to-be-sent data to the slave cluster based on the preset transmission strategy comprises: obtaining a target data amount of the to-be-sent data every interval of the sending time length; in a case where the target data amount is not less than the minimum sending data amount, sending the to-be-sent data to the slave cluster in batches.

2. The method of claim 1, wherein, the obtaining of the log data of the CDN comprises: obtaining first-type log data generated by an edge node of the CDN; and / or obtaining second-type log data generated by a third-party service node connected to the CDN.

3. The method of claim 1, wherein, the storing of the log data into the first message queue comprises: writing the log data into the first message queue and persisting to a non-volatile storage medium.

4. The method of claim 1, wherein, the transmission strategy comprises a maximum retransmission number; the sending of the to-be-sent data to the slave cluster based on the preset transmission strategy comprises: in a case where the to-be-sent data fails to be sent and a failure number is less than the maximum retransmission number, sending the to-be-sent data every interval of a preset retransmission time length; in a case where the to-be-sent data fails to be sent and the failure number is not less than the maximum retransmission number, marking the to-be-sent data as an abnormal log; and querying and deleting a local persistent copy corresponding to the abnormal log.

5. The method of claim 4, wherein, the method further comprises: in a case where a consumption model of the first message queue adopts an automatic confirmation mode, storing the to-be-sent data into a preset cache; and in response to a removal instruction for the to-be-sent data, removing the to-be-sent data from the cache, the removal instruction being used to represent that the to-be-sent data is an abnormal log or the to-be-sent data is sent successfully; in a case where the consumption model of the first message queue adopts a manual confirmation mode, in response to a removal instruction for the to-be-sent data, removing the to-be-sent data from the first message queue, the removal instruction being used to represent that the to-be-sent data is an abnormal log or the to-be-sent data is sent successfully.

6. The method of claim 1, wherein, the method further comprises: serializing and encoding the to-be-sent data to generate serialized data streams conforming to a preset standardized format; performing data compression processing on the serialized data streams based on a preset compression algorithm to generate compressed to-be-sent data.

7. A computer readable storage medium characterized by A computer readable storage medium has stored thereon computer instructions which, when executed by a processor, implement the steps of the method of any one of claims 1-6.

8. A computer program product, characterised in that, comprising computer program / instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1-6.

Images