Log storage method, computer readable storage medium and program product

Abstract

One or more embodiments of the invention provide a log storage method, a computer readable storage medium and a program product, the method is applied to a main cluster, and the main cluster is configured with a first message queue; the method comprises the following steps: acquiring log data of a content delivery network (CDN); storing the log data to the first message queue; and sending the log data in the first message queue to an auxiliary cluster, so that the auxiliary cluster stores the received log data to a second message queue configured by the auxiliary cluster.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of computer technology, and more particularly to a log storage method, a computer-readable storage medium, and a program product. Background Technology

[0002] Content Delivery Networks (CDNs), as intelligent virtual networks built on top of existing networks, generate hundreds of millions of log data points daily. These logs originate from both self-built global edge nodes and logs from multiple CDN vendors. To ensure high stability, security, and reliability in processing such massive amounts of data, the industry typically employs a dual-cluster deployment architecture with primary and backup clusters for redundancy. This architecture maintains the continuity of log processing and eventual data consistency even if one cluster experiences an anomaly, thus meeting enterprises' stringent requirements for high service availability and data reliability.

[0003] In related technologies, there are two main methods for implementing CDN log storage in primary and backup clusters: First, deploying a collection program on edge nodes to send logs in parallel to the receiving interfaces of both the primary and backup clusters via HyperText Transfer Protocol (HTTP). This method is commonly used for processing logs from self-built edge nodes. Second, building an Nginx service at the gateway layer and utilizing its traffic mirroring function to replicate and forward requests to the primary and backup clusters. This method is often used to handle logs returned from multiple CDN providers. However, the former consumes additional bandwidth and computing resources from the edge nodes, increasing system load; the latter not only increases the load on the receiving gateway but is also more susceptible to network fluctuations and carries the risk of a single point of failure. In short, regardless of the method used, the data received by the primary and backup clusters has a high probability of inconsistency. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a log storage method is proposed, applied to a primary cluster, the primary cluster being configured with a first message queue; the method includes: Obtain log data from the Content Delivery Network (CDN); The log data is stored in the first message queue; The log data in the first message queue is sent to the secondary cluster, so that the secondary cluster stores the received log data in the second message queue configured in the secondary cluster.

[0005] According to a second aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0006] According to a third aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as described in the first aspect.

[0007] As can be seen from the above embodiments, the solution in this specification, by configuring a first message queue on the main cluster and uniformly receiving CDN log data, and then having the main cluster forward the logs to the second message queue of the secondary cluster, effectively avoids the additional bandwidth and performance overhead caused by dual transmission at edge nodes in existing technologies. It also eliminates the load pressure, network sensitivity, and single point of failure risk introduced by relying on the gateway layer for request replication. Thus, while ensuring the continuity of log transmission and system reliability, it significantly reduces the probability of data inconsistency between the main and backup clusters. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of the architecture of a log storage system provided in an exemplary embodiment; Figure 2 This is a flowchart illustrating a log storage method provided in an exemplary embodiment; Figure 3 This is a schematic diagram of the architecture of another log storage system provided in an exemplary embodiment; Figure 4 This is a flowchart illustrating a service component execution method provided in an exemplary embodiment; Figure 5 This is a schematic diagram of the structure of a device provided in an exemplary embodiment; Figure 6 This is a schematic diagram of another log storage device provided in an exemplary embodiment. Detailed Implementation

[0009] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present invention.

[0010] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0011] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0012] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0013] Figure 1 This is a schematic diagram of the architecture of a traffic testing system provided in an exemplary embodiment. For example... Figure 1 As shown, the system may include a primary cluster 12 and a secondary cluster 14.

[0014] The primary cluster 12 is the core execution entity of the log storage method, and it is configured with a first message queue. The primary cluster 12 is responsible for directly acquiring raw log data from the CDN. This data source can cover self-built edge nodes or logs from one or more CDN providers. After acquiring the CDN logs, the primary cluster 12 does not need to immediately perform persistent storage or forwarding. Instead, it can first write the corresponding log data into its internal first message queue for buffering, thereby decoupling high-throughput data ingestion from subsequent processing, effectively handling data flow fluctuations, and avoiding system overload. Subsequently, the primary cluster 12 can independently and asynchronously send the log data in the first message queue to the secondary cluster 14. By having the primary cluster uniformly handle log reception and distribution, the performance overhead of edge nodes autonomously sending logs is eliminated, and the additional load and single point of failure risk caused by traffic replication at the receiving gateway layer are avoided.

[0015] Secondary cluster 14, acting as a redundant backup node, is configured with a second message queue. Instead of directly connecting to the CDN log source, secondary cluster 14 receives log data forwarded from primary cluster 12. Its core function is to reliably store data copies sent by the primary cluster. Once log data is received, secondary cluster 14 persistently stores it in its own second message queue. This approach ensures that secondary cluster 14 has a nearly identical data copy to primary cluster 12, but the entire process is completely transparent to the log data source, greatly simplifying the logic at the data collection end. Through this asynchronous replication architecture actively pushed by the primary cluster, secondary cluster 14 can complete data backup in a decoupled and reliable manner, forming a highly available, highly reliable log storage system with primary cluster 12 that maximizes eventual data consistency.

[0016] It's important to note that both the primary cluster 12 and the secondary cluster 14 are essentially distributed computing systems composed of multiple physical servers or virtual machines. The concept of a cluster lies in organizing multiple devices into a unified whole to jointly undertake data processing and storage tasks, thereby providing processing power, availability, and scalability far exceeding that of a single machine. Specifically, the multiple devices in the primary cluster 12 can work collaboratively to complete log acquisition, buffering, and forwarding tasks, internally distributing pressure through load balancing and other technologies. Similarly, the secondary cluster 14 also consists of multiple devices, with its second message queue distributed across these devices, collectively providing high-throughput, highly available persistent data storage capabilities, ensuring the security and reliability of data copies. This multi-device architecture enables the entire system to easily handle the challenge of processing billions of log data points, and effectively avoids single points of failure through redundancy design, ensuring service continuity.

[0017] Based on this, there are several optional architecture modes for the deployment relationship between the first message queue and the devices in the main cluster 12: One type is a centralized queue, where the first message queue exists as a middleware service independent of the devices within the main cluster, such as an independent Kafka or RocketMQ cluster. Each device in the main cluster acts as a producer, writing log data uniformly into this shared queue.

[0018] The second is local queues, where each device in the main cluster maintains a local message queue, such as a local disk file or an embedded database. This approach has a simple architecture and no central node bottleneck, but it requires coordination during data forwarding.

[0019] The third is the hybrid mode, which groups the devices in the main cluster, with each group sharing a message queue instance, thereby achieving a balance between performance and complexity.

[0020] This manual can be adjusted according to the actual scenario for the above various architecture modes. The relationship between the second message queue and the devices in sub-cluster 14 is similar, and an appropriate queue deployment strategy can be selected according to the actual scenario to achieve reliable and efficient data persistence and inter-cluster synchronization.

[0021] Figure 2 This is a flowchart illustrating an exemplary embodiment of a log storage method disclosed in this specification. The method is applied to a primary cluster, which is configured with a first message queue. The method may specifically include the following steps: Step S202: Obtain log data from the Content Delivery Network (CDN).

[0022] The aforementioned main cluster serves as a unified log access layer, responsible for receiving raw log data from the content delivery network. Data sources encompass a wide range of distributed nodes, including self-built global edge nodes and logs from multiple integrated CDN providers. Through highly available and high-concurrency receiving interfaces, the main cluster continuously collects massive daily log data streams reaching hundreds of millions of records, providing stable and reliable data input for subsequent processing.

[0023] Based on the data source, logs can be categorized into different types, such as: the first type of log data directly generated by CDN self-built edge nodes, and the second type of log data generated and transmitted back by service nodes of third-party CDN service providers. In practical applications, the main cluster can simultaneously or separately receive one or more of the above-mentioned log data types in combination. This specification does not restrict the specific source or combination of these log data.

[0024] Regarding the architecture implementation for data acquisition, the main cluster described in this manual can also support various optional log reception methods, including but not limited to: 1. Load-balanced reception: The data stream corresponding to the log data can first be evenly distributed to multiple devices in the main cluster through the load balancer, thereby realizing traffic distribution and parallel processing.

[0025] 2. Dedicated gateway-style reception: Dedicated devices such as gateways or log ingestors are set up within the main cluster to be responsible for receiving all log data in a unified manner, while other devices focus on subsequent processing tasks, thus achieving functional decoupling.

[0026] 3. Single entry point reception: In cases of small data volume or specific debugging scenarios, a single device in the main cluster can independently assume the responsibility of receiving logs.

[0027] Step S204: Store the log data in the first message queue.

[0028] After acquiring the raw log data, the main cluster can store it in the configured first message queue. By leveraging the buffering characteristics of the message queue, the traditional high-throughput data ingestion process can be effectively decoupled from subsequent storage and forwarding operations. In other words, regardless of whether the first message queue adopts a centralized, local, or hybrid deployment mode, this message queue-based temporary storage mechanism can effectively handle instantaneous fluctuations in the data stream, improve system throughput and stability, and avoid potential system overload caused by directly processing downstream pressure.

[0029] Furthermore, after successfully retrieving CDN log data, the producer process or device in the primary cluster can call the client interface of the first message queue to initiate a write request for the log data in the form of a "message," thus writing the log data into the first message queue. This request is sent over the network to the cluster where the first message queue service resides. Upon receiving this message, the first message queue service does not merely store it temporarily in memory but immediately initiates a persistence process. At this point, the server corresponding to the first message queue service can verify and classify the message, and then append it as a new record sequentially to the log file of its underlying associated non-volatile storage medium, such as a hard disk drive (HDD) or solid state drive (SSD). Only after confirming that the message has been successfully written to disk will the first message queue service return a success acknowledgment to the producer in the primary cluster, signifying that the log data has been reliably stored, and laying a solid foundation for subsequent stable synchronization to the secondary cluster.

[0030] Step S206: Send the log data in the first message queue to the secondary cluster, so that the secondary cluster stores the received log data in the second message queue configured in the secondary cluster.

[0031] The primary cluster can send log data from the first message queue to the secondary cluster. This step can be initiated by the primary cluster, while the secondary cluster does not need to interface with the data source; it only needs to receive and store the data in its second message queue. This design achieves decoupled log synchronization between the primary and secondary clusters, ensuring eventual consistency of data replicas while completely avoiding various performance and reliability issues caused by repeated transmissions at edge nodes or the gateway layer.

[0032] In one embodiment, log data can be consumed from the first message queue to obtain data to be sent, and simultaneously, the data to be sent is transmitted to the secondary cluster based on a preset transmission strategy. Specifically, a specific service or consumer group in the primary cluster can first consume the log data from the first message queue. After successfully retrieving and confirming the data, it is converted into data to be sent to the secondary cluster, i.e., the data to be sent. Subsequently, the primary cluster can transmit the data to the secondary cluster based on a preset transmission strategy. This transmission strategy can be flexibly configured to adapt to network fluctuations, maximize transmission throughput, and ensure the ultimate reliability of data transmission.

[0033] Based on the above transmission strategy, a transmission mechanism that combines periodicity and batch characteristics can be realized. This mechanism can effectively adapt to the synchronization requirements of different network environments and business scenarios through parameterized configuration, thereby achieving a balance between the real-time performance and throughput efficiency of data transmission.

[0034] In one embodiment, the aforementioned transmission strategy includes two core parameters: minimum data volume and transmission duration. After each transmission duration, the target data volume of the data to be transmitted is obtained. If the target data volume is not less than the minimum data volume, the data to be transmitted is sent in batches to the sub-cluster. If the minimum data volume threshold is not reached, the system can wait for the next transmission cycle, thus avoiding network overhead and performance degradation caused by frequently sending large numbers of small data packets. This embodiment uses a dual-judgment approach based on data volume and transmission duration. On the one hand, it ensures high-throughput batch transmission of data during peak data generation periods or when network conditions are good, maximizing bandwidth utilization and reducing system load. On the other hand, during periods of sparse data, it can gather an appropriate amount of data by waiting for a reasonable period, avoiding invalid or inefficient transmission due to insufficient data volume, thereby ensuring overall high efficiency and economy in data transmission.

[0035] The transmission strategy described in this specification may also include robust fault tolerance and cleanup mechanisms to ensure the robustness of data transmission. Specifically, these mechanisms can manage retry behavior in case of failure by defining a maximum number of retries.

[0036] In one embodiment, if the data to be sent fails to be transmitted and the number of failures is less than the maximum number of retransmissions, the data to be sent can be sent at preset retransmission intervals. This can be considered a retry process triggered when the primary cluster fails to send data to the secondary cluster. Specifically, if the condition that the cumulative number of failures for the data to be sent is less than the configured maximum number of retransmissions is met, the sending task corresponding to that data will not be immediately abandoned. Instead, it will wait for a preset retransmission interval before attempting to send the same batch of data again. This process is repeated until success or the retry limit is reached. The retransmission interval can be dynamically calculated using an exponential backoff algorithm or pre-input manually to achieve customization; this specification does not impose any restrictions on this.

[0037] In another embodiment, if the data to be sent fails to be transmitted and the number of failures is not less than the maximum number of retransmissions, the data to be sent can be marked as an abnormal log, and the local persistent copy corresponding to the abnormal log can be queried and deleted. If the number of transmission failures has reached or exceeded the maximum number of retransmissions, it indicates that the data may not be successfully delivered in the current environment due to reasons such as continuous network interruption, destination unavailability, or abnormal data format. In this case, a screenshot of the system where the main cluster resides marks this batch of data to be sent as an abnormal log. To further release storage resources and avoid the accumulation of invalid data, the following cleanup operation can be performed: querying and deleting the local copy corresponding to this batch of abnormal logs that is persisted in the first message queue or its associated storage. This embodiment ensures automatic recovery in the face of temporary failures, while performing clear failure isolation and resource reclamation on abnormal data that is confirmed to be unprocessable, preventing it from infinitely occupying system resources, and facilitating subsequent centralized auditing and problem investigation.

[0038] In the cleanup operation of the previous embodiment, in addition to the persisted local copy, the data of the first message queue also needs to be removed. This specification introduces different queue removal methods for different consumption models involved in the message queue to ensure the accuracy of message queue consumption management and the reliability of data processing, and to avoid data loss or duplicate processing that may be caused by mismatch between the consumption model and the data removal operation.

[0039] Specifically, the consumption model of the first message queue mentioned above can adopt an automatic confirmation mode or a manual confirmation mode, and each of these different modes has different corresponding processing situations.

[0040] In the first scenario, if the consumption model employs automatic acknowledgment, the system can temporarily store the acquired data to be sent in a pre-defined cache after consuming log data from the queue. Subsequently, the system must respond to a removal command to remove the corresponding data from the cache. This cache can be, for example, a Redis (Remote Dictionary Server) database or a local in-memory database. The removal command indicates that the data has been successfully sent to the secondary cluster or has been marked as an abnormal log. In this mode, the server corresponding to the message queue immediately marks the message as consumed due to the automatic acknowledgment mechanism, thus requiring external caching to ensure the traceability and reliability of data during processing.

[0041] In the second scenario, if the consumption model uses a manual acknowledgment mode, then an external cache is unnecessary compared to the first scenario. This is because after consuming data, the data to be sent remains in the message queue, though its state is not visible to other consumers. Only when the system responds to the removal command will it send an acknowledgment (ACK) to the message queue server, thus officially removing the message from the first message queue. This mode completely delegates data persistence and lifecycle management to the message queue itself, thereby simplifying the architecture.

[0042] In summary, by employing different mechanisms under the aforementioned different circumstances, the system corresponding to the main cluster can flexibly adapt to different message queue middleware and their confirmation models. While achieving abnormal data cleanup and successful data removal, it fundamentally ensures the accuracy of data processing semantics and avoids data loss or duplication.

[0043] It is worth mentioning that this manual also incorporates serialization and compression steps into the transmission process, thereby ensuring the efficiency and compatibility of data transmission, and performing necessary optimization processing on the log data to be transmitted before transmission. This processing can significantly reduce network transmission bandwidth consumption and improve the ability of data to interact across platforms.

[0044] In one embodiment, after the main cluster obtains the data to be sent but before it is actually sent, the data to be sent can be serialized and encoded to generate a serialized data stream that conforms to a preset standardized format; at the same time, the serialized data stream is compressed based on a preset compression algorithm to generate compressed data to be sent.

[0045] Specifically, the serialization encoding process described above transforms log data, which might otherwise exist as memory objects or specific data structures, into a serialized data stream conforming to a preset standardized format, such as a byte stream. This standardized format can include formats such as JavaScript Object Notation (JSON), eXtensible Markup Language (XML), Protocol Buffers, or the Avro Data Serialization System (Avro). In short, serialization not only eliminates the data's dependence on the runtime environment, enabling correct parsing across different systems, but also prepares a highly redundant and well-structured binary data foundation for subsequent compression steps.

[0046] Next, the main cluster can compress the serialized data stream using preset compression algorithms such as GNU Zip (GZIP), LZ4 Compression Algorithm (LZ4), Snappy Compression Algorithm (Snappy), or Zstandard Compression Algorithm (Zstandard). These compression algorithms significantly reduce the size of the data to be transmitted by eliminating redundant information, thus generating compressed data to be sent. This directly reduces network bandwidth consumption and transmission time, effectively saving costs and improving synchronization efficiency, especially in scenarios involving processing hundreds of millions of log data points daily.

[0047] Through the two-step processing of serialization and compression, the data to be sent maintains semantic integrity while minimizing its size, laying a solid foundation for efficient and economical batch transmission to the secondary cluster.

[0048] The secondary cluster's responsibility is to reliably receive and store log data from the primary cluster, ultimately achieving data consistency with it. Upon receiving data packets from the primary cluster over the network, it first decompresses and deserializes them to reconstruct the structured log data. Subsequently, the secondary cluster persistently stores the parsed raw log data as messages in its own configured second message queue. This second message queue possesses the same high availability and persistence features as the first message queue, ensuring that data is not lost on the secondary cluster side due to system restarts or other anomalies.

[0049] After successfully storing the log data in the second message queue, the secondary cluster completes the data persistence between the primary and secondary clusters. This step signifies that a log data entry has successfully passed through the complete pipeline from collection, temporary storage, and transmission in the primary cluster to reception and storage in the secondary cluster, achieving redundant data backup. Subsequently, other downstream processing programs in the secondary cluster, such as real-time computing engines or offline analysis systems, can consume data from the second message queue for further statistical analysis. Since all the above operations are based on the same reliable data copy—the log data from the primary cluster mentioned earlier—the eventual consistency of data processing results between the primary and secondary clusters is fundamentally guaranteed.

[0050] The following is combined with Figure 3 The complete process and system architecture of the above log storage method will be further explained. Figure 3 As shown, the system architecture clearly demonstrates the data flow and collaboration relationships between the main cluster, the secondary cluster, and related nodes.

[0051] First, during the data collection and storage phase, CDN log data from two main sources are uniformly received: firstly, access logs generated by edge nodes of the self-built CDN are reported to the receiving interface of the main cluster through the collection program; secondly, CDN logs returned by the converged CDN vendor are also aggregated to the main cluster. After receiving this data, the devices corresponding to the main cluster store it as messages in the configured first message queue for buffering. Taking a Kafka cluster as the first message queue as an example, this message queue provides a unified, high-throughput data source for all subsequent tasks such as real-time statistics, offline computing, and permanent disk storage. Subsequently, during the data synchronization and backup phase, dedicated service components deployed in the main cluster can continuously consume log data from the first message queue, Kafka. This component can use high-performance log synchronization tools such as Rsyslog to reliably send log data to the receiving gateway of the secondary cluster. This sending process is configured with a failure retry mechanism to ensure that it can continue to try even when network fluctuations occur until the data is successfully delivered to the secondary cluster, thereby strongly guaranteeing the eventual consistency of data between the main and backup clusters at the transport layer. After receiving the data, the secondary cluster stores it in its own second message queue, which is also based on Kafka.

[0052] Through the above process, all aggregated CDN access logs, after being stored in the primary cluster's Kafka queue, can be fully, orderly, and reliably synchronized to the secondary cluster and stored in the backup cluster's Kafka queue by this service component. This aims to provide the secondary cluster with source data that is completely consistent with the primary cluster for all subsequent statistical tasks, permanent disk storage, offline log generation, and secondary log distribution, fundamentally ensuring strong consistency of CDN logs between the primary and backup clusters, as well as consistency of statistical results between the two.

[0053] Furthermore, it is worth noting that, Figure 3 The service components described herein can be designed as distributed services, supporting horizontal scaling (scale-out) and distributed multi-node deployment. This avoids single points of failure, ensures high availability of the data synchronization process, and linearly increases processing capacity as the data volume grows. In other words, this solution efficiently solves the problems of data inconsistency between primary and backup clusters and excessive machine and performance resource consumption during the synchronization process in traditional solutions.

[0054] The following is combined with Figure 4 right Figure 3 The methods executed by the service components in the document will be introduced, such as... Figure 4 As shown, the method includes the following steps: Step S402: The service component is initialized and the HTTP transport configuration is loaded.

[0055] In one embodiment, the service component can first execute an initialization process upon startup. Its built-in HTTP parameter configuration module loads necessary HTTP transmission parameters from a local configuration file or a remote configuration center. These parameters may include, but are not limited to: the URL address of the secondary cluster receiving gateway, the authentication key (Token), the connection timeout, the content type (Content-Type), and parameters related to subsequent batch sending, such as the minimum data size to be sent, the sending duration, and the maximum number of retransmissions mentioned above. This step ensures that the service component has all the necessary information to initiate HTTP requests to the correct target, preparing for subsequent data sending.

[0056] In step S404, the service component consumes log data from the first message queue.

[0057] In one embodiment, the Kafka consuming module in the service component can start one or more consumers to continuously pull CDN log data from the first message queue Kafka of the main cluster. This module is responsible for maintaining the connection with the Kafka cluster, managing the consumption offset, and delivering the acquired raw log data stream to subsequent processing units, thereby providing a stable and ordered data source for the entire synchronization process.

[0058] Step S406: The service component compresses the consumed data.

[0059] In one embodiment, the data compression module in the service component can compress the raw log data consumed from Kafka to optimize transmission efficiency. This module can use a preset compression algorithm to convert batch log data into compressed binary data packets, significantly reducing data size, thereby reducing network bandwidth usage and improving the efficiency of subsequent HTTP transmissions.

[0060] In step S408, the service component stores the compressed data into the sending queue cache.

[0061] In one embodiment, the compressed data is not sent immediately, but is temporarily stored using a sending queue caching module of the service component. This caching can be implemented based on a memory queue to buffer the speed difference between production and consumption, absorb instantaneous traffic peaks, prevent overload of downstream sending processes, and support subsequent batch assembly and sending strategies.

[0062] Those skilled in the art will understand that the queue cache mentioned in this step is not the same as the preset cache mentioned earlier. The former serves as a working buffer for the queue, used to temporarily store data that has been consumed from Kafka, compressed, and is ready for batch transmission; the latter is a persistent state store, used only to address data loss issues in automatic acknowledgment mode. To avoid subsequent ambiguity, it is assumed that... Figure 4 The queue in the system uses a manual confirmation consumption mode.

[0063] In step S410, the service component assembles batch data from the cache and performs an HTTP send.

[0064] In one embodiment, the HTTP batch sending module of the service component retrieves multiple compressed data items from the sending queue cache according to a preset strategy and assembles them into a batch message body. Subsequently, the module uses the HTTP configuration loaded in step S402 to initiate an HTTP POST request to the receiving gateway address of the secondary cluster, sending the batch data out all at once. This batch processing mechanism greatly improves network throughput and resource utilization efficiency.

[0065] Step S412: The service component processes the sending result and performs timeout retransmission.

[0066] In one embodiment, the HTTP timeout retransmission module of the service component is responsible for handling the response to HTTP requests. If the transmission is successful, for example, receiving an "HTTP 200 OK" message, it confirms that the batch of data synchronization is complete. If a network timeout or server error occurs, such as receiving an HTTP 500 message, a retransmission mechanism is triggered. This module will wait for a period of time according to a preset retry policy, retrieve the original data from the sending queue cache, and resend it to the HTTP batch sending module until successful or the maximum number of retries is reached, thereby ensuring the ultimate reliability of data transmission.

[0067] Figure 5 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, memory 508, and non-volatile memory 510, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 502 reads the corresponding computer program from the non-volatile memory 510 into memory 508 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0068] Corresponding to the embodiments of the aforementioned log storage methods, this specification also provides an embodiment of a log storage device. Please refer to... Figure 6 The device is applied to a primary cluster, which is configured with a first message queue; the device includes: Log data acquisition unit 602 is used to acquire log data of the content delivery network CDN; Message queue storage unit 604 is used to store the log data in the first message queue; The log data sending unit 606 is used to send the log data in the first message queue to the secondary cluster, so that the secondary cluster stores the received log data in the second message queue configured in the secondary cluster.

[0069] Optionally, the log data acquisition unit 602 is specifically used for: Obtain the first type of log data generated by the edge nodes of the CDN; and / or, Obtain the second type of log data generated by the third-party service node that is connected to the CDN.

[0070] Optionally, the message queue storage unit 604 is specifically used for: The log data is written to the first message queue and persisted to a non-volatile storage medium.

[0071] Optionally, the log data sending unit 606 is specifically used for: Consume the log data from the first message queue to obtain data to be sent; The data to be sent is transmitted to the secondary cluster based on a preset transmission strategy.

[0072] Optionally, the transmission strategy includes a minimum data transmission amount and a transmission duration; the log data transmission unit 606 is specifically used for: After each interval of the specified transmission duration, the target data volume of the data to be sent is obtained; If the target data volume is not less than the minimum data volume to be sent, the data to be sent will be sent in batches to the sub-cluster.

[0073] Optionally, the transmission strategy includes a maximum number of retransmissions; the log data sending unit 606 is specifically used for: If the data to be sent fails to be sent and the number of failures is less than the maximum number of retransmissions, the data to be sent will be sent every preset retransmission interval. If the data to be sent fails to be sent and the number of failures is not less than the maximum number of retransmissions, the data to be sent is marked as an abnormal log; and the local persistent copy corresponding to the abnormal log is queried and deleted.

[0074] Optionally, the device further includes: The data removal unit is configured to store the data to be sent in a preset cache when the consumption model of the first message queue adopts an automatic confirmation mode; and to remove the data to be sent from the cache in response to a removal instruction for the data to be sent, wherein the removal instruction is used to indicate that the data to be sent is an abnormal log or that the data to be sent was successfully sent. When the consumption model of the first message queue adopts the manual confirmation mode, in response to the removal instruction for the data to be sent, the data to be sent is removed from the first message queue. The removal instruction is used to indicate that the data to be sent is an abnormal log or that the data to be sent was successfully sent.

[0075] Optionally, the device further includes: The data compression unit is used to perform serialization encoding processing on the data to be sent to generate a serialized data stream that conforms to a preset standardized format; The serialized data stream is compressed using a preset compression algorithm to generate compressed data to be sent.

[0076] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0077] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0078] While this invention contains numerous specific details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of particular inventions. Certain features described in the multiple embodiments of this invention may also be implemented in combination in a single embodiment. On the other hand, various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0079] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0080] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for storing logs, characterized in that, Applied to a primary cluster, wherein the primary cluster is configured with a first message queue; the method includes: Obtain log data from the Content Delivery Network (CDN); The log data is stored in the first message queue; The log data in the first message queue is sent to the secondary cluster, so that the secondary cluster stores the received log data in the second message queue configured in the secondary cluster.

2. The method according to claim 1, characterized in that, The acquisition of log data from the Content Delivery Network (CDN) includes: Obtain the first type of log data generated by the edge nodes of the CDN; and / or, Obtain the second type of log data generated by the third-party service node that is connected to the CDN.

3. The method according to claim 1, characterized in that, The step of storing the log data in the first message queue includes: The log data is written to the first message queue and persisted to a non-volatile storage medium.

4. The method according to claim 1, characterized in that, Sending the log data in the first message queue to the secondary cluster includes: Consume the log data from the first message queue to obtain data to be sent; The data to be sent is transmitted to the secondary cluster based on a preset transmission strategy.

5. The method according to claim 4, characterized in that, The transmission strategy includes a minimum data transmission amount and a transmission duration; the transmission strategy based on the preset transmission strategy sends the data to be sent to the sub-cluster, including: After each interval of the specified transmission duration, the target data volume of the data to be sent is obtained; If the target data volume is not less than the minimum data volume to be sent, the data to be sent will be sent in batches to the sub-cluster.

6. The method according to claim 4, characterized in that, The transmission strategy includes a maximum number of retransmissions; the transmission strategy based on a preset transmission strategy for sending the data to be sent to the secondary cluster includes: If the data to be sent fails to be sent and the number of failures is less than the maximum number of retransmissions, the data to be sent will be sent every preset retransmission interval. If the data to be sent fails to be sent and the number of failures is not less than the maximum number of retransmissions, the data to be sent is marked as an abnormal log; and the local persistent copy corresponding to the abnormal log is queried and deleted.

7. The method according to claim 6, characterized in that, The method further includes: When the consumption model of the first message queue adopts the automatic confirmation mode, the data to be sent is stored in a preset cache; and, in response to a removal instruction for the data to be sent, the data to be sent is removed from the cache, the removal instruction being used to indicate that the data to be sent is an abnormal log or that the data to be sent was successfully sent; When the consumption model of the first message queue adopts the manual confirmation mode, in response to the removal instruction for the data to be sent, the data to be sent is removed from the first message queue. The removal instruction is used to indicate that the data to be sent is an abnormal log or that the data to be sent was successfully sent.

8. The method according to claim 4, characterized in that, The method further includes: The data to be sent is serialized and encoded to generate a serialized data stream that conforms to a preset standardized format; The serialized data stream is compressed using a preset compression algorithm to generate compressed data to be sent.

9. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-8.

10. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-8.

Images