Cross-data center log processing system based on edge computing and mixed storage

Abstract

The invention discloses a cross-data-center log processing system based on edge computing and mixed storage, and aims to solve the problems that in a traditional cross-data-center log processing scheme in the prior art, bandwidth cost is high due to full-amount transmission, cold data occupies a large number of storage resources due to a centralized storage architecture, real-time query performance is limited, and data processing efficiency is high. The contradiction between the bandwidth cost and the real-time query efficiency cannot be solved simultaneously in the prior art. The system comprises an edge calculation layer, a central query layer and a reliability guarantee layer, wherein the edge computing layer is deployed in a plurality of edge machine rooms and is used for receiving and processing locally generated data, and the edge machine rooms are used for receiving and processing the locally generated data and storing cold data; the central query layer is deployed in a central computer room and used for processing query requests, and the reliability guarantee layer is deployed in an edge computer room and used for guaranteeing high availability and performance of the system; the problems are effectively solved.

Description

Technical Field

[0001] The present application relates to the field of distributed database technology, and in particular to a cross-data center log processing system. Background Art

[0002] In today's digital age, businesses and organizations often have their business systems distributed across multiple data centers, generating massive amounts of log data during operation. Log data is extremely valuable for monitoring system status, troubleshooting, and analyzing business trends. However, traditional log processing solutions face numerous challenges and issues.

[0003] In traditional solutions, all cross-data center log data must be transferred to a central computer room for centralized processing and storage. For example, terabytes of log data may need to be transferred daily, placing enormous strain on public network bandwidth and leading to high bandwidth costs. Such large-scale data transmission not only consumes significant network resources but can also lead to delays and packet loss during transmission, compromising the timeliness and integrity of log data.

[0004] At the same time, centralized storage architectures also have drawbacks. Cold data—raw logs that are infrequently accessed—occupies a significant amount of storage space on the central data center's storage devices, resulting in a waste of storage resources. However, when real-time queries of these logs are required, performance is limited by network latency, as the data is stored in the central data center. This network latency can result in extended query result return times, making it impossible to meet business requirements such as real-time monitoring and rapid troubleshooting.

[0005] While existing technologies, such as Kafka cross-datacenter synchronization and data compression transmission, attempt to optimize these issues, Kafka cross-datacenter synchronization can ensure real-time data to a certain extent, it cannot effectively address the bandwidth cost issues associated with large-scale data transmission. While data compression transmission can reduce the amount of data transmitted, it may introduce additional latency during the decompression process, and further compression has limited effect on some already highly compressed data formats. This makes it difficult to balance bandwidth costs and real-time query efficiency. Therefore, existing technologies cannot fundamentally resolve the conflict between bandwidth costs and real-time query efficiency. A new technical solution is urgently needed to address these challenges and achieve efficient, low-cost, and real-time cross-datacenter log processing and storage. Summary of the Invention

[0006] The present application provides a cross-data center log processing system based on edge computing and hybrid storage, a log processing method based on edge computing, and a log data query method, aiming to solve the existing technology in traditional cross-data center log processing solutions, where full transmission leads to high bandwidth costs, the centralized storage architecture causes cold data to occupy a large amount of storage resources and the real-time query performance is limited. The existing technology cannot simultaneously solve the contradiction between bandwidth costs and real-time query efficiency.

[0007] In a first aspect, a cross-data center log processing system based on edge computing and hybrid storage is provided, wherein the system includes an edge computing layer, a central query layer, and a reliability assurance layer; wherein,

[0008] The edge computing layer is deployed in multiple edge computer rooms to receive and process locally generated data. The edge computer rooms are responsible for receiving and processing locally generated data and storing cold data. The edge computing layer includes:

[0009] The data receiving and preprocessing module is used to receive locally generated log data in real time, perform aggregation calculations, and generate aggregation results;

[0010] A hot data upload module is used to synchronize the aggregation results to the hot data database in the central computer room; wherein the central computer room is responsible for storing and processing hot data and providing a fast query interface;

[0011] The cold data storage module is used to write the original log data into the local distributed storage system and perform distributed storage according to the sharding and replica strategies;

[0012] The central query layer is deployed in the central computer room to process query requests. The central query layer includes:

[0013] Thermal data query interface, used to directly access the thermal data database of the central computer room;

[0014] The cold data proxy engine is used to route query requests to the edge data center of the edge computing layer through distributed tables and merge the returned results;

[0015] The reliability assurance layer is deployed in the edge data center to ensure high availability and performance of the system. The reliability assurance layer includes:

[0016] The network routing optimization module uses a high availability mechanism in each shard within the edge data center to achieve automatic failover within the shard.

[0017] The shard load balancing module dynamically distributes query requests to each shard based on dynamic routing protocols or network load balancing mechanisms.

[0018] In the above solution, optionally, the edge computing layer further includes:

[0019] Use a stream processing engine to regularly collect statistics on the aggregation results and write the aggregation results into the hot data database in the central computer room;

[0020] Aggregate logs according to preset metrics and filter out noisy data.

[0021] In the above solution, further optionally, the preset indicator is a preset time window.

[0022] In the above solution, optionally, the distributed storage system of the cold data storage module adopts:

[0023] Sharding strategy, used to divide data into shards by time range;

[0024] Multi-copy protocol, based on the consistency protocol to ensure the consistency of copies;

[0025] Distribute cold data shards through distributed hashing algorithms to avoid single point bottlenecks;

[0026] The virtual IP drift mechanism is used to automatically redirect query requests to the healthy copy of the faulty shard after detecting a failure.

[0027] In the above solution, optionally, the hot data upload module synchronizes the aggregation result increments to the central computer room based on the version number.

[0028] In the above solution, optionally, the original log data of the edge computer room is distributed to the local cold data storage module through a distributed hash algorithm, and multiple copies are configured.

[0029] In a second aspect, a log processing method based on edge computing is provided. The method is applied to the edge computing layer, and the edge computing layer includes: a data receiving and preprocessing module, a hot data uploading module, and a cold data storage module. The method includes:

[0030] The data receiving and preprocessing module receives locally generated log data in real time, performs aggregation calculations, and generates aggregation results.

[0031] Through the hot data upload module, the aggregation results are synchronized to the hot data database of the central computer room, so that the central computer room can store and process hot data and provide a fast query interface;

[0032] The original log data is written to the local distributed storage system through the cold data storage module; specifically, it includes: dividing data shards by time range according to the sharding strategy, ensuring the consistency of replicas based on the multi-copy protocol and data consistency protocol, allocating cold data shards through the distributed hash algorithm to avoid single point bottlenecks, and using the virtual IP drift mechanism to automatically redirect query requests to the healthy copy of the faulty shard after detecting a fault.

[0033] In the above solution, optionally, a stream processing engine is used to regularly collect statistics on aggregation results and write them into the hot data database in the central computer room; logs are aggregated according to preset indicators and noise data is filtered to further optimize data processing and storage efficiency.

[0034] In a third aspect, a log data query method is provided. The method is applied to a central query layer, which is deployed in a central computer room and includes a hot data query interface and a cold data proxy engine. The method includes:

[0035] Receive user query requests and determine whether the query target is hot data or cold data;

[0036] When the query target is hot data, the hot data database in the central computer room is directly accessed through the hot data query interface;

[0037] When the query target is cold data, the cold data proxy engine receives the query request and routes the query request to multiple edge computer rooms through a distributed table. The multiple edge computer rooms accelerate the query in parallel, and the edge computer room where the target cold data is located is recorded as the target edge computer room. In the target edge computer room, the query request load is balanced to the shards of the distributed storage system through a dynamic routing protocol; in the ClickHouse cluster of each target edge computer room, the query request is distributed to each shard, and the shard independently processes the query request and returns the result; after the query is completed in multiple edge computer rooms, the cold data proxy engine is responsible for merging the results returned by multiple edge computer rooms, and finally feeding back the complete query results to the user.

[0038] In the above solution, optionally, when a node in the target edge data center fails, the high availability mechanism triggers the virtual IP drift mechanism, and the virtual IP automatically drifts from the failed node to another healthy node. The query request is automatically redirected to the healthy copy of the failed shard through the virtual IP;

[0039] The dynamic routing protocol again load balances the query requests to the shards of the distributed storage system.

[0040] Compared with the prior art, this application has at least the following beneficial effects:

[0041] Based on further analysis and research of existing technical problems, this application recognizes that in traditional cross-data center log processing solutions, full transmission leads to high bandwidth costs, and the centralized storage architecture causes cold data to occupy a large amount of storage resources and limits real-time query performance. The existing technology cannot simultaneously solve the contradiction between bandwidth costs and real-time query efficiency. By deploying an edge computing layer in the edge computer room, locally generated log data is received, pre-processed and aggregated, and only the aggregated hot data is synchronized to the central computer room, while the original cold data is stored in the local distributed storage system, the amount of data transmitted across data centers is effectively reduced, and the public network bandwidth cost is reduced; by deploying a central query layer in the central computer room, hot data query interface is used to quickly access hot data, and with the help of the cold data proxy engine, query requests are routed to the edge computer room, and the returned results are merged, thereby achieving efficient query of cold data and improving real-time query performance; by deploying a reliability assurance layer in the edge computer room, network routing optimization module and sharding load balancing module are adopted to ensure high availability and performance of the system, thereby achieving the effect of reducing bandwidth costs while improving real-time query efficiency, and solving the contradiction between bandwidth costs and real-time query efficiency in traditional solutions.

[0042] This application uses edge computing to achieve tiered storage of hot and cold data and dynamic routing query, reducing bandwidth consumption across computer rooms.

[0043] The technical effects that can be achieved by this application include: (1) bandwidth saving, only transmitting aggregated hot data (data volume reduced by more than 90%); (2) query efficiency improvement: local query latency of hot data is <100ms, and cross-computer room queries of cold data are accelerated through sharding and parallelization; (3) reliability enhancement: the sharding replica strategy + Keepalived / OSPF in computer rooms B and C achieves 99.99% availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of the system architecture of a cross-data center log processing system based on edge computing and hybrid storage is provided for one embodiment of the present application.

[0045] Figure 2 A schematic diagram of the internal architecture of an edge computer room provided for one embodiment of the present application.

[0046] Figure 3 A flowchart of a log processing method based on edge computing is provided for one embodiment of the present application.

[0047] Figure 4 A flowchart of a log data query method provided in one embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0049] In the description of this application: unless otherwise specified, "a plurality of" means two or more. Expressions such as "include", "comprising", "having" and the like also mean "not limited to" (certain units, components, materials, steps, etc.).

[0050] Existing cross-data center log processing has the following problems: (1) In traditional solutions, all cross-data center logs must be transmitted to the central computer room, resulting in high public network bandwidth costs (e.g., terabyte-level data transmission per day). (2) Under the centralized storage architecture, cold data (raw logs that are accessed infrequently) takes up a large amount of storage resources, while real-time query performance is limited by network latency. (3) Existing technologies (e.g., Kafka cross-data center synchronization and data compression transmission) cannot simultaneously resolve the contradiction between bandwidth costs and real-time query efficiency.

[0051] To solve the above problems, this application proposes a cross-data center log processing system based on edge computing and hybrid storage, which relates to the field of distributed database technology, and in particular to a cross-data center log transmission and query optimization system based on ClickHouse cluster. It realizes hierarchical storage and dynamic routing query of hot and cold data through edge computing, and reduces bandwidth consumption across computer rooms.

[0052] In one embodiment, reference Figure 1 , provides a cross-data center log processing system based on edge computing and hybrid storage, the system includes an edge computing layer, a central query layer and a reliability assurance layer; wherein,

[0053] The edge computing layer is deployed in multiple edge computer rooms to receive and process locally generated data. The edge computer rooms are responsible for receiving and processing locally generated data and storing cold data. The edge computing layer includes:

[0054] The data receiving and preprocessing module is used to receive locally generated log data in real time, perform aggregation calculations, and generate aggregation results;

[0055] A hot data upload module is used to synchronize the aggregation results to the hot data database in the central computer room; wherein the central computer room is responsible for storing and processing hot data and providing a fast query interface;

[0056] The cold data storage module is used to write the original log data to a local distributed storage system (for example, a ClickHouse cluster) and perform distributed storage according to the shard and replica strategies;

[0057] The central query layer is deployed in the central computer room to process query requests. The central query layer includes:

[0058] Thermal data query interface, used to directly access the thermal data database of the central computer room;

[0059] The cold data proxy engine is used to route query requests to the edge data center of the edge computing layer through distributed tables (for example, the Distributed engine) and merge the returned results;

[0060] The reliability assurance layer is deployed in the edge data center to ensure high availability and performance of the system. The reliability assurance layer includes:

[0061] The network routing optimization module uses a high availability mechanism (for example, the Keepalived mechanism) in each shard within the edge data center to achieve automatic failover within the shard.

[0062] The shard load balancing module dynamically distributes query requests to each shard based on a dynamic routing protocol (e.g., OSPF network) or a network load balancing mechanism.

[0063] In this embodiment, the system divides log processing into edge computing layer, central query layer and reliability assurance layer by combining edge computing with hybrid storage, and uses ClickHouse cluster to realize distributed storage and query optimization, solving the problems of high bandwidth cost, waste of storage resources and limited real-time query performance in traditional solutions.

[0064] Edge computing layer: The data reception and preprocessing module reduces the amount of data transmitted to the central computer room through preprocessing, reducing bandwidth consumption. Hot data upload module: Synchronizes aggregation results to the central computer room's hot data database. The central computer room is responsible for storing and processing hot data, providing a fast query interface to ensure efficient processing of frequently accessed data. The cold data storage module uses a tiered storage method to prevent cold data from occupying the central computer room's storage resources, while leveraging the distributed nature of ClickHouse to improve storage efficiency and query performance.

[0065] Centralized query layer: Since hot data is high-frequency data that has been aggregated and filtered, storing it in the central data center ensures low latency and high efficiency for queries. The cold data proxy engine enables distributed queries of cold data, avoiding the storage and query pressure associated with centrally storing cold data in the central data center.

[0066] Reliability assurance layer: The network routing optimization module ensures high system availability. Even if a shard fails, it automatically switches to another available shard, ensuring stable system operation. The shard load balancing module fully utilizes the computing and storage resources of each shard, avoiding resource waste and performance bottlenecks.

[0067] Through rational architectural design and technological innovation, this solution effectively addresses the high bandwidth costs, wasted storage resources, and limited real-time query performance issues inherent in traditional solutions. While reducing bandwidth costs, it improves storage efficiency and query performance, and enhances system reliability, providing an efficient, cost-effective, and reliable solution for large-scale, cross-data center log processing.

[0068] In addition, in the embodiments of the present application, the ClickHouse cluster, the Distributed engine, the consistent hashing algorithm, etc. are all exemplary illustrations of the present application. Users can flexibly select any other suitable components or algorithms to achieve the same or similar functions according to actual conditions. For example, other distributed databases (such as Apache Cassandra, Elasticsearch, etc.) can be used to replace the ClickHouse cluster; other distributed table engines (such as Apache ShardingSphere's sharding engine) can be used to replace the Distributed engine; or other distributed storage and load balancing algorithms (such as ZooKeeper's consistency algorithm, Redis's distributed algorithm, etc.) can be used to replace the consistent hashing algorithm. This application does not impose specific restrictions on the specific technology selection, and aims to provide a flexible and scalable cross-data center log processing architecture to meet the needs of different users in different scenarios.

[0069] In one embodiment, the edge computing layer further includes:

[0070] Use a stream processing engine (e.g., Flink stream processing engine) to regularly collect statistics on the aggregation results and write the aggregation results into a hot data database in the central computer room;

[0071] Aggregate logs according to preset metrics and filter out noisy data.

[0072] In one embodiment, the preset indicator is a preset time window.

[0073] In this embodiment, the stream processing engine optimizes the log processing process in the following ways:

[0074] Regularly collect statistics on aggregation results: further summarize and count the log data that has been aggregated by time window, and write the results to the hot data database in the central computer room to reduce data transmission volume and improve query performance.

[0075] Aggregate logs by preset metrics and filter out noisy data: Group and aggregate log data based on preset time windows, while filtering out useless noisy data to improve data quality and availability.

[0076] Dynamically adjust the time window: Dynamically adjust the size of the time window according to business needs and data traffic to ensure that the system can operate efficiently in different business scenarios.

[0077] The combination of these features enables the edge computing layer to efficiently process and transmit log data, while reducing system resource consumption and costs, providing a more optimized solution for cross-data center log processing.

[0078] In one embodiment, the distributed storage system of the cold data storage module adopts:

[0079] Sharding strategy, used to divide data into shards by time range;

[0080] Multi-copy protocol, based on the consistency protocol to ensure the consistency of copies;

[0081] Distribute cold data shards through distributed hashing algorithms (e.g., consistent hashing algorithms) to avoid single point bottlenecks;

[0082] The virtual IP drift mechanism is used to automatically redirect query requests to the healthy copy of the faulty shard after detecting a failure.

[0083] In this embodiment, by adopting a sharding strategy, a multi-copy protocol, a distributed hash algorithm, and a virtual IP drift mechanism, the distributed storage system of the cold data storage module can achieve the following goals:

[0084] Efficient storage: Data is stored in a dispersed manner through sharding strategy to improve storage efficiency.

[0085] High availability: The system is highly available through a multi-copy protocol and virtual IP drift mechanism. Even if a node fails, the system can still operate normally.

[0086] Data consistency: The consistency between replicas is guaranteed through the consistency protocol, ensuring the integrity and reliability of the data.

[0087] Load balancing: Distributed hashing algorithms are used to evenly distribute data to multiple nodes, avoiding single-point bottlenecks and improving system scalability and performance.

[0088] In one embodiment, the hot data upload module synchronizes the aggregation result increments to the central computer room based on the version number.

[0089] In this embodiment, in the cross-data center log processing system, the hot data upload module effectively solves the problems of high bandwidth cost and low synchronization efficiency caused by the traditional full synchronization method through an incremental synchronization mechanism based on version numbers.

[0090] In one embodiment, the original log data of the edge computer room is distributed to the local cold data storage module through a distributed hash algorithm, and multiple copies are configured.

[0091] In this embodiment, the system adopts an edge computing architecture to distribute data processing to multiple edge computer rooms, combining local cold data storage and central computer room hot data storage to achieve hierarchical management of data. The edge computer room uses a distributed hash algorithm to evenly distribute cold data to local storage nodes, configures multiple copies, and combines consistency protocols to ensure high availability and consistency of data.

[0092] The following describes the system solution of this application from another perspective:

[0093] This application proposes a hybrid architecture of "edge preprocessing + central proxy query", which includes the following core modules:

[0094] 1. Edge computing layer (deployed in computer rooms B and C):

[0095] Data reception and preprocessing module: Receives data from this computer room in real time and performs aggregate calculations (such as counting requests by time window).

[0096] Hot data upload module: synchronizes the aggregation results to the "hot data" database of computer room A.

[0097] Cold Data Storage: Raw logs are written to a local ClickHouse cluster and distributedly stored using sharding and replica strategies. The cold data storage cluster utilizes the following reliability mechanisms: ① Sharding: Data is sharded by time range; ② Multi-replica protocol: Replica consistency is ensured using the Raft algorithm; ③ Network routing: Automatic failover and load balancing are implemented using OSPF / Keepalived.

[0098] The edge computing layer also includes: a data preprocessing unit that aggregates logs by time window and filters out noise data; a hot data synchronization unit that synchronizes the aggregation result increments to the central computer room based on the version number.

[0099] 2. Central query layer (deployed in computer room A, which has a hot data storage module that receives and stores aggregated data from edge computer rooms);

[0100] Hot data query interface: Directly access the hot data storage of computer room A.

[0101] Cold data proxy engine: routes query requests to computer rooms B and C through a distributed table (Distributed engine) and merges and returns the results.

[0102] 3. Reliability assurance layer: edge data center

[0103] Network routing optimization: Each shard within the B and C computer rooms uses Keepalived to achieve automatic switching in the event of a fault within the shard.

[0104] Shard load balancing: Based on the OSPF network, load balances query requests to each shard;

[0105] The CK feature of the database distributes cold data shards through a consistent hashing algorithm to avoid single point bottlenecks.

[0106] Among them, the edge computing layer configuration (taking room B as an example):

[0107] Use tools such as the Flink stream processing engine to collect log metrics every 5 minutes and write them to the hot_metrics table in Data Center A.

[0108] The original log is allocated to cold data shards using the consistent hashing algorithm and written to the cold_logs table in Data Center B, with two copies configured.

[0109] Technical effects of this application:

[0110] 1. Bandwidth saving: Only aggregated hot data is transmitted (data volume is reduced by more than 90%).

[0111] 2. Improved query efficiency: The local query latency for hot data is less than 100ms, and cross-data center queries for cold data are accelerated through sharding and parallelization.

[0112] 3. Enhanced reliability: The shard replication strategy + Keepalived / OSPF within the B and C computer rooms achieves 99.99% availability.

[0113] In one embodiment, reference Figure 3 , provides a log processing method based on edge computing, the method is applied to the edge computing layer, the edge computing layer includes: a data receiving and preprocessing module, a hot data uploading module and a cold data storage module, the method includes:

[0114] The data receiving and preprocessing module receives locally generated log data in real time, performs aggregation calculations, and generates aggregation results.

[0115] Through the hot data upload module, the aggregation results are synchronized to the hot data database of the central computer room, so that the central computer room can store and process hot data and provide a fast query interface;

[0116] The original log data is written to the local distributed storage system through the cold data storage module; specifically, it includes: dividing data shards by time range according to the sharding strategy, ensuring the consistency of replicas based on the multi-copy protocol and data consistency protocol, allocating cold data shards through the distributed hash algorithm to avoid single point bottlenecks, and using the virtual IP drift mechanism to automatically redirect query requests to the healthy copy of the faulty shard after detecting a fault.

[0117] In this embodiment, the method achieves efficient processing and storage of locally generated log data through a clear division of labor among modules. The data reception and preprocessing module receives and aggregates log data in real time. After generating the aggregated results, the hot data upload module synchronizes them to the central computer room to meet real-time query requirements. At the same time, the original log data is written to the local distributed storage system through the cold data storage module. The method uses a sharding strategy, a multi-copy protocol, a distributed hash algorithm, and a virtual IP drift mechanism to optimize storage efficiency, improve data reliability, and ensure high availability. Overall, it takes into account efficiency, reliability, and flexibility, and is suitable for large-scale cross-data center log processing scenarios.

[0118] In one embodiment, a stream processing engine is used to regularly collect statistics on aggregation results and write them into the hot data database of the central computer room; logs are aggregated according to preset indicators and noise data is filtered to further optimize data processing and storage efficiency.

[0119] In this embodiment, the log processing process at the edge computing layer is further optimized by introducing a stream processing engine. The stream processing engine regularly collects statistics on the aggregation results and writes these statistics to the hot data database in the central computer room, thereby providing more efficient support for real-time query and analysis. At the same time, the stream processing engine also aggregates log data according to preset indicators (such as time window, log type, etc.) and filters out noisy data, further reducing the data volume and improving data quality and storage efficiency.

[0120] In one embodiment, reference Figure 1 、 Figure 2 and Figure 4 , provides a log data query method, which is applied to a central query layer deployed in a central computer room and includes a hot data query interface and a cold data proxy engine. The method includes:

[0121] Receive user query requests and determine whether the query target is hot data or cold data;

[0122] When the query target is hot data, the hot data database in the central computer room is directly accessed through the hot data query interface;

[0123] When the query target is cold data, the cold data proxy engine receives the query request and routes the query request to multiple edge computer rooms through a distributed table. The multiple edge computer rooms accelerate the query in parallel, and the edge computer room where the target cold data is located is recorded as the target edge computer room. In the target edge computer room, the query request load is balanced to the shards of the distributed storage system through a dynamic routing protocol; in the ClickHouse cluster of each target edge computer room, the query request is distributed to each shard, and the shard independently processes the query request and returns the result; after the query is completed in multiple edge computer rooms, the cold data proxy engine is responsible for merging the results returned by multiple edge computer rooms, and finally feeding back the complete query results to the user.

[0124] In this embodiment, a log data query method includes the steps of: receiving a query request and determining whether the target is hot data or cold data; if it is hot data, directly querying the central computer room storage; if it is cold data, distributing the query to the edge computer room shards through a distributed table, and merging and returning the results.

[0125] Center query code example:

[0126] --Query hot data (directly access computer room A);

[0127] SELECT avg(error_rate)FROM hot_metrics WHERE time>='2024-01-0100:00:00';

[0128] --Query cold data (agent to B / C computer room);

[0129] SELECT count(*)FROM distributed_cold_logs WHERE log_level='ERROR';

[0130] In this embodiment, when the query target is hot data, the system directly accesses the hot data database in the central computer room through the hot data query interface. This direct access method can quickly respond to query requests for high-frequency access data, ensuring low latency and high efficiency of queries. When the query target is cold data, the cold data proxy engine receives the query request and routes the query request to multiple edge computer rooms through a distributed table. This distributed query strategy takes advantage of edge computing and distributes the query load to multiple edge computer rooms, thereby accelerating the query process.

[0131] In the target edge data center, query requests are load-balanced across the distributed storage system's shards using a dynamic routing protocol. Each shard independently processes the query request and returns the results. This parallel processing approach further improves query efficiency and reduces query time. After multiple edge data centers complete the query, the cold data proxy engine consolidates the results returned by these centers and ultimately returns the complete query results to the user. This result consolidation mechanism ensures that users receive comprehensive and consistent query results.

[0132] In one embodiment, when a node in the target edge data center fails, the high availability mechanism triggers the virtual IP drift mechanism. The virtual IP automatically drifts from the failed node to another healthy node, and query requests are automatically redirected to the healthy copy of the failed shard through the virtual IP.

[0133] The dynamic routing protocol again load balances the query requests to the shards of the distributed storage system.

[0134] In this embodiment, upon detecting a node failure, the high availability mechanism automatically triggers virtual IP migration. The virtual IP automatically migrates from the failed node to a healthy replica, ensuring seamless handoff of query requests from the failed node to the healthy replica. Through virtual IP migration, query requests are automatically redirected to the healthy replica of the failed shard. This process is transparent to the user, requiring no user intervention and ensuring query continuity.

[0135] After the failure is recovered, the dynamic routing protocol will again load balance the query requests to the shards of the distributed storage system. This dynamic adjustment mechanism ensures load balancing of the system and avoids performance bottlenecks caused by single points of failure.

[0136] In this embodiment, the fault recovery process is as follows:

[0137] Scenario: A ClickHouse node in Data Center B goes down.

[0138] Recovery steps:

[0139] ①Keepalived triggers VIP (virtual IP) drift, and the query is automatically redirected to the healthy copy of the faulty shard.

[0140] ②OSPF protocol load balancing query request to the shard.

[0141] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A cross-data center log processing system based on edge computing and hybrid storage, characterized by: The system includes an edge computing layer, a central query layer and a reliability assurance layer; wherein, The edge computing layer is deployed in multiple edge computer rooms to receive and process locally generated data. The edge computer rooms are responsible for receiving and processing locally generated data and storing cold data. The edge computing layer includes: The data receiving and preprocessing module is used to receive locally generated log data in real time, perform aggregation calculations, and generate aggregation results; A hot data upload module is used to synchronize the aggregation results to the hot data database in the central computer room; wherein the central computer room is responsible for storing and processing hot data and providing a fast query interface; The cold data storage module is used to write the original log data into the local distributed storage system and perform distributed storage according to the sharding and replica strategies; The central query layer is deployed in the central computer room to process query requests. The central query layer includes: Thermal data query interface, used to directly access the thermal data database of the central computer room; The cold data proxy engine is used to route query requests to the edge data center of the edge computing layer through distributed tables and merge the returned results; The reliability assurance layer is deployed in the edge data center to ensure high availability and performance of the system. The reliability assurance layer includes: The network routing optimization module uses a high availability mechanism in each shard within the edge data center to achieve automatic failover within the shard. The shard load balancing module dynamically distributes query requests to each shard based on dynamic routing protocols or network load balancing mechanisms.

2. The cross-data center log processing system based on edge computing and hybrid storage according to claim 1 is characterized in that: The edge computing layer also includes: Use a stream processing engine to regularly collect statistics on the aggregation results and write the aggregation results into the hot data database in the central computer room; Aggregate logs according to preset metrics and filter out noisy data.

3. The cross-data center log processing system based on edge computing and hybrid storage according to claim 2 is characterized in that: The preset indicator is a preset time window.

4. The cross-data center log processing system based on edge computing and hybrid storage according to claim 1 is characterized in that: The distributed storage system of the cold data storage module adopts: Sharding strategy, used to divide data into shards by time range; Multi-copy protocol, based on the consistency protocol to ensure the consistency of copies; Distribute cold data shards through distributed hashing algorithms to avoid single point bottlenecks; The virtual IP drift mechanism is used to automatically redirect query requests to the healthy copy of the faulty shard after detecting a failure.

5. The cross-data center log processing system based on edge computing and hybrid storage according to claim 1 is characterized in that: The hot data upload module synchronizes the aggregation result increments to the central computer room based on the version number.

6. The cross-data center log processing system based on edge computing and hybrid storage according to claim 1 or 4, characterized in that: The original log data of the edge computer room is distributed to the local cold data storage module through the distributed hash algorithm, and multiple copies are configured.

7. A log processing method based on edge computing, characterized in that: The method is applied to the edge computing layer, which includes: a data receiving and preprocessing module, a hot data uploading module, and a cold data storage module. The method includes: The data receiving and preprocessing module receives locally generated log data in real time, performs aggregation calculations, and generates aggregation results. Through the hot data upload module, the aggregation results are synchronized to the hot data database of the central computer room, so that the central computer room can store and process hot data and provide a fast query interface; The original log data is written to the local distributed storage system through the cold data storage module; specifically, it includes: dividing data shards by time range according to the sharding strategy, ensuring the consistency of replicas based on the multi-copy protocol and data consistency protocol, allocating cold data shards through the distributed hash algorithm to avoid single point bottlenecks, and using the virtual IP drift mechanism to automatically redirect query requests to the healthy copy of the faulty shard after detecting a fault.

8. The log processing method based on edge computing according to claim 7, characterized in that: Use the stream processing engine to regularly collect statistics on aggregation results and write them into the hot data database in the central computer room; aggregate logs according to preset indicators and filter out noise data to further optimize data processing and storage efficiency.

9. A log data query method, characterized in that: The method is applied to a central query layer, which is deployed in a central computer room and includes a hot data query interface and a cold data proxy engine. The method includes: Receive user query requests and determine whether the query target is hot data or cold data; When the query target is hot data, the hot data database in the central computer room is directly accessed through the hot data query interface; When the query target is cold data, the cold data proxy engine receives the query request and routes the query request to multiple edge computer rooms through a distributed table. The multiple edge computer rooms accelerate the query in parallel, and the edge computer room where the target cold data is located is recorded as the target edge computer room. In the target edge computer room, the query request load is balanced to the shards of the distributed storage system through a dynamic routing protocol; in the ClickHouse cluster of each target edge computer room, the query request is distributed to each shard, and the shard independently processes the query request and returns the result; after the query is completed in multiple edge computer rooms, the cold data proxy engine is responsible for merging the results returned by multiple edge computer rooms, and finally feeding back the complete query results to the user.

10. The log data query method according to claim 9, characterized in that: When a node in the target edge data center fails, the high availability mechanism triggers the virtual IP drift mechanism. The virtual IP automatically drifts from the failed node to another healthy node, and query requests are automatically redirected to the healthy copy of the failed shard through the virtual IP. The dynamic routing protocol again load balances the query requests to the shards of the distributed storage system.

Images