A multi-tiered cross-cluster system for a SOC operations platform

Abstract

The application relates to the technical field of multi-cluster management, and provides a multi-level cross-cluster system for an SOC operation platform, a main system comprising: a reverse proxy module, which is used as a reverse proxy to receive and dispatch client query requests; an API gateway module, which is used to provide an API interface, process business logic and request scheduling; an A component, which is connected to the proxy layer of the subsystem and is used to preferentially query a main cluster according to query requests transmitted by the reverse proxy module and the API gateway module; and a B component, which is the proxy layer of the main system and is used to monitor a sub-cluster by adopting a basic heartbeat detection strategy and a fast fuse determination strategy. The application can significantly reduce network topology complexity under the premise of guaranteeing retrieval performance, greatly reduce errors by taking a harmonic mean of multiple independent statistics, realize global data source management and an abnormal self-recovery mechanism through the A component of multi-live deployment, and construct an end-to-end fault-tolerant system in combination with the dynamic proxy characteristics of the B component.

Description

Technical Field

[0001] This application relates to the field of multi-cluster management technology, specifically to a multi-level cross-cluster system for SOC operation platforms. Background Technology

[0002] In the multi-cluster management architecture design of a unified operation platform, cross-cluster retrieval is implemented based on an intelligent sharding routing mechanism between the upper-level coordination point and lower-level cluster nodes to achieve multi-level data retrieval. Existing technologies require configuring the entire list of lower-level cluster nodes into the upper-level cluster, which has drawbacks such as a large port exposure range and high network topology complexity.

[0003] For unique counting operators, Elasticsearch suffers from inaccurate query results in cross-cluster, multi-shard scenarios. When querying global Top-N results, existing Elasticsearch technologies control the number of candidate terms returned by each shard during local computation based on the `shard_size` parameter. These terms are then aggregated to the coordinating node for global sorting, ultimately retrieving the Top-size result. If the number of shards is large or data skew is severe, the coordinating node may not collect enough data to correctly calculate the global Top-N, leading to inaccurate global results. Increasing the query size and `shard_size` parameters can improve query accuracy to some extent, but it significantly increases the resource consumption of the Elasticsearch cluster.

[0004] In the passive response mode of traditional cluster failover, in a typical Elasticsearch multi-level cluster architecture, especially when the lower levels do not undertake computing tasks, query anomalies often occur. Summary of the Invention

[0005] To help solve the above-mentioned technical problems, this application provides a multi-level cross-cluster system for SOC operation platforms, adopting the following technical solution: A multi-level cross-cluster system for a SOC operation platform, comprising a main system and at least one subsystem, wherein the main system includes a main cluster, and the subsystems include sub-clusters, wherein the main cluster and the sub-clusters are connected through a preset coordination node, and the main system includes: The reverse proxy module is used as a reverse proxy to receive and dispatch client query requests; The API gateway module is used to provide API interfaces, handle business logic, and schedule requests. Component A connects to the proxy layer of the subsystem and is used to prioritize querying the main cluster based on query requests transmitted by the reverse proxy module and API gateway module. When the main cluster response time exceeds the preset response time threshold or returns an exception, Component A automatically triggers the sub-cluster query by triggering the proxy layer of the subsystem and releases the occupied memory resources. Component B is the proxy layer of the main system. It is used to monitor the sub-cluster using a basic heartbeat detection strategy and a fast circuit breaker decision strategy. When a sub-cluster is detected to be unavailable, it will automatically remove the unavailable sub-cluster from the query route and reconnect to the unavailable sub-cluster after the cluster is restored through a hot reloading mechanism. The alarm aggregation module is used to obtain the original alarms from all subsystems from the distributed message queue module in the main system, generate the final alarm according to the alarm merging rules, and store the final alarm in the main cluster.

[0006] Preferably, both the main system and the subsystem include: The data collector is used to collect node logs and send them to the correlation analysis module; The correlation analysis module is used to generate original alarms according to preset rules and send them to the upper-level distributed message queue module.

[0007] Preferably, the preset coordination node is port 9300.

[0008] Preferably, component A queries the cluster in the following way: Step A1: When a client initiates an aggregation query request, the main cluster and sub-cluster transmit the aggregation query request to the cluster shards. Each cluster shard creates an aggregation query request object, which initializes the aggregator build factory by parsing the request parameters. Step A2: Create an aggregator through an aggregator factory. The aggregator is used to collect data. Step A3: Execute the HLL++ algorithm on each shard using the aggregator to calculate and store the HLL++ hash value; Step A4: After each shard is calculated, the HLL++ hash values ​​of all shards are merged through the coordinating node to obtain the cardinality estimation result, and the cardinality estimation result is returned to the client.

[0009] Preferably, step A1 includes: the request parameters include 9200 request parameters and 9300 request parameters.

[0010] Preferably, component A, component B, and the data collector constitute a data center, and the reverse proxy module, API gateway module, and correlation analysis module constitute an integrated brain platform. The multi-level cross-cluster system for the SOC operation platform is configured in the following way: B1: Open ports to the integrated brain platform through the data center, configure source address, destination address and port for data access and inter-module communication; B2: For core functional modules, modify configuration files and build images, including: uploading and decompressing upgrade packages, modifying configuration files of corresponding modules, cloning and modifying configuration files according to the IP address of the data center to be connected, building images and publishing configurations on the Nacos page, switching to the server in the corresponding data center to restart the module in the background, and checking logs to confirm normal startup; B3: Log in to the server of the integrated brain platform and register the ES cluster of the data center to the integrated brain platform; B4: Integrate the SAE core clusters of each data center into the unified platform management, edit the SAE monitoring configuration file, and add the service names of the SAE core clusters to be discovered; B5: Update the SAE monitor by replacing the class files and restarting the service to view an overview of the analytics engines in each data center.

[0011] Preferably, B1 includes: opening ports 9111, 9015, 7777, 8765, 9200 and 9300 to the integrated brain platform through the data center.

[0012] Preferably, the multi-level cross-cluster system for the SOC operation platform further includes a security event engine and a scheduling engine. The security event engine is used for the detection and response to security events, and the scheduling engine is used for the scheduling and execution of tasks.

[0013] In summary, this application has the following advantages: In this application, the main ES cluster node of the upper-level SOC platform only needs to establish a dedicated TCP reliable connection with the specific coordination node 9300 port of each lower-level sub-ES cluster, and construct a virtual data view by using the Cross-Cluster Search (CCS) feature, by configuring cluster.remote on the upper-level main cluster.<cluster_alias> The `.seeds` directive points to the seed nodes of the sub-cluster. Combined with access control via `indices:data / read / search` and a seed node liveness detection mechanism (using a sniffer), cross-cluster joint queries are achieved. This solution overcomes the limitations of traditional fully interconnected architectures, reducing the port exposure range from N×(N-1) to linear growth. Simultaneously, it controls query routing through the `preference` parameter, significantly reducing network topology complexity while ensuring retrieval performance.

[0014] This application employs the HLL (HyperLogLog) algorithm to address the issue of inaccurate query results in Elasticsearch scenarios involving multiple shards across clusters. It maps elements to binary values ​​using a hash function and estimates the cardinality by counting the maximum number of leading zeros in the hash value. By using multiple independent statistical summaries and averages, the error is significantly reduced.

[0015] This application adopts a two-level ES cluster architecture, which distributes business data across upper and lower level ES nodes. It implements global data source management and anomaly self-healing mechanism through multi-active deployment of component A, and builds an end-to-end fault-tolerant system by combining the dynamic proxy characteristics of component B. Attached Figure Description

[0016] Figure 1 This is a schematic block diagram of one embodiment of a multi-level cross-cluster system for a SOC operation platform according to this application; Figure 2 This is a schematic diagram of the query cluster based on the HLL algorithm in this application. Detailed Implementation

[0017] The present application will be further described below with reference to the accompanying drawings. The structure and principle of the present application are very clear to those skilled in the art. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

[0018] Elasticsearch Cluster is a core feature of Elasticsearch that allows multiple Elasticsearch nodes to work together to form a distributed system. Here is a brief introduction to Elasticsearch Cluster: Node: An Elasticsearch instance that can be added to a cluster.

[0019] Master Node: Responsible for managing the cluster's metadata, such as index creation, deletion, node addition, and node departure.

[0020] Data Node: Responsible for storing data and performing search and indexing operations.

[0021] Client Node: Does not store data; primarily used to route requests to the correct node.

[0022] Data is divided into multiple shards, distributed across different nodes, improving data storage and query performance. Replica shards provide redundant data storage, ensuring no data loss in the event of node failure. Cluster storage capacity and query performance can be expanded by adding new nodes. Data and query requests are automatically distributed among the nodes in the cluster, ensuring efficient system operation. Data is divided into shards, each with multiple replicas distributed across different nodes. The nodes in the cluster elect a master node to manage the cluster's metadata. When performing queries or indexing operations, requests are routed to the node containing the relevant data. If a node or shard becomes unavailable, the cluster automatically routes requests to other available replica shards.

[0023] Figure 1 This is a schematic block diagram of an embodiment of a multi-level cross-cluster system for a SOC operation platform according to this application. The multi-level cross-cluster system for a SOC operation platform includes a main system and at least one subsystem. In this embodiment, it has one main system and two subsystems. The main system includes a main cluster, and the subsystems include sub-clusters. The main cluster and the sub-clusters are connected via a preset coordinating node (port 9300). The main system includes: Reverse proxy module, corresponding Figure 1 Nginx (Reverse Proxy) is used as a reverse proxy to receive and dispatch client query requests.

[0024] The API Gateway module, corresponding to the API Gateway (Webserver / Node) in the diagram, is used to provide API interfaces, handle business logic, and schedule requests.

[0025] Component A corresponds to the DAL query engine (Data Access Layer) in the diagram. Component A connects to the proxy layer of the subsystem and is used to prioritize querying the main cluster based on query requests transmitted by the reverse proxy module and API gateway module. When the main cluster response time exceeds the preset response time threshold or returns an exception, Component A automatically triggers the sub-cluster query by triggering the proxy layer of the subsystem and releases the occupied memory resources.

[0026] Component B corresponds to the ES Proxy in the diagram, the ES proxy layer. Component B is the proxy layer of the main system, used to monitor sub-clusters using basic heartbeat detection and fast circuit breaker determination strategies. When a sub-cluster is detected as unavailable, it automatically removes the unavailable sub-cluster from the query route and reconnects it to the unavailable sub-cluster after the main system recovers through a hot reloading mechanism.

[0027] The alarm aggregation module is used to collect alarms from the distributed message queue module in the main system (corresponding to...). Figure 1 The system retrieves the original alarms from all subsystems in the Kafka module, generates the final alarms according to the alarm merging rules, and stores the final alarms in the main cluster.

[0028] In addition, both the main system and the subsystems include: The data collector is used to collect node logs and send them to the correlation analysis module.

[0029] Association analysis module (corresponding) Figure 1 The correlation analysis engine in the module is used to generate original alarms according to preset rules and send them to the upper-level distributed message queue module.

[0030] Figure 1 In Elasticsearch, CCS stands for Cross-Cluster Search. Cross-cluster search is a feature that allows users to query and retrieve data from one or more remote clusters within a single cluster. To use CCS, one or more remote clusters must first be configured in the local cluster. Each remote cluster has a name and a list of seed nodes used to discover nodes in the remote cluster that are eligible to act as gateway nodes. Nodes performing CCS do not join the remote cluster; instead, they connect to the remote cluster in a lightweight manner to execute federated search requests.

[0031] Working principle: Configuring a remote cluster: A remote cluster can be configured within the local cluster using the `cluster.remote` setting. For example, add the following configuration to the `elasticsearch.yml` file: cluster.remote.cluster_one.seeds:["host1:9300", "host2:9300"] Here, `cluster_one` is an alias for the remote cluster, and the `seeds` list specifies the addresses of some nodes in the remote cluster used for initial connection. After configuring the remote cluster, you can use the standard Elasticsearch query API to perform cross-cluster searches. When querying, specify the cluster alias to search; the local cluster will forward the query to the specified remote cluster and retrieve the results from it. If the query involves multiple clusters, the local cluster will be responsible for aggregating all results and returning them to the client.

[0032] Figure 1 HA (High Availability) in Elasticsearch means that, through the HA mechanism, the Elasticsearch cluster can automatically switch over when a node fails, thereby ensuring service continuity.

[0033] also, Figure 1 Other submodules (such as MySQL, Redis, HA Agent, Supervisor, MetricBeat, Monitor, etc.) play a supporting role in the system. They are responsible for tasks such as data storage, caching, high availability management, process monitoring, alerting, metric collection, and system monitoring, which will not be elaborated on here.

[0034] The system also includes a security event engine and a scheduling engine. The security event engine is used to detect and respond to security events, and the scheduling engine is used to schedule and execute tasks.

[0035] Specifically, the original AB component used a typical serial query mechanism. To ensure query efficiency and accuracy after data integration, we logically adjusted the query relationship between the AB components. Component A integrates serial priority query, adaptive timeout control, and resource reclamation mechanisms. When executing query tasks, it establishes a serial link according to the priority principle of the upper-level cluster. When the upper-level cluster responds with a timeout (default 60 seconds, supports dynamic configuration) or returns an exception, it automatically triggers the switching of the lower-level cluster and releases the occupied memory resources to avoid system-level blocking caused by single point of failure. At the same time, it monitors cluster topology changes in real time through an embedded state awareness engine to achieve dynamic isolation of faulty nodes and elastic expansion of healthy nodes. Component B, as the ES proxy layer, uses a dual-cycle detection strategy to perform health monitoring on the lower-level cluster, including basic heartbeat detection (60 seconds / time) and fast circuit breaker judgment (10-second timeout threshold). When a cluster is detected as unavailable, it is automatically removed from the query route, and a hot-reload mechanism enables seamless reconnection after the cluster recovers. Meanwhile, in the multi-level deployment architecture, each lower-level node deploys a data collector and a correlation analysis module. The data collector collects the node's logs and sends them to the correlation analysis module. The correlation analysis module generates raw alerts according to the configured rules. All lower-level nodes send the raw alerts to the upper-level Kafka. The upper-level alert aggregation module retrieves all the raw alerts from Kafka, generates threat alerts according to the alert merging rules, and stores them uniformly in the upper-level Elasticsearch storage. Finally, the upper-level system completes the unified query and handling of threat alerts.

[0036] This solution breaks through the passive response mode of traditional cluster failover. Through multi-dimensional anomaly perception, tiered circuit breaker strategy and real-time resource reclamation mechanism, it effectively improves the availability of the system in complex query scenarios while ensuring cross-cluster data consistency, and successfully avoids the avalanche effect caused by cascading failures.

[0037] Component A queries the cluster in the following ways: Step A1: When a client initiates an aggregation query request, the main cluster and sub-cluster transmit the aggregation query request to the cluster shards. Each cluster shard creates an aggregation query request object, which initializes the aggregator build factory by parsing the request parameters.

[0038] Step A2: Create an aggregator through the aggregator factory. The aggregator is used to collect data.

[0039] Step A3: Execute the HLL++ algorithm on each shard using the aggregator to calculate and store the HLL++ hash value.

[0040] Step A4: After each shard is calculated, the HLL++ hash values ​​of all shards are merged through the coordinating node to obtain the cardinality estimation result, which is then returned to the client. Steps A1 to A4 correspond in sequence. Figure 2 The four modules.

[0041] Specifically, multiple Elasticsearch shards (Shard 1, Shard 2, ..., Shard N) each receive aggregation query requests from the coordinating node. When a user initiates an aggregation query request, Elasticsearch pushes this request down to all shards. Each shard creates an aggregation query request object, CopyCardinalityAggregationBuilder, to process the data on that shard. CopyCardinalityAggregationBuilder parses the request parameters, including parameters 9200 and 9300. The aggregator building factory, CopyCardinalityAggregatorFactory, is then initialized.

[0042] Within each shard, data is divided into buckets (Bucket 1, Bucket 2, ..., Bucket M), and each bucket corresponds to an HLL++ hash value calculation process. On each shard, the CopyCardinalityAggregatorFactory creates a concrete aggregator, CopyCardinalityAggregator. The CopyCardinalityAggregator is responsible for collecting data and calculating the HLL++ hash value for each bucket. HLL++ is an improved version of the HLL algorithm, used for more accurate cardinality estimation. The calculated hash values ​​are stored in hilArray, which is an ObjectArray. <hyperloglogplus>An array of type [type]. In the Coordinating Node section, the results of all shards are aggregated and merged using the HLL++ algorithm to obtain the final cardinality estimate. After each shard is computed, the Coordinating Node collects the results of all shards. The Coordinating Node uses a CopyInternalCardinality object to merge the HLL++ hash values ​​of all shards. CopyInternalCardinality merges the intermediate aggregation results of all shards using the doReduce method to obtain the final cardinality estimate. The final result is converted to JSON format using the doXContentBody method and sent to the user.

[0043] In the multi-cluster management architecture design of a unified operation platform, cross-cluster retrieval is implemented based on an intelligent sharding routing mechanism between the upper-level coordination point and lower-level cluster nodes to achieve multi-level data retrieval. By default, the entire list of lower-level cluster nodes needs to be configured in the upper-level cluster. In development practice, this can be optimized so that the main ES cluster node of the upper-level SOC platform only needs to establish a dedicated reliable TCP connection with the specific coordination node 9300 port of each lower-level sub-ES cluster. By leveraging the Cross-Cluster Search (CCS) feature, a virtual data view can be constructed by configuring cluster.remote on the upper-level main cluster.<cluster_alias> The `.seeds` directive points to the seed nodes of the sub-cluster. Combined with access control via `indices:data / read / search` and a seed node liveness detection mechanism (using a sniffer), cross-cluster joint queries are achieved. This solution overcomes the limitations of traditional fully interconnected architectures, reducing the port exposure range from N×(N-1) to linear growth. Simultaneously, it controls query routing through the `preference` parameter, significantly reducing network topology complexity while ensuring retrieval performance.

[0044] Furthermore, based on the HLL algorithm, and with unified log query as the core, the cascading deployment architecture was adjusted to consist of three main modules: The global metadata management hub uses a distributed key-value storage engine to achieve full data source topology management and dynamically maintain key metadata information including node status, index mapping, and permission policies.

[0045] The intelligent routing coordination layer, based on an adaptive load balancing algorithm, supports intelligent request distribution and result aggregation.

[0046] Elastic Data Service Cluster (EDS) constructs a heterogeneous data storage system by hybrid deployment of self-built clusters and Elasticsearch clusters.

[0047] Especially in cross-cluster retrieval, the system has deeply optimized the native Elasticsearch mechanism and innovatively implemented a four-level processing flow. The specific optimization logic is as follows: When a user initiates a federated query, the local coordinating node parses the request topology through the metadata repository and distributes the standardized query instructions to each associated cluster in parallel. During the independent execution of retrieval by each cluster, query optimization can be performed based on the cluster's replication strategy, sharding rules, and other configurations. The result return phase adopts a streaming processing mechanism, which dynamically adjusts the data transmission sequence through a priority queue; Finally, the coordinating node performs intelligent deduplication, relevance sorting and other post-processing to generate a unified result set.

[0048] Compared to traditional hierarchical systems, this architecture has three major breakthroughs: 1) The independently developed anomaly isolation mechanism uses heartbeat detection and asynchronous retry strategies to automatically trigger backup index switching when a timeout is detected in the lower-level cluster (threshold is configurable), ensuring the continuity of query services; 2) The system availability reaches over 99.9%, meeting stability requirements; 3) It supports dynamic expansion and allows multiple lower-level nodes to access the cluster.< / hyperloglogplus>

Claims

1. A multi-level cross-cluster system for a SOC operation platform, the multi-level cross-cluster system for the SOC operation platform comprising a main system and at least one subsystem, the main system comprising a main cluster, and the subsystem comprising a sub-cluster, characterized in that, The main cluster and the sub-cluster are connected through a preset coordination node. The main system includes: The reverse proxy module is used as a reverse proxy to receive and dispatch client query requests; The API gateway module is used to provide API interfaces, handle business logic, and schedule requests. Component A is the DAL query engine, which connects to the proxy layer of the subsystem. It is used to prioritize querying the main cluster based on the query requests transmitted by the reverse proxy module and the API gateway module. When the main cluster response time exceeds the preset response time threshold or returns an exception, Component A automatically triggers the sub-cluster query by triggering the proxy layer of the subsystem and releases the occupied memory resources. Component B is the ES proxy layer of the main system. It is used to monitor sub-clusters using basic heartbeat detection and fast circuit breaker determination strategies. When a sub-cluster is detected to be unavailable, it will automatically remove the unavailable sub-cluster from the query route and reconnect the unavailable sub-cluster after the cluster recovers through a hot reloading mechanism. The alarm aggregation module is used to obtain the original alarms from all subsystems from the distributed message queue module in the main system, generate the final alarm according to the alarm merging rules, and store the final alarm in the main cluster.

2. The multi-level cross-cluster system for SOC operation platform according to claim 1, characterized in that, Both the main system and the subsystem include: The data collector is used to collect node logs and send them to the correlation analysis module; The correlation analysis module is used to generate original alarms according to preset rules and send them to the upper-level distributed message queue module.

3. The multi-level cross-cluster system for SOC operation platform according to claim 1, characterized in that, The preset coordination node is port 9300.

4. The multi-level cross-cluster system for SOC operation platform according to claim 1, characterized in that, Component A queries the cluster in the following ways: Step A1: When a client initiates an aggregation query request, the main cluster and sub-cluster transmit the aggregation query request to the cluster shards. Each cluster shard creates an aggregation query request object, which initializes the aggregator build factory by parsing the request parameters. Step A2: Create an aggregator through an aggregator factory. The aggregator is used to collect data. Step A3: Execute the HLL++ algorithm on each shard using the aggregator to calculate and store the HLL++ hash value; Step A4: After each shard is calculated, the HLL++ hash values ​​of all shards are merged through the coordinating node to obtain the cardinality estimation result, and the cardinality estimation result is returned to the client.

5. The multi-level cross-cluster system for SOC operation platform according to claim 4, characterized in that, Step A1 includes: the request parameters include 9200 request parameters and 9300 request parameters.

6. The multi-level cross-cluster system for SOC operation platform according to claim 1, characterized in that, Component A, Component B, and the data collector constitute a data center. The reverse proxy module, API gateway module, and correlation analysis module constitute an integrated brain platform. The multi-level cross-cluster system for the SOC operation platform is configured in the following way: B1: Open ports to the integrated brain platform through the data center, configure source address, destination address and port for data access and inter-module communication; B2: For core functional modules, modify configuration files and build images, including: uploading and decompressing upgrade packages, modifying configuration files of corresponding modules, cloning and modifying configuration files according to the IP address of the data center to be connected, building images and publishing configurations on the Nacos page, switching to the server in the corresponding data center to restart the module in the background, and checking logs to confirm normal startup; B3: Log in to the server of the integrated brain platform and register the ES cluster of the data center to the integrated brain platform; B4: Integrate the SAE core clusters of each data center into the unified platform management, edit the SAE monitoring configuration file, and add the service names of the SAE core clusters to be discovered; B5: Update the SAE monitor by replacing the class files and restarting the service to view an overview of the analytics engines in each data center.

7. The multi-level cross-cluster system for SOC operation platform according to claim 6, characterized in that, The B1 includes: opening ports 9111, 9015, 7777, 8765, 9200 and 9300 to the integrated brain platform through the data center.

8. The multi-level cross-cluster system for SOC operation platform according to claim 1, characterized in that, The multi-level cross-cluster system for the SOC operation platform also includes a security event engine and a scheduling engine. The security event engine is used for the detection and response to security events, and the scheduling engine is used for the scheduling and execution of tasks.

Images