Large-scale storage and computing resource dynamic recombination system, method, equipment and medium

Abstract

The invention relates to the technical field of cloud computing resource management, and discloses a large-scale storage and computing resource dynamic recombination system, method, equipment and medium, which comprises a global resource sensing module, real-time convergence of computing and storage resource states of a main data center, a standby data center and an edge node and real-time reorganization of the computing and storage resource states of the main data center, the standby data center and the edge node. Constructing a global resource topology containing a physical position and a network delay attribute; a calculation mode is adaptively selected according to the cost estimation of the task by using a storage and calculation collaborative scheduling module; through an ETL pipeline based on a directed acyclic graph and a security policy injection mechanism based on an abstract syntax tree, standardized access and endogenous security protection of multi-source heterogeneous data are realized, and zero data loss of key services is guaranteed in combination with a cross-data-center synchronous double-writing technology. According to the method, the limitation of physical regions on computing power is effectively broken, elastic flow and peak load shifting of resources of the whole network are realized, and the query analysis performance of large-scale security data and the utilization rate of system resources are improved.

Description

Technical Field

[0001] This invention relates to the field of cloud computing resource management technology, and in particular to a large-scale storage and computing resource dynamic reorganization system, method, device and medium. Background Technology

[0002] With the acceleration of digital transformation, cybersecurity threats are becoming more sophisticated, covert, and automated. To address these challenges, Network Security Situational Awareness (NSSA) systems have emerged. The core of NSSA lies in collecting, processing, analyzing, and visualizing massive amounts of data within the network from a macroscopic perspective, enabling the "perception, understanding, and prediction" of security threats.

[0003] However, existing NSSA systems have the following problems when dealing with large-scale data: Data silos and fragmentation: In large enterprises or national networks, data is often scattered across different geographical locations (such as provincial branches), different network areas (such as office networks and production networks), and different storage media. Traditional centralized big data platforms attempt to aggregate all data into a single center, but when faced with tens of TB or even PB of full traffic logs daily, wide area network bandwidth becomes an insurmountable bottleneck. Furthermore, the data formats (Syslog, JSON, Binary) output by security devices from different vendors (firewalls, IDS, WAF) vary significantly, making cross-domain correlation analysis extremely difficult.

[0004] Rigidity and Utilization Imbalance of Storage and Computing Resources: Traditional Hadoop or MPP (Massively Parallel Processing) database architectures often adopt a "memory-compute integration" model, where compute nodes and storage nodes are physically bound together. In this architecture, when storage capacity needs to be increased, compute nodes must be added simultaneously, and vice versa. In cybersecurity scenarios, compliant storage of historical data (cold data) requires massive disk space but low computational demands; however, during real-time analysis of sudden attacks (such as DDoS), computational resource demands surge while storage I / O pressure is relatively low. The memory-compute integration architecture cannot flexibly handle these dynamically changing loads, leading to significant resource idleness and waste. Summary of the Invention

[0005] In view of the aforementioned existing problems, the present invention provides a system, method, device and medium for dynamic reorganization of large-scale storage and computing resources.

[0006] Therefore, the technical problem solved by this invention is that edge nodes are typically resource-constrained and heterogeneous (using a mix of ARM and x86). Existing scheduling strategies for task scheduling between edge environments and resource-rich cloud centers are mostly static, lacking a "reorganization" mechanism that can dynamically incorporate edge computing power into the global resource pool based on real-time network status and node load.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a large-scale storage and computing resource dynamic reorganization system, comprising: a resource awareness module, a storage-computing collaborative scheduling module, a data processing module, and a disaster recovery collaboration module; The resource awareness module is used to collect the computing and storage resource status of the main data center, backup data center and edge nodes in real time through a multi-level heartbeat mechanism, and to construct a global resource topology that includes physical location and network latency attributes. The data processing module is used to achieve standardized access to multi-source data and transparent injection of security policies by utilizing an ETL processing pipeline based on a directed acyclic graph and an SQL parser and rewriter based on an abstract syntax tree. The storage-computing collaborative scheduling module is connected to the data processing module and is used to dynamically select the computing mode based on the cost estimation of the query task. The disaster recovery coordination module is used to maintain strong data consistency between the primary and backup data centers and to perform metadata switching in the event of a failure.

[0008] As a preferred embodiment of the large-scale storage and computing resource dynamic reorganization system described in this invention, the multi-level heartbeat mechanism includes: edge computing nodes and physical nodes inside the data center sending heartbeat packets to the local control center of their respective regions at a first frequency to report real-time load information; The local control center aggregates the received node status within the region to generate a local resource view containing regional resource fingerprints and idle resource summaries; The global control center pulls resource summaries from each local control center at a second frequency lower than the first frequency, or receives state change pushes from local controllers, and constructs a global resource topology.

[0009] As a preferred embodiment of the large-scale storage and computing resource dynamic reorganization system described in this invention, the SQL of the abstract syntax tree includes intercepting the user's abstract syntax tree before generating the physical query plan; Based on preset permission rules, filtering predicates or de-identification functions are automatically injected into the abstract syntax tree to achieve transparent injection of security policies.

[0010] As a preferred embodiment of the large-scale storage and computing resource dynamic reorganization system described in this invention, the step of dynamically selecting the computing mode based on the cost estimation of the query task includes: The virtual cluster scheduling mode responds to highly complex cross-domain correlation analysis tasks by dynamically selecting idle computing nodes across regions and logically reorganizing them into a virtual cluster, and remotely accessing off-site storage shards through a virtual file system. The direct forwarding scheduling mode responds to local aggregation tasks with high data volumes by distributing query plans, including predicate pushdown, to the edge nodes where the data resides for local execution, and only aggregating statistical results.

[0011] As a preferred embodiment of the large-scale storage and computing resource dynamic reorganization system described in this invention, the directed acyclic graph ETL processing pipeline includes abstracting the data standardization process into a directed acyclic graph, which consists of data source nodes, transformation operator nodes, output nodes, and flow edges connecting each node. Secure data is accessed at the data source node via Kafka message queues, TCP / UDP socket streams, Syslog log streams, and HTTP / HTTPS protocols; Configure parsing operators, transformation operators, and enrichment operators in the transformation operator node to perform JSON / regular expression / key-value pair parsing, data type conversion, time formatting, IP address normalization, and data enrichment based on the threat intelligence database; Perform streaming logic control, including data splitting, filtering, and field mapping, to uniformly map raw log fields to entities and attributes of a standard security data model.

[0012] As a preferred embodiment of the large-scale storage and computing resource dynamic reorganization system described in this invention, the step of maintaining strong data consistency between the primary and backup data centers includes writing data to the storage layers of the primary and backup data centers in parallel when a write transaction is received. A two-phase commit protocol is used, and transactions are committed only after successful confirmations are received from all target storage nodes across availability zones.

[0013] As a preferred embodiment of the large-scale storage and computing resource dynamic reorganization system described in this invention, the construction of a global resource topology including physical location and network latency attributes includes defining a unified resource descriptor, including computing attributes, storage attributes and network attributes. When an edge node starts up, the agent process automatically scans local hardware information and sends a registration request to the local control center of its region. The global control center divides resources into different logical groups based on business needs, including a default shared pool, a dedicated protection pool, and a tidal resource pool.

[0014] This invention provides a method for dynamic reorganization of large-scale storage and computing resources.

[0015] As a preferred embodiment of the large-scale storage and computing resource dynamic reorganization method described in this invention, it includes: collecting the computing and storage resource status of the main data center, backup data center and edge nodes in real time through a resource awareness module, and constructing a global resource topology that includes physical location and network latency attributes; Through the data processing module, a directed acyclic graph-based ETL processing pipeline is used to achieve standardized access to multi-source heterogeneous data, and an abstract syntax tree-based SQL parser and rewriter is used to achieve transparent injection of security policies. The in-memory computing collaborative scheduling module dynamically selects the computing mode based on the cost estimate of the query task and executes the query task. The disaster recovery and collaboration module maintains strong data consistency between primary and backup data centers and performs metadata switching in the event of a failure.

[0016] The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of a large-scale storage and computing resource dynamic reorganization system.

[0017] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a large-scale storage and computing resource dynamic reorganization system.

[0018] The beneficial effects of this invention are as follows: By using global resource awareness and virtual cluster technology, computing resources that are physically dispersed across provincial companies or edge nodes are logically reorganized into a unified resource pool. The system can dynamically select idle edge computing power to participate in large-scale analysis tasks at the center based on the task load, thus achieving "peak shaving and valley filling" of computing power.

[0019] A dual-mode adaptive scheduling based on cost estimation is proposed. For high-bandwidth aggregation tasks, it automatically switches to direct forwarding mode; for complex association analysis, it automatically switches to virtual cluster mode. This dynamic optimization mechanism can minimize network bandwidth consumption while ensuring query response speed.

[0020] By employing a cross-datacenter synchronous dual-write mechanism, every transaction is ensured to be written simultaneously to both the primary and backup centers. Furthermore, combined with a read-write separation strategy, the backup center handles read traffic, activating idle resources and enhancing the overall system's concurrency capabilities. In the event of a disaster, rapid metadata switching ensures quick business recovery. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the working modules of a large-scale storage and computing resource dynamic reorganization system provided in one embodiment of the present invention. Detailed Implementation

[0023] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0024] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a large-scale storage and computing resource dynamic reorganization system, including: a resource awareness module, a storage-computing collaborative scheduling module, a data processing module, and a disaster recovery collaboration module; The resource awareness module is used to collect the computing and storage resource status of the main data center, backup data center and edge nodes in real time through a multi-level heartbeat mechanism, and to construct a global resource topology that includes physical location and network latency attributes. The data processing module is used to achieve standardized access to multi-source data and transparent injection of security policies by utilizing an ETL processing pipeline based on a directed acyclic graph and an SQL parser and rewriter based on an abstract syntax tree. The storage-computing collaborative scheduling module is connected to the data processing module and is used to dynamically select the computing mode based on the cost estimate of the query task; The disaster recovery and coordination module is used to maintain strong data consistency between primary and backup data centers and perform metadata switching in the event of a failure.

[0025] It should be noted that the system adopts a "two-site, three-center" physical node deployment approach. At the physical level, the primary data center undertakes core computing, storage, and index building tasks; the off-site backup data center is in hot standby or dual-active state, capable of sharing read traffic and preparing to take over services; edge nodes distributed across various locations are responsible for real-time data collection and preliminary processing. Logically, the system adopts a front-end / back-end separation design, achieving a combination of centralized management and distributed scheduling through a global and local two-level control center. The global control center is responsible for metadata management, resource view construction, and cross-domain query planning; the local control center is responsible for resource registration, task scheduling, and status monitoring within its region; and the back-end computing and storage nodes focus on data shard storage and the execution of specific operators. This design decouples the control plane from the data plane, enabling the system to balance the uniformity of centralized management with the scalability of distributed execution, thereby achieving the goals of high availability, high performance, and elastic resource scheduling.

[0026] Example 2 is an embodiment of the present invention. Based on the above embodiment, a large-scale storage and computing resource dynamic reorganization system is provided.

[0027] Furthermore, in this embodiment, the resource awareness module is used to collect the computing and storage resource status of the main data center, backup data center, and edge nodes in real time through a multi-level heartbeat mechanism, and to construct a global resource topology that includes physical location and network latency attributes. Specific steps include: For physically discrete resources deployed in a distributed manner, this embodiment describes a method for abstracting them into a logically schedulable resource pool, specifically including: Define a set of uniform resource descriptors, including: Computational attributes: CPU architecture (x86 / ARM), number of cores, clock speed, and AVX instruction set support.

[0028] Storage attributes: capacity, media type (SSD / HDD), I / O throughput, local data shard list.

[0029] Network attributes: Region, IP address, round-trip delay to the main center.

[0030] When an edge node starts up, its agent process automatically scans local hardware information and sends a registration request to the local control center of its region.

[0031] State aggregation is achieved through a multi-level heartbeat mechanism: The first level involves physical nodes sending lightweight heartbeats to the local control center at a high frequency (e.g., once per second) to report real-time load (CPU load, memory load).

[0032] The second level involves the local control center aggregating resources within the region and generating a resource summary. For example, instead of reporting the status of each machine, it reports "Region A: 500 idle compute cores, 100TB of storage remaining".

[0033] The third level is the global control center, which maintains a network-wide resource status table. This table not only records the static attributes of resources but also dynamically updates their "availability scores." For example, when the network latency jitter in a certain province exceeds a threshold, the priority of nodes in that province participating in global computation is automatically reduced.

[0034] In an optional embodiment, the generated resource summary may also periodically (e.g., every 30 seconds) batch summarize the heartbeat data received during the period, calculating and reporting only the average or peak value for that time period.

[0035] In another optional embodiment, the generated resource summary can also continuously receive heartbeats, reporting to the global control center only when there is a significant change in the aggregation status of resources within the region. When the change exceeds a threshold, or when a node joins / leaves the resource pool, the latest resource summary is reported.

[0036] The global control center divides resources into different logical groups based on business needs: Default shared pool: Contains all available resources for temporary querying.

[0037] Dedicated protection pool: A specific group of high-performance nodes is marked as "high protection" and is dedicated to real-time threat detection services to ensure that they are not interfered with by other tasks.

[0038] Tidal Resource Pool: Automatically identifies idle edge office PCs or low-load servers at night and includes them in the "Tidal Pool" for handling large-scale offline reporting tasks that are not real-time.

[0039] Furthermore, in this embodiment, the data processing module utilizes an ETL processing pipeline based on a directed acyclic graph and an SQL parser / rewriter based on an abstract syntax tree to achieve standardized access to multi-source data and transparent injection of security policies. Specific steps include: To address the need for multi-source data access from different sources, a standardized processing method based on a directed acyclic graph (DAG) ETL pipeline is adopted, specifically including: The pipeline pattern is used to abstract the data standardization process into a directed acyclic graph (DAG), which consists of data source nodes, transformation operator nodes, output nodes, and flow edges connecting the nodes. The data source nodes are compatible with a variety of network security protocols, including high-throughput Kafka message queues, real-time TCP / UDP socket streams, standard Syslog log streams, and HTTP / HTTPS requests.

[0040] The transformation operator node is configured with parsing operators, transformation operators, and enrichment operators. It supports JSON structured parsing, regular expression extraction, key-value pair parsing, and delimiter splitting. It can flexibly handle unstructured and semi-structured logs and provides a variety of data processing functions, including: Type conversion: string to integer / floating-point type, etc.

[0041] Time processing: Standardized timestamp format.

[0042] Network attribute processing: IP address formatting (IPv4 / IPv6 conversion, mask calculation).

[0043] Logical processing: conditional statements (If-Else), string encoding conversion, etc.

[0044] The output nodes convert the processed data into an efficient columnar storage format (such as Parquet or Orc) and write it to the storage cluster.

[0045] To eliminate semantic ambiguity in data caused by different devices, a standard data model is established. This model includes unified entities, attributes, relationships, and constraints. Regardless of the device from which the source data originates, key fields are mapped to standard fields after passing through the ETL pipeline. For example, the source IP field for all devices is mapped to `src_ip`, and the attack time is mapped to `event_time`. This ensures that upper-layer analysis units do not need to be aware of the differences in the underlying devices, achieving data standardization and consistency.

[0046] In an optional embodiment, standardization can also be performed by pre-defining a separate field mapping configuration file for each type or each data source device. When the ETL pipeline processes data, it looks up the corresponding configuration file based on the source identifier of the data packet (such as device type or IP), and maps the source field names to intermediate field names according to that file. When the upper-layer application queries, it still needs to know the device type corresponding to the data being queried and its specific set of intermediate field names.

[0047] In another alternative embodiment, standardization can also be performed by storing the data in the storage layer according to its original field names and structure, but attaching an explicit "data source type" label to each record. During querying, by creating "views" for different data source types or using a unified query function (UDF), the differentiated field names are converted into logical field names expected by the business layer at the SQL level, achieving semantic consistency during queries.

[0048] Transparent injection of security policies is achieved by using an SQL parser rewriter based on an abstract syntax tree (AST). Apache Calcite or similar technologies are used to transform the SQL statements of the user's query plan into an AST.

[0049] Before generating the execution plan, the AST is processed in depth, and security filtering conditions (such as AND department_id = 'user_dept') are automatically added to the WHERE clause of the AST according to user permissions to achieve data isolation.

[0050] Alarm records marked as "whitelisted" or "false alarms" are automatically filtered out at the AST level; for sensitive fields (such as mobile phone numbers and ID cards), desensitization functions (such as mask(phone)) are automatically wrapped; the rewritten AST is sent to the optimizer, which combines the scheduling mode to generate a distributed physical plan and distributes it to each computing node for execution.

[0051] Furthermore, in this embodiment, the in-memory computing collaborative scheduling module is connected to the data processing module and is used to dynamically select the computing mode based on the cost estimate of the query task. Specific steps include: For the processed query plan, the storage-computing collaboration module dynamically selects the execution mode based on the estimated query cost. The execution modes include virtual cluster scheduling mode and direct forwarding scheduling mode, where: Virtual cluster mode is a global scheduling strategy for complex analysis scenarios. In this mode, the global control center logically maps physically distributed backend nodes managed by local control centers in various provinces into a unified virtual resource pool through VFS (Virtual File System) or a unified namespace. When users initiate complex tasks such as cross-provincial joint queries or network-wide data aggregation, the global control center generates a distributed execution plan and directly schedules the backend nodes in the relevant provinces. The system performs cross-regional data exchange and parallel computing. This mode breaks down physical geographical limitations, achieves complete resource pooling, provides extremely high computational flexibility, and can support the most complex correlation analysis tasks.

[0052] However, due to the data transmission across wide area networks, network overhead is significant, query latency is relatively high, and high network bandwidth is required. It is suitable for scenarios with high requirements for data integrity and relevance, such as cross-provincial business analysis, network-wide data aggregation, and complex report generation. For example, if a user needs to query "all hosts that have accessed malicious IP X in the past 24 hours," it involves cross-provincial data aggregation. The global control center receives a query request and analyzes it to find that it involves data from the entire network. Query cost estimation indicates that the main center's computing power is insufficient, but edge nodes in provinces A and B have a large amount of idle CPU. The global control center dynamically binds some nodes from the main center with idle nodes in provinces A and B, forming a temporary "virtual computing cluster." The scheduler divides the query plan into multiple shards. The shards allocated to nodes in province A may need to process data stored in province C. Nodes in province A remotely read the data shards from province C via the backbone network through a virtual file system or object storage interface. To reduce network overhead, the system prioritizes scheduling nodes in province A to process province A data, only performing remote reading when local resources are insufficient. Intermediate results are computed in parallel by each node and finally converged to the main center for return.

[0053] In an optional embodiment, the joint operation of cross-provincial data can also be further optimized after forming a virtual cluster. When the scheduler allocates task shards, it no longer prioritizes data locality. Instead, it allocates shards entirely based on the real-time computing resource availability (such as the number of idle CPU cores) of each provincial node, assigning task shards to the most idle node. Regardless of its physical location, this node uniformly reads the data shards it needs to process remotely through the virtual file system.

[0054] In another alternative embodiment, the joint operation of cross-provincial data can also be pre-established and fixed during the logical partitioning phase of the virtual cluster, establishing and fixing the binding relationship between computing nodes and data storage regions. For example, it can be explicitly specified that nodes from province A in the virtual cluster are only assigned to process data shards from province A, and nodes from province B are only assigned to process data shards from province B.

[0055] Direct forwarding is an edge computing strategy for localized data. In this mode, when a user sends a query request to the global control center, the center first parses the SQL statement to identify the geographical location of the data. If it finds that the data required for the query is entirely located in a specific region, the global control center does not perform complex global planning but forwards the SQL request unchanged to the local control center in the corresponding region. The local control center in that region completes all resource scheduling and computing tasks within its local cluster, returning only a very small final result set to the global control center, which then aggregates and returns it to the user. This mode maximizes the utilization of edge computing capabilities, achieving "data not leaving the domain," and significantly reducing WAN bandwidth consumption and response latency.

[0056] However, this mode cannot utilize idle computing power in remote locations and is limited to processing queries with strong data locality, not supporting cross-domain correlation analysis. It is suitable for scenarios with extremely high real-time requirements and clearly defined data ranges, such as local report queries, real-time monitoring dashboards, and simple point queries. For example, if headquarters needs to calculate "the number of attacks intercepted in each province today," the data volume is large but the calculation logic is simple, and the data for each province is independent: The global control center parses the SQL and identifies it as a type of aggregate query, with the data partitioned by province.

[0057] Instead of pulling raw logs, the scheduler pushes subqueries containing filtering conditions (Where) and aggregation functions (Group By) directly down to the local control centers of each province. The local control centers of each province use their local computing power to execute the queries on their local storage. Each province only returns a very small statistical result (such as a single line of numbers), and the global control center performs the final summation operation.

[0058] Furthermore, in this embodiment, the disaster recovery coordination module is used to maintain strong data consistency between the primary and backup data centers and perform metadata switching in the event of a failure. Specific steps include: At the data synchronization level, the system employs a synchronous dual-write mechanism based on "two-phase commit," providing a robust foundation for data consistency in disaster recovery. When a client initiates a write request, the primary data center control center distributes it in parallel to the backend storage clusters of both the primary and backup data centers. Subsequently, the system executes a rigorous two-phase commit protocol: during the preparation phase, data is written to the write-ahead logs of each center; only after receiving successful confirmation from both centers does the control center commit the transaction and return success to the client. This mechanism fundamentally guarantees real-time strong consistency of the underlying data in the primary and backup data centers, paving the way for subsequent rapid failover.

[0059] The system achieves efficient read / write separation and second-level failover at the business access layer. During daily operation, the backup data center is not idle; its computing resources are fully utilized to handle read traffic. The global control center intelligently distributes query requests to the fastest-responding center based on real-time load, thereby improving overall performance and resource utilization. When the primary center fails, since the data is already synchronized and ready in the backup center, the failover process mainly involves the rapid reconstruction of the master-slave metadata relationship, eliminating the need for large-scale data migration. This allows the entire business recovery process to be controlled within seconds, significantly reducing service interruption time.

[0060] In an optional embodiment, intelligent query request distribution can also employ a statically configured priority order to distribute read requests. All read requests are always prioritized and sent to the primary data center; requests are only forwarded to the backup data center if the primary data center is unreachable or explicitly overloaded.

[0061] In another optional embodiment, intelligent distribution of query requests can also be achieved by using a simple polling method to alternately distribute the received read requests to the primary data center and the backup data center.

[0062] Example 3 is the third embodiment of the present invention, which differs from the previous two embodiments in that: This embodiment also provides a method for dynamic reorganization of large-scale storage and computing resources, including: collecting the computing and storage resource status of the main data center, backup data center and edge nodes in real time through a resource awareness module, and constructing a global resource topology that includes physical location and network latency attributes; Through the data processing module, a directed acyclic graph-based ETL processing pipeline is used to achieve standardized access to multi-source heterogeneous data, and an abstract syntax tree-based SQL parser and rewriter is used to achieve transparent injection of security policies. The in-memory computing collaborative scheduling module dynamically selects the computing mode based on the cost estimate of the query task and executes the query task. The disaster recovery and collaboration module maintains strong data consistency between primary and backup data centers and performs metadata switching in the event of a failure.

[0063] This embodiment also provides an electronic device, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, thereby realizing a large-scale storage and computing resource dynamic reorganization system as proposed in the above embodiment.

[0064] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements a large-scale storage and computing resource dynamic reorganization system as proposed in the above embodiments.

[0065] The storage medium proposed in this embodiment and the system for dynamic reorganization of large-scale storage and computing resources proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0066] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0067] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A large-scale storage and computing resource dynamic reorganization system, characterized in that: It includes a resource awareness module, a storage-computing collaborative scheduling module, a data processing module, and a disaster recovery collaboration module; The resource awareness module is used to collect the computing and storage resource status of the main data center, backup data center and edge nodes in real time through a multi-level heartbeat mechanism, and to construct a global resource topology that includes physical location and network latency attributes. The data processing module is used to achieve standardized access to multi-source data and transparent injection of security policies by utilizing an ETL processing pipeline based on a directed acyclic graph and an SQL parser and rewriter based on an abstract syntax tree. The storage-computing collaborative scheduling module is connected to the data processing module and is used to dynamically select the computing mode based on the cost estimation of the query task. The disaster recovery coordination module is used to maintain strong data consistency between the primary and backup data centers and to perform metadata switching in the event of a failure.

2. The large-scale storage and computing resource dynamic reorganization system as described in claim 1, characterized in that: The multi-level heartbeat mechanism includes edge computing nodes and physical nodes inside the data center sending heartbeat packets to the local control center of their respective regions at a first frequency and reporting real-time load information. The local control center aggregates the received node status within the region to generate a local resource view containing regional resource fingerprints and idle resource summaries; The global control center pulls resource summaries from each local control center at a second frequency lower than the first frequency, or receives state change pushes from local controllers, and constructs a global resource topology.

3. The large-scale storage and computing resource dynamic reorganization system as described in claim 2, characterized in that: The SQL of the abstract syntax tree includes intercepting the user's abstract syntax tree before generating the physical query plan; Based on preset permission rules, filtering predicates or de-identification functions are automatically injected into the abstract syntax tree to achieve transparent injection of security policies.

4. The large-scale storage and computing resource dynamic reorganization system as described in claim 3, characterized in that: The dynamic selection of the calculation mode based on the cost estimation of the query task includes, The virtual cluster scheduling mode responds to highly complex cross-domain correlation analysis tasks by dynamically selecting idle computing nodes across regions and logically reorganizing them into a virtual cluster, and remotely accessing off-site storage shards through a virtual file system. The direct forwarding scheduling mode responds to local aggregation tasks with high data volumes by distributing query plans, including predicate pushdown, to the edge nodes where the data resides for local execution, and only aggregating statistical results.

5. A large-scale storage and computing resource dynamic reorganization system as described in claim 4, characterized in that: The ETL processing pipeline of the directed acyclic graph includes abstracting the data standardization process into a directed acyclic graph, which consists of data source nodes, transformation operator nodes, output nodes, and flow edges connecting each node. Secure data is accessed at the data source node via Kafka message queues, TCP / UDP socket streams, Syslog log streams, and HTTP / HTTPS protocols; Configure parsing operators, transformation operators, and enrichment operators in the transformation operator node to perform JSON / regular expression / key-value pair parsing, data type conversion, time formatting, IP address normalization, and data enrichment based on the threat intelligence database; Perform streaming logic control, including data splitting, filtering, and field mapping, to uniformly map raw log fields to entities and attributes of a standard security data model.

6. The large-scale storage and computing resource dynamic reorganization system as described in claim 5, characterized in that: Maintaining strong data consistency between the primary and backup data centers includes writing data to the storage layers of both the primary and backup data centers in parallel when a write transaction is received. A two-phase commit protocol is used, and transactions are committed only after successful confirmations are received from all target storage nodes across availability zones.

7. A large-scale storage and computing resource dynamic reorganization system as described in claim 6, characterized in that: The construction of a global resource topology that includes physical location and network latency attributes includes defining a unified resource descriptor, including computing attributes, storage attributes, and network attributes. When an edge node starts up, the agent process automatically scans local hardware information and sends a registration request to the local control center of its region. The global control center divides resources into different logical groups based on business needs, including a default shared pool, a dedicated protection pool, and a tidal resource pool.

8. A large-scale storage and computing resource dynamic reorganization system, employing the large-scale storage and computing resource dynamic reorganization system as described in any one of claims 1 to 7, characterized in that, include: The resource awareness module collects the computing and storage resource status of the main data center, backup data center and edge nodes in real time, and constructs a global resource topology that includes physical location and network latency attributes. Through the data processing module, a directed acyclic graph-based ETL processing pipeline is used to achieve standardized access to multi-source heterogeneous data, and an abstract syntax tree-based SQL parser and rewriter is used to achieve transparent injection of security policies. The in-memory computing collaborative scheduling module dynamically selects the computing mode based on the cost estimate of the query task and executes the query task. The disaster recovery and collaboration module maintains strong data consistency between primary and backup data centers and performs metadata switching in the event of a failure.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of a large-scale storage and computing resource dynamic reorganization system as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of a large-scale storage and computing resource dynamic reorganization system as described in any one of claims 1 to 7.

Images