Dual-mode storage and calculation cooperative scheduling method and system

Abstract

The invention discloses a dual-mode storage and calculation cooperative scheduling method and system, and relates to the technical field of data processing, and the method comprises the following steps: mode 1, accessing a network through a virtual cluster mode, saving calculation resources, accessing the network through a virtual file system VFS, saving storage resources, carrying out unified scheduling on a network-level front end FE, and obtaining global metadata and calculation node information by the FE through a topological structure; a distributed query plan is generated, and flexible scheduling is carried out according to computing resource loads and data storage distribution in the scheduling process; and mode 2, the network FE directly forwards a query language to each province FE, each province FE automatically schedules resources to generate a query plan, the query plan returns a query result to the network FE after execution is completed, and the network FE summarizes the query result and returns the query result to the user. According to the invention, through dual-mode scheduling, when the network delay is high or the resource load is heavy, the scheduling mode can be automatically switched to a more efficient scheduling mode, the resource utilization of the edge node and the central data center is effectively considered, and the cross-level resource optimization distribution is realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a dual-mode storage and computing collaborative scheduling method and system. Background Art

[0002] Traditional distributed computing systems typically use a single scheduling mode, which fails to balance resource utilization efficiency across multiple layers (e.g., edge nodes and central data centers). In existing technologies, centralized scheduling modes are prone to network bandwidth bottlenecks, while decentralized scheduling struggles to achieve global resource optimization, resulting in high query latency and low resource utilization. This invention aims to propose a dual-mode storage and computing collaborative scheduling method and system to address the shortcomings of single-mode scheduling. Summary of the Invention

[0003] In view of the above-mentioned problems, the present invention is proposed.

[0004] Therefore, the problem to be solved by the present invention is that in the prior art, the centralized scheduling mode easily leads to network bandwidth bottlenecks, while the decentralized scheduling mode is difficult to achieve global resource optimization, resulting in high query latency and low resource utilization.

[0005] To solve the above technical problems, the present invention provides the following technical solutions: a dual-mode storage and computing collaborative scheduling method, which includes: Mode 1, wherein Mode 1 specifically includes opening up the network and saving computing resources through a virtual cluster, opening up the network and saving storage resources through a virtual file system VFS, and the network-level front-end FE performs unified scheduling. The FE obtains global metadata and computing node information through the topology structure, generates a distributed query plan, and flexibly schedules according to the computing resource load and data storage distribution during the scheduling process; Mode 2, wherein Mode 2 specifically includes: the network FE directly forwards the query language to each provincial FE, and each provincial FE schedules resources to generate a query plan. After execution, the query results are returned to the network FE, which summarizes and returns them to the user.

[0006] As an optimal solution for the dual-mode storage and computing collaborative scheduling method described in the present invention, Mode 1 also includes integrating network-level and provincial storage resources through the virtual file system VFS, uniformly generating distributed query plans by the network-level front-end FE, and globally scheduling computing tasks. Mode 1 supports cross-availability zone AZ data calculation.

[0007] As an optimal solution for the dual-mode storage and computing collaborative scheduling method described in the present invention, Mode 2 also includes that the network-level FE forwards the query request to the provincial-level FE, and the provincial-level node independently performs local calculations and returns the results. Only the final result needs to be transmitted, reducing cross-AZ data traffic.

[0008] As an optimal solution for the dual-mode storage and computing collaborative scheduling method described in the present invention, a dynamic mode switching mechanism and a metadata synchronization and disaster recovery mechanism are set for mode 1 and mode 2; the dynamic mode switching mechanism includes automatically selecting the optimal scheduling mode based on real-time indicators of network delay, data distribution and resource load.

[0009] As a preferred solution of the dual-mode storage and computing collaborative scheduling method described in the present invention, the method of obtaining the network delay includes calculating the average delay within the same AZ and across AZs by detecting the communication delay across AZs; the method of obtaining the data distribution includes counting the proportion of required data in provincial nodes by querying the data localization ratio involved; the method of obtaining the resource load includes collecting CPU, memory and IO utilization data within provincial nodes, and collecting query complexity and result set size indicators, assigning weights to each indicator, and determining the mode to be used after calculating the total score.

[0010] As a preferred solution of the dual-mode storage and computing collaborative scheduling method described in the present invention, the dynamic mode switching mechanism also includes hard rules with the highest priority; the hard rules include, if the provincial node goes down or the network is completely interrupted, forced switching to mode 1, cross-AZ scheduling; the query contains cross-provincial JOIN operations, when the data distribution is <30% local, forced mode 1.

[0011] As an optimal solution for the dual-mode storage and computing collaborative scheduling method described in the present invention, the metadata synchronization and disaster recovery mechanism includes synchronizing data across availability zones (AZs) through read-write separation technology. During the data writing process, the system writes data to the backup AZ cluster in real time. Only when the write operation of the backup AZ is successfully completed is the transaction that has been successfully written committed. When the primary AZ fails, the FE obtains the data distribution information of the backup AZ node from the metadata of the backup AZ. Based on the data distribution information, the FE schedules the computing nodes of the backup AZ, thereby ensuring smooth data switching.

[0012] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the dual-mode storage and computing collaborative scheduling method described above are implemented.

[0013] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the dual-mode storage and computing collaborative scheduling method described above.

[0014] The beneficial effects of the present invention are as follows: the virtual cluster scheduling of mode 1 designed by the present invention can flexibly schedule computing resources, optimize according to load and data distribution, and improve overall resource utilization; the direct forwarding scheduling of mode 2 designed significantly reduces data transmission across availability zones (AZs), and only transmits the final results, effectively reducing network bandwidth consumption.

[0015] In addition, the present invention uses dual-mode scheduling to ensure that it can automatically switch to a more efficient scheduling mode when network latency is high or resource load is heavy. It supports both cross-AZ computing for complex analysis scenarios and localized computing for efficient queries, effectively balancing the resource utilization of edge nodes and central data centers, and achieving cross-level resource optimization allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 This is a structural diagram of Mode 1 of a dual-mode storage and computing collaborative scheduling method in Example 1.

[0018] Figure 2 This is a structural diagram of Mode 2 of a dual-mode storage and computing collaborative scheduling method in Example 1.

[0019] Figure 3 This is a diagram of the metadata synchronization and disaster recovery mechanism of a dual-mode storage and computing collaborative scheduling method in Example 1. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0022] Example 1, reference Figure 1-Figure 3 , which is the first embodiment of the present invention, provides a dual-mode storage and computing collaborative scheduling method including: Mode 1: connecting the network and saving computing resources through a virtual cluster, and connecting the network and saving storage resources through a VFS (virtual file system). Figure 1As shown, the network FE (network-level front-end) performs unified scheduling. The FE uses the topology to obtain global metadata and compute node information and generates distributed query plans. This scheduling process allows for flexible scheduling based on computing resource load and data storage distribution. This model allows the network to directly perform data computations across availability zones (AZs), providing more flexible computational analysis. However, this increases network latency and bandwidth consumption, and involves extensive inter-AZ data read and write operations, potentially degrading query performance.

[0023] Mode 1 (Virtual Cluster Scheduling) integrates network-level and provincial-level storage resources through a virtual file system (VFS). The network-level front end (FE) generates a unified distributed query plan and globally schedules computing tasks. This mode supports cross-AZ data computation and is suitable for complex analytical scenarios.

[0024] Mode 2: The network FE directly forwards the query language to each provincial FE, which then dispatches resources to generate a query plan. After execution, the query results are returned to the network FE, which summarizes and returns them to the user. This mode does not involve large amounts of data flow between AZs. Only the final results are transmitted back to the network point, reducing network bandwidth consumption and improving query performance. However, the data needs to be calculated locally and cannot be cross-regional. Figure 2 As shown, meta refers to metadata and BE refers to the network-level backend.

[0025] In mode 2 (direct forwarding scheduling), the network-level FE forwards the query request to the provincial-level FE, and the provincial-level node independently performs local calculations and returns the results. Only the final result needs to be transmitted, significantly reducing cross-AZ data traffic.

[0026] The present invention optimizes cross-level resource scheduling and supports both virtual clustering and direct forwarding modes to meet the computing needs of different scenarios. The solutions include:

[0027] Reduce network bandwidth consumption: Reduce cross-availability zone (AZ) data transmission through localized computing, improving query performance.

[0028] Support dynamic load balancing: Dynamically select the scheduling mode based on real-time resource load to achieve optimal allocation of computing and storage resources.

[0029] The solution of this invention addresses the shortcomings of Model 1: network latency and bandwidth consumption, which involve large amounts of data read and written between AZs and can lead to degraded query performance. It also addresses the shortcomings of Model 2: data must be computed locally and cannot be transferred across zones. This solution achieves efficient query performance and allows cross-zone data computation.

[0030] The solution specifically includes adding a dynamic mode switching mechanism and metadata synchronization and disaster recovery mechanism on the basis of the two working modes to optimize the usage effect.

[0031] The dynamic mode switching mechanism automatically selects the optimal scheduling mode based on real-time metrics such as network latency, data distribution, and resource load. Network latency is measured by detecting inter-AZ communication latency and calculating the average latency within and across AZs. Data distribution is measured by calculating the localization ratio of the data involved in the query and calculating the proportion of the required data in provincial nodes. Resource load is measured by collecting CPU, memory, and I / O utilization data within provincial nodes, as well as query complexity and result set size metrics. Weights are assigned to each metric and the overall score is calculated to determine the optimal mode.

[0032] In addition, some hard rules (with the highest priority) are set to ensure the normal implementation of the function, including:

[0033] 1. If the provincial node goes down or the network is completely interrupted, it is forced to switch to mode 1 (cross-AZ scheduling).

[0034] 2. The query contains cross-province JOIN operations. When the data distribution is less than 30% local, force mode 1.

[0035] The metadata synchronization and disaster recovery mechanism includes the system achieving data synchronization across availability zones (AZs) through read-write separation technology. During the data writing process, the system will write the data to the backup AZ cluster in real time. Only when the write operation of the backup AZ is successfully completed will the transaction be committed. Figure 3 As shown in the figure, IngestManager is an application used to simplify data import and management, BE-Ingest refers to backend ingestion, BE-Store refers to backend storage, and Data refers to data transmission.

[0036] When a primary AZ fails, the FE retrieves data distribution information about the backup AZ's nodes from the backup AZ's metadata. Based on this information, the FE schedules the compute nodes in the backup AZ, enabling smooth data failover. Because cross-AZ backups occur during the data write process, the system can synchronize data down to the table level. This allows the system to configure different backup policies for different tables to meet the needs of diverse business scenarios.

[0037] Example 2 is the second embodiment of the present invention, which is different from the first embodiment in that: a system for a dual-mode storage and computing collaborative scheduling method includes: if the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0038] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0039] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0040] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0041] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A dual-mode storage and computing collaborative scheduling method, characterized by: include, Mode 1: This mode specifically includes connecting the network and saving computing resources through virtual clusters, and using a virtual file system (VFS) to connect the network and save storage resources. The network-level front-end (FE) performs unified scheduling. The FE obtains global metadata and computing node information through the topology structure and generates a distributed query plan. During the scheduling process, flexible scheduling is performed based on computing resource load and data storage distribution. Mode 2: Mode 2 specifically includes that the network FE directly forwards the query language to each provincial FE, and each provincial FE independently dispatches resources to generate a query plan. After execution, the query results are returned to the network FE, which summarizes them and returns them to the user.

2. The dual-mode storage and computing collaborative scheduling method according to claim 1, characterized in that: Mode 1 also includes integrating network-level and provincial storage resources through the virtual file system VFS, uniformly generating distributed query plans by the network-level front-end FE, and globally scheduling computing tasks. Mode 1 supports cross-availability zone AZ data computing.

3. The dual-mode storage and computing collaborative scheduling method according to claim 2, characterized in that: Mode 2 also includes that the network-level FE forwards the query request to the provincial-level FE, and the provincial-level node independently performs local calculations and returns the results. Only the final result needs to be transmitted, reducing cross-AZ data traffic.

4. The dual-mode storage and computing collaborative scheduling method according to claim 3, characterized in that: Setting a dynamic mode switching mechanism and metadata synchronization and disaster recovery mechanism for Mode 1 and Mode 2; The dynamic mode switching mechanism includes automatically selecting the optimal scheduling mode based on real-time indicators of network latency, data distribution and resource load.

5. The dual-mode storage and computing collaborative scheduling method according to claim 4, characterized in that: The network delay is obtained by detecting the inter-AZ communication delay and calculating the average delay within the same AZ and across AZs. The data distribution is obtained by querying the localization ratio of the data involved and calculating the proportion of the required data in the provincial nodes; The resource load is obtained by collecting CPU, memory and IO usage data within provincial nodes, and collecting query complexity and result set size indicators, assigning weights to each indicator, and determining the mode to be used after calculating the total score.

6. The dual-mode storage and computing collaborative scheduling method according to claim 5, characterized in that: The dynamic mode switching mechanism also includes a hard rule with the highest priority; The hard rules include: if a provincial node goes down or the network is completely interrupted, it is forced to switch to mode 1, cross-AZ scheduling; The query contains cross-province JOIN operations and when the data distribution is <30% local, mode 1 is enforced.

7. The dual-mode storage and computing collaborative scheduling method according to claim 6, characterized in that: The metadata synchronization and disaster recovery mechanism includes synchronizing data across availability zones (AZs) through read-write separation technology. During data writing, the system writes data to the backup AZ cluster in real time. Only when the write operation in the backup AZ is successfully completed will the transaction be committed. When a failure occurs in the primary AZ, the FE obtains data distribution information of the backup AZ nodes from the metadata of the backup AZ. Based on the data distribution information, the FE schedules the computing nodes of the backup AZ to ensure smooth data switching.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the processor implements the steps of a dual-mode storage and computing collaborative scheduling method described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a dual-mode storage and computing collaborative scheduling method described in any one of claims 1 to 7 are implemented.

Images