A cross-computer room cluster control method and device

Abstract

The present application provides a cross-computer room cluster control method and device, the method comprising: receiving at least one target task sent by at least one computer room; determining the target computer room corresponding to each target task in at least one target task according to the task information contained in each target task in at least one target task; the task information includes the computer room identifier where the task input data is located, the computer room identifier where the task output data is located, and the task type; assigning each target task in at least one target task to the corresponding target computer room, so that each target task in at least one target task is executed in the corresponding target computer room. The cross-computer room cluster control method and architecture provided by the present application realizes the sharing of computing resources and storage space across computer rooms by means of wavelength division interconnection, thereby reducing the stacking of switches and reducing network jitter.

Description

Technical Field

[0001] The present application relates to the field of distributed clustering, and in particular to a cross-computer room cluster control method and device. Background Art

[0002] Hadoop, an open source software born in the era of big data, is widely used in data storage and computing systems of all sizes. Since its birth, Hadoop has been widely used in many industries such as the Internet, telecommunications, and finance. Its throughput, reliability, and ease of use have been widely recognized by the industry.

[0003] However, the traditional cluster architecture designed based on Hadoop has the following disadvantages: 1) The capacity of a single computer room is limited. When the computer space in the data center / computer room is full, the cluster cannot be expanded, which cannot meet the demand for data growth; 2) When multiple different clusters are set up for multiple computer rooms, data in different fields cannot be related to each other. The above data needs to be synchronized or transferred to the same cluster before subsequent calculations; 3) The resources used for computing or storage in multiple different clusters are independent of each other and cannot be reasonably allocated or shared. 4) Setting up clusters in multiple computer rooms consumes too high operation and maintenance costs; 5) If there is a fault in a computer room, the business stability of the computer room is difficult to guarantee.

[0004] In view of this, it is necessary to design a new cross-computer room cluster control method to solve the problem that the data between computer rooms are independent and cannot be interconnected, and the operation and maintenance costs are too high. Summary of the invention

[0005] The present application provides a cross-computer room cluster control method and device, which are used to solve the problem that data between computer rooms are independent and cannot be interconnected and used, and the operation and maintenance cost is too high.

[0006] In a first aspect, the present application provides a cross-computer room cluster control method, the method comprising:

[0007] receiving at least one target task sent by at least one computer room;

[0008] Determine a target computer room corresponding to each target task in the at least one target task according to task information contained in each target task in the at least one target task; the task information includes an identifier of a computer room where task input data is located, an identifier of a computer room where task output data is located, and a task type;

[0009] Allocate each target task in the at least one target task to the corresponding target computer room, so that each target task in the at least one target task is executed in the corresponding target computer room.

[0010] As a possible implementation, the method further includes:

[0011] A wavelength division multiplexing device is set in each of the at least one computer room, and the wavelength division multiplexing device set in each of the at least one computer room forms a network topology structure, so that the network of each of the at least one computer room is shared through the wavelength division multiplexing device.

[0012] As a possible implementation, the method further includes:

[0013] Receiving heartbeat information, data storage information, and computing resource information sent by each of at least one computer room, and determining data storage conditions and computing resource usage conditions of each of the at least one computer room;

[0014] Based on the data storage situation and computing resource usage situation of each computer room in the at least one computer room, a shared storage space and a shared computing resource queue are established.

[0015] As a possible implementation manner, after establishing the shared storage space, the method further includes:

[0016] The same name space is defined for data in the same computer room, and the data in the shared storage space is allocated to each computer room in the at least one computer room for storage according to a set data allocation strategy, wherein the set data allocation strategy includes: data service type, data location information and data domain.

[0017] As a possible implementation, establishing the shared computing resource queue includes:

[0018] Based on the computing resource usage of each computer room in the at least one computer room, re-dividing the sum of the computing resources of each computer room in the at least one computer room into a plurality of computing resource blocks according to a set resource block size;

[0019] At least one column of computing resource queue is established using the multiple computing resource blocks.

[0020] As a possible implementation manner, determining a target computer room corresponding to each target task in the at least one target task according to the task information included in each target task in the at least one target task includes:

[0021] Determine, according to the task information included in the target task, a computer room where target task input data corresponding to the target task is located and a computer room where target task output data is located;

[0022] When the computer room where the target task input data is located meets the task execution condition, the computer room where the target task input data is located is determined as the target computer room, wherein the task execution condition includes at least one of the following: the bandwidth of the computer room is greater than a set bandwidth threshold, the computing resources of the computer room are greater than a set computing resource threshold, and the storage space of the computer room is greater than a set storage space threshold;

[0023] When the computer room where the target task output data is located meets the task execution condition, determining the computer room where the target task output data is located as the target computer room;

[0024] When the computer room where the target task input data is located and the computer room where the target task output data is located both meet the task execution condition, compare the computer room computing resources of the computer room where the target task input data is located with the computer room computing resources of the computer room where the target task output data is located, and determine the computer room with greater computer room computing resources as the target computer room;

[0025] When the computer room where the target task input data is located and the computer room where the target task output data is located do not meet the task execution condition, the computer room with the largest computer room bandwidth is determined from other computer rooms in the at least one computer room as the target computer room.

[0026] Using the cross-computer room cluster control method provided by this application, in terms of network, the cross-computer room cluster can realize cross-computer room network sharing through wavelength division interconnection, reduce the mutual stacking of switches, and then reduce network jitter, thereby building a remote multi-active Hadoop cluster, enhancing the robustness of the business, and reducing operation and maintenance costs; in terms of storage, the storage sharing of the cross-computer room cluster can be used in a federal mode, thereby dividing multiple namespaces, achieving physical isolation and logical unification, and making reasonable data segmentation strategies according to the actual resource conditions of each computer room and various types of business, so as to well control the network traffic across computer rooms and improve processing performance; in terms of computing, cross-computer room cluster computing can divide a larger shared resource pool into multiple minimum resource blocks, thereby using fragmented idle computing resources in remote computer rooms. In terms of task allocation, the task allocation of the cross-computer room cluster can be carried out through the cross-computer room task scheduling strategy, and the resource allocation and scheduling within the cross-computer room are carried out, so as to improve the robustness of the big data system and the system operation capability, and optimize resource allocation.

[0027] In a second aspect, the present application provides a cross-machine room cluster control device, the device comprising:

[0028] The task receiving module is used to: receive at least one target task sent by at least one computer room;

[0029] A computer room determination module, used to: determine a target computer room corresponding to each target task in the at least one target task according to task information contained in each target task in the at least one target task; the task information includes an identifier of a computer room where task input data is located, an identifier of a computer room where task output data is located, and a task type;

[0030] The task allocation module is used to allocate each target task in the at least one target task to the corresponding target computer room, so that each target task in the at least one target task is executed in the corresponding target computer room.

[0031] As a possible implementation manner, the device further includes:

[0032] The network sharing module is used to: set up a wavelength division multiplexing device in each of the at least one computer room, and the wavelength division multiplexing device set up in each of the at least one computer room constitutes a network topology structure so that the network of each of the at least one computer room is shared through the wavelength division multiplexing device.

[0033] As a possible implementation manner, the device further includes:

[0034] The shared space and computing resource module is used to: receive heartbeat information, data storage information and computing resource information sent by each computer room in at least one computer room, and determine the data storage status and computing resource usage status of each computer room in the at least one computer room;

[0035] Based on the data storage situation and computing resource usage situation of each computer room in the at least one computer room, a shared storage space and a shared computing resource queue are established.

[0036] As a possible implementation manner, the device further includes:

[0037] The storage space processing module is used to define the same name space for data in the same computer room, and allocate the data in the shared storage space to each computer room in the at least one computer room for storage according to a set data allocation strategy. The set data allocation strategy includes: data service type, data location information and data field.

[0038] As a possible implementation mode, the shared space and computing resource module is specifically used to: based on the computing resource usage of each computer room in the at least one computer room, re-divide the sum of the computing resources of each computer room in the at least one computer room into multiple computing resource blocks according to a set resource block size;

[0039] At least one column of computing resource queue is established using the multiple computing resource blocks.

[0040] As a possible implementation, the task allocation module is specifically used to:

[0041] Determine, according to the task information included in the target task, a computer room where target task input data corresponding to the target task is located and a computer room where target task output data is located;

[0042] When the computer room where the target task input data is located meets the task execution condition, the computer room where the target task input data is located is determined as the target computer room, wherein the task execution condition includes at least one of the following: the bandwidth of the computer room is greater than a set bandwidth threshold, the computing resources of the computer room are greater than a set computing resource threshold, and the storage space of the computer room is greater than a set storage space threshold;

[0043] When the computer room where the target task output data is located meets the task execution condition, determining the computer room where the target task output data is located as the target computer room;

[0044] When the computer room where the target task input data is located and the computer room where the target task output data is located both meet the task execution condition, compare the computer room computing resources of the computer room where the target task input data is located with the computer room computing resources of the computer room where the target task output data is located, and determine the computer room with greater computer room computing resources as the target computer room;

[0045] When the computer room where the target task input data is located and the computer room where the target task output data is located do not meet the task execution condition, the computer room with the largest computer room bandwidth is determined from other computer rooms in the at least one computer room as the target computer room.

[0046] In a third aspect, the present application provides a cross-computer room cluster control system, the system comprising: at least one computer room and any cross-computer room cluster control device according to the second aspect;

[0047] Each of the at least one computer room is configured to receive a target task and send the target task to the cross-computer room cluster control device.

[0048] In a fourth aspect, the present application provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0049] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flow chart of steps of a cross-computer room cluster control method;

[0051] Figure 2It is a schematic diagram of the structure of a YARN resource cluster management architecture;

[0052] Figure 3 The figure is a schematic diagram of the structure of a cross-computer room cluster control device. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Currently, there are several disadvantages in Hadoop clusters: 1) The capacity of a single computer room is limited, and the cluster cannot be expanded when the machine is full; 2) Data in different fields in multiple clusters cannot be linked to each other; 3) The resources responsible for computing / storage in multiple clusters are independent of each other; 4) It is too costly to set up clusters in multiple computer rooms; 5) When a failure occurs in a computer room, the business of that computer room is difficult to stabilize.

[0055] In view of this, the present application provides a cross-computer room cluster control method and device, which realizes the sharing of computing resources and storage space across computer room networks through wavelength division interconnection, thereby reducing the stacking of switches and reducing network jitter.

[0056] Below, some terms involved in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0057] (1) Hadoop: It is a software platform for developing and running large-scale data processing. It is an open source software framework implemented in Java by Apache, which enables distributed computing of massive data in a cluster composed of a large number of computers.

[0058] (2) Data Federation: It is one of the more mature enterprise data integration methods. Data federation can uniformly access any digital information represented in any format (structured or unstructured) in any information storage.

[0059] (3) View File System (ViewFS): View File System provides a method to manage multiple Hadoop file system namespaces (namespace volumes), which is particularly important in the scenario of multiple namespaces (NameNode) and multiple namespaces in federated mode. ViewFS is similar to the client mount table under some Unix / Unix-like operating systems (Linux). It can be used to create not only personalized namespace views, but also global views. Each cluster has a file Hadoop system with multiple namespaces integrated in federated mode.

[0060] (4) Heartbeat Message: A message sent from a source to a receiver that allows the receiver to determine if and when the source fails or terminates. Usually, heartbeat messages are sent from the time the source is started until the source is shut down, during which time the source will continuously send periodic or repetitive messages. When the receiver does not receive a message within a certain message receiving cycle, the receiver may believe that the source has been shut down, failed, or is currently unavailable. Heartbeat messages may often be used for high availability or fault tolerance purposes.

[0061] (5) Yet Another Resource Negotiator (YARN): It is a new Hadoop resource manager. It is a general resource management system that can provide unified resource management and scheduling for upper-layer applications. Its introduction has brought great benefits to the cluster in terms of utilization, unified resource management and data sharing. The basic idea of ​​YARN is to separate the two main functions of JobTracker (resource management and job scheduling / monitoring). The main method is to create a global ResourceManager (cluster resource manager, RM) and several ApplicationMasters (application hosts, AM) for applications.

[0062] (6) Access layer: refers to the part of the network that is directly connected to or accessed by users. The access layer uses transmission media such as optical fiber, twisted pair, coaxial cable, and wireless access technology to connect to users and allocate services and bandwidth. The purpose of the access layer is to allow end users to connect to the network. Therefore, access layer switches have the characteristics of low cost and high port density.

[0063] (7) Aggregation layer: It is the information aggregation point of a building complex or community. It is a network device that connects the access layer and the core layer. It provides data aggregation, transmission, management, and distribution processing for the access layer. The aggregation layer provides policy-based connections for the access layer, such as address merging, protocol filtering, routing services, and authentication management. Network segmentation and network isolation can prevent problems in certain network segments from spreading and affecting the core layer. The aggregation layer can also provide interconnection between access layer virtual networks, control and limit access to the core layer from the access layer, and ensure the security and stability of the core layer.

[0064] (8) Namespace: Also known as "namespace", it is a form of code organization that uses namespaces to classify and distinguish different code functions. Namespaces are used to organize and reuse code. Just like the name means, the reason why such a thing came into being is that the number of words available to humans is too small, and it is impossible for all variables in programs written by different people to have no duplicate names. For libraries, this problem is particularly serious. If two people write library files with the same name and variables or functions appear (inevitable), there will be problems in using them. In order to solve this problem, the concept of namespace is introduced so that unnecessary conflicts will not occur.

[0065] like Figure 1 As shown, the cross-computer room cluster control method according to an embodiment of the present invention includes the following steps:

[0066] Step S101, receiving at least one target task sent by at least one computer room;

[0067] Step S102, determining the target computer room corresponding to each target task in the at least one target task according to the task information contained in each target task in the at least one target task; the task information includes the computer room identifier where the task input data is located, the computer room identifier where the task output data is located, and the task type.

[0068] Step S103: assign each target task in the at least one target task to the corresponding target computer room, so that each target task in the at least one target task is executed in the corresponding target computer room.

[0069] The execution subject of the embodiment of the present invention can be a central server (also called name node) NameNode, etc., which is responsible for describing data storage location, historical data, resource search, file records, etc., or it can be ETL (Extract-Transform-Load, data warehouse).

[0070] It should be noted that the embodiments of the present invention can be applied to 2G networks, 3G networks, 4G networks, 5G networks, 6G networks, fixed networks, and other network structures that can be used to transmit data.

[0071] The execution subject of the embodiment of the present invention can realize network sharing, storage sharing, and computing sharing, so that data from multiple professions and fields can be interconnected, thereby realizing data unification and sharing, improving the stability of platform operation, and enhancing the robustness of business.

[0072] In order to realize network sharing, storage sharing, and computing sharing, and to connect data from multiple disciplines and fields, a cross-computer room cluster control system needs to build a cross-computer room shared network:

[0073] As a possible implementation, a method for sharing a network across computer rooms may include: setting up a wavelength division multiplexing device in each of the at least one computer room, and forming a network topology structure by the wavelength division multiplexing device set up in each of the at least one computer room, so that the network of each of the at least one computer room is shared through the wavelength division multiplexing device.

[0074] Under the premise of cross-computer room communication and sharing, each of the multiple computer rooms can realize network sharing through the servers or network devices therein. Specifically, dual uplink load balancing can be used to realize high availability of nodes and high-bandwidth interconnection functions between computer rooms. At present, in network redundancy protection, equipment redundancy protection can be adopted. Equipment redundancy protection includes traditional optical line terminal equipment (OLT) trunk optical fiber redundancy protection, main control disk redundancy protection, OLT equipment uplink disk link aggregation protection, power supply equipment redundancy protection, etc. The dual uplink method can improve the reliability of the network by switching the primary and backup links in the event of a link failure. Under normal circumstances, the traffic passes through the primary link. Once the primary link fails, the traffic is switched to the backup link in time to improve the reliability of network transmission.

[0075] After realizing the high availability of nodes and the high-bandwidth interconnection function in the computer room, a high-performance network switch can be used in the gateway to deploy the network structure according to a simple two-layer architecture (access layer-aggregation layer), thereby reducing stacking and latency, and making the network sharing effect better. In the embodiment of the present application, the high-performance network switch can be used, but not limited to, to construct the above-mentioned network architecture, and can also construct a three-layer or multi-layer architecture with a better architecture. Those skilled in the art should know that no further details will be given here.

[0076] Each of the multiple computer rooms uses physical wavelength division equipment to achieve network interconnection. Among them, communication optical fiber has a large information capacity. Multiple optical signals of different wavelengths can be transmitted simultaneously and independently inside an optical fiber. Each optical signal can be used as an independent carrier to carry different information content, and the transmission capacity can be easily increased or decreased by increasing or decreasing the number of wavelengths. This technology of simultaneously transmitting multiple optical signals of different wavelengths in an optical fiber is wavelength division multiplexing technology. Wavelength division multiplexing is frequency division multiplexing in the optical domain. Due to its large-capacity transmission capacity and smooth expansion characteristics, it is widely used in modern optical communication networks. The wavelength division equipment can be RE (Radio Equipment, radio frequency equipment) and so on, which is not limited here. Each of the multiple computer rooms uses physical wavelength division equipment to build a star network topology structure, thereby realizing network sharing across computer rooms in different locations, and can also unify the export gateway to ensure the control of network traffic and the security of network data. The network architecture constructed by the above-mentioned multiple computer rooms can also be a bus topology structure, a ring topology structure, a tree topology structure, a mesh topology structure, a hybrid topology structure, and so on.

[0077] The cross-computer room cluster network of the above embodiment adopts wavelength division technology, which can open up high transmission rate (such as 2*100GE) channels between multiple computer rooms. The switch adopts access layer and aggregation layer structure, thereby reducing stacking and establishing a low-latency, high-bandwidth network infrastructure for Hadoop clusters.

[0078] After building a cross-computer room sharing network, it is necessary to share storage space and computing space between computer rooms. The computer room storage sharing method may include the following steps: receiving heartbeat information and data storage information sent by each computer room in at least one computer room, determining the data storage status of each computer room in the at least one computer room; and establishing a shared storage space based on the data storage status of each computer room in the at least one computer room.

[0079] First, multiple namespaces can be configured in a cross-computer room cluster, and namespaces can be allocated to different computer rooms to share all storage resources in the computer room. Storage metadata management roles can be deployed in each namespace to manage data storage information in the computer room. Specifically, all metadata nodes in the metadata cluster composed of multiple metadata management roles provide metadata services for the global namespace, and metadata nodes establish a global consistency cache and / or a shared lock to build a real-time application cluster through the cross-computer room shared network established by the above embodiment. The data performance of the entire metadata cluster can be horizontally expanded, and all metadata nodes are in an active state. The failure or disconnection of any of the multiple metadata nodes will not affect the metadata service of the entire cluster.

[0080] In addition, the metadata node in each computer room can actively report heartbeat information. The heartbeat information indicates the operating status of the monitored object that reports the heartbeat information within the heartbeat cycle. In this application, the monitored object is the metadata node arranged in each computer room. The metadata node in each computer room can actively report the heartbeat information and also report the data storage information. By analyzing the data storage information, abnormal storage nodes can be eliminated in time to obtain the real location of the data storage.

[0081] For example, data storage information may include type identification information and corresponding storage location information. By storing the data storage information corresponding to the data, operations such as reading, modifying, and deleting can be performed on the stored data. If a piece of data needs to be deleted, the identification information of the data to be deleted is sent to the metadata server. The metadata server determines the network address information and storage location information corresponding to the identification information based on the data storage information stored by itself, and deletes the data corresponding to the identification information in the storage location indicated by the determined storage location information through the storage space corresponding to the determined network address information. When the metadata server finds that an abnormality occurs corresponding to a certain piece of data, the data corresponding to the identification information in the storage location indicated by the determined storage location information can be deleted based on the stored data storage information.

[0082] In addition, multiple storage meta-management roles can be deployed in one namespace, so that metadata shared between each storage meta-management role can be synchronized in seconds to ensure data consistency and high availability. The metadata stored between each namespace can achieve physical isolation of cross-data center clusters while ensuring logical unification.

[0083] As a possible implementation, after establishing the shared storage space, the method further includes: defining the same name space for data in the same computer room, and allocating the data in the shared storage space to each computer room in the at least one computer room for storage according to a set data allocation strategy, wherein the set data allocation strategy includes: data service type, data location information, and data domain.

[0084] Specifically, multiple storage metadata management roles can make reasonable data partitioning strategies based on the actual resource situation of the local computer room. For example, data in the shared storage space can be divided into different namespaces for management according to data business types, thereby reducing unnecessary redundant interactions of storage metadata and network traffic consumption across machines.

[0085] The data storage method in the cross-computer room cluster in the above embodiment adopts federal technology and ViewFS to divide the stored data according to region and business, and uses multiple namespaces for management, which effectively reduces the computing traffic across computer rooms, and realizes the physical isolation of data of multiple devices across multiple computer rooms between clusters, logical unification, transparency to users, and significant maintenance costs.

[0086] The method for computer room computing resources may include the following steps: receiving heartbeat information and computing resource information sent by each computer room in at least one computer room, and determining the computing resource usage of each computer room in the at least one computer room; and establishing a shared computing resource queue based on the computing resource usage of each computer room in the at least one computer room.

[0087] First, a computing resource management role can be deployed in each computer room in the cross-computer room cluster to manage the computing resource information in each computer room, and the computing nodes in each computer room actively report the heartbeat information and computing resource information. Among them, the heartbeat information is not described in detail here, see the description of the above embodiment. The computing resource information is used to characterize the computing power of the computing equipment in the computer room. The computing resource information can also be a computing power specification. The computing power specification can be a general unit of computing power, such as one trillion floating-point operations per second (TFLOPS), or one trillion integer operations per second. The computing resource information can also include specified hardware specifications, such as a certain hardware model. Different hardware models correspond to different computing power specifications. According to the reported computing resource information, a unified computing resource pool is constructed. In addition, multiple computing resource roles are deployed in the cross-computer room cluster, distributed in multiple different computer rooms, to achieve one master and multiple backups, to prevent single computer room node failures, thereby affecting the calculation. Specifically, after the computing resource pool is divided into multiple minimum unit resource blocks, they are put back into the computing resource pool, and the resource usage is reported to the computing resource management role in real time. For example, the ratio of the minimum unit resource blocks in the computing resource pool to the used resource blocks and the unused resource blocks. Optionally, multiple resource queues can be divided on the computing resource pool, and through reasonable arrangement, multiple minimum unit resource blocks in the resource pool can be combined and used. Exemplarily, a computing resource usage request can be received first, and the computing resource usage request can be queued in a resource queue, and the scheduler can schedule according to load conditions, request types, and other attributes. In the above embodiment, by constructing a unified computing resource pool and integrating fragmented idle resources, the maximum computing power of the cross-computer room cluster is significantly improved compared to the maximum computing power of the existing architecture.

[0088] After cross-computer room network sharing, storage space sharing, and computing resource sharing, the resource cluster management receives at least one target task sent by at least one computer room. According to the task information contained in each of the at least one target task, the target computer room corresponding to each of the at least one target task is determined; the task information includes the computer room identifier where the task input data is located, the computer room identifier where the task output data is located, and the task type. Each of the at least one target task is assigned to the corresponding target computer room, so that each of the at least one target task is executed in the corresponding target computer room.

[0089] Among them, resource cluster management can be based on the YARN resource cluster management architecture, see Figure 2 As shown, Figure 2 The diagram is a structural diagram of a YARN resource cluster management architecture. Through YARN's resource cluster management (unified scheduling), multiple independent YARN clusters can be managed and monitored in a unified manner, thereby providing basic resource information for the scheduling logic, and also registering bandwidth information between various computer rooms. In addition, the underlying YARN cluster submission entry is not open, and all received tasks are submitted to a unified resource scheduling engine. Exemplarily, the resource scheduling engine can interact with other monitoring modules based on machine learning algorithms to optimize the task allocation method.

[0090] As a possible implementation manner, determining a target computer room corresponding to each target task in the at least one target task according to the task information included in each target task in the at least one target task includes the following methods:

[0091] According to the task information included in the target task, a computer room where the target task input data corresponding to the target task is located and a computer room where the target task output data is located are determined.

[0092] Method 1: When the computer room where the target task input data is located meets the task execution conditions, the computer room where the target task input data is located is determined as the target computer room, wherein the task execution conditions include at least one of the following: the computer room bandwidth is greater than a set bandwidth threshold, the computer room computing resources are greater than a set computing resource threshold, and the computer room storage space is greater than a set storage space threshold.

[0093] In this scenario, since the computer room where the input data of the target task is located meets the task execution conditions, in order to reduce unnecessary data interaction, the computer room where the input data of the target task is located can be directly ordered to execute the target task. After the target task is executed in the computer room where the input data of the target task is located, the target task execution result is sent to the computer room where the output data of the target task is located, thereby minimizing the data flow between computer rooms.

[0094] Method 2: When the computer room where the target task output data is located meets the task execution condition, the computer room where the target task output data is located is determined as the target computer room.

[0095] In this scenario, since the computer room where the output data of the target task is located meets the task execution conditions, in order to reduce unnecessary data interaction, the computer room where the output data of the target task is located can receive the input data of the target task, and the computer room where the output data of the target task is located can directly execute the target task, thereby minimizing the data flow between computer rooms.

[0096] Method three: When the computer room where the target task input data is located and the computer room where the target task output data is located both meet the task execution conditions, compare the computer room computing resources of the computer room where the target task input data is located with the computer room computing resources of the computer room where the target task output data is located, and determine the computer room with larger computer room computing resources as the target computer room.

[0097] In this scenario, since the computer room where the target task input data is located and the computer room where the target task output data is located both meet the task execution conditions, in order to improve resource utilization, the computer room with higher computing resources / bandwidth between the computer room where the target task input data is located and the computer room where the target task output data is located is selected to execute the target task.

[0098] Mode 4: When the computer room where the target task input data is located and the computer room where the target task output data is located do not meet the task execution condition, determine the computer room with the largest computer room bandwidth from other computer rooms in the at least one computer room as the target computer room.

[0099] In this scenario, because the computer room where the target task input data is located and the computer room where the target task output data is located do not meet the task execution conditions, in order to activate the fragmented and idle computing / storage resources in multiple computer rooms, the computer room with the largest bandwidth in the at least one computer room is determined as the target computer room from the other computer rooms, thereby improving resource utilization.

[0100] Using the cross-computer room cluster control method provided by this application, in terms of network, the cross-computer room cluster can realize cross-computer room network sharing through wavelength division interconnection, reduce the mutual stacking of switches, and then reduce network jitter, thereby building a remote multi-active Hadoop cluster, enhancing the robustness of the business, and reducing operation and maintenance costs; in terms of storage, the storage sharing of the cross-computer room cluster can be used in a federal mode, thereby dividing multiple namespaces, achieving physical isolation and logical unification, and making reasonable data segmentation strategies according to the actual resource conditions of each computer room and various types of business, so as to well control the network traffic across computer rooms and improve processing performance; in terms of computing, cross-computer room cluster computing can divide a larger shared resource pool into multiple minimum resource blocks, thereby using fragmented idle computing resources in remote computer rooms. In terms of task allocation, the task allocation of the cross-computer room cluster can be carried out through the cross-computer room task scheduling strategy, and the resource allocation and scheduling within the cross-computer room are carried out, so as to improve the robustness of the big data system and the system operation capability, and optimize resource allocation.

[0101] Based on the same inventive concept, refer to Figure 3 As shown, the embodiment of the present application further provides a cross-machine room cluster control device 300, and the device 300 includes:

[0102] The task receiving module 301 is used to: receive at least one target task sent by at least one computer room;

[0103] The computer room determination module 302 is used to: determine the target computer room corresponding to each target task in the at least one target task according to the task information contained in each target task in the at least one target task; the task information includes the computer room identifier where the task input data is located, the computer room identifier where the task output data is located, and the task type;

[0104] The task allocation module 303 is used to allocate each target task in the at least one target task to the corresponding target computer room, so that each target task in the at least one target task is executed in the corresponding target computer room.

[0105] As a possible implementation manner, the device 300 further includes:

[0106] The network sharing module 304 is used to: set up a wavelength division multiplexing device in each of the at least one computer room, and the wavelength division multiplexing device set up in each of the at least one computer room constitutes a network topology structure so that the network of each of the at least one computer room is shared through the wavelength division multiplexing device.

[0107] As a possible implementation manner, the device further includes:

[0108] The shared space and computing resource module 305 is used to: receive the heartbeat information, data storage information and computing resource information sent by each computer room in at least one computer room, and determine the data storage status and computing resource usage status of each computer room in the at least one computer room;

[0109] Based on the data storage situation and computing resource usage situation of each computer room in the at least one computer room, a shared storage space and a shared computing resource queue are established.

[0110] As a possible implementation manner, the device 300 further includes:

[0111] The storage space processing module 306 is used to define the same name space for the data in the same computer room, and allocate the data in the shared storage space to each computer room in the at least one computer room for storage according to the set data allocation strategy. The set data allocation strategy includes: data service type, data location information and data field.

[0112] As a possible implementation method, the shared space and computing resource module 305 is specifically used to: based on the computing resource usage of each computer room in the at least one computer room, re-divide the sum of the computing resources of each computer room in the at least one computer room into multiple computing resource blocks according to a set resource block size; and use the multiple computing resource blocks to establish at least one column of computing resource queues.

[0113] As a possible implementation, the task allocation module 303 is specifically used to:

[0114] According to the task information contained in the target task, determine the computer room where the target task input data corresponding to the target task is located and the computer room where the target task output data is located; when the computer room where the target task input data is located meets the task execution condition, determine the computer room where the target task input data is located as the target computer room, wherein the task execution condition includes at least one of the following: the computer room bandwidth is greater than a set bandwidth threshold, the computer room computing resources are greater than a set computing resource threshold, and the computer room storage space is greater than a set storage space threshold; when the computer room where the target task output data is located meets the task execution condition, the target task output data is located. The target computer room is determined according to the computer room where the target task input data is located; when the computer room where the target task input data is located and the computer room where the target task output data is located both meet the task execution condition, the computer room computing resources of the computer room where the target task input data is located are compared with the computer room computing resources of the computer room where the target task output data is located, and the computer room with larger computer room computing resources is determined as the target computer room; when the computer room where the target task input data is located and the computer room where the target task output data is located both do not meet the task execution condition, the computer room with the largest computer room bandwidth is determined from other computer rooms in the at least one computer room as the target computer room.

[0115] Based on the same inventive concept, an embodiment of the present application further provides a cross-computer room cluster control system, the system comprising: at least one computer room and the cross-computer room cluster control device 300 described in the above embodiment.

[0116] Based on the same inventive concept, an embodiment of the present application further provides an electronic device for cross-computer room cluster control, comprising: a memory and a processor; wherein the memory is used to store a computer program;

[0117] The processor is used to execute the program in the memory to implement the method described in the above embodiment.

[0118] Based on the same inventive concept, an embodiment of the present application further provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in the above embodiment is implemented.

[0119] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0120] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0123] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A cross-computer room cluster control method, characterized in that: The method comprises: A wavelength division multiplexing device is arranged in each of the at least one computer room, and the wavelength division multiplexing device arranged in each of the at least one computer room forms a star network topology structure with a two-layer architecture and double uplink load balancing, so that the network of each of the at least one computer room is shared through the wavelength division multiplexing device; receiving at least one target task sent by the at least one computer room; Determine a target computer room corresponding to each target task in the at least one target task according to task information contained in each target task in the at least one target task; the task information includes an identifier of a computer room where task input data is located, an identifier of a computer room where task output data is located, and a task type; Allocating each target task in the at least one target task to the corresponding target computer room, so that each target task in the at least one target task is executed in the corresponding target computer room; Wherein, the method further comprises: Receiving heartbeat information, data storage information, and computing resource information sent by each of at least one computer room, and determining data storage conditions and computing resource usage conditions of each of the at least one computer room; Based on the data storage situation and computing resource usage situation of each computer room in the at least one computer room, a shared storage space and a shared computing resource queue are established.

2. The method according to claim 1, characterized in that After establishing the shared storage space, the method further includes: The same name space is defined for data in the same computer room, and the data in the shared storage space is allocated to each computer room in the at least one computer room for storage according to a set data allocation strategy, wherein the set data allocation strategy includes: data service type, data location information and data domain.

3. The method according to claim 1, characterized in that Establishing the shared computing resource queue includes: Based on the computing resource usage of each computer room in the at least one computer room, re-dividing the sum of the computing resources of each computer room in the at least one computer room into a plurality of computing resource blocks according to a set resource block size; At least one column of computing resource queue is established using the multiple computing resource blocks.

4. The method according to claim 1, characterized in that: Determining a target computer room corresponding to each target task in the at least one target task according to task information included in each target task in the at least one target task includes: Determine, according to the task information included in the target task, a computer room where target task input data corresponding to the target task is located and a computer room where target task output data is located; When the computer room where the target task input data is located meets the task execution condition, the computer room where the target task input data is located is determined as the target computer room, wherein the task execution condition includes at least one of the following: the bandwidth of the computer room is greater than a set bandwidth threshold, the computing resources of the computer room are greater than a set computing resource threshold, and the storage space of the computer room is greater than a set storage space threshold; When the computer room where the target task output data is located meets the task execution condition, determining the computer room where the target task output data is located as the target computer room; When the computer room where the target task input data is located and the computer room where the target task output data is located both meet the task execution condition, compare the computer room computing resources of the computer room where the target task input data is located with the computer room computing resources of the computer room where the target task output data is located, and determine the computer room with greater computer room computing resources as the target computer room; When the computer room where the target task input data is located and the computer room where the target task output data is located do not meet the task execution condition, the computer room with the largest computer room bandwidth is determined from other computer rooms in the at least one computer room as the target computer room.

5. A cross-computer room cluster control device, characterized in that: The device comprises: The network sharing module is used to: set a wavelength division multiplexing device in each of the at least one computer room, and the wavelength division multiplexing device set in each of the at least one computer room forms a star network topology structure with a two-layer architecture and double uplink load balancing, so that the network of each of the at least one computer room is shared through the wavelength division multiplexing device; The task receiving module is used to: receive at least one target task sent by the at least one computer room; A computer room determination module, used to: determine a target computer room corresponding to each target task in the at least one target task according to task information contained in each target task in the at least one target task; the task information includes an identifier of a computer room where task input data is located, an identifier of a computer room where task output data is located, and a task type; A task allocation module, used to: allocate each target task in the at least one target task to the corresponding target computer room, so that each target task in the at least one target task is executed in the corresponding target computer room; The shared space and computing resource module is used to: receive heartbeat information, data storage information and computing resource information sent by each computer room in at least one computer room, determine the data storage status and computing resource usage of each computer room in the at least one computer room; and establish a shared storage space and a shared computing resource queue based on the data storage status and computing resource usage of each computer room in the at least one computer room.

6. The cross-machine room cluster control device according to claim 5, characterized in that: The device also includes: The storage space processing module is used to define the same name space for data in the same computer room, and allocate the data in the shared storage space to each computer room in the at least one computer room for storage according to a set data allocation strategy. The set data allocation strategy includes: data service type, data location information and data field.

7. The cross-machine room cluster control device according to claim 5, characterized in that: The shared space and computing resource module is specifically used to: based on the computing resource usage of each computer room in the at least one computer room, re-divide the sum of the computing resources of each computer room in the at least one computer room into a plurality of computing resource blocks according to a set resource block size; At least one column of computing resource queue is established using the multiple computing resource blocks.

8. The cross-machine room cluster control device according to claim 5, characterized in that: The task allocation module is specifically used for: Determine, according to the task information included in the target task, a computer room where target task input data corresponding to the target task is located and a computer room where target task output data is located; When the computer room where the target task input data is located meets the task execution condition, the computer room where the target task input data is located is determined as the target computer room, wherein the task execution condition includes at least one of the following: the bandwidth of the computer room is greater than a set bandwidth threshold, the computing resources of the computer room are greater than a set computing resource threshold, and the storage space of the computer room is greater than a set storage space threshold; When the computer room where the target task output data is located meets the task execution condition, determining the computer room where the target task output data is located as the target computer room; When the computer room where the target task input data is located and the computer room where the target task output data is located both meet the task execution condition, compare the computer room computing resources of the computer room where the target task input data is located with the computer room computing resources of the computer room where the target task output data is located, and determine the computer room with greater computer room computing resources as the target computer room; When the computer room where the target task input data is located and the computer room where the target task output data is located do not meet the task execution condition, the computer room with the largest computer room bandwidth is determined from other computer rooms in the at least one computer room as the target computer room.

9. A cross-computer room cluster control system, characterized in that: The system comprises: at least one computer room and a cross-computer room cluster control device as claimed in any one of claims 5 to 8; Each of the at least one computer room is configured to: receive a target task and send the target task to the cross-computer room cluster control device; The cross-computer room cluster control device includes: The network sharing module is used to: set a wavelength division multiplexing device in each of the at least one computer room, and the wavelength division multiplexing device set in each of the at least one computer room forms a star network topology structure with a two-layer architecture and double uplink load balancing, so that the network of each of the at least one computer room is shared through the wavelength division multiplexing device; The task receiving module is used to: receive at least one target task sent by the at least one computer room; A computer room determination module, used to: determine a target computer room corresponding to each target task in the at least one target task according to task information contained in each target task in the at least one target task; the task information includes an identifier of a computer room where task input data is located, an identifier of a computer room where task output data is located, and a task type; A task allocation module, used to: allocate each target task in the at least one target task to the corresponding target computer room, so that each target task in the at least one target task is executed in the corresponding target computer room; The shared space and computing resource module is used to: receive heartbeat information, data storage information and computing resource information sent by each computer room in at least one computer room, determine the data storage status and computing resource usage of each computer room in the at least one computer room; and establish a shared storage space and a shared computing resource queue based on the data storage status and computing resource usage of each computer room in the at least one computer room.

10. A computer storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images