Big data processing analysis cluster dynamic load balancing method and device based on cloud architecture and medium

Abstract

The invention discloses a dynamic load balancing method and device for a big data processing analysis cluster based on a cloud architecture and a medium, belongs to the technical field of cloud computing, and aims to solve the technical problem that the cluster cannot dynamically add or delete computing and storage resources in real time during each data processing analysis in the past. According to the technical scheme, the method comprises the steps that a cloud platform based on OpenStack is established, a control node is additionally erected, when a user needs MapReduce service, a pre-installed computing environment and specific configuration in the cloud platform are selected, a request is sent to the control node, a virtual host is erected through the cloud platform virtualization technology, a system mirror image where Spark is installed is started, and the MapReduce service is established. A control node sends a command, uploads a configuration file and starts a Spark service, information is interacted through an internal network of the cloud platform, starting of the Spark platform is completed, and then a MapReduce computing service can be provided.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a cloud architecture-based big data processing and analysis cluster dynamic load balancing method, device and medium. Background Art

[0002] MapReduce is a parallel programming model of the Spark distributed computing platform. It can run on a large number of PC computer nodes to form a distributed computing cluster. However, this method lacks flexibility and security. At this time, the cloud platform becomes the best choice: deploy the MapReduce service on the cloud platform and integrate it into a convenient and secure system.

[0003] Currently, there is a new technology that can provide MapReduce services on the cloud platform, Amazon Hadoop managed service operation architecture (Amazon Elastic MapReduce, hereinafter referred to as EMR). EMR provides easy-to-use and easy-to-expand MapReduce services. However, EMR over-encapsulates some services, reducing user availability and service reuse, resulting in resource waste and low flexibility. Therefore, there are many disadvantages such as too few user-controllable parameters, inability to obtain computing service logs, and the need to manually set configuration files for each run. Summary of the invention

[0004] The technical task of the present invention is to provide a cloud-based big data processing and analysis cluster dynamic load balancing method, equipment and medium to solve the problem that the cluster cannot dynamically and real-time add or delete computing and storage resources during each data processing and analysis.

[0005] The technical task of the present invention is achieved in the following way: a dynamic load balancing method for a big data processing and analysis cluster based on a cloud architecture, which is to build a cloud platform based on OpenStack and set up a control node. When a user needs MapReduce service, he selects the pre-installed computing environment and specific configuration in the cloud platform, sends a request to the control node, uses the cloud platform virtualization technology to set up a virtual host, starts the system image with Spark installed, sends commands through the control node, uploads configuration files, starts the Spark service, and completes the startup of the Spark platform through the internal network interaction of the cloud platform, and then provides MapReduce computing services.

[0006] Preferably, the control node is used to receive user input, send control commands to the cloud platform, and output results.

[0007] Better yet, the OpenStack-based cloud platform is built on the cloud platform physical cluster to provide IaaS services, and then the Spark cluster is automatically built on the cloud platform physical cluster to provide MapReduce distributed computing services.

[0008] Preferably, the cloud platform physical cluster includes keystone nodes, Nova nodes, Glance nodes, and Swift nodes;

[0009] Among them, the Keystone node is used for key verification;

[0010] Nova nodes are computing nodes and controllers for cloud organizations, providing a tool to deploy clouds, including running instances, managing networks, and controlling users;

[0011] Glance nodes provide services for discovering, registering, and obtaining virtual machine images;

[0012] Swift Node is a scalable object storage system.

[0013] Preferably, the MapReduce computing service is started as follows:

[0014] The monitoring node accepts user input and records user requirements, including the number of nodes, CPU, memory, storage space, system environment, and detailed configuration parameters of Spark.

[0015] The monitoring node sends a request to the OpenStack-based cloud platform, applies for virtual machine resources according to user needs, and creates a virtual machine cluster with the Spark environment pre-installed on the OpenStack cloud platform;

[0016] Assign a public IP address to the master node in the virtual machine cluster to facilitate direct access to the master node by system monitoring nodes outside the OpenStack-based cloud platform;

[0017] The monitoring node creates a configuration file directory, generates the configuration files required for Spark startup, and modifies the configuration files based on user input;

[0018] The monitoring node stores a daemon program, which needs to be run in the master node of the virtual machine. The daemon program is used to receive instructions from the monitoring node, and the monitoring node uploads the daemon program to the master node of the virtual machine. The instructions received by the daemon program include creating a Spark cluster and running MapReduce tasks.

[0019] The monitoring node uploads the configured configuration file directory to the master node of the virtual machine to start the Spark service.

[0020] The node starts a daemon program for interacting with the monitoring node.

[0021] Preferably, when the instruction received by the daemon is to create a Spark cluster, the specific instructions are as follows:

[0022] The virtual machine cluster starts the Spark service according to the received configuration file;

[0023] The master node of the virtual machine looks for slave nodes and synchronizes configurations according to the configuration files. The slave nodes include NameNode, DataNode, Jobtracker, and Tasktracker to establish a Spark cluster.

[0024] The Spark cluster formats the HDFS (Hadoop Distributed File System) on which the Spark service depends and completes the startup.

[0025] Preferably, the daemon receives instructions to run a MapReduce task, as follows:

[0026] Start the Spark service;

[0027] Since a large amount of data in the system is stored in the storage node Swift in the cloud platform, and MapReduce only supports access to the file system in HDFS format, a middle layer for reading and writing Swift data is built to read and write the Swift file system;

[0028] Read the data files required for calculation and the algorithms required by the user from the Swift file system, assign them to each node, and start MapReduce calculation;

[0029] The calculation results are saved and returned to the monitoring node, and finally displayed to the user, completing the calculation of the MapReduce service.

[0030] A cloud-based big data processing and analysis cluster dynamic load balancing system, the system includes a control node and a cloud platform built by a physical computer cluster;

[0031] The control node is used to receive user requests, identify the requests, and send instructions to the cloud platform according to the user's request, and further control the nodes of the virtual machine through the cloud platform to build a Spark MapReduce cluster or perform efficient calculations;

[0032] The cloud platform is built on a physical computer cluster, which includes keystone nodes, Nova nodes, Glance nodes and Swift nodes. Keystone nodes are used for key verification. Nova nodes are computing nodes, which are controllers for cloud organizations and provide a tool for deploying clouds, including running instances, managing networks and controlling users. Glance nodes provide virtual machine image discovery, registration and acquisition services. Swift nodes are a scalable object storage system.

[0033] An electronic device comprising: a memory and at least one processor;

[0034] Wherein, the memory stores computer-executable instructions;

[0035] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the above-mentioned cloud-based architecture-based big data processing and analysis cluster dynamic load balancing method.

[0036] A computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the above-mentioned cloud-based architecture-based big data processing and analysis cluster dynamic load balancing method is implemented.

[0037] The cloud-based big data processing and analysis cluster dynamic load balancing method, device and medium of the present invention have the following advantages:

[0038] (1) The present invention runs MapReduce on a cloud platform and uses the characteristics of cloud computing virtualization to provide a flexible, convenient, secure and fast MapReduce cluster service;

[0039] (II) The present invention provides MapReduce services based on a cloud platform. First, a cloud platform based on OpenStack is built, and a control node is set up separately. When a user needs a service, he or she can select a computing environment and a specific configuration pre-installed in the cloud platform, send a request to the control node, use the cloud platform virtualization technology, set up a virtual host, start the system image with Spark installed, send commands through the control node, upload configuration files, start the Spark service, and complete the startup of the Spark platform through the internal network of the cloud platform. Then, MapReduce computing services can be provided;

[0040] (III) The present invention adopts a cloud-based approach to provide Spark services, and builds Spark clusters on computing nodes virtualized by the cloud platform. Users can set the number of startup nodes, CPU, memory, storage space and other parameters as needed, and have the function of temporarily adding or deleting nodes. Compared with building Spark services on physical clusters, cloud-based Spark services are more convenient and flexible. Users can choose computing environments on demand without complicated configurations or changing the physical cluster architecture, and can conveniently control Spark clusters. On the other hand, physical cluster-based Spark services cannot effectively control files of multiple users, which is prone to potential security risks. Cloud-based Spark services use virtual technology to isolate files well, ensure the privacy of files between users, eliminate the risk of illegal access, and improve the security of the system. In general, the reusability of services is improved, and they have strong scalability, high flexibility and high security.

[0041] (IV) The present invention adopts a cloud-based method to build a MapReduce computing cluster. Relying on the virtualization technology of the cloud platform, a MapReduce cluster is built on the virtualized computing nodes, which can make the Spark MapReduce computing cluster more flexible and easy to expand, and improve the security in the case of multiple users;

[0042] (V) The present invention solves the problem that the cluster cannot dynamically and real-time add or delete computing and storage resources each time data processing and analysis is performed in the past, and when resources are added or released, the data processing and analysis tasks can be dynamically and real-time adjusted according to the resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The present invention is further described below in conjunction with the accompanying drawings.

[0044] Attached Figure 1 A flowchart of a dynamic load balancing method for a big data processing and analysis cluster based on a cloud architecture;

[0045] Attached Figure 2 This is a structural diagram of the dynamic load balancing system for big data processing and analysis clusters based on cloud architecture. DETAILED DESCRIPTION

[0046] The following detailed description is made of the cloud architecture-based big data processing and analysis cluster dynamic load balancing method, device and medium of the present invention with reference to the drawings and specific embodiments of the specification.

[0047] Embodiment 1:

[0048] As attached Figure 1As shown, this embodiment provides a method for dynamic load balancing of a big data processing and analysis cluster based on cloud architecture. The method is to build a cloud platform based on OpenStack and set up a control node. When a user needs MapReduce service, the pre-installed computing environment and specific configuration in the cloud platform are selected, and a request is sent to the control node. A virtual host is set up using the cloud platform virtualization technology, and a system image with Spark installed is started. Commands are sent through the control node, configuration files are uploaded, and Spark services are started. Messages are exchanged through the internal network of the cloud platform to complete the startup of the Spark platform, and then MapReduce computing services can be provided.

[0049] The control node in this embodiment is used to receive user input, send control commands to the cloud platform, and output results.

[0050] The OpenStack-based cloud platform in this embodiment is built on a cloud platform physical cluster to provide IaaS services, and then a Spark cluster is automatically built on the cloud platform physical cluster to provide MapReduce distributed computing services.

[0051] The cloud platform physical cluster in this embodiment includes keystone nodes, Nova nodes, Glance nodes, and Swift nodes;

[0052] Among them, the Keystone node is used for key verification;

[0053] Nova nodes are computing nodes and controllers for cloud organizations, providing a tool to deploy clouds, including running instances, managing networks, and controlling users;

[0054] Glance nodes provide services for discovering, registering, and obtaining virtual machine images;

[0055] Swift Node is a scalable object storage system.

[0056] In this embodiment, the MapReduce computing service is started as follows:

[0057] S1. The monitoring node accepts user input and records user requirements, including the number of nodes, CPU, memory, storage space, system environment, and detailed configuration parameters of Spark.

[0058] S2, the monitoring node sends a request to the OpenStack-based cloud platform, applies for virtual machine resources according to user needs, and creates a virtual machine cluster with the Spark environment pre-installed on the OpenStack cloud platform;

[0059] S3. Allocate a public IP address to the master node in the virtual machine cluster to facilitate direct access to the master node by system monitoring nodes outside the OpenStack-based cloud platform;

[0060] S4. The monitoring node creates a configuration file directory, generates the configuration file required for Spark startup, and modifies the configuration file based on user input;

[0061] S5. A daemon is stored in the monitoring node. The daemon needs to be run in the master node of the virtual machine. The daemon is used to receive instructions from the monitoring node. The monitoring node uploads the daemon to the master node of the virtual machine. The instructions received by the daemon include creating a Spark cluster and running MapReduce tasks.

[0062] S6. The monitoring node uploads the configured configuration file directory to the master node of the virtual machine to start the Spark service.

[0063] S7. The node starts a daemon program to interact with the monitoring node.

[0064] In this embodiment, when the instruction received by the daemon is to create a Spark cluster, the details are as follows:

[0065] ① The virtual machine cluster starts the Spark service according to the received configuration file;

[0066] ② The master node of the virtual machine finds the slave nodes and synchronizes the configuration according to the configuration file. The slave nodes include NameNode nodes, DataNode nodes, Jobtracker nodes and Tasktracker nodes to establish a Spark cluster;

[0067] ③The Spark cluster formats the HDFS (Hadoop Distributed File System) on which the Spark service depends and completes the startup.

[0068] In this embodiment, when the instruction received by the daemon is to run a MapReduce task, the specific instructions are as follows:

[0069] ① Start the Spark service;

[0070] ② Since a large amount of data in the system is stored in the storage node Swift in the cloud platform, and MapReduce only supports access to the file system in HDFS format, at this time, a middle layer for reading and writing Swift data is built to read and write the Swift file system;

[0071] ③ Read the data files required for calculation and the algorithms required by the user in the Swift file system, assign them to each node, and start MapReduce calculation;

[0072] ④ Save the calculation results and return them to the monitoring node, and finally display them to the user to complete the calculation of the MapReduce service.

[0073] Embodiment 2:

[0074] As attached Figure 2 As shown, this embodiment provides a cloud-based big data processing and analysis cluster dynamic load balancing system, which includes a control node and a cloud platform built by a physical computer cluster;

[0075] The control node is used to receive user requests, identify the requests, and send instructions to the cloud platform according to the user's request, and further control the nodes of the virtual machine through the cloud platform to build a Spark MapReduce cluster or perform efficient calculations;

[0076] The cloud platform is built on a physical computer cluster, which includes keystone nodes, Nova nodes, Glance nodes and Swift nodes. Keystone nodes are used for key verification. Nova nodes are computing nodes, which are controllers for cloud organizations and provide a tool for deploying clouds, including running instances, managing networks and controlling users. Glance nodes provide virtual machine image discovery, registration and acquisition services. Swift nodes are a scalable object storage system.

[0077] As attached Figure 1 As shown in the figure, the system workflow is demonstrated by taking the completion of the user starting the MapReduce computing service as an example, including the following steps:

[0078] (1) The monitoring node accepts user input and records user requirements, including the number of nodes, CPU, memory, storage space, system environment, and Spark's detailed configuration parameters;

[0079] (2) The monitoring node sends a request to the cloud platform to apply for virtual machine resources according to user needs. The cloud platform creates a virtual machine cluster with the Spark environment pre-installed;

[0080] (3) Allocate a public IP address to the master node in the virtual machine cluster to facilitate direct access to the master node by system monitoring nodes outside the cloud platform;

[0081] (4) The monitoring node creates a configuration file directory, generates the configuration file required for Spark startup, and modifies the configuration file based on user input;

[0082] (5) The monitoring node stores a daemon program, which needs to be run in the master node of the virtual machine. It is responsible for receiving instructions from the monitoring node, such as starting the Spark service, starting MapReduce calculations, etc. At this time, the monitoring node will upload the daemon program to the master node of the virtual machine;

[0083] (6) The monitoring node uploads the configuration file directory set in step (4) to the master node to start the Spark service;

[0084] (7) The master node starts the daemon received in step (5) to interact with the system monitoring node.

[0085] (8) The daemon starts receiving commands, as follows:

[0086] When the command is to create a Spark cluster:

[0087] ① The virtual machine cluster starts the Spark service according to the configuration file received in step (6);

[0088] ② The master node searches for slave nodes and synchronizes configurations based on the configuration files, including NameNode, DataNode, Jobtracker, and Tasktracker, to establish a Spark cluster;

[0089] ③ The cluster formats the HDFS (Hadoop Distributed File System) on which the Spark service depends;

[0090] ④Complete the startup.

[0091] When the command is to run a MapReduce task:

[0092] ① Start the Spark service;

[0093] ② Since a large amount of data in the system is stored in the storage node Swift in the cloud platform, and MapReduce only supports access to the file system in HDFS format, at this time, a middle layer for reading and writing Swift data is built to read and write the Swift file system;

[0094] ③ Read the data files required for calculation and the algorithm required by the user in Swift, assign them to each node, and start MapReduce calculation;

[0095] ④Save the calculation results, return them to the monitoring node, and finally display them to the user;

[0096] ⑤Calculation completed.

[0097] Embodiment 3:

[0098] This embodiment also provides an electronic device, including: a memory and at least one processor;

[0099] Wherein, the memory stores computer-executable instructions;

[0100] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the cloud-based architecture-based big data processing and analysis cluster dynamic load balancing method described in any one of the present inventions.

[0101] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0102] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0103] Embodiment 4:

[0104] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the cloud architecture-based big data processing and analysis cluster dynamic load balancing method in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0105] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0106] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer by a communication network.

[0107] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0108] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cloud-based big data processing and analysis cluster dynamic load balancing method, characterized in that: The method is to build a cloud platform based on OpenStack and set up a control node. When the user needs MapReduce service, he selects the pre-installed computing environment and specific configuration in the cloud platform, sends a request to the control node, uses the cloud platform virtualization technology to set up a virtual host, starts the system image with Spark installed, sends commands through the control node, uploads configuration files, starts the Spark service, and completes the startup of the Spark platform through the internal network of the cloud platform. Then, MapReduce computing services can be provided.

2. The method for dynamic load balancing of a big data processing and analysis cluster based on cloud architecture according to claim 1, characterized in that: The control node is used to receive user input, send control commands to the cloud platform, and output results.

3. The method for dynamic load balancing of a big data processing and analysis cluster based on cloud architecture according to claim 1 or 2, characterized in that: The OpenStack-based cloud platform is built on the cloud platform physical cluster to provide IaaS services, and then automatically builds a Spark cluster on the cloud platform physical cluster to provide MapReduce distributed computing services.

4. The method for dynamic load balancing of a big data processing and analysis cluster based on cloud architecture according to claim 3 is characterized in that: The cloud platform physical cluster includes keystone nodes, Nova nodes, Glance nodes, and Swift nodes; Among them, the Keystone node is used for key verification; Nova nodes are computing nodes and controllers for cloud organizations, providing a tool to deploy clouds, including running instances, managing networks, and controlling users; Glance nodes provide services for discovering, registering, and obtaining virtual machine images; Swift Node is a scalable object storage system.

5. The method for dynamic load balancing of a big data processing and analysis cluster based on cloud architecture according to claim 4 is characterized in that: The MapReduce computing service startup is as follows: The monitoring node accepts user input and records user requirements, including the number of nodes, CPU, memory, storage space, system environment, and detailed configuration parameters of Spark. The monitoring node sends a request to the OpenStack-based cloud platform, applies for virtual machine resources according to user needs, and creates a virtual machine cluster with the Spark environment pre-installed on the OpenStack cloud platform; Assign a public IP address to the master node in the virtual machine cluster to facilitate direct access to the master node by system monitoring nodes outside the OpenStack-based cloud platform; The monitoring node creates a configuration file directory, generates the configuration files required for Spark startup, and modifies the configuration files based on user input; The monitoring node stores a daemon program, which needs to be run in the master node of the virtual machine. The daemon program is used to receive instructions from the monitoring node, and the monitoring node uploads the daemon program to the master node of the virtual machine. The instructions received by the daemon program include creating a Spark cluster and running MapReduce tasks. The monitoring node uploads the configured configuration file directory to the master node of the virtual machine to start the Spark service. The node starts a daemon program for interacting with the monitoring node.

6. The method for dynamic load balancing of a big data processing and analysis cluster based on cloud architecture according to claim 5, characterized in that: When the daemon receives the instruction to create a Spark cluster, the details are as follows: The virtual machine cluster starts the Spark service according to the received configuration file; The master node of the virtual machine looks for slave nodes and synchronizes configurations according to the configuration files. The slave nodes include NameNode, DataNode, Jobtracker, and Tasktracker to establish a Spark cluster. The Spark cluster formats the HDFS on which the Spark service depends and starts up.

7. The cloud-based big data processing and analysis cluster dynamic load balancing method according to claim 6, characterized in that: The instructions received by the daemon are as follows when running a MapReduce task: Start the Spark service; Build an intermediate layer for reading and writing Swift data, which is used to read and write the Swift file system; Read the data files required for calculation and the algorithms required by the user from the Swift file system, assign them to each node, and start MapReduce calculation; The calculation results are saved and returned to the monitoring node, and finally displayed to the user, completing the calculation of the MapReduce service.

8. A cloud-based big data processing and analysis cluster dynamic load balancing system, characterized in that: The system includes control nodes and a cloud platform built by a physical computer cluster; The control node is used to receive user requests, identify the requests, and send instructions to the cloud platform according to the user's request, and further control the nodes of the virtual machine through the cloud platform to build a Spark MapReduce cluster or perform efficient calculations; The cloud platform is built on a physical computer cluster, which includes keystone nodes, Nova nodes, Glance nodes and Swift nodes. Keystone nodes are used for key verification. Nova nodes are computing nodes, which are controllers for cloud organizations and provide a tool for deploying clouds, including running instances, managing networks and controlling users. Glance nodes provide virtual machine image discovery, registration and acquisition services. Swift nodes are a scalable object storage system.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the cloud-based architecture-based big data processing and analysis cluster dynamic load balancing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the cloud-based big data processing and analysis cluster dynamic load balancing method as described in any one of claims 1 to 8 is implemented.

Images