A cloud platform-based large-scale parallel processing data node dynamic expansion method

Abstract

The present application relates to a kind of based on cloud platform large-scale parallel processing data node dynamic expansion method, belong to large-scale parallel processing application field.The detector of the present application judges whether to trigger coordinator to complete the dynamic expansion of data node according to the usage of Slave node and threshold value, coordinator reads the expansion parameter (boolean isScaleOut, boolean isScaleCPU, boolean isScaleMem, boolean isScaleDisk) of incoming, when vertical expansion, coordinator calls resource allocation interface to complete the vertical expansion of existing Slave node CPU, memory, disk.When horizontal expansion, Java application calls resource allocation interface to complete new Slave node resource initialization, calls installation script to complete new Slave node initial installation, configuration, so as to realize MPP data node dynamic horizontal expansion.The innovation of the present application lies in that the method of automatic detection and automatic expansion data node is used, the method that MPP is carried out dynamic vertical expansion and dynamic horizontal expansion data node when query response is slow in large data analysis application is realized, and the concurrent processing capacity of MPP and the dynamic scalability of MPP are improved.

Description

Technical Field

[0001] This invention belongs to the field of large-scale parallel processing applications, specifically relating to a method for dynamically expanding data nodes for large-scale parallel processing based on a cloud platform. Background Technology

[0002] With the increasing application of big data technology, the storage and analysis of massive amounts of data has become a hot research topic. Faced with massive amounts of structured data, traditional relational databases struggle to meet computational performance requirements for aggregation queries and multi-dimensional data analysis. Single-machine scale-up quickly reaches its bottleneck, and SMP (Symmetric Multiprocessing) architectures are not easily scalable and cannot meet performance requirements in terms of CPU computation and I / O throughput. This has led to the emergence of large-scale parallel data processing (MLP) technologies. MPP can run on ordinary PCs and supports scale-out horizontal scaling, achieving performance targets for rapid computation of petabyte-scale data through distributed storage and parallel computing on each node. This invention primarily focuses on how to quickly identify MPP bottlenecks and solve performance issues through rapid scale-out in non-internet network environments.

[0003] Currently, in non-Internet network environments, identifying and scaling MPP performance bottlenecks primarily relies on manual data node expansion based on monitoring system alarms or user feedback regarding slow responses. This method requires coordination and collaboration among operations and maintenance personnel, system troubleshooters, resource allocation personnel, and MPP deployment personnel to complete the data node expansion. Since this involves multiple roles, problems at any stage can affect the timeliness of problem resolution and application stability. To address this issue, this invention employs automated detection and automated data node expansion methods. This enables dynamic vertical and horizontal scaling of data nodes in MPP during big data analytics applications when query responses are slow, thereby improving MPP's concurrent processing capabilities and dynamic scalability. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] The technical problem to be solved by this invention is how to provide a method for dynamically expanding data nodes based on a cloud platform for large-scale parallel processing, so as to solve the problem that MPP cannot automatically expand when a bottleneck occurs.

[0006] (II) Technical Solution

[0007] To address the aforementioned technical problems, this invention proposes a method for dynamically expanding data nodes in large-scale parallel processing based on a cloud platform. This method includes the following steps:

[0008] Step 1: The client periodically sends SQL query requests to the management node (Master) in the MPP.

[0009] Step 2: The Master node parses the SQL request to generate an SQL execution plan, which is then sent to all data nodes (Slaves) for parallel execution. Each Slave node executes the SQL and returns the execution results to the Master node. The Master node then aggregates the query results from all Slave nodes and returns the total amount of data in the execution plan and the query response time to the Client.

[0010] Step 3: The Client makes a judgment based on the total amount of data and the query response time. When the response time is greater than the set threshold, the Client activates the detector to perform load testing on the Slave node.

[0011] Step 4: The detector parses the CPU log files, memory log files, and disk log files of each Slave node to obtain the CPU utilization, memory utilization, and disk utilization. If the utilization exceeds the set load threshold, the coordinator is triggered to complete the dynamic expansion of the data nodes.

[0012] Step 5: The coordinator calls the resource allocation interface based on the parameters passed in by the detector to realize the dynamic vertical expansion of the Slave node's CPU, memory, and disk, or calls the resource allocation interface to complete the creation of new Slave resources, initialization of new Slave nodes, and configuration of node interconnection, thereby realizing the horizontal dynamic expansion of MPP data nodes.

[0013] Furthermore, in step one, the client uses JavaDataBase Connectivity (JDBC) technology to connect to the MPP management node (Master), and sends a specific SQL query request to the Master node at preset intervals. The query SQL is to perform conditional filtering, grouping, aggregation, and sorting queries on the data in a single large table, and waits for the Master node to execute and return the total amount of data in the SQL physical execution plan and the response time of the complex single-table query.

[0014] Furthermore, the preset time is 5 minutes.

[0015] Furthermore, in step two, the Master node selects the execution path with the lowest cost based on the global Slave cost optimization strategy and distributes the physical execution plan of each Slave node to the Slave nodes for parallel execution through global node interconnection.

[0016] Furthermore, in step two, the Slave node uses its local computing power to perform calculations on the local data according to the physical execution plan and returns the results to the Master node. The Master node then summarizes the results from all Slave nodes through simple calculations and returns the total amount of data in the physical execution plan and the SQL query response time to the Client.

[0017] Furthermore, in step three, the query response time threshold is 50ms when the total query data volume is less than 100,000, 200ms when the total query data volume is less than 1,000,000 but greater than or equal to 100,000, and 1,000ms when the total query data volume is less than 10,000,000 but greater than or equal to 1,000,000.

[0018] Furthermore, in step four, the detector uses a Java application to read the CPU information from the / proc / stat file of each Slave node, the memory information from the / proc / meminfo file of the Slave node, and the disk information from the / proc / net / dev file of the Slave node, and performs parsing and calculation to obtain the number of CPU cores and CPU utilization, total memory and memory utilization, and total disk capacity and disk utilization.

[0019] Further, in step four, the detector calculates the overall utilization rate by weighting the indicators (overall utilization rate = ((Slave1 CPU utilization rate + Slave2 CPU utilization rate + ... + SlaveN CPU utilization rate) / N*1.2 + (Slave1 memory utilization rate + Slave2 memory utilization rate + ... + SlaveN memory utilization rate) / N*1.2 + (Slave1 disk utilization rate + Slave2 disk utilization rate + ... + SlaveN disk utilization rate) / N*0.6) / 3). When the overall utilization rate indicator exceeds the overall utilization rate, the overall utilization rate is calculated. When the CPU utilization rate reaches the threshold, the coordinator is triggered to complete the horizontal dynamic expansion of the slave node. Otherwise, the individual indicators of the slave are compared. When the number of CPU cores of the slave node is less than 64 and the CPU utilization rate is greater than the CPU utilization rate threshold, the CPU of the slave is expanded. When the total memory of the slave node is less than 512G and the memory utilization rate is greater than the memory utilization rate threshold, the memory of the slave is expanded. When the total storage of the slave is less than 10T and the disk utilization rate is greater than the disk utilization rate threshold, the disk of the slave is expanded. When all individual indicators have been expanded to the maximum resources, the horizontal expansion of the slave node will be carried out.

[0020] Furthermore, in step five, the parameters passed to the detector are (boolean isScaleOut, boolean isScaleCPU, boolean isScaleMem, boolean isScaleDisk).

[0021] Furthermore, during vertical scaling, the isScaleOut parameter is false, and the coordinator calls the resource allocation interface to complete the vertical scaling of the existing Slave node's CPU, memory, and disk. When the isScaleCPU parameter is true, vertical scaling of the CPU is performed; when the isScaleMem parameter is true, vertical scaling of the memory is performed; and when the isScaleDisk parameter is true, vertical scaling of the disk is performed. During horizontal scaling, the isScaleOut parameter is true, and the Java application calls the resource allocation interface to complete the resource initialization of the new Slave node, calls the installation script to complete the initial installation of the new Slave node, and calls the script command to complete the network initialization of the new Slave node and configure node interconnection, thereby realizing dynamic horizontal scaling of MPP data nodes.

[0022] (III) Beneficial Effects

[0023] This invention proposes a method for dynamically scaling data nodes based on massively parallel processing (MPP) on a cloud platform. In one embodiment of this method, the coordinator reads the input scaling parameters (boolean isScaleOut, boolean isScaleCPU, boolean isScaleMem, boolean isScaleDisk). During vertical scaling (isScaleOut is false), the coordinator calls the resource allocation interface to scale the existing Slave node's CPU (isScaleCPU is true), memory (isScaleMem is true), and disk (isScaleDisk is true). During horizontal scaling (isScaleOut is true), the Java application calls the resource allocation interface to initialize the new Slave node's resources, calls the installation script to install the new Slave node, and calls script commands to initialize the new Slave node's network and configure node interconnection, thereby achieving dynamic horizontal scaling of MPP data nodes.

[0024] The innovation of this invention lies in the use of automated detection and automated expansion of data nodes. This enables MPP to dynamically expand vertically and horizontally when query responses are slow in big data analysis applications, thereby improving MPP's concurrent processing capabilities and dynamic scalability. Attached Figure Description

[0025] Figure 1 This is a flowchart of the dynamic expansion method for data nodes based on massively parallel processing (MPP) on a cloud platform according to the present invention.

[0026] Figure 2 This is a flowchart of the MPP server performance bottleneck detection process of the present invention;

[0027] Figure 3 This is a flowchart illustrating the dynamic expansion process of the MPP server in this invention. Detailed Implementation

[0028] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0029] This invention aims to solve the problem of MPP (Multi-Level Processing) nodes failing to automatically expand when a bottleneck occurs, and proposes a method for dynamically detecting bottlenecks and automatically expanding MPP data nodes. The technical solution described in this invention requires the following steps:

[0030] Step 1: The client periodically sends SQL query requests to the management node (Master) in the MPP.

[0031] Step 2: The Master node parses the SQL request to generate an SQL execution plan, which is then sent to all data nodes (Slaves) for parallel execution. Each Slave node executes the SQL and returns the execution results to the Master node. The Master node then aggregates the query results from all Slave nodes and returns the total amount of data in the execution plan and the query response time to the Client.

[0032] Step 3: The Client makes a judgment based on the total amount of data and the query response time. When the response time is greater than the set threshold, the Client activates the detector to perform load testing on the Slave node.

[0033] Step 4: The detector parses the CPU log files, memory log files, and disk log files of each Slave node to obtain the CPU utilization, memory utilization, and disk utilization. If the utilization exceeds the set load threshold, the coordinator is triggered to complete the dynamic expansion of the data nodes.

[0034] Step 5: The coordinator calls the resource allocation interface based on the parameters passed in by the detector to realize the dynamic vertical expansion of the Slave node's CPU, memory, and disk, or calls the resource allocation interface to complete the creation of new Slave resources, initialization of new Slave nodes, and configuration of node interconnection, thereby realizing the horizontal dynamic expansion of MPP data nodes.

[0035] In one embodiment of the present invention, a method for dynamically expanding data nodes based on massively parallel processing (MPP) on a cloud platform, the client uses Java Database Connectivity (JDBC) technology to connect to the MPP management node (Master). At preset intervals, the client sends specific SQL query requests to the Master node. The query SQL involves conditional filtering, grouping, aggregation, and sorting of data in a single large table. The client waits for the Master node to execute and return the total amount of data in the SQL physical execution plan and the response time for the complex single-table query. The preset interval is 5 minutes.

[0036] In one embodiment of the present invention, a method for dynamic scaling of data nodes based on massively parallel processing (MPP) on a cloud platform, the Master node selects the execution path with the lowest cost for each Slave node based on a global Slave cost optimization strategy. The physical execution plan is distributed to the Slave nodes in parallel through global node interconnection. The Slave nodes use their local computing power to perform calculations on local data according to the physical execution plan and return the results to the Master node. The Master node summarizes the results of all Slave nodes through simple calculations and returns the total amount of data in the physical execution plan and the SQL query response time to the Client.

[0037] In one embodiment of the present invention, a method for dynamically expanding data nodes based on massively parallel processing (MPP) on a cloud platform, the Client receives the total amount of data and the SQL query response time returned by the Master node, and makes a judgment based on the query data amount and response threshold. When the response time of the total query data amount is greater than the response threshold time, the detector is called to perform load detection on the Slave node.

[0038] Table 1: Comparison of Query Data Volume and Response Threshold

[0039]

[0040]

[0041] In one embodiment of the present invention, a method for dynamically expanding data nodes based on massively parallel processing (MPP) on a cloud platform, the detector uses a Java application to read CPU information from the / proc / stat file of each Slave node, memory information from the / proc / meminfo file of the Slave node, and disk information from the / proc / net / dev file of the Slave node. The detector then performs parsing and calculations to obtain the number of CPU cores and CPU utilization, total memory and memory utilization, and total disk capacity and disk utilization. The overall utilization rate is obtained by weighting the above indicators (Overall utilization rate = ((Slave1 CPU utilization rate + Slave2 CPU utilization rate + ... + SlaveN CPU utilization rate) / N * 1.2 + (Slave1 memory utilization rate + Slave2 memory utilization rate)). The formula is: (SlaveN memory utilization + ... + SlaveN disk utilization) / N * 1.2 + (Slave1 disk utilization + Slave2 disk utilization + ... + SlaveN disk utilization) / N * 0.6) / 3). When the overall utilization rate exceeds the overall utilization rate threshold, the coordinator is triggered to complete the horizontal dynamic expansion of the Slave nodes. Otherwise, individual Slave metrics are compared. If the number of CPU cores on the Slave node is less than 64 and the CPU utilization rate is greater than the CPU utilization rate threshold, the Slave's CPU is expanded. If the total memory on the Slave node is less than 512GB and the memory utilization rate is greater than the memory utilization rate threshold, the Slave's memory is expanded. If the total storage on the Slave node is less than 10TB and the disk utilization rate is greater than the disk utilization rate threshold, the Slave's disk is expanded. When all individual metrics have been expanded to the maximum resource level, horizontal expansion of the Slave nodes will be performed. The resource utilization rate threshold table is as follows.

[0042] Table 2 Resource Usage Threshold Comparison Table

[0043]

[0044] In a method for dynamic scaling of data nodes based on massively parallel processing (MPP) on a cloud platform according to the present invention, the coordinator reads the scaling parameters (boolean isScaleOut, boolean isScaleCPU, boolean isScaleMem, boolean isScaleDisk) passed in by the detector. When scaling vertically (isScaleOut parameter value is false), the coordinator calls the resource allocation interface to complete the vertical scaling of the CPU (isScaleCPU parameter value is true), memory (isScaleMem parameter value is true), and disk (isScaleDisk parameter value is true) of the existing Slave node. When scaling horizontally (isScaleOut parameter value is true), the Java application calls the resource allocation interface to complete the resource initialization of the new Slave node, calls the installation script to complete the initial installation of the new Slave node, and calls the script command to complete the network initialization of the new Slave node and configure the node interconnection, thereby realizing the dynamic horizontal scaling of MPP data nodes.

[0045] The innovation of this invention lies in the use of automated detection and automated expansion of data nodes. This enables MPP to dynamically expand vertically and horizontally when query responses are slow in big data analysis applications, thereby improving MPP's concurrent processing capabilities and dynamic scalability.

[0046] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for dynamically expanding data nodes based on massively parallel processing on a cloud platform, characterized in that, The method includes the following steps: Step 1: The client periodically sends SQL query requests to the management node Master in the MPP. Step 2: The Master node parses the SQL request to generate an SQL execution plan, which is then sent to all Slave data nodes for parallel execution. Each Slave node executes the SQL and returns the execution results to the Master node. The Master node then aggregates the query results from all Slave nodes and returns the total amount of data in the execution plan and the query response time to the Client. Step 3: The Client makes a judgment based on the total amount of data and the query response time. When the response time is greater than the set threshold, the Client activates the detector to perform load testing on the Slave node. Step 4: The detector parses the CPU log files, memory log files, and disk log files of each Slave node to obtain the CPU utilization, memory utilization, and disk utilization. If the utilization exceeds the set load threshold, the coordinator is triggered to complete the dynamic expansion of the data nodes. Step 5: The coordinator calls the resource allocation interface based on the parameters passed in by the detector to realize the dynamic vertical expansion of the Slave node's CPU, memory, and disk, or calls the resource allocation interface to complete the creation of new Slave resources, initialization of new Slave nodes, and configuration of node interconnection, thereby realizing the horizontal dynamic expansion of MPP data nodes. in, In step four, the detector calculates the overall utilization rate by weighting the indicators: Overall Utilization Rate = ((Slave1 CPU Utilization Rate + Slave2 CPU Utilization Rate + ... + SlaveN CPU Utilization Rate) / N 1.2 + (Slave1 memory usage + Slave2 memory usage + ... + SlaveN memory usage) / N 1.2 + (Slave1 disk utilization + Slave2 disk utilization + ... + SlaveN disk utilization) / N 0.6) / 3, when the overall utilization rate exceeds the overall utilization rate threshold, the coordinator is triggered to complete the horizontal dynamic expansion of the Slave node. Otherwise, the individual indicators of the Slave are compared. When the number of CPU cores of the Slave node is less than 64 and the CPU utilization rate is greater than the CPU utilization rate threshold, the CPU of the Slave is expanded. When the total memory of the Slave node is less than 512G and the memory utilization rate is greater than the memory utilization rate threshold, the memory of the Slave is expanded. When the total storage of the Slave is less than 10T and the disk utilization rate is greater than the disk utilization rate threshold, the disk of the Slave is expanded. When all individual indicators have been expanded to the maximum resources, the horizontal expansion of the Slave node will be carried out.

2. The method for dynamically expanding data nodes based on massively parallel processing on a cloud platform as described in claim 1, characterized in that, In step one, the client uses JDBC technology to connect to the MPP management node Master and sends an SQL query request to the Master node at preset intervals. The query SQL is to perform conditional filtering, grouping, aggregation, and sorting queries on the data in a single large table. The client waits for the Master node to execute and return the total amount of data in the SQL physical execution plan and the response time of the complex single-table query.

3. The method for dynamically expanding data nodes based on massive parallel processing on a cloud platform as described in claim 2, characterized in that, The preset time is 5 minutes.

4. The method for dynamically expanding data nodes based on massive parallel processing on a cloud platform as described in claim 1, characterized in that, In step two, the Master node selects the execution path with the lowest cost based on the global Slave cost optimization strategy, and distributes the physical execution plan of each Slave node to the Slave nodes in parallel through global node interconnection for parallel execution.

5. The method for dynamically expanding data nodes based on massive parallel processing on a cloud platform as described in claim 4, characterized in that, In step two, the Slave node uses its local computing power to perform calculations on the local data according to the physical execution plan and returns the results to the Master node. The Master node then performs simple calculations to summarize the results from all Slave nodes and returns the total amount of data in the physical execution plan and the SQL query response time to the Client.

6. The method for dynamically expanding data nodes based on massively parallel processing on a cloud platform as described in claim 1, characterized in that, In step three, the query response time threshold is 50ms when the total query data volume is less than 100,000, 200ms when the total query data volume is less than 1,000,000 but greater than or equal to 100,000, and 1,000ms when the total query data volume is less than 10,000,000 but greater than or equal to 1,000,000.

7. The method for dynamically expanding data nodes based on massively parallel processing on a cloud platform as described in any one of claims 1-6, characterized in that, In step four, the detector uses a Java application to read the CPU information from the / proc / stat file of each Slave node, the memory information from the / proc / meminfo file of the Slave node, and the disk information from the / proc / net / dev file of the Slave node. The detector then performs parsing and calculation to obtain the number of CPU cores and CPU utilization, total memory and memory utilization, and total disk capacity and disk utilization.

8. The method for dynamically expanding data nodes based on massive parallel processing on a cloud platform as described in claim 7, characterized in that, In step five, the parameters passed to the detector are (boolean isScaleOut, boolean isScaleCPU, boolean isScaleMem, boolean isScaleDisk).

9. The method for dynamically expanding data nodes based on massive parallel processing on a cloud platform as described in claim 8, characterized in that, When scaling vertically, if the isScaleOut parameter is false, the coordinator calls the resource allocation interface to scale the CPU, memory, and disk of the existing slave node vertically. If the isScaleCPU parameter is true, CPU scaling is performed vertically; if the isScaleMem parameter is true, memory scaling is performed vertically; and if the isScaleDisk parameter is true, disk scaling is performed vertically. When scaling horizontally, if the isScaleOut parameter is true, the Java application calls the resource allocation interface to initialize the resources of the new slave node, calls the installation script to complete the initial installation of the new slave node, and calls script commands to initialize the network of the new slave node and configure node interconnection, thereby realizing dynamic horizontal scaling of MPP data nodes.

Images