Distributed data processing method

Abstract

The invention relates to the technical field of data processing, and particularly discloses a distributed data processing method which comprises the steps of establishing a distributed node network, monitoring node states, analyzing node operation loads, judging node load states, judging data processing priorities, distributing data processing tasks and optimizing distributed processing nodes. A distributed node network is established, an area covered by a data processing system is divided into a plurality of processing areas, and running state parameters of nodes on servers in the processing areas are monitored in real time and comprise the CPU utilization rate, the memory utilization rate, the network bandwidth occupancy rate and the disk space utilization rate. According to the method, the fault or performance problem of the node can be found and processed in time, and meanwhile, through a refined task allocation and node optimization strategy, each server resource in the distributed processing area can be efficiently utilized to a great extent, so that the priority processing of a core task and the stable operation of the overall system are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a distributed data processing method. Background Art

[0002] Data processing is generally divided into centralized data processing and distributed data processing. The biggest problem with centralized data processing is that the data processing speed is slow, and at the same time, the performance requirements of the computer are relatively high. As the performance of personal computers has been greatly improved and their use has become more and more popular, it has become possible to distribute processing power to all computers on the network. Distributed computing is a concept opposite to centralized computing. The data of distributed computing can be distributed over a large area.

[0003] There are several common problems with existing data processing systems: First, resource utilization is low, with some nodes being in a low-load state for a long time, while other nodes are overloaded, resulting in low processing efficiency; second, task allocation lacks intelligence and cannot be reasonably allocated based on multiple factors such as task priority, node load status, and data characteristics, resulting in delays in the processing of important tasks and affecting the overall system performance; third, node failures or load fluctuations are handled passively, and there is a lack of effective monitoring and early warning mechanisms. Once a problem occurs, it is difficult to adjust in time, which may lead to data processing interruptions or a significant drop in efficiency. Summary of the invention

[0004] The object of the present invention is to provide a distributed data processing method to solve the above-mentioned background problems.

[0005] The object of the present invention can be achieved by the following technical solution: A distributed data processing method comprises the following steps:

[0006] Step 1: Establish a distributed node network: Divide the area covered by the data processing system into several processing areas, and set up several servers inside each processing area, and deploy each distributed node on the corresponding server in each processing area;

[0007] Step 2: Node status monitoring: Real-time monitoring of the running status of each node on each server corresponding to each distributed processing area is performed to obtain the running status parameters of each node on each server corresponding to each distributed processing area;

[0008] Step 3: Node operation load analysis: Comprehensively calculate and analyze the CPU usage, memory usage, network bandwidth usage, and disk space usage of each node on each server in each distributed processing area to obtain a comprehensive load evaluation coefficient for each node on each server in each distributed processing area.

[0009] Step 4, node load status determination: obtain the comprehensive load evaluation coefficient classification threshold of each node on each server in each distributed processing area from the system database, and compare and analyze the comprehensive load evaluation coefficient of each node on each server in each distributed processing area and the comprehensive load evaluation coefficient classification threshold of each node on each server in each distributed processing area;

[0010] Step 5: Data processing priority determination: extract the data features to be processed, combine the data features to be processed and the load status of each node on each server in each distributed processing area into a mapping table, assign a task allocation priority to each combination, and formulate a decision rule for determining the task allocation priority;

[0011] Step 6: Data processing task allocation: high-priority tasks are allocated to low-load nodes on the corresponding servers in each distributed processing area. Under the premise of ensuring that high-priority tasks are processed first, medium-priority tasks are allocated to nodes on the corresponding servers in each distributed processing area. Finally, low-priority tasks are scheduled for processing during system idle time.

[0012] Step 7, distributed processing node optimization: Real-time monitoring and acquisition of the comprehensive load assessment coefficient of each node on each server in each distributed processing area, and dividing the monitoring cycle into several monitoring periods. When a node on each server in each distributed processing area frequently experiences delays or excessive load, a load abnormality signal is generated and the task allocation weight of the node is adjusted.

[0013] Preferably, in step 1, the nodes on the corresponding servers in the distributed processing areas are interconnected via high-speed Ethernet, optical fiber network and dedicated network.

[0014] Preferably, in step 2, the operating status parameters of each node on each server corresponding to each distributed processing area include CPU usage, memory usage, network bandwidth usage and disk space usage.

[0015] Preferably, in step 3, the specific method of comprehensively calculating and analyzing the CPU usage, memory usage, network bandwidth usage, and disk space usage of each node on each server corresponding to each distributed processing area is as follows:

[0016] Get the m CPU usage rates of each node on each server corresponding to each distributed processing area from time point t1 to time point t2, sum them up and divide them by m to get the mean CPU usage rate of each node on each server corresponding to each distributed processing area. According to the mean CPU usage rate of each node on each server corresponding to each distributed processing area, use the variance calculation formula to calculate the variance of CPU usage rate of each node on each server corresponding to each distributed processing area. Then calculate the square root of the variance of CPU usage rate of each node on each server corresponding to each distributed processing area to get the standard deviation of CPU usage rate of each node on each server corresponding to each distributed processing area. Finally, divide the result of subtracting the CPU usage rate of each node on each server corresponding to each distributed processing area from the mean CPU usage rate by the standard deviation of CPU usage rate to get the standard value of CPU usage rate of each node on each server corresponding to each distributed processing area, which is recorded as CN i , i represents the number of each node, i = 1, 2, ..., n, n represents the total number of node numbers;

[0017] Similarly, the standard values ​​of memory usage, network bandwidth usage, and disk space usage of each node on each server in each distributed processing area are calculated and recorded as MN i NN i and DN i ;

[0018] According to the formula Calculate the comprehensive evaluation value of each node on each server in each distributed processing area a1, a2, a3, and a4 represent the weight factors corresponding to the set CPU usage, memory usage, network bandwidth usage, and disk space usage, respectively;

[0019] According to the formula Calculate the CPU usage index value of each node on each server in each distributed processing area Where α represents the set attenuation factor, 0<α<1. Similarly, the index values ​​corresponding to the memory usage rate, network bandwidth usage rate and disk space usage rate of each node on each server in each distributed processing area are calculated and recorded as and

[0020] By comprehensively calculating the standard values ​​corresponding to the CPU utilization, memory utilization, network bandwidth occupancy and disk space utilization of each node on each server corresponding to each distributed processing area and the comprehensive evaluation value of each node on each server corresponding to each distributed processing area, the comprehensive load evaluation coefficient of each node on each server corresponding to each distributed processing area is obtained.

[0021] Preferably, the comprehensive load evaluation coefficient of each node on each server in each distributed processing area is specifically calculated and analyzed as follows:

[0022] According to the formula Calculate the comprehensive load assessment coefficient γ of each node on each server in each distributed processing area, where β is represented by the set adjustment factor, 0<β<1, and w1, w2, w3, and w4 are the weight factors corresponding to the set standard values ​​of CPU utilization, memory utilization, network bandwidth utilization, and disk space utilization, respectively.

[0023] Preferably, in step 4, the comprehensive load evaluation coefficients of each node on each server in each distributed processing area and the classification thresholds of the comprehensive load evaluation coefficients of each node on each server in each distributed processing area are compared and analyzed. The specific comparative analysis method is as follows:

[0024] Obtain the classification thresholds of the comprehensive load evaluation coefficients of each node on each server corresponding to each distributed processing area from the system database, which are denoted as k1 and k2, respectively, where 0<k1<k2<1;

[0025] If γ < k1, it means that the node has sufficient resources to process new tasks and is recorded as a low-load node;

[0026] If k1<γ<k2, it means that the resource usage of the node is relatively reasonable, and it is recorded as a medium-load node;

[0027] If γ>k2, it means that the node resources are close to saturation and it is recorded as a high-load node.

[0028] Preferably, in step five, the data characteristics include data type, data size, data timeliness requirement and data processing complexity.

[0029] Preferably, in step seven, by obtaining the number of nodes on the corresponding servers in each distributed processing area within a monitoring period that have load abnormality signals, denoted as g, the number of nodes g on the corresponding servers in each distributed processing area within a monitoring period that have load abnormality signals and the number n of nodes on the corresponding servers in each distributed processing area are compared and calculated. If the calculation result is greater than 0.8, it means that the load margin of each node on the corresponding servers in each distributed processing area within the monitoring period is insufficient, and a node expansion optimization signal is generated. The processing system increases the number of nodes on the corresponding servers in each distributed processing area within the monitoring period by 50% of the current number of nodes. After the new node joins the system, the load balance is automatically adjusted through the load balancer to evenly distribute the data processing tasks to each node. If the calculation result is less than 0.4, it means that the load margin of each node on the corresponding servers in each distributed processing area within the monitoring period is sufficient, and a node contraction optimization signal is generated. The processing system reduces the number of nodes on the corresponding servers in each distributed processing area within the monitoring period by 20% of the current number of nodes.

[0030] Beneficial effects of the present invention:

[0031] The present invention divides the area covered by the data processing system into multiple processing areas by establishing a distributed node network, and deploys different types of servers in each processing area, including physical servers, virtual servers and cloud servers. These servers are interconnected through high-speed Ethernet, optical fiber networks and dedicated networks, so that even if the network in a certain area fails or suffers a natural disaster, the nodes in other areas can still work normally, ensuring the continuity of data processing and improving the reliability and fault tolerance of the system. At the same time, by real-time monitoring of the operating status parameters of the nodes on each server in each processing area, including CPU utilization, memory utilization, network bandwidth occupancy and disk space utilization, node failures or performance problems can be discovered and handled in a timely manner, thereby further optimizing the operating efficiency and stability of the system.

[0032] The present invention can obtain the comprehensive load evaluation coefficient of each node by comprehensively calculating and analyzing the CPU utilization rate, memory utilization rate, network bandwidth occupancy rate and disk space utilization rate of each node on each server in each distributed processing area; further, by calculating the mean, variance and standard deviation of these resource utilization rates of each node, the standard value of each resource utilization rate of each node can be calculated; then based on these standard values ​​and weight factors, the index value and comprehensive evaluation value of each resource utilization rate of each node are calculated; finally, by comprehensively calculating the standard value, comprehensive evaluation value and adjustment factor of each resource utilization rate of each node, the comprehensive load evaluation coefficient of each node can be obtained; this coefficient can help the system accurately identify the load state of each node, and each node can be divided into low-load nodes, medium-load nodes and high-load nodes, wherein the low-load node indicates that the node has sufficient resources to process tasks and the system performance is good, the medium-load node indicates that the node resource usage is relatively reasonable but the performance change needs to be paid attention to, and the high-load node indicates that the node resources are close to saturation, the system performance may be affected, and measures need to be taken to reduce the load, thereby realizing effective management and optimization of the node operation load.

[0033] The present invention comprehensively considers the data type, data size, data timeliness requirement and data processing complexity of the data to be processed, and combines it with the real-time load status of each node on each server corresponding to each processing area to generate a mapping table, and determines the priority of task allocation based on this table. The system pays special attention to the task allocation of high-timeliness data, and gives higher priority regardless of its data characteristics and node load status, or gives high priority processing when processing high-complexity data under low load, and adjusts the priority when the load is high. At the same time, by collecting actual effect data such as task processing time and node load changes in real time, the mapping table and decision rules are dynamically optimized to ensure effective use of resources, so as to achieve more efficient distributed data processing, ensure the quality and speed of data processing, and improve the operating efficiency and response speed of the entire system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present invention will be further described below in conjunction with the accompanying drawings.

[0035] Figure 1 It is a flow chart of the distributed data processing method of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] See also Figure 1 As shown, the present invention is a distributed data processing method, comprising the following steps:

[0038] Step 1: Establish a distributed node network: Divide the area covered by the data processing system into several processing areas, and set up several servers inside each processing area. Deploy each distributed node on the corresponding server in each processing area, and connect each node to each other through high-speed Ethernet, optical fiber network and dedicated network. By deploying distributed nodes in different geographical locations, if the network in one area fails or suffers from natural disasters, the nodes in other areas can still work normally, which can ensure the continuity of data processing and effectively improve the reliability and fault tolerance of the system.

[0039] It should be noted that servers include physical servers, virtual servers, and cloud servers. According to actual needs and budget, you can choose the most suitable server type. By setting up different types of servers in each processing area, physical servers have higher performance and stability, and virtual servers and cloud servers have higher flexibility and scalability. Using a variety of different types of servers can improve data processing efficiency to a certain extent.

[0040] Step 2, node status monitoring: real-time monitoring of the running status of each node on each server corresponding to each distributed processing area is performed to obtain the running status parameters of each node on each server corresponding to each distributed processing area. The running status parameters of each node on each server corresponding to each distributed processing area include CPU usage, memory usage, network bandwidth usage and disk space usage;

[0041] It should be noted that by real-time monitoring of the operating status of each node on the corresponding server in each distributed processing area, node failures or performance problems can be discovered in a timely manner, and corresponding measures can be taken to deal with them, such as reallocating tasks, repairing faulty nodes, etc.

[0042] In a specific embodiment, the present invention divides the area covered by the data processing system into multiple processing areas by establishing a distributed node network, and deploys different types of servers in each processing area, including physical servers, virtual servers and cloud servers. These servers are interconnected through high-speed Ethernet, fiber optic networks and dedicated networks, so that even if the network in a certain area fails or suffers a natural disaster, the nodes in other areas can still work normally, ensuring the continuity of data processing and improving the reliability and fault tolerance of the system. At the same time, by real-time monitoring of the operating status parameters of the nodes on each server in each processing area, including CPU utilization, memory utilization, network bandwidth occupancy and disk space utilization, node failures or performance problems can be discovered and handled in a timely manner, thereby further optimizing the operating efficiency and stability of the system.

[0043] Step 3, node operation load analysis: extract the values ​​of CPU usage, memory usage, network bandwidth occupancy and disk space usage from the operation status parameters of each node on each server corresponding to each distributed processing area, and obtain the CPU usage, memory usage, network bandwidth occupancy and disk space usage of each node on each server corresponding to each distributed processing area; by comprehensively calculating and analyzing the CPU usage, memory usage, network bandwidth occupancy and disk space usage of each node on each server corresponding to each distributed processing area, obtain the comprehensive load evaluation coefficient of each node on each server corresponding to each distributed processing area;

[0044] Furthermore, the comprehensive load evaluation coefficient of each node on each server in each distributed processing area is calculated and analyzed as follows:

[0045] Get the m CPU usage rates of each node on each server corresponding to each distributed processing area from time point t1 to time point t2, sum them up and divide them by m to get the mean CPU usage rate of each node on each server corresponding to each distributed processing area. According to the mean CPU usage rate of each node on each server corresponding to each distributed processing area, use the variance calculation formula to calculate the variance of CPU usage rate of each node on each server corresponding to each distributed processing area. Then calculate the square root of the variance of CPU usage rate of each node on each server corresponding to each distributed processing area to get the standard deviation of CPU usage rate of each node on each server corresponding to each distributed processing area. Finally, divide the result of subtracting the CPU usage rate of each node on each server corresponding to each distributed processing area from the mean CPU usage rate by the standard deviation of CPU usage rate to get the standard value of CPU usage rate of each node on each server corresponding to each distributed processing area, which is recorded as CN i , i represents the number of each node, i = 1, 2, ..., n, n represents the total number of node numbers;

[0046] Similarly, the standard values ​​of memory usage, network bandwidth usage, and disk space usage of each node on each server in each distributed processing area are calculated and recorded as MN i NN i and DN i ;

[0047] It should be noted that, for the calculation of the CPU usage variance of each node on each server corresponding to each distributed processing area, the difference between each time point and the CPU usage mean is calculated, the difference is squared, and the squared difference results of each time point are summed and divided by m to obtain the CPU usage variance of each node on each server corresponding to each distributed processing area. The variances of memory usage, network bandwidth usage, and disk space usage of each node on each server corresponding to each distributed processing area are calculated in the same way.

[0048] According to the formula Calculate the comprehensive evaluation value of each node on each server in each distributed processing area a1, a2, a3, and a4 represent the weight factors corresponding to the set CPU usage, memory usage, network bandwidth usage, and disk space usage, respectively;

[0049] According to the formula Calculate the CPU usage index value of each node on each server in each distributed processing area Where α represents the set attenuation factor, 0<α<1. Similarly, the index values ​​corresponding to the memory usage rate, network bandwidth usage rate and disk space usage rate of each node on each server in each distributed processing area are calculated and recorded as and

[0050] The comprehensive load assessment coefficient of each node on each server in each distributed processing area is obtained by comprehensively calculating the standard values ​​of the CPU utilization, memory utilization, network bandwidth utilization and disk space utilization of each node on each server in each distributed processing area and the comprehensive assessment value of each node on each server in each distributed processing area. The specific calculation and analysis method is as follows:

[0051] According to the formula Calculate the comprehensive load evaluation coefficient γ of each node on each server in each distributed processing area, where β represents the set adjustment factor, 0<β<1, and w1, w2, w3, and w4 represent the weight factors corresponding to the set standard values ​​of CPU utilization, memory utilization, network bandwidth utilization, and disk space utilization, respectively;

[0052] Step 4: Determine the node load status: Obtain the classification thresholds of the comprehensive load evaluation coefficients of each node on each server in each distributed processing area from the system database, which are denoted as k1 and k2, respectively, where 0<k1<k2<1;

[0053] It should be noted that for the setting of the classification thresholds of the comprehensive load assessment coefficients of each node on each server in each distributed processing area, some typical load ranges are determined by analyzing historical load data, and then based on the typical load ranges, low load, medium load and high load thresholds are set, which are represented by k1 and k2 respectively.

[0054] The comprehensive load evaluation coefficients of each node on each server in each distributed processing area and the classification thresholds of the comprehensive load evaluation coefficients of each node on each server in each distributed processing area are compared and analyzed. The specific comparison and analysis method is as follows:

[0055] If γ<k1, it means that the node has sufficient resources to process new tasks, the system performance is good, and the task processing time is short, so it is recorded as a low-load node;

[0056] If k1<γ<k2, it means that the resource usage of the node is relatively reasonable, but as the load increases, it may be necessary to pay attention to the change in performance, and the task processing time may be slightly extended. It is recorded as a medium-load node;

[0057] If γ>k2, it means that the node resources are close to saturation, the system performance may be affected, the task processing time may be significantly extended, and measures need to be taken to reduce the load. It is recorded as a high-load node;

[0058] In a specific embodiment, the present invention calculates the comprehensive load evaluation coefficient of each node by analyzing parameters such as CPU utilization, memory utilization, network bandwidth occupancy and disk space utilization of each server node in the distributed processing area, and then evaluates the load status of each node. Specifically, firstly, the mean and variance of each parameter of each node in a specified time period are calculated, and then the standard deviation and normalized value are calculated based on the variance. A comprehensive evaluation is performed based on the set weight factor to finally obtain the comprehensive load evaluation coefficient. Then, low, medium and high load classification thresholds are set according to historical load data, and the comprehensive load evaluation coefficient of each node is compared with them. Based on this, the load status of each node is judged, whether the resources are sufficient and the performance is good (low-load node), the resource usage is reasonable and performance changes need to be paid attention to (medium-load node), or the resources are close to saturation and affect performance (high-load node), thereby realizing effective judgment and optimization of the node load status in the distributed processing system.

[0059] Step 5: Determine the data processing priority: extract the data features to be processed, which include data type, data size, data timeliness requirements and data processing complexity. The data types are classified into structured data, semi-structured data and unstructured data. Combine the data features to be processed and the load status of each node on each server in each distributed processing area into a mapping table, assign a task allocation priority to each combination, and formulate a decision rule for determining the task allocation priority.

[0060] The data size is classified as follows: data less than 10MB is considered small data, data between 10MB and 100MB is considered medium data, and data greater than 100MB is considered big data.

[0061] The data timeliness requirements are classified as follows: data that needs to be completed within 30 seconds is considered high timeliness, data that needs to be completed within 30 minutes is considered medium timeliness, and data that has no time requirement is considered low timeliness.

[0062] The classification of data processing complexity is as follows: those that only require data filtering and statistics are judged as simple processing, those that involve complex algorithms but do not require a lot of calculations are judged as medium processing, and those that involve simulation learning model training are judged as complex processing;

[0063] The characteristics of the data to be processed and the load status of each node on each server in each distributed processing area are combined into a mapping table. The specific method of the mapping table is shown in the following table;

[0064]

[0065] Decision rules are also developed to prioritize task assignments, including but not limited to:

[0066] 1. When data has high timeliness requirements, a higher priority is given to task allocation regardless of data characteristics and node load status;

[0067] 2. When processing data with high complexity, if the node is in a low-load state, it will be given a high priority; if the node is in a high-load state, the priority will be lowered;

[0068] It should be noted that as the system runs and the data processing situation changes, the mapping table and decision rules are dynamically adjusted and optimized at any time. By collecting actual effect data of task allocation, such as task processing time, node load changes, etc., if it is found that the priority of task allocation of certain combinations is unreasonable, the mapping table can be adjusted, and according to changes in business needs, the importance of data features and the impact of node load status can be re-evaluated, and the weights and priority judgment criteria of decision rules can be adjusted. By establishing a mapping table or decision rules, the priority of task allocation can be determined more systematically according to different data features and node load status, thereby achieving more efficient distributed data processing.

[0069] In a specific embodiment, the present invention constructs a mapping table by comprehensively considering the data type, size, timeliness requirement and processing complexity of the data to be processed, as well as the current load status of each server and node in the distributed processing environment, and assigns a task allocation priority to each data processing task accordingly; specifically, when the data has a high timeliness requirement, regardless of other conditions, the task will be given a higher priority for processing. For data with high processing complexity, when the node is in a low load state, the task will be given a high priority; on the contrary, if the node is in a high load state, the task priority will be reduced. In addition, by continuously collecting data such as actual task processing time and node load changes, the system can dynamically adjust the mapping table and decision rules to ensure the rationality and effectiveness of the allocation strategy, thereby realizing efficient and orderly processing of data of different types and sizes in a distributed environment, meeting the data timeliness and processing complexity requirements under different business needs, and significantly improving the overall efficiency and performance of data processing.

[0070] Step 6: Data processing task allocation: Assign high-priority tasks to low-load nodes on servers in distributed processing areas. On the premise of ensuring that high-priority tasks are processed first, assign medium-priority tasks to nodes on servers in distributed processing areas. Finally, arrange low-priority tasks to be processed during system idle time, such as at night or other periods when low-load nodes are available.

[0071] Step 7, distributed processing node optimization: Real-time monitoring is performed on the comprehensive load evaluation coefficient of each node on each server in each distributed processing area, and the monitoring cycle is divided into several monitoring periods. When a node on each server in each distributed processing area frequently experiences delays or excessive load, a load abnormality signal is generated, and the task allocation weight of the node is adjusted;

[0072] The number of nodes with abnormal load signals on the corresponding servers in each distributed processing area within a monitoring period is obtained, which is recorded as g. The number of nodes g with abnormal load signals on the corresponding servers in each distributed processing area within a monitoring period is compared with the number of nodes n on the corresponding servers in each distributed processing area. If the calculation result is greater than 0.8, it means that the load margin of each node on the corresponding servers in each distributed processing area within the monitoring period is insufficient, and a node expansion optimization signal is generated. The processing system increases the number of nodes on the corresponding servers in each distributed processing area within the monitoring period by 50% of the current number of nodes. After the new node joins the system, the load balance is automatically adjusted through the load balancer to evenly distribute the data processing tasks to each node. If the calculation result is less than 0.4, it means that the load margin of each node on the corresponding servers in each distributed processing area within the monitoring period is sufficient, and a node contraction optimization signal is generated. The processing system reduces the number of nodes on the corresponding servers in each distributed processing area within the monitoring period by 20% of the current number of nodes.

[0073] In a specific embodiment, the present invention can efficiently utilize the resources of each server in the distributed processing area to a large extent through refined task allocation and node optimization strategies, thereby ensuring the priority processing of core tasks and the stable operation of the overall system; first, the system will allocate high-priority tasks to low-load nodes on each server and process them immediately; then allocate medium-priority tasks to other nodes to minimize delays; finally, low-priority tasks are scheduled to be processed during system idle periods, such as at night or periods with low node loads. During the distributed processing process, the system will also monitor the comprehensive load of each node in real time. Once a node is found to be frequently delayed or overloaded, the load pressure is alleviated by adjusting the task allocation weight, especially In addition, the system will count the number of nodes with abnormal load signals within a monitoring period and compare it with the total number of nodes. If the ratio exceeds 80%, it indicates that the load is relatively tight, and the system will trigger a node expansion optimization signal to increase the number of existing nodes by 50% to enhance processing capabilities, and automatically adjust through the load balancer to ensure uniformity of task distribution; when the ratio is lower than 40%, it indicates that the system load is relatively light, and the system will trigger a node contraction optimization signal to reduce the current number of nodes by 20% to make more efficient use of existing resources. By dynamically adjusting the number of nodes and intelligent task allocation, the system can effectively respond to the processing needs of tasks of different priorities, and continuously optimize resource utilization during operation to ensure the stability and efficiency of the overall system performance.

[0074] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A distributed data processing method, comprising the following steps: Step 1: Establish a distributed node network; Step 2: Node status monitoring; Step 3: Node operation load analysis: Comprehensively calculate and analyze the CPU usage, memory usage, network bandwidth usage, and disk space usage of each node on each server in each distributed processing area to obtain a comprehensive load evaluation coefficient for each node on each server in each distributed processing area. Step 4, node load status determination: obtain the comprehensive load evaluation coefficient classification threshold of each node on each server in each distributed processing area from the system database, and compare and analyze the comprehensive load evaluation coefficient of each node on each server in each distributed processing area and the comprehensive load evaluation coefficient classification threshold of each node on each server in each distributed processing area; Step 5: Data processing priority determination: extract the data features to be processed, combine the data features to be processed and the load status of each node on each server in each distributed processing area into a mapping table, assign a task allocation priority to each combination, and formulate a decision rule for determining the task allocation priority; Step 6: Data processing task allocation: high-priority tasks are allocated to low-load nodes on the corresponding servers in each distributed processing area. Under the premise of ensuring that high-priority tasks are processed first, medium-priority tasks are allocated to nodes on the corresponding servers in each distributed processing area. Finally, low-priority tasks are scheduled for processing during system idle time. Step 7, distributed processing node optimization: Real-time monitoring and acquisition of the comprehensive load assessment coefficient of each node on each server in each distributed processing area, and dividing the monitoring cycle into several monitoring periods. When a node on each server in each distributed processing area frequently experiences delays or excessive load, a load abnormality signal is generated and the task allocation weight of the node is adjusted.

2. The distributed data processing method according to claim 1, characterized in that: The specific process of step one is: divide the area covered by the data processing system into several processing areas, and set up several servers inside each processing area, and deploy each distributed node on the corresponding servers in each processing area, wherein the nodes on the corresponding servers in each distributed processing area are interconnected through high-speed Ethernet, optical fiber network and dedicated network.

3. The distributed data processing method according to claim 1, characterized in that: The specific process of step two is: real-time monitoring of the operating status of each node on each server corresponding to each distributed processing area, and obtaining the operating status parameters of each node on each server corresponding to each distributed processing area, where the operating status parameters include CPU utilization, memory utilization, network bandwidth utilization and disk space utilization.

4. The distributed data processing method according to claim 1, characterized in that: In step 3, the specific method of comprehensively calculating and analyzing the CPU usage, memory usage, network bandwidth usage, and disk space usage of each node on each server in each distributed processing area is as follows: Get the m CPU usage rates of each node on each server corresponding to each distributed processing area from time point t1 to time point t2, sum them up and divide them by m to get the mean CPU usage rate of each node on each server corresponding to each distributed processing area. According to the mean CPU usage rate of each node on each server corresponding to each distributed processing area, use the variance calculation formula to calculate the variance of CPU usage rate of each node on each server corresponding to each distributed processing area. Then calculate the square root of the variance of CPU usage rate of each node on each server corresponding to each distributed processing area to get the standard deviation of CPU usage rate of each node on each server corresponding to each distributed processing area. Finally, divide the result of subtracting the CPU usage rate of each node on each server corresponding to each distributed processing area from the mean CPU usage rate by the standard deviation of CPU usage rate to get the standard value of CPU usage rate of each node on each server corresponding to each distributed processing area, which is recorded as CN i , i represents the number of each node; Similarly, the standard values ​​of memory usage, network bandwidth usage, and disk space usage of each node on each server in each distributed processing area are calculated and recorded as MN i NN i and DN i ; According to the formula Calculate the comprehensive evaluation value of each node on each server in each distributed processing area a1, a2, a3, and a4 represent the weight factors corresponding to the set CPU usage, memory usage, network bandwidth usage, and disk space usage, respectively; According to the formula Calculate the CPU usage index value of each node on each server in each distributed processing area Where α represents the set attenuation factor, 0<α<1. Similarly, the index values ​​corresponding to the memory usage rate, network bandwidth usage rate and disk space usage rate of each node on each server in each distributed processing area are calculated and recorded as and By comprehensively calculating the standard values ​​corresponding to the CPU utilization, memory utilization, network bandwidth occupancy and disk space utilization of each node on each server corresponding to each distributed processing area and the comprehensive evaluation value of each node on each server corresponding to each distributed processing area, the comprehensive load evaluation coefficient of each node on each server corresponding to each distributed processing area is obtained.

5. The distributed data processing method according to claim 4, characterized in that: The comprehensive load evaluation coefficient of each node on each server in each distributed processing area is calculated and analyzed as follows: According to the formula The comprehensive load assessment coefficient γ of each node on each server in each distributed processing area is calculated, where β is represented by the set adjustment factor, and w1, w2, w3, and w4 are weight factors corresponding to the set standard values ​​of CPU utilization, memory utilization, network bandwidth utilization, and disk space utilization, respectively.

6. The distributed data processing method according to claim 5, characterized in that: In step 4, the comprehensive load evaluation coefficients of each node on each server in each distributed processing area and the classification thresholds of the comprehensive load evaluation coefficients of each node on each server in each distributed processing area are compared and analyzed. The specific comparative analysis method is as follows: Obtain the classification thresholds of the comprehensive load evaluation coefficients of each node on each server corresponding to each distributed processing area from the system database, which are denoted as k1 and k2, respectively, where 0<k1<k2<1; If γ < k1, it means that the node has spare resources to process new tasks, and it is recorded as a low-load node; If k1<γ<k2, it means that the resource usage of the node is reasonable, and it is recorded as a medium-load node; If γ>k2, it means that the node resources are close to saturation and it is recorded as a high-load node.

7. The distributed data processing method according to claim 1, characterized in that: In step five, data characteristics include data type, data size, data timeliness requirements, and data processing complexity.

8. The distributed data processing method according to claim 1, characterized in that: In step seven, by obtaining the number of nodes with load abnormality signals on the corresponding servers in each distributed processing area within a monitoring period, denoted as g, the number of nodes g with load abnormality signals on the corresponding servers in each distributed processing area within a monitoring period is compared with the number of nodes n on the corresponding servers in each distributed processing area. If the calculation result is greater than 0.8, it means that the load margin of each node on the corresponding servers in each distributed processing area within the monitoring period is insufficient, and a node expansion optimization signal is generated. The processing system increases the number of nodes on the corresponding servers in each distributed processing area within the monitoring period by 50% of the current number of nodes. After the new node joins the system, the load balance is automatically adjusted through the load balancer to evenly distribute the data processing tasks to each node. If the calculation result is less than 0.4, it means that the load margin of each node on the corresponding servers in each distributed processing area within the monitoring period is sufficient, and a node contraction optimization signal is generated. The processing system reduces the number of nodes on the corresponding servers in each distributed processing area within the monitoring period by 20% of the current number of nodes.

Images