A Dynamic Load Balancing Adjustment Method for Big Data Processing Integrating Deep Learning

Abstract

This invention discloses a dynamic load balancing adjustment method for big data processing that integrates deep learning. The method comprises the following steps: Step S1: Collecting basic parameters of edge nodes and calculating communication and collaboration metrics between edge nodes; Step S2: Constructing an initial polyhedron structure based on the basic parameters, communication metrics, and collaboration metrics of the edge nodes; Step S3: Obtaining real-time load data of the edge nodes and predicting future load data of the edge nodes based on the real-time load data using an LSTM model; Step S4: Optimizing the initial polyhedron structure based on the future load data using the Transformer deep learning algorithm to obtain an optimized polyhedron structure; Step S5: When an edge node experiences overload, formulating an adjustment strategy based on the structural parameters of the optimized polyhedron structure and performing load balancing adjustments based on the adjustment strategy. By constructing a dynamically changing polyhedron structure, the method reflects changes in edge node load and inter-node relationships, enabling timely and accurate dynamic load balancing adjustments.

Description

Technical Field

[0001] This invention relates to the field of load balancing technology, and in particular to a dynamic adjustment method for load balancing in big data processing that integrates deep learning. Background Technology

[0002] With the rapid development of big data technology and edge computing, the number of edge nodes is constantly increasing. The computing power, storage and bandwidth resources required by various tasks on edge nodes exhibit dynamic fluctuation characteristics. Load balancing has become the key to ensuring the stable and efficient operation of edge computing systems. Existing edge node load balancing methods usually first collect the basic resource parameters of edge nodes, evaluate the communication status and cooperation capabilities between nodes individually, then use simple statistical models or traditional algorithms to predict load changes based on real-time load data, and finally formulate static or semi-dynamic load scheduling strategies based on the load prediction results or node overload trigger signals to realize task diversion and resource allocation for overloaded nodes.

[0003] However, existing load balancing methods still have significant drawbacks: First, the accuracy of load prediction is insufficient. Traditional prediction algorithms struggle to accurately capture the dynamic fluctuations in edge node loads and cannot predict future load trends, leading to lagging scheduling strategies. Second, they fail to comprehensively quantify and consider the communication and collaboration between nodes, making it difficult to match the optimal node combination during scheduling and affecting load migration efficiency. Third, their dynamic adjustment capabilities are weak. The lack of efficient algorithms to support the coordinated optimization of load prediction and scheduling strategies results in unreasonable resource allocation and untimely task migration when overloaded nodes occur, easily leading to resource waste or system lag, and failing to meet the high-efficiency load balancing requirements of big data processing in edge computing scenarios. Summary of the Invention

[0004] In view of this, the present invention proposes a dynamic adjustment method for load balancing in big data processing that integrates deep learning. By constructing a dynamic polyhedron structure, the method reflects the changes in node load and the changes in the relationship between nodes, thereby enabling dynamic adjustment of the load.

[0005] The technical solution of this invention is implemented as follows:

[0006] A dynamic load balancing method for big data processing that integrates deep learning includes the following steps:

[0007] Step S1: Collect the basic parameters of the edge nodes and calculate the communication and collaboration metrics between the edge nodes;

[0008] Step S2: Construct the initial polyhedron structure based on the basic parameters, communication metrics, and collaboration metrics of the edge nodes;

[0009] Step S3: Obtain real-time load data of edge nodes, and predict future load data of edge nodes based on real-time load data using an LSTM model;

[0010] Step S4: Use the Transformer deep learning algorithm to optimize the initial polyhedron structure based on future load data to obtain an optimized polyhedron structure;

[0011] Step S5: When an overload occurs at an edge node, an adjustment strategy is formulated based on the structural parameters of the optimized polyhedron structure, and load balancing is performed based on the adjustment strategy.

[0012] Preferably, step S1 includes the following steps:

[0013] Step S11: Collect the node computing power, storage capacity, network bandwidth, deployment location, resource type and resource surplus of each edge node as basic parameters;

[0014] Step S12: Use network detection tools to detect communication between edge nodes and measure network latency, bandwidth and packet loss rate between edge nodes as communication indicators.

[0015] Step S13: Calculate the collaboration index between edge nodes using the Pearson correlation coefficient algorithm based on the resource type and resource surplus of each edge node.

[0016] Preferably, the specific steps of step S13 are as follows:

[0017] The resource types of edge nodes are divided into advantageous resources and disadvantageous resources, and the degree of surplus of advantageous and disadvantageous resources is quantified.

[0018] The Pearson correlation coefficient algorithm is used to calculate the correlation coefficient between the advantageous and disadvantageous resources of two edge nodes, and the correlation coefficient is output as a collaboration index.

[0019] Preferably, the specific steps of step S2 are as follows:

[0020] Step S21: Map the edge nodes to three-dimensional space and determine the initial coordinates based on the deployment location of the edge nodes;

[0021] Step S22: Adjust the initial coordinates of the edge nodes based on communication and collaboration metrics to make closely related edge nodes adjacent in the polyhedron.

[0022] Step S23: Determine the vertices, edges, and face structure of the polyhedron based on the adjusted coordinates of the edge nodes, and construct the initial polyhedron structure.

[0023] Preferably, the specific steps of step S22 are as follows:

[0024] After standardizing the communication and collaboration metrics respectively, assigning them different weights, and summing them, the adjustment parameters are obtained.

[0025] The initial coordinates of the edge nodes are adjusted based on the adjustment parameters to make closely related edge nodes adjacent in the polyhedron.

[0026] Preferably, step S3 includes the following specific steps:

[0027] Step S31: Collect real-time load data of edge nodes and perform standardized preprocessing on the real-time load data;

[0028] Step S32: Input the preprocessed real-time load data into the pre-trained LSTM model;

[0029] Step S33: Output the future load data of each edge node within the preset future time period from the LSTM model.

[0030] Preferably, step S4 includes the following specific steps:

[0031] Step S41: Construct a multimodal feature vector containing the node coordinates of the edge nodes in the initial polyhedral structure, future load data, communication indicators, and collaboration indicators;

[0032] Step S42: Input the multimodal feature vectors into the Transformer encoder layer for deep feature fusion;

[0033] Step S43: Calculate the mutual influence weight matrix between edge nodes using a multi-head attention mechanism, and calculate the node coordinate adjustment vector based on the mutual influence weight matrix and future load data;

[0034] Step S44: Apply the node coordinate adjustment vector to adjust the edge nodes and reconstruct the initial polyhedron structure to form an optimized polyhedron structure.

[0035] Preferably, the specific steps of calculating the node coordinate adjustment vector based on the mutual influence weight matrix and future load data in step S43 are as follows: calculate the expected load change of each edge node according to the future load data and the mutual influence weight matrix, map the expected load change to the geometric displacement direction and magnitude of the edge node in the polyhedron, and project it onto the feasible adjustment space to output the final node coordinate adjustment vector.

[0036] Preferably, step S5 includes the following specific steps:

[0037] Step S51: Monitor the load data of edge nodes in real time. When an overload is detected, identify and optimize all faces where the overloaded edge node is located in the polyhedral structure.

[0038] Step S52: Collect load data of other edge nodes on the surface, and determine the migration node and migration amount based on the load data of the overloaded edge node and the load data of other edge nodes on the surface;

[0039] Step S53: Split the tasks of the overloaded edge nodes and allocate them to the corresponding migration nodes according to the migration amount.

[0040] Preferably, the specific steps in step S52 for determining the migration node and migration amount based on the load data of the overloaded edge node and the load data of other edge nodes on the surface are as follows:

[0041] The resource redundancy is calculated based on the load data of other edge nodes on the surface, and the overload is calculated based on the load data of overloaded edge nodes. Edge nodes whose resource redundancy meets the overload are recorded as migration nodes.

[0042] Calculate the Euclidean distance between overload edge nodes and migration nodes in the optimized polyhedron, use the Euclidean distance as the allocation ratio, allocate the overload amount based on the allocation ratio, and obtain the migration amount.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] This invention provides a dynamic load balancing adjustment method for big data processing that integrates deep learning. By collecting basic parameters of edge nodes and calculating communication and collaboration indicators, it can achieve comprehensive quantification of node relationships, solving the problems of evaluating node status separately and matching the optimal node combination in existing methods. Then, based on the relevant content of the edge nodes, an initial polyhedral structure is constructed, so that the relationship between nodes can be intuitively represented by geometric distance and topological connection. At the same time, each face of the polyhedral structure can be regarded as a collaborative unit, and the nodes on the face can efficiently collaborate to process tasks, so as to quickly achieve dynamic load balancing adjustment.

[0045] The initial polyhedral structure is not static. With the help of the LSTM model, the future load data of the edge nodes can be accurately predicted. Then, the Transformer algorithm can optimize the initial polyhedral structure based on the future load data. The resulting optimized polyhedral structure can dynamically reflect the changes in node load and the changes in the relationship between nodes. Combined with the optimized structural parameters, an overload adjustment strategy can be formulated to enhance the dynamic adaptation capability of load balancing, ensure that the tasks of overloaded nodes are distributed in a timely and reasonable manner, reduce resource waste and system lag, improve the load balancing efficiency and stability of the edge computing system in big data processing scenarios, and meet the requirements of high-efficiency load balancing. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only preferred embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of a big data processing load balancing dynamic adjustment method that integrates deep learning, according to the present invention.

[0048] Figure 2 This is a flowchart of step S1 of a big data processing load balancing dynamic adjustment method that integrates deep learning according to the present invention.

[0049] Figure 3 This is a flowchart of step S2 of a big data processing load balancing dynamic adjustment method that integrates deep learning according to the present invention.

[0050] Figure 4 This is a flowchart of step S3 of a big data processing load balancing dynamic adjustment method that integrates deep learning according to the present invention.

[0051] Figure 5 This is a flowchart of step S4 of a big data processing load balancing dynamic adjustment method that integrates deep learning according to the present invention.

[0052] Figure 6 This is a flowchart of step S5 of a big data processing load balancing dynamic adjustment method that integrates deep learning according to the present invention. Detailed Implementation

[0053] To better understand the technical content of this invention, a specific embodiment is provided below, and the invention will be further described in conjunction with the accompanying drawings.

[0054] See Figures 1 to 6 This invention provides a dynamic adjustment method for load balancing in big data processing that integrates deep learning, comprising the following steps:

[0055] Step S1: Collect the basic parameters of the edge nodes and calculate the communication and collaboration metrics between the edge nodes;

[0056] Step S2: Construct the initial polyhedron structure based on the basic parameters, communication metrics, and collaboration metrics of the edge nodes;

[0057] Step S3: Obtain real-time load data of edge nodes, and predict future load data of edge nodes based on real-time load data using an LSTM model;

[0058] Step S4: Use the Transformer deep learning algorithm to optimize the initial polyhedron structure based on future load data to obtain an optimized polyhedron structure;

[0059] Step S5: When an overload occurs at an edge node, an adjustment strategy is formulated based on the structural parameters of the optimized polyhedron structure, and load balancing is performed based on the adjustment strategy.

[0060] This invention discloses a dynamic load balancing adjustment method for big data processing that integrates deep learning. This method is applicable to load balancing adjustments of edge nodes in big data scenarios. By collecting basic parameters of edge nodes deployed in different locations, the basic conditions of the edge nodes can be determined, such as computing power and resource types. These basic parameters can roughly determine the processing and load capacity of the edge nodes. Different edge nodes communicate with each other to transmit data. In addition to communication, edge nodes also cooperate with each other. When high task volume leads to overload, idle edge nodes can provide cooperation to overloaded edge nodes, preventing system paralysis due to edge node overload and inability to process data normally. Therefore, after collecting the basic parameters of the edge nodes, it is also necessary to calculate the communication and cooperation indicators between edge nodes to determine the communication reachability and resource complementarity between them.

[0061] After collecting basic parameters, communication metrics, and collaboration metrics, an initial polyhedron structure can be constructed. The edge nodes are the vertices of the polyhedron, and the lines connecting the edge nodes form the edges of the polyhedron, representing the communication and collaboration relationships between the edge nodes. The faces formed by the edges are regarded as collaboration units. When any edge node in a face is overloaded, the edge nodes on the face can efficiently collaborate to process tasks, migrating the tasks of the overloaded edge node to other edge nodes. This helps to consider the collaboration relationships between nodes during load balancing, rather than just the load of a single node. The constructed polyhedron structure allows the relationships between nodes (such as communication delay and collaboration frequency) to be intuitively represented through geometric distance and topological connections.

[0062] The polyhedral structure is not static; it can be dynamically updated according to a preset period. After real-time collection of edge node load data, the LSTM model can predict the future load data of the edge nodes based on the real-time load data, thereby determining whether an overload event will occur in the future. This compensates for the shortcomings of traditional algorithms, such as insufficient load prediction accuracy and lagging scheduling strategies. Then, the Transformer deep learning algorithm is introduced, which can dynamically optimize the initial polyhedral structure based on future load data. The resulting optimized polyhedral structure can dynamically reflect changes in node load and relationships between nodes, providing a geometric basis for load balancing adjustments. Then, edge nodes are monitored in real time. When an overload occurs, structural parameters can be extracted from the optimized polyhedral structure, including the relationships between edge nodes, the positions and computing power of other edge nodes on the same face, etc. Based on this, an adjustment strategy is formulated, and then load balancing adjustments are performed based on the adjustment strategy to avoid prolonged overload of a single edge node. The polyhedral structure can quickly build relationships between edge nodes and can be dynamically adjusted based on future load data, directly reflecting changes between edge nodes. When constructing adjustment strategies based on this, load balancing adjustments can be achieved accurately and timely, ensuring the normal operation of data processing.

[0063] Preferably, step S1 includes the following steps:

[0064] Step S11: Collect the node computing power, storage capacity, network bandwidth, deployment location, resource type and resource surplus of each edge node as basic parameters;

[0065] Step S12: Use network detection tools to detect communication between edge nodes and measure network latency, bandwidth and packet loss rate between edge nodes as communication indicators.

[0066] Step S13: Calculate the collaboration index between edge nodes using the Pearson correlation coefficient algorithm based on the resource type and resource surplus of each edge node.

[0067] The actual deployment location of edge nodes provides the initial basis for the construction of the initial polyhedral structure, while the computing power of the nodes determines the amount of data they can carry. When the load of an edge node exceeds its computing power, an overload will occur. Similarly, when the storage capacity and network bandwidth are exceeded, an overload will also occur.

[0068] After collecting the basic parameters of the edge nodes, it is necessary to calculate the communication indicators between the edge nodes. This involves using network probing tools to detect communication between each pair of edge nodes, ensuring that the communication links between edge nodes are reachable and stable. At the same time, network latency, bandwidth, and packet loss rate between edge nodes are measured as communication indicators. If the network latency between edge nodes is high, the correlation between the two edge nodes is low, and the corresponding workload will be reduced when load balancing is adjusted later. In addition, based on the resource type and resource surplus of the edge nodes, subsequent collaboration indicators can be calculated. The collaboration indicators are calculated using the Pearson correlation coefficient method. The specific steps are as follows: the resource types of the edge nodes are divided into advantageous resources and disadvantageous resources, and the surplus of advantageous and disadvantageous resources is quantified.

[0069] The Pearson correlation coefficient algorithm is used to calculate the correlation coefficient between the advantageous and disadvantageous resources of two edge nodes, and the correlation coefficient is output as a collaboration index.

[0070] Different edge nodes possess both advantageous and disadvantageous resources. If the advantageous resource of edge node A is the disadvantageous resource of edge node B, it indicates a complementary and cooperative relationship between the two edge nodes. Therefore, advantageous and disadvantageous resources can be categorized based on the resource types of edge nodes, and the abundance of these resources can be converted into quantitative values ​​in the 0-1 range. Then, the Pearson correlation coefficient method is introduced to calculate the Pearson correlation coefficient between the advantageous resource variable and the disadvantageous resource variable of the two edge nodes. The larger the Pearson correlation coefficient, the stronger the cooperation between the two edge nodes. In subsequent task migration, more tasks can be migrated to the corresponding edge nodes.

[0071] Preferably, the specific steps of step S2 are as follows:

[0072] Step S21: Map the edge nodes to three-dimensional space and determine the initial coordinates based on the deployment location of the edge nodes;

[0073] Step S22: Adjust the initial coordinates of the edge nodes based on communication and collaboration metrics to make closely related edge nodes adjacent in the polyhedron.

[0074] Step S23: Determine the vertices, edges, and face structure of the polyhedron based on the adjusted coordinates of the edge nodes, and construct the initial polyhedron structure.

[0075] The structure of a polyhedron is mainly composed of vertices, edges formed by vertices, and faces formed by edges. When constructing a polyhedron, it is necessary to start from the most basic vertices. Therefore, the number of vertices can be determined first based on the number of edge nodes. Then, the edge nodes are mapped into three-dimensional space. The initial coordinates are determined first based on the physical deployment position of the edge nodes. However, the initial coordinates are not fixed. The correlation between edge nodes can be determined based on communication and cooperation indicators. The initial coordinates of the edge nodes are adjusted according to the different correlations so that the edge nodes can be closely adjacent in the polyhedron. Finally, the vertex, edge, and face structure of the polyhedron can be determined based on the adjusted coordinates, thereby constructing the initial polyhedron structure.

[0076] Preferably, the specific steps of step S22 are as follows:

[0077] After standardizing the communication and collaboration metrics respectively, assigning them different weights, and summing them, the adjustment parameters are obtained.

[0078] The initial coordinates of the edge nodes are adjusted based on the adjustment parameters to make closely related edge nodes adjacent in the polyhedron.

[0079] Communication metrics and collaboration metrics represent whether communication between edge nodes is smooth and whether resources can complement each other. After obtaining the communication metrics and collaboration metrics, they are standardized to the same dimension, assigned different weights, and summed. The summation result is the adjustment parameter. The larger the adjustment parameter, the higher the correlation between the two edge nodes. Therefore, the two edge nodes need to be adjusted to be closer. Based on this, the reciprocal of the adjustment parameter is taken as the distance between edge nodes, and the initial coordinates of the edge nodes are adjusted so that closely related edge nodes can be adjacent in the polyhedron.

[0080] Preferably, step S3 includes the following specific steps:

[0081] Step S31: Collect real-time load data of edge nodes and perform standardized preprocessing on the real-time load data;

[0082] Step S32: Input the preprocessed real-time load data into the pre-trained LSTM model;

[0083] Step S33: Output the future load data of each edge node within the preset future time period from the LSTM model.

[0084] After the initial polyhedral structure is constructed, it needs to be adjusted according to the actual load. However, if adjustments are made based on the current load data, it is obviously too late to handle overload situations. Therefore, the dynamic adjustment of the polyhedral structure needs to take into account whether overload will occur in the future. Thus, it is necessary to predict the load status of the edge nodes. This invention uses an LSTM model to predict future load data. The real-time load data of the collected edge nodes is preprocessed, including standardization, outlier removal, and noise reduction. The processed real-time load data can be input into the LSTM model, which predicts the future load data within 5-30 minutes. The set time period can be determined according to the actual data processing situation. For example, when there are many data processing tasks, the time period can be shortened, and when there are few data processing tasks, the time period can be appropriately extended.

[0085] Preferably, step S4 includes the following specific steps:

[0086] Step S41: Construct a multimodal feature vector containing the node coordinates of the edge nodes in the initial polyhedral structure, future load data, communication indicators, and collaboration indicators;

[0087] Step S42: Input the multimodal feature vectors into the Transformer encoder layer for deep feature fusion;

[0088] Step S43: Calculate the mutual influence weight matrix between edge nodes using a multi-head attention mechanism, and calculate the node coordinate adjustment vector based on the mutual influence weight matrix and future load data;

[0089] Step S44: Apply the node coordinate adjustment vector to adjust the edge nodes and reconstruct the initial polyhedron structure to form an optimized polyhedron structure.

[0090] After predicting future load data, the initial polyhedron structure can be adjusted and optimized based on this data to adapt to load changes over a fixed time period. First, structural parameters, including the coordinates of each edge node, are obtained from the initial polyhedron structure. Then, the node coordinates, future load data, communication metrics, and collaboration metrics are standardized and concatenated into a multimodal feature vector. This multimodal feature vector is input into a trained Transformer deep learning algorithm. The encoder in the Transformer algorithm performs deep feature fusion and calculates the mutual influence weight matrix between edge nodes through its multi-head attention mechanism, achieving global information interaction and focused optimization. Finally, the Transformer algorithm outputs the mutual influence weight matrix. Based on this matrix and future load data, a node coordinate adjustment vector is calculated. This vector is used to adjust the edge nodes, changing the positions of vertices on the initial polyhedron structure and thus altering its shape. Simultaneously, the size and position of the faces acting as collaborative units on the polyhedron also change synchronously, ultimately forming an optimized polyhedron structure. This optimized structure dynamically reflects changes in node load and relationships between nodes.

[0091] Preferably, the specific steps of calculating the node coordinate adjustment vector based on the mutual influence weight matrix and future load data in step S43 are as follows: calculate the expected load change of each edge node according to the future load data and the mutual influence weight matrix, map the expected load change to the geometric displacement direction and magnitude of the edge node in the polyhedron, and project it onto the feasible adjustment space to output the final node coordinate adjustment vector.

[0092] When calculating the node coordinate adjustment vector, the future load change is obtained by subtracting the current load data from the future load data. Multiplying the future load change by the mutual influence weight matrix yields the expected load change. The expected load change quantifies the amount of task each node needs to undertake or transfer due to changes in the overall system load distribution, providing a numerical basis for subsequent geometric adjustments. Finally, the expected load change is mapped into the geometric displacement direction and magnitude, projected into three-dimensional space, and the final node coordinate adjustment vector is output. Based on the node coordinate adjustment vector, the initial coordinates of the edge nodes can be adjusted to form an optimized polyhedral structure.

[0093] Preferably, step S5 includes the following specific steps:

[0094] Step S51: Monitor the load data of edge nodes in real time. When an overload is detected, identify and optimize all faces where the overloaded edge node is located in the polyhedral structure.

[0095] Step S52: Collect load data of other edge nodes on the surface, and determine the migration node and migration amount based on the load data of the overloaded edge node and the load data of other edge nodes on the surface;

[0096] Step S53: Split the tasks of the overloaded edge nodes and allocate them to the corresponding migration nodes according to the migration amount.

[0097] After the optimized polyhedral structure is formed, each edge node can be monitored in real time within a preset time period. When the load data of an edge node exceeds a preset threshold, it indicates that the edge node is overloaded. At this time, the overloaded node can be recorded as an overloaded edge node, and the tasks it receives need to be distributed. Therefore, firstly, all faces where the overloaded edge node is located are identified from the optimized polyhedral structure. Other edge nodes on the face are candidate edge nodes. However, not all candidate edge nodes can accept task migration. It is necessary to determine which edge nodes can be used as migration nodes based on the load data of the overloaded edge node and the load data of the candidate edge nodes. The migration amount of different migration nodes will also be different. Finally, the tasks of the overloaded edge node can be migrated to the migration nodes on the face to achieve dynamic adjustment of load balancing.

[0098] Preferably, the specific steps in step S52 for determining the migration node and migration amount based on the load data of the overloaded edge node and the load data of other edge nodes on the surface are as follows:

[0099] The resource redundancy is calculated based on the load data of other edge nodes on the surface, and the overload is calculated based on the load data of overloaded edge nodes. Edge nodes whose resource redundancy meets the overload are recorded as migration nodes.

[0100] Calculate the Euclidean distance between overload edge nodes and migration nodes in the optimized polyhedron, use the Euclidean distance as the allocation ratio, allocate the overload amount based on the allocation ratio, and obtain the migration amount.

[0101] When selecting migration nodes, the overload amount needs to be calculated based on the load data of the overloaded edge node. Then, the resource surplus needs to be calculated based on the load data of other edge nodes on the same plane as the overloaded edge node. Only when the resource surplus meets the overload amount will it be determined as a migration node. There are multiple migration nodes, and the migration amount is not migrated evenly. Instead, the Euclidean distance between the overloaded edge node and the migration node needs to be calculated. The closer the Euclidean distance, the larger the allocation ratio, and thus the more migration amount is obtained. Finally, after determining each migration node and its corresponding migration amount, the tasks of the overloaded node are split and migrated according to the migration amount to complete the dynamic adjustment of load balancing.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic load balancing adjustment method for big data processing integrating deep learning, characterized in that, Includes the following steps: Step S1: Collect the basic parameters of the edge nodes and calculate the communication and collaboration metrics between the edge nodes; Step S2: Construct the initial polyhedron structure based on the basic parameters, communication metrics, and collaboration metrics of the edge nodes; Step S3: Obtain real-time load data of edge nodes, and predict future load data of edge nodes based on real-time load data using an LSTM model; Step S4: Use the Transformer deep learning algorithm to optimize the initial polyhedron structure based on future load data to obtain an optimized polyhedron structure; Step S5: When an overload occurs at an edge node, an adjustment strategy is formulated based on the structural parameters of the optimized polyhedron structure, and load balancing is performed based on the adjustment strategy.

2. The method for dynamic load balancing adjustment of big data processing based on deep learning according to claim 1, characterized in that, The specific steps of step S1 include: Step S11: Collect the node computing power, storage capacity, network bandwidth, deployment location, resource type and resource surplus of each edge node as basic parameters; Step S12: Use network detection tools to detect communication between edge nodes and measure network latency, bandwidth and packet loss rate between edge nodes as communication indicators. Step S13: Calculate the collaboration index between edge nodes using the Pearson correlation coefficient algorithm based on the resource type and resource surplus of each edge node.

3. The method for dynamic load balancing adjustment of big data processing based on deep learning according to claim 2, characterized in that, The specific steps of step S13 are as follows: The resource types of edge nodes are divided into advantageous resources and disadvantageous resources, and the degree of surplus of advantageous and disadvantageous resources is quantified. The Pearson correlation coefficient algorithm is used to calculate the correlation coefficient between the advantageous and disadvantageous resources of two edge nodes, and the correlation coefficient is output as a collaboration index.

4. The method for dynamic load balancing adjustment of big data processing based on deep learning according to claim 1, characterized in that, The specific steps of step S2 are as follows: Step S21: Map the edge nodes to three-dimensional space and determine the initial coordinates based on the deployment location of the edge nodes; Step S22: Adjust the initial coordinates of the edge nodes based on communication and collaboration metrics to make closely related edge nodes adjacent in the polyhedron. Step S23: Determine the vertices, edges, and face structure of the polyhedron based on the adjusted coordinates of the edge nodes, and construct the initial polyhedron structure.

5. The method for dynamic load balancing adjustment of big data processing based on deep learning according to claim 4, characterized in that, The specific steps of step S22 are as follows: After standardizing the communication and collaboration metrics respectively, assigning them different weights, and summing them, the adjustment parameters are obtained. The initial coordinates of the edge nodes are adjusted based on the adjustment parameters to make closely related edge nodes adjacent in the polyhedron.

6. The method for dynamic load balancing adjustment of big data processing based on deep learning according to claim 1, characterized in that, The specific steps of step S3 include: Step S31: Collect real-time load data of edge nodes and perform standardized preprocessing on the real-time load data; Step S32: Input the preprocessed real-time load data into the pre-trained LSTM model; Step S33: Output the future load data of each edge node within the preset future time period from the LSTM model.

7. The method for dynamic load balancing adjustment of big data processing based on deep learning according to claim 1, characterized in that, The specific steps of step S4 include: Step S41: Construct a multimodal feature vector containing the node coordinates of the edge nodes in the initial polyhedral structure, future load data, communication indicators, and collaboration indicators; Step S42: Input the multimodal feature vectors into the Transformer encoder layer for deep feature fusion; Step S43: Calculate the mutual influence weight matrix between edge nodes using a multi-head attention mechanism, and calculate the node coordinate adjustment vector based on the mutual influence weight matrix and future load data; Step S44: Apply the node coordinate adjustment vector to adjust the edge nodes and reconstruct the initial polyhedron structure to form an optimized polyhedron structure.

8. The method for dynamic load balancing adjustment of big data processing based on deep learning according to claim 7, characterized in that, The specific steps in step S43 for calculating the node coordinate adjustment vector based on the mutual influence weight matrix and future load data are as follows: calculate the expected load change of each edge node according to the future load data and the mutual influence weight matrix, map the expected load change to the geometric displacement direction and magnitude of the edge node in the polyhedron, and project it onto the feasible adjustment space to output the final node coordinate adjustment vector.

9. The method for dynamic load balancing adjustment of big data processing based on deep learning according to claim 1, characterized in that, The specific steps of step S5 include: Step S51: Monitor the load data of edge nodes in real time. When an overload is detected, identify and optimize all faces where the overloaded edge node is located in the polyhedral structure. Step S52: Collect load data of other edge nodes on the surface, and determine the migration node and migration amount based on the load data of the overloaded edge node and the load data of other edge nodes on the surface; Step S53: Split the tasks of the overloaded edge nodes and allocate them to the corresponding migration nodes according to the migration amount.

10. The method for dynamic load balancing adjustment of big data processing based on deep learning according to claim 9, characterized in that, The specific steps in step S52 for determining the migration node and migration amount based on the load data of the overloaded edge node and the load data of other edge nodes on the surface are as follows: The resource redundancy is calculated based on the load data of other edge nodes on the surface, and the overload is calculated based on the load data of overloaded edge nodes. Edge nodes whose resource redundancy meets the overload are recorded as migration nodes. Calculate the Euclidean distance between overload edge nodes and migration nodes in the optimized polyhedron, use the Euclidean distance as the allocation ratio, allocate the overload amount based on the allocation ratio, and obtain the migration amount.

Images