Method for optimizing load balancing by exporting historical data and readable storage medium

Abstract

The present invention discloses a method for optimizing load balancing by exporting historical data and a readable storage medium, comprising the following steps: a client receives request data and obtains the data volume of the request data; the request data is sliced ​​and the sliced ​​request data is distributed to each node server; the system load and performance index are obtained and the basic data segmentation is calculated; the segmented request data of the node server with a load higher than the system load is distributed to the node server with a load lower than the system load in real time; and a complete data export or query result is generated. In the technical solution of the present invention, the adverse effects of query operations of a large amount of data on system performance are effectively avoided, query requests are distributed to multiple node servers in parallel, and then the query results returned by each node are integrated, and the query efficiency and system load capacity are greatly improved by parallel query and data integration; the size of the sliced ​​query data of each node server is dynamically adjusted in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of historical data export, and in particular to a method for optimizing load balancing by exporting historical data and a readable storage medium. Background Art

[0002] With the rapid growth of data scale, the export and query operations of large-scale historical data in database management systems have become the main challenge of system performance issues, especially in terms of load balancing; traditional load balancing methods usually only consider the overall load distribution, but ignore the different data sizes, resulting in query operations with larger data volumes being more time-consuming and resource-intensive. Therefore, a method that can dynamically adjust the size of request data segments is needed to optimize the system load balancing performance. Summary of the invention

[0003] The main purpose of the present invention is to provide a method for optimizing load balancing of historical data export and a readable storage medium, aiming to solve the problem that the export and query of existing large-scale historical data is time-consuming and resource-intensive.

[0004] To achieve the above object, the method for optimizing load balancing by exporting historical data proposed in the present invention comprises the following steps:

[0005] The client receives request data for exporting or querying historical data, and obtains the data volume of the requested data;

[0006] Slice the request data according to the number of node servers and variables, divide the request data into a plurality of slice request data of equal size, and distribute the slice request data to each node server;

[0007] Obtaining system load and performance indicators, calculating and obtaining basic data segments, dividing the slice request data into multiple segment request data according to the basic data segments and processing them according to the time scale;

[0008] Obtain the load and performance indicators of each node server and system in real time, and distribute the segmented request data of the node server with a load higher than the system load to the node server with a load lower than the system load;

[0009] The results returned by each of the node servers are integrated to generate complete data export or query results to a specified storage medium.

[0010] Preferably, the step of receiving request data for exporting or querying historical data by the client and obtaining the data volume of the request data includes:

[0011] The client receives request data for exporting or querying historical data;

[0012] The data volume of the requested data is obtained according to the request type of the requested data and the time when the client receives the request.

[0013] Preferably, before the steps of obtaining the system load and performance index, calculating and obtaining the basic data segment, dividing the slice request data into a plurality of segment request data according to the basic data segment and processing them according to the time scale, the steps include:

[0014] Get system load and performance metrics;

[0015] The size of the basic data segment is obtained through the formula: D=k1*y+k2*load+k3*(y^2)+k4*(load^2)+k5*(y*load), where: D represents the size of the basic data segment, y represents the time scale, load represents the system load and performance index; k1, k2, k3, k4 and k5 all represent coefficients;

[0016] The segmented request data is segmented into a plurality of segmented request data according to the basic data segment and processed according to a time scale.

[0017] Preferably, before the step of segmenting the shard request data into a plurality of segment request data according to the basic data segments and processing them according to the time scale, the step includes:

[0018] Determine whether the size of the shard request data is greater than the size of the basic data shard;

[0019] If not, no segmentation is performed and the data is processed directly;

[0020] If so, the segment request data is segmented into a plurality of segment request data according to the basic data segment and processed according to the time scale.

[0021] Preferably, the step of segmenting the slice request data into a plurality of segment request data according to the basic data segments and processing them according to the time scale includes:

[0022] Calculate the total size of the data requested by the sharding request;

[0023] Determine the number of shards N = ceil (T / D), where: D represents the size of the basic data segment, and T represents the total size of the shard request data;

[0024] The slice request data is divided into N segment request data, and processed according to the time scale.

[0025] Preferably, the step of acquiring the load and performance index of each node server and the system in real time and distributing the segmented request data of the node server with a load higher than the system load to the node server with a load lower than the system load comprises:

[0026] Obtain the load and performance indicators of each node server and system in real time, as well as the initial values ​​of data segment size and time scale;

[0027] According to the formula , adjust the size and time scale of each shard request data;

[0028] in: Indicates the size or time scale of the new shard request data of the z-th node server. The zth node server represents the size or time scale of the requested data segment at the initial moment, and α represents the impact parameter of the system load and performance index on the data segment size and time scale; Represents the system load and performance indicators of the z-th node server; β represents the impact parameter of data segment size and time scale on system load and performance indicators; γ represents the linear relationship coefficient between system load and performance indicators and data segment size and time scale; δ represents the impact parameter of the initial value of data segment size and time scale on system load and performance indicators; ε represents the impact parameter of the fluctuation of system load and performance indicators on data segment size and time scale; φ represents the impact parameter of the adjustment of data segment size and time scale on system load and performance indicators.

[0029] Preferably, the step of dividing the request data into N segmented request data comprises:

[0030] Obtaining the starting time, ending time and size of the basic data segment of the slice request data;

[0031] =(ba) / N*[f(a)+f(a+h)+f(a+2h)+...+f(bh)]; where: a represents the starting time of the shard request data; b represents the end time of the shard request data; h represents the time period of each basic request data; N represents the number of shards, represents the estimated completion time and required computing resources for the zth shard request data, and f(a) represents the computing resource estimation function required for data for h periods starting at time a.

[0032] Preferably, the step of acquiring the load and performance index of each node server and the system in real time and distributing the segmented request data of the node server with a load higher than the system load to the node server with a load lower than the system load comprises:

[0033] Real-time calculation obtains the real-time load and performance indicators of each node server;

[0034] =λ*(CPU_Usage+Memory_Usage+Network_Usage+Disk_Usage)+μ*(Request_Rate / Response_Time);

[0035] in: represents the load status of the zth node server; λ and μ represent constant coefficients related to system performance; CPU_Usage represents CPU usage; Memory_Usage represents memory usage; Network_Usage represents network usage; Disk_Usage represents disk usage; Request_Rate represents request rate; Response_Time represents response time;

[0036] Calculate system load based on real-time load and performance indicators of each node server;

[0037] The segmented request data of the node server whose load is higher than the system load is distributed to the node server whose load is lower than the system load.

[0038] Preferably, the step of integrating the query results returned by each node server to generate a complete query result includes:

[0039] Integrate the result data returned by each node server:

[0040] ;

[0041] in: Indicates the complete query result; Indicates the query result returned by the i-th server;

[0042] Encrypting the result data returned by each of the node servers using a data encryption algorithm;

[0043] The encrypted result data is compressed by a data compression algorithm;

[0044] Generate complete query results and transfer them to the specified storage medium.

[0045] A readable storage medium includes a computer program, and when the computer program is executed by a processor, the method for optimizing load balancing by exporting historical data as described in any one of the above items is implemented.

[0046] In the technical solution of the present invention, data is sharded and segmented according to the data size and the number of node servers to effectively avoid the adverse effects of query operations with large data volumes on system performance, query requests are distributed in parallel to multiple node servers, and then the query results returned by each node are integrated, parallel query and data integration greatly improve query efficiency and system load capacity; the size of the sharded query data of each node server is dynamically adjusted in real time: according to the real-time system load and performance indicators, the present invention can effectively optimize the load balancing performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0048] Figure 1 A flow chart of a method for optimizing load balancing by deriving historical data of the present invention.

[0049] Figure 2 A flowchart of one embodiment of a method for optimizing load balancing by deriving historical data according to the present invention.

[0050] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0053] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0054] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0055] Please refer to Figure 1-Figure 2 The present invention proposes a method for optimizing load balancing by exporting historical data, comprising the following steps:

[0056] S100, the client receives request data for exporting or querying historical data, and obtains the data volume of the requested data;

[0057] S200, slicing the request data according to the number of node servers and variables, dividing the request data into a plurality of slicing request data of equal size, and distributing the slicing request data to each node server;

[0058] S300, obtaining system load and performance indicators, calculating and obtaining basic data segments, dividing the slice request data into multiple segment request data according to the basic data segments and processing them according to the time scale;

[0059] S400, obtaining the load and performance index of each node server and system in real time, and distributing the segmented request data of the node server whose load is higher than the system load to the node server whose load is lower than the system load;

[0060] S500, integrating the results returned by each of the node servers to generate complete data export or query results to a designated storage medium.

[0061] In the technical solution of the present invention, data is sharded and segmented according to the data size and the number of node servers to effectively avoid the adverse effects of query operations with large data volumes on system performance, query requests are distributed in parallel to multiple node servers, and then the query results returned by each node are integrated, parallel query and data integration greatly improve query efficiency and system load capacity; the size of the sharded query data of each node server is dynamically adjusted in real time: according to the real-time system load and performance indicators, the present invention can effectively optimize the load balancing performance of the system.

[0062] In another embodiment of the present invention, the step S100 includes:

[0063] S110, the client receives request data for exporting or querying historical data;

[0064] S120: Obtain the data volume of the requested data according to the request type of the requested data and the time when the client receives the request.

[0065] Specifically, receiving a query request: When the system receives a historical data export or query request, the amount of data in the request is recorded. Assuming that the amount of data in the request is D, the request type is T (can be export or query), and the time when the system receives the request is t. Then, a mathematical model can be established to describe this process: D=∫(T(t)×H(t))dt; where f is a function that represents the impact of the request type T and the receiving time t on the data size D. The specific form of this function can be defined and adjusted according to the actual situation.

[0066] In another embodiment of the present invention, the step S300 includes:

[0067] S310, obtaining system load and performance indicators;

[0068] S320, obtain the size of the basic data segment through the formula: D=k1*y+k2*load+k3*(y^2)+k4*(load^2)+k5*(y*load), where: D represents the size of the basic data segment, y represents the time scale, load represents the system load and performance index; k1, k2, k3, k4 and k5 all represent coefficients;

[0069] S330, dividing the slice request data into a plurality of segment request data according to the basic data segments and processing them according to a time scale.

[0070] Specifically, this equation is a multiple linear regression equation, where k1-k5 are the coefficients of the linear regression model, which can be estimated by methods such as the least squares method. Load represents the system load and performance indicators, which is an independent variable that can be measured and adjusted according to actual conditions. Through this equation, we can adjust the size and time scale of the data segment according to the system load and performance indicators to achieve the purpose of optimizing the export of historical data.

[0071] In another embodiment of the present invention, before the step S330, the following steps are included:

[0072] S331, determining whether the size of the shard request data is greater than the size of the basic data shard;

[0073] If not, execute step S332 and process directly without segmentation;

[0074] If yes, execute step S333 to segment the fragment request data into multiple fragment request data according to the basic data segments and process them according to the time scale.

[0075] Specifically, when the shard request data is small, it does not need to be split.

[0076] In another embodiment of the present invention, the step S330 includes:

[0077] S334, calculating and obtaining the total data size of the shard request data;

[0078] S335, determining the number of shards N=ceil(T / D), where: D represents the size of the basic data segment, and T represents the total data size of the shard request data;

[0079] S336, dividing the slice request data into N segment request data, and processing them according to the time scale.

[0080] Specifically, during processing, the time scale is used to sequentially process from one end, and when the fragment request data is adjusted, the segment request data is sequentially adjusted at the other end, and ceil is a rounding function.

[0081] In another embodiment of the present invention, the step S330 includes:

[0082] S337, obtaining the starting time, ending time and size of the basic data segment of the fragment request data;

[0083] S338, =(ba) / N*[f(a)+f(a+h)+f(a+2h)+...+f(bh)]; where: a represents the starting time of the shard request data; b represents the end time of the shard request data; h represents the width of each basic request data; N represents the number of shards, represents the estimated completion time and required computing resources for the zth shard request data, and f(a) represents the computing resource estimation function required for data for h periods starting at time a.

[0084] S339, the slice request data is divided into N segment request data, and processed according to the time scale.

[0085] Specifically, the starting time and the ending time of each segmented request data can be calculated, and the estimated completion time and the required computing resources of each segmented request data can be obtained, thereby obtaining the completion time and the required computing resources of each segmented request data. When adjusting each segmented request data, the adjusted completion time and the required computing resources of each segmented request data can be obtained, thereby reasonably allocating the number of segmented request data of each segmented request data.

[0086] In another embodiment of the present invention, the step of S400 includes:

[0087] S410, obtaining load and performance indicators of each node server and system, as well as initial values ​​of data segment size and time scale in real time;

[0088] S420, according to the formula , adjust the size and time scale of each shard request data;

[0089] in: Indicates the size or time scale of the new shard request data of the z-th node server. The zth node server represents the size or time scale of the requested data segment at the initial moment, and α represents the impact parameter of the system load and performance index on the data segment size and time scale; Represents the system load and performance indicators of the z-th node server; β represents the impact parameter of data segment size and time scale on system load and performance indicators; γ represents the linear relationship coefficient between system load and performance indicators and data segment size and time scale; δ represents the impact parameter of the initial value of data segment size and time scale on system load and performance indicators; ε represents the impact parameter of the fluctuation of system load and performance indicators on data segment size and time scale; φ represents the impact parameter of the adjustment of data segment size and time scale on system load and performance indicators.

[0090] Specifically, the size and time scale of each query data segment are dynamically adjusted according to the real-time system load and performance indicators. Based on some key performance indicators, such as response time, throughput, CPU usage, etc., the size and time scale of the shard request data are dynamically adjusted, which can improve query efficiency, shorten query time, and also improve the system's load capacity. The number and allocation strategy of parallel queries can be dynamically adjusted according to actual conditions to achieve a load balancing effect for optimizing data export.

[0091] In another embodiment of the present invention, the step of S400 includes:

[0092] S410, obtaining real-time load and performance indicators of each node server through real-time calculation;

[0093] =λ*(CPU_Usage+Memory_Usage+Network_Usage+Disk_Usage)+μ*(Request_Rate / Response_Time); where: represents the load status of the zth node server; λ and μ represent constant coefficients related to system performance; CPU_Usage represents CPU usage; Memory_Usage represents memory usage; Network_Usage represents network usage; Disk_Usage represents disk usage; Request_Rate represents request rate; Response_Time represents response time;

[0094] S520, calculating the system load according to the real-time load and performance index of each node server;

[0095] S530, allocating the segmented request data of the node server whose load is higher than the system load to the node server whose load is lower than the system load.

[0096] Specifically, a weighted summation method is used to calculate the load status of the node server, taking into account the influence of multiple performance indicators. At the same time, weight coefficients λ and μ are used to adjust the importance of different indicators.

[0097] In another embodiment of the present invention, the step of S500 includes:

[0098] S510, integrating the result data returned by each of the node servers:

[0099] ;

[0100] in: Indicates the complete query result; Indicates the query result returned by the i-th server;

[0101] S520, encrypting the result data returned by each of the node servers using a data encryption algorithm;

[0102] S530, compressing the encrypted result data using a data compression algorithm;

[0103] S540: Generate complete query results and transmit them to a designated storage medium.

[0104] Specifically, the data export process = storage medium capacity * (shard size / transmission rate) + data encryption algorithm + data compression algorithm, where the storage medium capacity represents the total capacity of the storage medium, the shard size represents the amount of data in each shard, and the transmission rate represents the speed of data transmission; the data encryption algorithm is used to ensure the security of data transmission; the data compression algorithm is used to reduce the size of data transmission.

[0105] A readable storage medium, characterized in that the readable storage medium includes a computer program, and when the computer program is executed by a processor, the method for optimizing load balancing by exporting historical data as described in any of the above items is implemented.

[0106] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structural changes made using the contents of the present invention's specification and drawings, or directly / indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A method for optimizing load balancing by exporting historical data, characterized in that: The steps include: The client receives request data for exporting or querying historical data, and obtains the data volume of the requested data; Slice the request data according to the number of node servers and variables, divide the request data into a plurality of slice request data of equal size, and distribute the slice request data to each node server; Get system load and performance metrics; The size of the basic data segment is obtained through the formula: D=k1*y+k2*load+k3*(y^2)+k4*(load^2)+k5*(y*load), where: D represents the size of the basic data segment, y represents the time scale, load represents the system load and performance index; k1, k2, k3, k4 and k5 all represent coefficients; Dividing the segment request data into a plurality of segment request data according to the basic data segments and processing them according to a time scale; Obtaining system load and performance indicators, calculating and obtaining basic data segments, dividing the slice request data into multiple segment request data according to the basic data segments and processing them according to the time scale; Obtain the load and performance indicators of each node server and system in real time, as well as the initial values ​​of data segment size and time scale; According to the formula , adjust the size and time scale of each shard request data; in: Indicates the size or time scale of the new shard request data of the z-th node server. represents the size or time scale of the request data segment at the initial moment of the z-th node server, and α represents the influencing parameter of the system load and performance index on the data segment size and time scale; represents the system load and performance index of the z-th node server; β represents the influence parameter of data segment size and time scale on system load and performance index; γ represents the linear relationship coefficient between system load and performance index and data segment size and time scale; δ represents the influence parameter of the initial value of data segment size and time scale on system load and performance index; ε represents the influence parameter of the fluctuation of system load and performance index on data segment size and time scale; φ represents the influence parameter of adjustment of data segment size and time scale on system load and performance index; Integrate the results returned by each of the node servers to generate complete data export or query results to a specified storage medium.

2. The method for optimizing load balancing by exporting historical data as claimed in claim 1, characterized in that: The step of receiving request data for exporting or querying historical data by the client and obtaining the data volume of the request data comprises: The client receives request data for exporting or querying historical data; The data volume of the requested data is obtained according to the request type of the requested data and the time when the client receives the request.

3. The method for optimizing load balancing by exporting historical data as claimed in claim 1, characterized in that: Before the step of dividing the shard request data into a plurality of shard request data according to the basic data segments and processing them according to the time scale, the method includes: Determine whether the size of the shard request data is greater than the size of the basic data shard; If not, no segmentation is performed and the data is processed directly; If so, the segment request data is segmented into a plurality of segment request data according to the basic data segment and processed according to the time scale.

4. The method for optimizing load balancing by exporting historical data as claimed in claim 1, characterized in that: The step of dividing the slice request data into a plurality of segment request data according to the basic data segments and processing them according to the time scale includes: Calculate the total size of the data requested by the sharding request; Determine the number of shards N = ceil (T / D), where: D represents the size of the basic data segment, T represents the total size of the shard request data, and ceil represents the rounding function; The slice request data is divided into N segment request data, and processed according to the time scale.

5. The method for optimizing load balancing by exporting historical data as claimed in claim 1, characterized in that: The step of dividing the request data into N segmented request data comprises: Obtaining the starting time, ending time and size of the basic data segment of the slice request data; ; Where: a represents the starting time of the shard request data; b represents the end time of the shard request data; h represents the time period of each basic request data; N represents the number of shards, represents the estimated completion time and required computing resources for the zth shard request data, and f(a) represents the computing resource estimation function required for data for h periods starting at time a.

6. The method for optimizing load balancing by exporting historical data as claimed in claim 1, characterized in that: The step of acquiring the load and performance index of each node server and the system in real time and distributing the segmented request data of the node server with a load higher than the system load to the node server with a load lower than the system load comprises: Real-time calculation obtains the real-time load and performance indicators of each node server; ； in: represents the load status of the zth node server; λ and μ represent constant coefficients related to system performance; CPU_Usage represents CPU usage; Memory_Usage represents memory usage; Network_Usage represents network usage; Disk_Usage represents disk usage; Request_Rate represents request rate; Response_Time represents response time; Calculate system load based on real-time load and performance indicators of each node server; The segmented request data of the node server whose load is higher than the system load is distributed to the node server whose load is lower than the system load.

7. The method for optimizing load balancing by deriving historical data as described in any one of claims 1 to 6, characterized in that: The step of integrating the query results returned by each of the node servers to generate a complete query result includes: Integrate the result data returned by each node server: ； in: Indicates the complete query result; Indicates the query result returned by the i-th node server; Encrypting the result data returned by each of the node servers using a data encryption algorithm; The encrypted result data is compressed by a data compression algorithm; Generate complete query results and transfer them to the specified storage medium.

8. A readable storage medium, characterized in that: The readable storage medium includes a computer program, and when the computer program is executed by a processor, the method for optimizing load balancing by exporting historical data as described in any one of claims 1 to 7 is implemented.

Images