Data dynamic allocation method, device and storage medium

Abstract

The application discloses a data dynamic allocation method, device and storage medium. The data dynamic allocation method comprises the following steps: in response to a load balancing request for a data receiving node, obtaining the database data quantity corresponding to each pre-allocation database, the current data receivable quantity of each data receiving node, and the database allocated by each data receiving node in a previous allocation period; determining a first allocation result based on the database data quantity corresponding to each pre-allocation database and the current data receivable quantity of each data receiving node; determining a second allocation result based on the database allocated by each data receiving node in the previous allocation period; and obtaining a dynamic allocation result by combining the first allocation result and the second allocation result. The data dynamic allocation method can effectively reduce the switching of databases and data receiving nodes, reduce the performance consumption of data receiving nodes, and more quickly complete the load balancing of data receiving nodes while achieving the load balancing among the data receiving nodes.

Description

Technical Field

[0001] This application relates to the field of resource management technology, and in particular to a method, device and storage medium for dynamic data allocation. Background Technology

[0002] In current data processing scenarios, there are a large number of databases and nodes used for analyzing or storing data. It is necessary to allocate databases to these nodes for analysis or storage.

[0003] Because the amount of data in the database changes over time, and the number of nodes also increases or decreases dynamically, uneven load distribution among nodes often occurs in data processing scenarios. Summary of the Invention

[0004] This application provides at least one method, device, and storage medium for dynamic data allocation.

[0005] The first aspect of this application provides a dynamic data allocation method, the method comprising: in response to a load balancing request for a data receiving node, obtaining the database data volume corresponding to each pre-allocated database, the current data receivable volume of each data receiving node, and the databases allocated to each data receiving node in the previous allocation cycle; wherein, the database data volume is used to characterize the amount of data contained in the pre-allocated database, and the data receivable volume is used to characterize the amount of data that the data receiving node can receive; determining a first allocation result for each data receiving node based on the database data volume corresponding to each pre-allocated database and the current data receivable volume of each data receiving node; and determining a second allocation result for each data receiving node based on the databases allocated to each data receiving node in the previous allocation cycle; and combining the first allocation result and the second allocation result to obtain a dynamic allocation result between each pre-allocated database and each data receiving node.

[0006] In one embodiment, obtaining the amount of database data corresponding to each pre-allocated database includes: obtaining the actual amount of data currently contained in the pre-allocated database; predicting the amount of data contained in the pre-allocated database within a preset time period based on the actual amount of data, and obtaining the amount of database data; wherein the preset time period is after the current time.

[0007] In one embodiment, based on the actual data volume, the amount of data contained in the pre-allocated database within a preset time period is predicted to obtain the database data volume. This includes: statistically analyzing the changes in the data volume of the pre-allocated database and calculating the difference in the maximum data received by each data receiving node; using the actual data volume, the changes in data volume, and the difference in the received data volume, the amount of data contained in the pre-allocated database within the preset time period is predicted to obtain the database data volume.

[0008] In one embodiment, determining the first allocation result for each data receiving node based on the amount of database data corresponding to each pre-allocated database and the current data receiving capacity of each data receiving node includes: sorting each pre-allocated database in descending order based on the amount of database data corresponding to each pre-allocated database to obtain a database sorting result; using the database sorting result to select a target allocation database from each pre-allocated database; selecting a target data receiving node from each data receiving node based on the current data receiving capacity of each data receiving node; and using the target allocation database and the target data receiving node as the first allocation result.

[0009] In one embodiment, the target allocation database is selected from each pre-allocated database using the database sorting results, including: obtaining the number of databases selected; and selecting the pre-allocated database that ranks higher in the database sorting results based on the number of databases selected, thereby obtaining the target allocation database.

[0010] In one embodiment, obtaining the number of databases selected includes: detecting the amount of database data corresponding to the remaining pre-allocated databases; and calculating the number of databases selected based on the amount of database data corresponding to the remaining pre-allocated databases and the current data receiving capacity of each data receiving node.

[0011] In one embodiment, determining the second allocation result for each data receiving node based on the database allocated by each data receiving node in the previous allocation period includes: using the database allocated by each data receiving node in the previous allocation period as the affinity database corresponding to each data receiving node in the current allocation period; selecting the affinity database corresponding to each data receiving node in each pre-allocated database to obtain the second allocation result.

[0012] In one embodiment, by combining the first allocation result and the second allocation result, a dynamic allocation result between each pre-allocated database and each data receiving node is obtained, including: if the first allocation result and the second allocation result of the data receiving node have the same pre-allocated database, then the same pre-allocated database is allocated to the data receiving node; if the first allocation result and the second allocation result of the data receiving node do not have the same database, then the pre-allocated database with the largest amount of database data in the first allocation result is allocated to the data receiving node.

[0013] A second aspect of this application provides a data dynamic allocation apparatus, comprising: a request response module, configured to respond to a load balancing request for a data receiving node, and obtain the database data volume corresponding to each pre-allocated database, the current data receivable volume of each data receiving node, and the databases allocated to each data receiving node in the previous allocation cycle; wherein the database data volume is used to characterize the amount of data contained in the pre-allocated database, and the data receivable volume is used to characterize the amount of data that the data receiving node can receive; an initial allocation module, configured to determine a first allocation result for each data receiving node based on the database data volume corresponding to each pre-allocated database and the current data receivable volume of each data receiving node; and to determine a second allocation result for each data receiving node based on the databases allocated to each data receiving node in the previous allocation cycle; and a result generation module, configured to combine the first allocation result and the second allocation result to obtain a dynamic allocation result between each pre-allocated database and each data receiving node.

[0014] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the aforementioned dynamic data allocation method.

[0015] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described dynamic data allocation method.

[0016] The above scheme, in response to load balancing requests for data receiving nodes, obtains the database data volume corresponding to each pre-allocated database, the current data receiving capacity of each data receiving node, and the databases allocated to each data receiving node in the previous allocation cycle. Based on the database data volume corresponding to each pre-allocated database and the current data receiving capacity of each data receiving node, it determines the first allocation result for each data receiving node. Based on the databases allocated to each data receiving node in the previous allocation cycle, it determines the second allocation result for each data receiving node. Combining the first and second allocation results, it obtains the dynamic allocation result between each pre-allocated database and each data receiving node. This scheme can effectively reduce the switching between databases and data receiving nodes, reduce node performance consumption, and complete node load balancing faster while achieving load balancing between data receiving nodes.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0019] Figure 1 This is a schematic diagram of the implementation environment of a solution provided in one embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating a dynamic data allocation method in an exemplary embodiment of this application;

[0021] Figure 3 This is a flowchart illustrating a database allocation process, as shown in an exemplary embodiment of this application.

[0022] Figure 4 This is a schematic diagram illustrating database allocation in an exemplary embodiment of this application;

[0023] Figure 5 This is a schematic diagram illustrating database allocation as shown in another exemplary embodiment of this application;

[0024] Figure 6 This is a block diagram illustrating a data dynamic allocation apparatus according to an exemplary embodiment of this application;

[0025] Figure 7 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of this application;

[0026] Figure 8 This is a schematic diagram illustrating the structure of a computer-readable storage medium, as shown in an exemplary embodiment of this application. Detailed Implementation

[0027] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0028] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0029] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0030] The data dynamic allocation method provided in the embodiments of this application will be described below.

[0031] Please refer to Figure 1The diagram illustrates an implementation environment provided in one embodiment of this application. This implementation environment may include a node set 110 and a database set 120. The node set 110 contains n data receiving nodes, and the database set 120 contains m databases.

[0032] The database contains a collection of object data of any type, such as device information, time, and location data; the data receiving node is used to analyze or store one or more databases in the database collection 120.

[0033] The data dynamic allocation method provided in this application embodiment can be executed by any data receiving node in the node set 110, or by a data allocation module 130 that is independently deployed relative to the node set 110. This application does not limit the execution subject of the data allocation module 130.

[0034] Please see Figure 2 , Figure 2 This is a flowchart illustrating a dynamic data allocation method in an exemplary embodiment of this application. This dynamic data allocation method can be applied to... Figure 1 The implementation environment shown is specifically executed by the data allocation module within that implementation environment. It should be understood that this method can also be applied to other exemplary implementation environments and executed by devices in other implementation environments; this embodiment does not limit the implementation environment to which the method is applicable.

[0035] like Figure 2 As shown, the dynamic data allocation method includes at least steps S210 to S230, which are described in detail below:

[0036] Step S210: In response to the load balancing request for the data receiving node, obtain the database data volume corresponding to each pre-allocated database, the current data receivable volume of each data receiving node, and the databases allocated to each data receiving node in the previous allocation cycle; wherein, the database data volume is used to characterize the amount of data contained in the pre-allocated database, and the data receivable volume is used to characterize the amount of data that the data receiving node can receive.

[0037] Data receiving nodes can store or analyze the allocated database. For example, data receiving nodes can provide business services such as data query services and data statistics services based on the allocated database.

[0038] In real-world applications, the amount of data in the database and the number of data receiving nodes can vary, which can lead to uneven load distribution among the data receiving nodes. Therefore, it is necessary to flexibly adjust the database allocation strategy according to the actual situation.

[0039] A load balancing request is used to request load balancing among the current data receiving nodes.

[0040] For example, load balancing requests can be generated automatically and periodically, such as generating a load balancing request at preset intervals.

[0041] For example, a load balancing request can also be generated after a specified event is detected, such as calculating the load of each data receiving node. If the load of any data receiving node is greater than a preset load threshold, a load balancing request is generated. Alternatively, a load balancing request is generated if one or more of the following are detected: an increase or decrease in data receiving nodes; a change in the data processing capability of data receiving nodes (e.g., due to hardware or software factors); an increase or decrease in databases; or a change in the amount of data in a database that exceeds a preset threshold.

[0042] Based on the load balancing request, obtain the database data volume corresponding to each pre-allocated database, the current data receivable volume of each data receiving node, and the database allocated to each data receiving node in the previous allocation cycle. Among them, the database data volume is used to represent the amount of data contained in the pre-allocated database, and the data receivable volume is used to represent the amount of data that the data receiving node can receive. It can be the total amount of data that the data receiving node can receive, or the amount of data that the data receiving node currently has remaining receivable volume.

[0043] It should be noted that the data receiving node may be a newly created node. If the data receiving node is a newly created node, then the database allocated to it in the previous allocation cycle was empty; if the data receiving node is not a newly created node, then the data receiving node may have already been allocated a database in the previous allocation cycle. The previous allocation cycle refers to the time period before the load balancing request was received.

[0044] It should also be noted that the pre-allocated database obtained refers to the database that can be used at present. It may contain newly added databases, as well as databases that have been allocated to data receiving nodes in the previous allocation cycle and have not been deleted.

[0045] For example, suppose there are n pre-allocated databases in the current allocation cycle, and let any pre-allocated database be denoted as c. i , i∈{1,2,3...,n}.

[0046] Step S220: Based on the amount of database data corresponding to each pre-allocated database and the current amount of data that each data receiving node can receive, determine the first allocation result of each data receiving node; and based on the databases allocated to each data receiving node in the previous allocation cycle, determine the second allocation result of each data receiving node.

[0047] Based on the amount of data in each pre-allocated database and the current data receiving capacity of each data receiving node, the first allocation result for each data receiving node is determined.

[0048] Optionally, the first allocation result may contain one or more pre-allocated databases.

[0049] For example, based on the amount of database data corresponding to each pre-allocated database, each pre-allocated database is allocated to each data receiving node, so that the difference in load between the data receiving nodes is less than a preset load difference threshold, thus obtaining the first allocation result for each data receiving node. The load of each data receiving node can be calculated based on the amount of database data already allocated to the data receiving node and the remaining data that the data receiving node can receive.

[0050] For example, since different data receiving nodes have different data processing capability preferences, such as some data receiving nodes having higher performance for floating-point operations and some data receiving nodes having better performance for integer operations, the performance parameters of the data receiving nodes for each pre-allocated database can be detected. Then, by combining the database data volume corresponding to each pre-allocated database and its performance parameters with each data receiving node, each pre-allocated database is allocated to each data receiving node, so that the difference between the loads of each data receiving node is less than a preset load difference threshold, and the sum of the performance parameters of each pre-allocated database is greater than a preset performance threshold.

[0051] The method for determining the first allocation result of the data receiving node based on the amount of data in each pre-allocated database and the current data receiving capacity of each data receiving node can be flexibly determined for different application scenarios, and this application does not limit it.

[0052] Based on the databases allocated by each data receiving node in the previous allocation cycle, determine the second allocation result for each data receiving node.

[0053] In the second allocation result of the data receiving node, there may be no database. For example, if the data receiving node is a newly added data receiving node, then the data receiving node was not allocated a database in the previous allocation cycle, and the second allocation result will be empty.

[0054] The second allocation result of the data receiving node may also contain one or more databases.

[0055] For example, if the data receiving node is not a newly added data receiving node, and this data receiving node allocated multiple databases in the previous allocation cycle, and none of these databases have been deleted, then all of these databases can be used as the second allocation result for this data receiving node. Alternatively, these databases can be filtered. Filtering methods could include: selecting databases whose data volume exceeds a preset data volume threshold to obtain the second allocation result for this data receiving node; detecting the performance parameters of the data receiving node for each database, and then selecting databases whose performance parameters exceed a preset performance threshold to obtain the second allocation result for this data receiving node; or combining the database's data volume and the data receiving node's performance parameters for each database to calculate a selection recommendation index for each database, and then selecting databases whose selection recommendation index exceeds a preset index threshold to obtain the second allocation result for this data receiving node.

[0056] For example, the databases allocated to each data receiving node in the previous allocation period are used as the affinity databases corresponding to each data receiving node in the current allocation period; the affinity databases corresponding to each data receiving node in each pre-allocated database are selected to obtain the second allocation result.

[0057] The method for determining the second allocation result of the data receiving node based on the database allocated by the data receiving node in the previous allocation cycle can be flexibly determined for different application scenarios, and this application does not limit it.

[0058] Step S230: Combine the first allocation result and the second allocation result to obtain the dynamic allocation result between each pre-allocated database and each data receiving node.

[0059] The first allocation result is obtained by considering the data volume dimension, which can ensure the load balance of the data receiving nodes. The second allocation result is obtained by considering the databases allocated to the data receiving nodes in the previous allocation cycle, which can reduce the amount of data migration. Combining the first and second allocation results, the dynamic allocation result between each pre-allocated database and each data receiving node is obtained. This dynamic allocation result can comprehensively consider the load balance of the data receiving nodes and the amount of data migration, thereby improving the effectiveness of database allocation.

[0060] It should be noted that different scenarios have different load balancing requirements and data migration requirements. The method of combining the first allocation result and the second allocation result to obtain the final dynamic allocation result can be flexibly determined according to the actual situation. This application does not limit this.

[0061] The following are examples illustrating some embodiments of this application.

[0062] In some implementations, step S210 involves obtaining the amount of database data corresponding to each pre-allocated database, including:

[0063] Step S211: Obtain the actual amount of data currently contained in the pre-allocated database.

[0064] Actual data volume refers to the actual amount of data currently contained in the pre-allocated database.

[0065] For example, let the pre-allocated database be denoted as c. i , i∈{1,2,3...,n}, the actual amount of data pre-allocated in the database. This can be expressed as Formula 1:

[0066]

[0067] In Formula 1, v(t) represents the pre-allocated database c. i The data ingestion rate is a function of time t, where p is the pre-allocated database c. i The data retention period refers to the retention period range, which means that data whose entry time is within the retention period range will be retained, while data that is outside the retention period range will be deleted after it expires.

[0068] Step S212: Based on the actual data volume, predict the amount of data to be pre-allocated in the database within a preset time period to obtain the database data volume.

[0069] Based on the actual amount of data in the pre-allocated database, predict the amount of data the pre-allocated database will contain within a preset time period, which refers to a future time period after the current time.

[0070] For example, the changes in the amount of data in the pre-allocated database are statistically analyzed, and the difference in the amount of data between the actual amounts of data in each pre-allocated database is calculated. Using the actual amounts of data, the changes in amounts of data, and the differences in amounts of data, the amount of data contained in the pre-allocated database within a preset time period is predicted, and the amount of data in the database is obtained.

[0071] Record the changes in the amount of data in the pre-allocated database to statistically analyze the changes in the amount of data in the pre-allocated database. These changes reflect the trend of data changes in the amount of data in the pre-allocated database, such as a decrease in the amount of data, an increase in the amount of data, the rate of decrease in the amount of data, or the rate of increase in the amount of data.

[0072] The difference in data volume between the actual data volumes of each pre-allocated database can be obtained by calculating the difference between the maximum and minimum actual data volumes in each pre-allocated database.

[0073] By using the actual data volume, data volume changes, and data volume differences, the amount of data to be pre-allocated in the database within a preset time period is predicted, and the database data volume is obtained.

[0074] For example, let's say the pre-allocated database c i The actual data volume is expressed as The formula for predicting the amount of data to be pre-allocated in the database within a preset time period can be expressed as Formula 2:

[0075]

[0076] In formula 2, Indicates the pre-allocation of database c i The amount of data obtained from the prediction; This represents the maximum actual amount of data across all pre-allocated databases. This represents the minimum actual amount of data in all pre-allocated databases; r represents the difference between the maximum number of databases that can be allocated to data receiving nodes of the same type, where data receiving nodes of the same type can be data receiving nodes with the same data processing capability preference; This indicates that the pre-allocated database c is used. i The changes in data volume are predicted to indicate changes in data volume within a preset time period.

[0077] Specifically, The calculation method can be expressed as Formula 3:

[0078]

[0079] In Formula 3, p represents the pre-allocated database c. i The data retention time range; q is the length of the preset time period. Combining formulas 1, 2, and 3, we get formula 4:

[0080]

[0081] Based on Formula 4, the amount of data to be pre-allocated in the database within a preset time period is predicted, and the amount of database data is obtained.

[0082] For example, other methods can also be used to predict the amount of data contained in the pre-allocated database within a preset time period. For example, based on the statistical changes in the amount of data in the pre-allocated database, the amount of data in the pre-allocated database within the preset time period can be directly calculated, and then the maximum amount of data within the preset time period can be selected as the amount of database data. This application does not limit the method of predicting the amount of database data.

[0083] By predicting the amount of data in the database, the number of times data changes in the database can trigger the reallocation of data receiving nodes within a certain period of time, thereby reducing the need for database migration.

[0084] Of course, in addition to obtaining the library data volume through prediction, the actual data volume can also be directly used as the library data volume to reduce the required computing resources.

[0085] Then, based on the amount of data in each pre-allocated database, the current amount of data that each data receiving node can receive, and the databases allocated to each data receiving node in the previous allocation cycle, the corresponding pre-allocated database is allocated to each data receiving node.

[0086] It should be noted that when allocating pre-allocated databases to data receiving nodes, it is necessary to ensure that each pre-allocated database is assigned a data receiving node, and that the total amount of database data allocated to all databases by the data receiving node is less than the maximum amount of data that can be allocated to that data receiving node.

[0087] For example, given n pre-allocated databases and m data receiving nodes, any data receiving node can be represented as a. j Let j∈{1,2,3,...,m}, and denote the data receiving node a. j The allocated database set is The database sets allocated to each data receiving node satisfy Data receiving node a j The total amount of data in all allocated databases is If data receiving node a j The corresponding maximum number of allocatable data is A. j ,but

[0088] The following example illustrates how to allocate a pre-allocated database to each data receiving node.

[0089] In some implementations, step S220 determines the first allocation result for each data receiving node based on the amount of database data corresponding to each pre-allocated database and the current data receiving capacity of each data receiving node, including:

[0090] Step S221: Based on the size of the database data corresponding to each pre-allocated database, sort each pre-allocated database in descending order to obtain the database sorting result.

[0091] Step S222: Using the database sorting results, select the target allocation database from each pre-allocated database.

[0092] For example, the number of databases selected is obtained; based on the number of databases selected, the pre-allocated databases that rank higher in the database sorting results are selected to obtain the target allocated database.

[0093] For example, if the number of databases selected is k, then the k pre-allocated databases that rank first in the database sorting results are selected to obtain the target allocated database.

[0094] The number of databases selected can be preset or calculated flexibly according to the actual situation. For example, the number of databases selected can be calculated based on at least one of the following: the amount of database data corresponding to the currently remaining pre-allocated databases, the amount of data that each data receiving node can currently receive.

[0095] For example: Detect the amount of database data corresponding to the remaining pre-allocated databases; based on the amount of database data corresponding to the remaining pre-allocated databases and the current data receiving capacity of each data receiving node, calculate the number of databases to be selected.

[0096] Specifically, the number of database selections can be calculated using Formula 5:

[0097]

[0098] In Formula 5, A j Indicates data receiving node a j The corresponding maximum number of data that can be allocated; For data receiving node a j The sum of the total amount of data in all the allocated databases; the difference is the data receiving node a. j The current data receiving node's data receiving capacity; This indicates selecting the minimum amount of data that can be received from each data receiving node; c VF `max` represents the maximum amount of data in the remaining pre-allocated database.

[0099] Step S223: Based on the current data receiving capacity of each data receiving node, select the target data receiving node from among the data receiving nodes.

[0100] If the data receiving node with the largest data receiving capacity is selected as the target data receiving node, the data receiving node with a data receiving capacity greater than the preset receiving capacity threshold can also be selected as the target data receiving node.

[0101] Step S224: The target allocation database and the target data receiving node are used as the first allocation result.

[0102] It should be noted that the first allocation result obtained at this time is the first allocation result of the current stage. After the database allocation of the current stage is completed by combining the second allocation result, there may still be unallocated pre-allocated databases. For example, there are still unselected pre-allocated databases in step S222. Steps S220 and S230 need to be executed iteratively until all pre-allocated databases are allocated.

[0103] Specifically, in conjunction with steps S221 to S224, an example is given to illustrate the specific implementation method of obtaining the dynamic allocation results between each pre-allocated database and each data receiving node by combining the first allocation result and the second allocation result.

[0104] For example, taking the databases allocated to each data receiving node in the previous allocation period as the affinity databases corresponding to each data receiving node in the current allocation period, and selecting the affinity databases corresponding to each data receiving node in each pre-allocated database to obtain the second allocation result, please refer to [example example]. Figure 3 , Figure 3 A database allocation flowchart is shown as an exemplary embodiment of this application, such as Figure 3 As shown, it includes:

[0105] Step 310: Based on the size of the database data corresponding to each pre-allocated database, sort each pre-allocated database in descending order to obtain the database sorting result;

[0106] Step 320: Has the pre-allocation of the database been completed? If not, proceed to steps 330 to 360; if yes, proceed to step 370.

[0107] Step 330: Filter the target allocation database from the pre-allocated database;

[0108] Step 340: Select the target data receiving node from the data receiving nodes;

[0109] Step 350: Select the appropriate database from the target allocation database based on the affinity database of the target data receiving node;

[0110] Step 360: Has the target database allocation been completed? If not, return to step 340; otherwise, return to step 320.

[0111] Step 370: Combine the corresponding adaptation database for each target allocation database to obtain the dynamic allocation results between each pre-allocated database and each data receiving node.

[0112] The process of selecting an adaptation database from the target allocation database based on the affinity database of the target data receiving node may include: if the first allocation result and the second allocation result of the data receiving node have the same pre-allocated database, then the same pre-allocated database is assigned as the adaptation database to the data receiving node; if the first allocation result and the second allocation result of the data receiving node do not have the same database, then the pre-allocated database with the largest amount of data in the first allocation result is assigned as the adaptation database to the data receiving node.

[0113] Selecting an appropriate database from the target allocation database based on the affinity database of the target data receiving node may further include: calculating a first allocation recommendation value between the pre-allocated database and the target data receiving node based on the amount of database data in the target allocation database and / or the amount of data the target data receiving node can receive in the first allocation result, wherein the larger the amount of data the target data receiving node can receive and the larger the amount of database data in the pre-allocated database in the first allocation result, the higher the first allocation recommendation value; calculating a second allocation recommendation value between the pre-allocated database and the target data receiving node based on the amount of database data in the pre-allocated database and / or the performance parameters between the target data receiving node and the pre-allocated database in the second allocation result, wherein the higher the processing efficiency of the pre-allocated database in the second allocation result, the higher the second allocation recommendation value; then, performing a weighted calculation on the first allocation recommendation value and the second allocation recommendation value of the pre-allocated database to obtain a comprehensive allocation recommendation value for each pre-allocated database in the first allocation result relative to the target data receiving node, and selecting the pre-allocated database with the highest comprehensive allocation recommendation value in the first allocation result as the appropriate database to be allocated to the data receiving node.

[0114] In some implementations, the difference in the number of databases allocated between different data receiving nodes can be limited to avoid a large number of databases being allocated to the same data receiving node.

[0115] For example, there are currently 2 data receiving nodes and 20 pre-allocated databases. One pre-allocated database has 100 database entries, while the other 19 pre-allocated databases have 0 database entries. If the difference in the number of databases allocated is not limited, one data receiving node will be allocated one database with 100 database entries, while the other data receiving node will be allocated 19 databases with 0 database entries. If the databases with 100 database entries are subsequently increased (e.g., the database with 100 database entries increases to 120, and the other databases with 0 database entries all increase to 20), then the difference in the number of databases between the two data receiving nodes will be 260, resulting in a significant gap in the number of databases between the two data receiving nodes.

[0116] However, if the difference in the number of allocated databases is limited, such as not exceeding 6, then one data receiving node might be allocated one database with 100 data points and six databases with 0 data points, while another data receiving node might be allocated 13 databases with 0 data points. If the data points of each database are subsequently increased—for example, the database with 100 data points increases to 120, and the other databases with 0 data points all increase to 20—then the difference in the number of databases between the two data receiving nodes will be 20, thus reducing the gap in the number of databases between them.

[0117] The difference in the number of databases allocated between each data receiving node can be preset by technicians based on experience, or it can be flexibly calculated according to the actual application scenario. For example, if an initial difference in the number of databases is preset, the difference in the number of databases between each data receiving node is detected after the databases are allocated according to the initial difference in the number of databases. If the difference in the number of databases is greater than a preset threshold, the initial difference in the number of databases is reduced; otherwise, it remains unchanged. This application does not limit the method for determining the difference in the number of databases allocated between each data receiving node.

[0118] Next, using steps 310 to 370 as an example, we will explain the dynamic data allocation method in detail.

[0119] Assume there are currently data receiving nodes a1, a2, and a3, each with a maximum allocable data size of 30 (A1 = A2 = A3 = 30), and none of these nodes were allocated a database in the previous allocation cycle. Additionally, there are pre-allocated databases c1, c2, c3, c4, c5, c6, c7, and c8, with predicted database sizes of [database sizes to be filled in].

[0120] Based on the size of the database corresponding to each pre-allocated database, the databases are sorted in descending order. The sorted database results are shown in Table 1.

[0121]

[0122] Table 1

[0123] If the pre-allocation of databases is incomplete, select the target database from the pre-allocated databases in the database pool. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 A schematic diagram illustrating database allocation as shown in an exemplary embodiment of this application, such as Figure 4 As shown, the number of database selections is calculated. The selected target allocation databases include c2, c7, and c6. c2, c7, and c6 are added to a temporary queue, and c2, c7, and c6 are removed from the database pool.

[0124] Select the target data receiving node. Since the data receiving capacity and load of the current target data receiving nodes are consistent, any data receiving node can be randomly selected as the target data receiving node, or a data receiving node with an affinity database in the temporary queue can be selected as the target data receiving node. In this case, a1 is selected as the target data receiving node.

[0125] The appropriate database is selected from the target allocation database based on the affinity database of the target data receiving node. Since none of the current data receiving nodes were allocated a database in the previous allocation cycle, this is the first allocation. Therefore, none of the data receiving nodes have an affinity database. Thus, a1 selects the pre-allocated database c2 with the largest amount of database data in the temporary queue as the appropriate database and removes c2 from the temporary queue.

[0126] If the target allocation database still exists in the temporary queue, the allocation process continues, and a new target data receiving node is selected. The data receiving capacity and / or load of each data receiving node are calculated. The target data receiving node is selected based on the data receiving capacity and / or load. For example, if the data receiving capacity and load of data receiving nodes a2 and a3 are the same, and neither has a corresponding affinity database in the temporary queue, then a2 is randomly selected as the target data receiving node.

[0127] Since none of the current data receiving nodes were allocated a database in the previous allocation cycle, and this is the first allocation, none of the data receiving nodes have an affinity database. Therefore, a2 selects the pre-allocated database c7 with the largest amount of database data in the temporary queue as the adaptation database and removes c7 from the temporary queue.

[0128] Similarly, a3 selects the pre-allocated database c6 with the largest amount of database data in the temporary queue as the adaptation database, and removes c6 from the temporary queue.

[0129] At this point, it is determined that the target allocation database in the temporary queue has been fully allocated, while the pre-allocated database has not been fully allocated. The number of databases to be selected is then recalculated. The selected target allocation databases include c1, c4, c5, and c3. Allocation continues sequentially in the manner described above. See the allocation details below. Figure 4 This process continues until the pre-allocated databases are fully allocated. The corresponding adapted databases for a1 are c2, c5, and c3; the corresponding adapted databases for a2 are c7 and c4; and the corresponding adapted databases for a3 are c6, c1, and c8. This completes the current database allocation.

[0130] After a period of time, the amount of data in the database changes over time. The results of the database changes over time are shown in Table 2 below:

[0131]

[0132] Table 2

[0133] The amount of data that each data receiving node can receive after the amount of data in the database changes over time is shown in Table 3 below:

[0134] <![CDATA[a1]]> <![CDATA[c2 c5 c3]]> 3 <![CDATA[a2]]> <![CDATA[c7 c4]]> 14 <![CDATA[a3]]> <![CDATA[c6 c1 c8]]> 15

[0135] Table 3

[0136] As shown in Table 3, the load on data receiving node a1 is significantly higher than that on other data receiving nodes. This triggers a load balancing request to reallocate the database. It should be noted that besides uneven load distribution among data receiving nodes, load balancing requests can also be triggered by the detection of additions or removals of data receiving nodes or databases.

[0137] Since there are no additions or subtractions of data receiving nodes and databases, there are currently data receiving nodes a1, a2, and a3, and pre-allocated databases c1, c2, c3, c4, c5, c6, c7, and c8. Their sorting results at the current moment are shown in Table 2.

[0138] If the pre-allocation of databases is incomplete, select the target database from the pre-allocated databases in the database pool. Please refer to [link to relevant documentation]. Figure 5 , Figure 5 A schematic diagram illustrating database allocation as shown in an exemplary embodiment of this application, such as Figure 5 As shown, the number of database selections is calculated. The selected target allocation databases include c5, c2, and c7. c5, c2, and c7 are added to a temporary queue, and c5, c2, and c7 are removed from the database pool.

[0139] Select the target data receiving node. Since the data receiving capacity and load of the current target data receiving nodes are consistent, any data receiving node can be randomly selected as the target data receiving node, or a data receiving node with an affinity database in the temporary queue can be selected as the target data receiving node. In this case, a1 is selected as the target data receiving node.

[0140] The appropriate database is selected from the target allocation database based on the affinity database of the target data receiving node. If there are affinity databases c5 and c2 corresponding to data receiving node a1 in the temporary queue, then a1 selects the pre-allocated database c5 with the largest amount of data in the affinity database as the appropriate database and removes c5 from the temporary queue.

[0141] If the target allocation database still exists in the temporary queue, the allocation process continues, and a new target data receiving node is selected. The data receiving capacity and / or load of each data receiving node are calculated. The target data receiving node is selected based on this capacity and / or load. For example, if data receiving nodes a2 and a3 have the same data receiving capacity and load, and an affinity database c7 corresponding to data receiving node a2 exists in the temporary queue, then a2 is selected as the target data receiving node.

[0142] a2 selects the affinity database c7 in the temporary queue as the adaptation database and removes c7 from the temporary queue.

[0143] Similarly, a3 selects the pre-allocated database c2 with the largest amount of database data in the temporary queue as the adaptation database, and removes c2 from the temporary queue.

[0144] At this point, it is determined that the target allocation database in the temporary queue has been fully allocated, while the pre-allocated database has not been fully allocated. The number of databases to be selected is then recalculated. The selected target allocation databases include c6 and c3. Allocation continues sequentially in the manner described above. See the allocation details below. Figure 5 This process continues until the pre-allocated databases are fully allocated. The corresponding adapted databases for a1 are c5, c4, and c8; for a2, c7 and c3; and for a3, c2, c6, and c1. This completes the current database allocation.

[0145] After allocation, the data receiving capacity corresponding to a1 is 10, the data receiving capacity corresponding to a2 is 13, and the data receiving capacity corresponding to a3 is 9, which reduces the load difference between the data receiving nodes. In addition, compared with the database allocation results of the previous period, it can be seen that a1 needs to load databases c4 and c8 and unload databases c2 and c3; a2 needs to load database c3 and unload database c4; a3 needs to load database c2 and unload database c8.

[0146] In some implementations, the first allocation result can also be the average amount of data to be allocated. Then, combining the first allocation result and the second allocation result, the dynamic allocation result between each pre-allocated database and each data receiving node can include: comparing the current allocated data amount of the data receiving node with the average amount of data to be allocated. If the allocated data amount is greater than the average amount of data to be allocated, the database with the smallest outgoing data amount is selected from the databases already allocated by the data receiving node as the redistribution database, and the data receiving node with the largest value of the average amount of data to be allocated minus the allocated data amount is selected as the database redistribution node. The redistribution database is removed from the data receiving node and assigned to the database redistribution node. The above steps are repeated until each data receiving node meets at least one of the load balancing condition and the data migration amount condition. The load balancing condition and the data migration amount condition can be flexibly set according to the actual application scenario.

[0147] For example, in the previous allocation cycle, the adapted databases corresponding to a1 include c2, c5, and c3; the adapted databases corresponding to a2 include c7 and c4; and the adapted databases corresponding to a3 include c6, c1, and c8, thus completing the current database allocation.

[0148] After a period of time, the amount of data in the database changes over time, as shown in Table 2. After the amount of data in the database changes over time, the data receiving capacity of each data receiving node becomes as shown in Table 3, leading to an imbalance in the load among the data receiving nodes. This triggers a load balancing request to reallocate the database. The load balancing condition is set as follows: the difference between the maximum and minimum allocated data volume of a data receiving node is less than 3; the data migration condition is that the number of databases to be migrated is less than 6.

[0149] Based on the current data volume of each pre-allocated database, the average data volume to be allocated to each data receiving node is calculated to be 19. Since the current allocated data volume of a1 is greater than the average data volume to be allocated, the database c3 with the smallest outgoing data volume is selected from the databases already allocated to a1 as the reassigned database. The data receiving node a3 with the largest value of the average data volume to be allocated minus the allocated data volume is selected as the database reassignment node. c3 is removed from a1 and assigned to a3.

[0150] At this time, the allocated data volume of a1 is 19, the allocated data volume of a2 is 16, and the allocated data volume of a3 is 23. The current allocated data volume of a3 is greater than the average data volume that should be allocated. Therefore, the database c8 with the smallest outgoing data volume is selected from the databases allocated by a3 as the redistribution database, and the data receiving node a2 with the largest value of the average data volume that should be allocated minus the allocated data volume is selected as the database redistribution node. c8 is removed from a3 and assigned to a2.

[0151] At this point, the allocated data volume of a1 is 19, the allocated data volume of a2 is 19, and the allocated data volume of a3 is 20. The difference between the maximum allocated data volume and the minimum allocated data volume is less than 3, which satisfies the load balancing condition, and the allocation ends.

[0152] The data dynamic allocation method provided in this application, in response to load balancing requests for data receiving nodes, obtains the database data volume corresponding to each pre-allocated database, the current data receivable volume of each data receiving node, and the databases allocated to each data receiving node in the previous allocation cycle. Based on the database data volume corresponding to each pre-allocated database and the current data receivable volume of each data receiving node, a first allocation result for each data receiving node is determined. Based on the databases allocated to each data receiving node in the previous allocation cycle, a second allocation result for each data receiving node is determined. Combining the first and second allocation results, a dynamic allocation result between each pre-allocated database and each data receiving node is obtained. This method can effectively reduce the switching between databases and data receiving nodes, reduce node performance consumption, and complete node load balancing faster, while achieving load balancing between data receiving nodes.

[0153] Figure 6 This is a block diagram illustrating a data dynamic allocation apparatus as shown in an exemplary embodiment of this application. Figure 6 As shown, the exemplary data dynamic allocation device 600 includes: a request response module 610, an initial allocation module 620, and a result generation module 630. Specifically:

[0154] The request response module 610 is used to respond to load balancing requests for data receiving nodes and obtain the database data volume corresponding to each pre-allocated database, the current data receivable volume of each data receiving node, and the databases allocated to each data receiving node in the previous allocation cycle; wherein, the database data volume is used to represent the amount of data contained in the pre-allocated database, and the data receivable volume is used to represent the amount of data that the data receiving node can receive.

[0155] The initial allocation module 620 is used to determine the first allocation result of each data receiving node based on the amount of database data corresponding to each pre-allocated database and the current data receiving capacity of each data receiving node; and to determine the second allocation result of each data receiving node based on the databases allocated to each data receiving node in the previous allocation cycle.

[0156] The result generation module 630 is used to combine the first allocation result and the second allocation result to obtain the dynamic allocation result between each pre-allocated database and each data receiving node.

[0157] It should be noted that the data dynamic allocation device and the data dynamic allocation method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the data dynamic allocation device provided in the above embodiments can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0158] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 700 includes a memory 701 and a processor 702. The processor 702 is used to execute program instructions stored in the memory 701 to implement the steps in any of the above-described embodiments of the dynamic data allocation method. In a specific implementation scenario, the electronic device 700 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 700 may also include mobile devices such as laptops and tablets, which are not limited here.

[0159] Specifically, processor 702 controls itself and memory 701 to implement the steps in any of the above-described embodiments of the dynamic data allocation method. Processor 702 can also be referred to as a Central Processing Unit (CPU). Processor 702 may be an integrated circuit chip with signal processing capabilities. Processor 702 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 702 can be implemented using integrated circuit chips.

[0160] Please see Figure 8 , Figure 8 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 800 stores program instructions 810 that can be executed by a processor. The program instructions 810 are used to implement the steps in any of the above-described embodiments of the dynamic data allocation method.

[0161] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0162] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0163] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0164] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for dynamic data allocation, characterized in that, The method includes: In response to a load balancing request for a data receiving node, the database data volume corresponding to each pre-allocated database, the current data receivable volume of each data receiving node, and the databases allocated to each data receiving node in the previous allocation cycle are obtained; wherein, the database data volume is used to characterize the amount of data contained in the pre-allocated database, and the data receivable volume is used to characterize the amount of data that the data receiving node can receive. Based on the amount of database data corresponding to each pre-allocated database and the current data receiving capacity of each data receiving node, a first allocation result for each data receiving node is determined; and based on the databases allocated to each data receiving node in the previous allocation cycle, a second allocation result for each data receiving node is determined. By combining the first allocation result and the second allocation result, a dynamic allocation result between each pre-allocated database and each data receiving node is obtained.

2. The method according to claim 1, characterized in that, The step of obtaining the amount of database data corresponding to each pre-allocated database includes: Obtain the actual amount of data currently contained in the pre-allocated database; Based on the actual data volume, the amount of data contained in the pre-allocated database within a preset time period is predicted to obtain the database data volume; wherein, the preset time period is after the current time.

3. The method according to claim 2, characterized in that, The step of predicting the amount of data contained in the pre-allocated database within a preset time period based on the actual data volume, and obtaining the database data volume, includes: The changes in the amount of data in the pre-allocated databases are statistically analyzed, and the difference in the amount of data between the actual amounts of data in each pre-allocated database is calculated. By using the actual data volume, the data volume change, and the data volume difference, the amount of data contained in the pre-allocated database within a preset time period is predicted, and the database data volume is obtained.

4. The method according to claim 1, characterized in that, The determination of the first allocation result for each data receiving node based on the amount of database data corresponding to each pre-allocated database and the current data receiving capacity of each data receiving node includes: Based on the size of the database data corresponding to each pre-allocated database, the pre-allocated databases are sorted in descending order to obtain the database sorting result. Using the database sorting results, the target allocation database is selected from each of the pre-allocated databases; Based on the current data receiving capacity of each data receiving node, a target data receiving node is selected from the data receiving nodes. The target allocation database and the target data receiving node are used as the first allocation result.

5. The method according to claim 4, characterized in that, The step of using the database sorting results to filter out the target allocation database from the various pre-allocated databases includes: Get the number of database entries selected; Based on the number of databases selected, the pre-allocated database that ranks first in the database sorting results is selected to obtain the target allocated database.

6. The method according to claim 5, characterized in that, The process of obtaining the number of database selections includes: Detect the amount of database data corresponding to the currently remaining pre-allocated database; Based on the amount of database data corresponding to the remaining pre-allocated databases and the current data receiving capacity of each data receiving node, the number of databases to be selected is calculated.

7. The method according to claim 1, characterized in that, The determination of the second allocation result for each data receiving node based on the database allocated by each data receiving node in the previous allocation cycle includes: The databases allocated to each data receiving node in the previous allocation period are used as the affinity databases corresponding to each data receiving node in the current allocation period. Select the affinity database corresponding to each data receiving node in each of the pre-allocated databases to obtain the second allocation result.

8. The method according to any one of claims 1 to 7, characterized in that, The step of combining the first allocation result and the second allocation result to obtain the dynamic allocation result between each pre-allocated database and each data receiving node includes: If the first allocation result and the second allocation result of the data receiving node have the same pre-allocated database, then the same pre-allocated database is allocated to the data receiving node. If the first allocation result and the second allocation result of the data receiving node do not have the same database, then the pre-allocated database with the largest amount of data in the first allocation result will be allocated to the data receiving node.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.

Images