Data transfer method and device, equipment, storage medium and program product

Abstract

The invention provides a data transfer method and device, equipment, a storage medium and a program product, and relates to the technical field of computers.The method comprises the steps that transfer coefficients of a cloud database are determined according to service information of the cloud database, the transfer coefficient is related to the priority level of at least one transfer task in the cloud database and the average read-write rate of the at least one transfer task; according to the business information and resource utilization information of at least one host machine in the cloud database, determining a computing power factor, the computing power factor being used for measuring a computing power balance degree and a computing power matching degree between the at least one transfer task and the at least one host machine; and transferring data in each transfer task in the at least one transfer task to the corresponding host machine based on the transfer coefficient and the computing power factor. According to the embodiment of the invention, the problem of cluster load imbalance can be solved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data transfer method, device, equipment, storage medium and program product. Background Art

[0002] Mobile Cloud Database uses binary log files (Binlog) to synchronize master and backup data, restore backup data, and perform off-site data disaster recovery. Currently, Mobile Cloud Database uses scheduled task processing in the Binlog transfer strategy, that is, when the transfer preconditions are met, the scheduled task is started to schedule the instance's Binlog to the appropriate node in the cluster. As the cluster size gradually increases, there may be multiple instances of Binlog that need to be transferred at the same time, resulting in cluster load imbalance. Summary of the invention

[0003] The embodiments of the present application provide a data transfer method, apparatus, device, storage medium and program product to solve the problem of cluster load imbalance caused by the need to transfer Binlogs of multiple instances at the same time.

[0004] To solve the above technical problems, this application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a data transfer method, the method comprising:

[0006] Determine a transfer coefficient of the cloud database according to the business information of the cloud database, wherein the transfer coefficient is related to the priority level of at least one transfer task in the cloud database and the average read and write rate of the at least one transfer task;

[0007] Determine a computing power factor according to the business information and resource utilization information of at least one host machine in the cloud database, where the computing power factor is used to measure a degree of computing power balance and a degree of computing power matching between the at least one transfer task and the at least one host machine;

[0008] Based on the transfer coefficient and the computing power factor, data in each transfer task in the at least one transfer task is transferred to the corresponding host machine.

[0009] Optionally, the transfer coefficient includes a first transfer coefficient, and determining the transfer coefficient of the cloud database according to the business information of the cloud database includes:

[0010] Acquire the business information, wherein the business information includes user level information corresponding to each user to which each transfer task in the at least one transfer task belongs;

[0011] According to the user level information, the priority level of each transfer task in the at least one transfer task is divided to determine the first transfer coefficient, where the first transfer coefficient is related to the priority level of at least one transfer task in the cloud database.

[0012] Optionally, the transfer coefficient includes a second transfer coefficient, and determining the transfer coefficient of the cloud database according to the business information of the cloud database includes:

[0013] Acquire the business information, the business information including first information corresponding to at least one historical transfer task in the cloud database, and disk capacity required for each of the at least one transfer task to be stored in each of the at least one host machine; wherein the first information includes a first parameter, task duration, and an average read and write rate of the transfer task, the first parameter being a parameter related to the size of a file generated by reading and writing the historical transfer task;

[0014] Based on the first parameter, the task duration, the average read and write rate of the transfer task, and the disk capacity required for each of the at least one transfer task to be stored in each of the at least one host machine, the second transfer coefficient is determined, and the second transfer coefficient is related to the average read and write rate of the at least one transfer task.

[0015] Optionally, determining the computing power factor according to the business information and resource utilization information of at least one host machine in the cloud database includes:

[0016] Obtain resource utilization information of at least one host machine in the cloud database; wherein the resource utilization information includes a first utilization rate of each host machine in the at least one host machine and a second utilization rate corresponding to the at least one host machine, the first utilization rate being a utilization rate of at least one of a central processing unit, a memory, and a network bandwidth by the host machine, and the second utilization rate being an average utilization rate of at least one of a central processing unit, a memory, and a network bandwidth by the at least one host machine;

[0017] Acquire the service information, wherein the service information includes the number of host machine types corresponding to the at least one host machine, the idle amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one host machine, and the demand amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one transfer task;

[0018] Determine a first computing power factor according to the first utilization rate and the second utilization rate;

[0019] Determine a second computing power factor according to the number of host machine types corresponding to the at least one host machine, the idle amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one host machine, and the demand amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one transfer task;

[0020] Determine the computing power factor based on the first computing power factor and the second computing power factor.

[0021] Optionally, transferring data in each of the at least one transfer task to the corresponding host machine based on the transfer coefficient and the computing power factor includes:

[0022] Determining an initialization heuristic according to an average task transfer rate of the at least one transfer task in the business information and a to-be-transferred amount of the at least one transfer task;

[0023] In the case where the first transfer task is assigned to the first host machine, a second pheromone is obtained based on the first pheromone update, wherein the first pheromone is the pheromone concentration on the path where the first transfer task is transferred to the first host machine, the first transfer task is any one of the at least one transfer task, and the first host machine is any one of the at least one host machine;

[0024] Determining a probability of allocating the first transfer task to each of the at least one host machine based on the initialization heuristic, the second pheromone, the transfer coefficient, and the computing power factor;

[0025] Determine, according to the probability of the first transfer task being assigned to each of the at least one host machine, a second host machine corresponding to the first transfer task, the second host machine being a host machine with the highest probability of receiving the first transfer task assignment among the at least one host machine;

[0026] The data in the first transfer task is transferred to the second host machine.

[0027] Optionally, the updating of the second pheromone based on the first pheromone includes:

[0028] determining a first pheromone according to a utilization rate of a central processing unit by the first host machine, a utilization rate of a memory by the first host machine, a task transfer rate of the first host machine, and a capacity of the first transfer task;

[0029] The first pheromone is updated based on a preset volatility factor to obtain the second pheromone.

[0030] In a second aspect, the embodiment of the present application further provides a data transfer device, including:

[0031] A first determination module is used to determine a transfer coefficient of the cloud database according to business information of the cloud database, wherein the transfer coefficient is related to a priority level of at least one transfer task in the cloud database and an average read and write rate of the at least one transfer task;

[0032] A second determination module is used to determine a computing power factor according to the business information and resource utilization information of at least one host machine in the cloud database, wherein the computing power factor is used to measure a degree of computing power balance and a degree of computing power matching between the at least one transfer task and the at least one host machine;

[0033] A task transfer module is used to transfer the data in each transfer task in the at least one transfer task to the corresponding host machine based on the transfer coefficient and the computing power factor.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data transfer method described in the first aspect above.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data transfer method described in the first aspect are implemented.

[0036] According to a fifth aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0037] In the embodiment of the present application, the transfer coefficient of the cloud database is defined through the business information of the cloud database. The transfer coefficient is related to the priority level of at least one transfer task in the cloud database and the average read and write rate of at least one transfer task. A computing power factor determined by the resource utilization information of at least one host machine in the cloud database is also defined. The computing power factor is determined based on the degree of computing power balance and computing power matching between at least one transfer task and the at least one host machine. In this way, a transfer coefficient is proposed according to the business characteristics of the cloud database, and a computing power factor is defined according to the resource load balance measurement between the transfer task and the host machine. Through the determined transfer coefficient and computing power factor, the data in each transfer task in at least one transfer task is transferred to the corresponding host machine, which reduces the scheduling time of the transfer task, makes the cluster load more even, and improves the operating efficiency of the task transfer allocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 It is a flow chart of a data transfer method provided in an embodiment of the present application;

[0040] Figure 2 It is a structural schematic diagram of a data transfer device provided in an embodiment of the present application;

[0041] Figure 3 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] See also Figure 1 , Figure 1 This is one of the flow charts of a data transfer method provided in an embodiment of the present application, the method comprising the following steps:

[0044] Step 101: Determine a transfer coefficient of the cloud database according to business information of the cloud database, wherein the transfer coefficient is related to a priority level of at least one transfer task in the cloud database and an average read and write rate of the at least one transfer task.

[0045] It is worth mentioning that the transfer task in this application can be a binary log file (binlog), which has three formats: row format (Row), statement format (Statement), and mixed format (Mixed). Since the transfer task in the historical instance of the cloud database is related to the cloud business volume on the production line and the amount of data generated per unit time that needs to be transferred, and is also related to the storage space size of the cloud data, it is necessary to comprehensively consider the actual business processing of the cloud database, and combine historical transfer tasks, cloud data volume, cloud storage and other business information to set the transfer coefficient.

[0046] It should be understood that the transfer coefficient in the present application can be a metric for evaluating and optimizing data dump tasks in a cloud database. Each transfer task can correspond to a priority (or priority level), which reflects the importance of the task. Transfer tasks with high priority levels can be given priority in resource allocation and scheduling to ensure the timely transfer of data in important transfer tasks. At the same time, the transfer coefficient can be related to the average read and write rate of the transfer task, and the average read and write rate can be the average speed of the read and write operations of the transfer task in the cloud database within a certain period of time. When transferring the transfer task, the read and write rate will affect the performance and efficiency of the transfer task transfer. A higher average read and write rate can indicate that data can be processed faster in the cloud database to support rapid backup or migration.

[0047] In specific implementations, the transfer coefficient can be used as a comprehensive indicator and calculated by combining the priority level and average read and write rate of the transfer task. For example, a transfer task with a high priority and a higher average read and write rate will obtain a larger transfer coefficient, and thus be given priority when the task transfer is assigned.

[0048] Step 102: Determine a computing power factor based on the business information and resource utilization information of at least one host machine in the cloud database, where the computing power factor is used to measure the degree of computing power balance and computing power matching between the at least one transfer task and the at least one host machine.

[0049] It should be noted that the cloud database in this application can be managed through Kubenate, and the actual business of the cloud database instance is carried by the host machine, and multiple transfer tasks in the cloud database are carried, such as Binlog transfer tasks, automatic backup tasks, cluster alarm tasks, etc. Therefore, the initiation of the transfer task needs to measure the resource usage of the current host machine, which may include the usage of the central processing unit (CPU), memory, network bandwidth, etc. In order to maximize the resource utilization of each host machine in the cloud database cluster, m independent transfer tasks can be scheduled to n different host machines, where m and n are both greater than 1, to maximize the use of host machine resources, achieve cluster load balancing, and meet the quality of service (QoS) requirements of each transfer task as much as possible.

[0050] In the embodiment of the present application, the degree of balance and matching of computing power between multiple transfer tasks and multiple host machines can be measured by defining a computing power factor. That is, both the computing power balance of the entire host cluster and the computing power matching between multiple host machines and multiple transfer tasks can be considered. It is worth mentioning that the computing power factor in this application is dynamically changed. With the subscription and unsubscription of instances of the entire Kubenate cluster, and the replacement of other task series (backup tasks, scheduled tasks, etc.), the number of hosts will change dynamically over time, and the computing power balance of the host will change. At the same time, during the transfer task scheduling process, as the transfer task is actually scheduled and assigned to transfer, the computing power matching between the transfer task and the host will also change. Therefore, the computing power factor in this application is dynamically changing.

[0051] Step 103: Based on the transfer coefficient and the computing power factor, transfer the data in each transfer task in the at least one transfer task to the corresponding host machine.

[0052] Specifically, the transfer task in the present application can be regarded as an ant being assigned to a corresponding host machine. Combined with the characteristic that ants leave pheromones on the path they pass through during movement to transmit information, the process of determining the corresponding host machine for the transfer task is substituted into the ant colony algorithm. Through the transfer coefficient and computing power factor, the ant colony algorithm is continuously iterated to fully consider the time changes and the actual situation of task transfer, improve the matching rate and scheduling efficiency of the transfer task transfer scheduling, and improve the utilization efficiency of resources in the cloud database.

[0053] In this way, the embodiment of the present application can optimize the probability formula of the ant colony algorithm based on the ant colony algorithm through the transfer coefficient and computing power factor, so as to reduce the time consumption of the ant colony algorithm task scheduling, achieve cluster load balancing, and improve the transfer task transfer allocation efficiency.

[0054] Optionally, the transfer coefficient includes a first transfer coefficient, and determining the transfer coefficient of the cloud database according to the business information of the cloud database includes:

[0055] Acquire the business information, wherein the business information includes user level information corresponding to each user to which each transfer task in the at least one transfer task belongs;

[0056] According to the user level information, the priority level of each transfer task in the at least one transfer task is divided to determine the first transfer coefficient, where the first transfer coefficient is related to the priority level of at least one transfer task in the cloud database.

[0057] In some embodiments, the transfer coefficient may include a first transfer coefficient, which is related to the priority level of each transfer task in the cloud database, and the values ​​of the first transfer coefficient corresponding to different priority levels may be different. For example, the coefficient values ​​corresponding to the first transfer coefficient may be 1, 0.8, 0.6, and 0.4, corresponding to four different priority levels, which may be specifically expressed as:

[0058] C rs ={1.0,0.8,0.6,0.4};

[0059] Among them, C rs Indicates the first transfer coefficient of the sth transfer task on the rth host.

[0060] Specifically, based on the existing production characteristics and operating status of the cloud database, the users of the cloud database can be used as an evaluation dimension for task transfer, and the user groups can be divided into levels. The user levels are divided according to the importance of different customers, and the different transfer tasks corresponding to different users also have different priorities. For example, the user level is divided into four levels: gold, silver, bronze, and general. The transfer tasks corresponding to users with a general level also have a lower priority level. In this way, by dividing multiple transfer tasks according to priority levels and determining the first transfer coefficient corresponding to each transfer task, it is convenient to fully consider the business status and user needs when allocating and scheduling transfer tasks, and improve the matching rate and scheduling efficiency of transfer task transfer scheduling.

[0061] Optionally, the transfer coefficient includes a second transfer coefficient, and determining the transfer coefficient of the cloud database according to the business information of the cloud database includes:

[0062] Acquire the business information, the business information including first information corresponding to at least one historical transfer task in the cloud database, and disk capacity required for each of the at least one transfer task to be stored in each of the at least one host machine; wherein the first information includes a first parameter, task duration, and an average read and write rate of the transfer task, the first parameter being a parameter related to the size of a file generated by reading and writing the historical transfer task;

[0063] Based on the first parameter, the task duration, the average read and write rate of the transfer task, and the disk capacity required for each of the at least one transfer task to be stored in each of the at least one host machine, the second transfer coefficient is determined, and the second transfer coefficient is related to the average read and write rate of the at least one transfer task.

[0064] In some embodiments, the computing power of multiple transfer tasks in the cloud database is related to the actual business of the cloud database, and the number of transactions per second (TPS) can be used to measure the actual business demand of the cloud database. Specifically, the second transfer coefficient can be determined according to the following formula:

[0065]

[0066] Among them, W rs Indicates the second transfer coefficient of the sth transfer task on the rth host;

[0067] r0 represents the alert ratio of the transfer task in the cloud database;

[0068] μ0 represents the constant coefficient, i.e., the first parameter, which is related to the file size generated by the read-write history transfer task;

[0069] M rs Indicates the disk capacity required when the sth transfer task is assigned to the rth host machine;

[0070] T sum It indicates the average time required from the end of the previous historical transfer task allocation in the cloud database to the start of the current transfer task allocation, that is, the task duration;

[0071] TPS rs It indicates the average read and write rate when the sth transfer task is assigned to the rth host machine, that is, the average read and write rate of the transfer task.

[0072] In this way, the second transfer coefficient can be determined by the above formula, which is convenient for fully considering the task transfer status when performing transfer task allocation and scheduling, and improving the matching rate and scheduling efficiency of transfer task transfer scheduling.

[0073] Optionally, determining the computing power factor according to the business information and resource utilization information of at least one host machine in the cloud database includes:

[0074] Obtain resource utilization information of at least one host machine in the cloud database; wherein the resource utilization information includes a first utilization rate of each host machine in the at least one host machine and a second utilization rate corresponding to the at least one host machine, the first utilization rate being a utilization rate of at least one of a central processing unit, a memory, and a network bandwidth by the host machine, and the second utilization rate being an average utilization rate of at least one of a central processing unit, a memory, and a network bandwidth by the at least one host machine;

[0075] Acquire the service information, wherein the service information includes the number of host machine types corresponding to the at least one host machine, the idle amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one host machine, and the demand amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one transfer task;

[0076] Determine a first computing power factor according to the first utilization rate and the second utilization rate;

[0077] Determine a second computing power factor according to the number of host machine types corresponding to the at least one host machine, the idle amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one host machine, and the demand amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one transfer task;

[0078] Determine the computing power factor based on the first computing power factor and the second computing power factor.

[0079] In some other embodiments, by obtaining resource utilization information and service information of each host in the cloud database, the first computing power factor N can be obtained by the following formula, which is specifically expressed as:

[0080]

[0081] in, Indicates the host machine; Respectively represent the host machine First utilization of CPU, memory and network bandwidth; Respectively represent the second utilization of CPU, memory and network bandwidth of the host cluster.

[0082] It can be seen that the value of the first computing power factor can be N∈(0,1), that is, the lower the value, the more evenly balanced the computing power of the cluster is. With the scheduling and allocation of the transfer tasks in the entire host cluster, and the replacement of other tasks (backup tasks, scheduled tasks, etc.), N will change dynamically over time.

[0083] In addition, during the transfer task allocation process, it is necessary to calculate the matching degree between the computing power required for the transfer task and the computing power of the host in the K8S cluster. With the transfer task The computing power matching degree can be established by introducing the Euclidean function. The second computing power factor can be specifically expressed as:

[0084]

[0085] Wherein, S represents the number of types of host machine resources (in the embodiment of the present application, the influence of the central processing unit, memory and network bandwidth on the host machine is taken into account, so S=3); Indicates the host machine The idle amount of the Sth resource; Indicates the demand of the transfer task for the sth resource.

[0086] In this way, the smaller the second computing power factor between the host and the transfer task, that is, the smaller the computing power matching degree, the more the host can adapt to the performance requirements of the transfer task. It is worth mentioning that during the transfer task scheduling process, the second computing power factor also changes dynamically.

[0087] In a specific embodiment, after determining the first computing power factor and the second computing power factor, that is, determining the computing power balance and computing power matching between the host machine and the transfer task in the cloud database, the computing power factor at time t can be defined as follows: It is expressed as:

[0088]

[0089] Where N(t) represents the first computing power factor at time t; It represents the second computing power factor between the computing power required for data transfer at time t and the host in the cloud database cluster; ψ represents the load balancing computing power coefficient.

[0090] In summary, the dynamic computing power factor is determined through the above embodiment, combining the resource utilization of each host machine and the data utilization of the cluster host machine, and comprehensively considering the cluster computing power load, the computing power required for the transfer task and the computing power required for each host machine, so that the cluster load is more even, the rationality of the transfer task allocation is improved, and the matching rate and scheduling efficiency of the transfer task transfer scheduling are improved.

[0091] Optionally, transferring data in each of the at least one transfer task to the corresponding host machine based on the transfer coefficient and the computing power factor includes:

[0092] Determining an initialization heuristic according to an average task transfer rate of the at least one transfer task in the business information and a to-be-transferred amount of the at least one transfer task;

[0093] In the case where the first transfer task is assigned to the first host machine, a second pheromone is obtained based on the first pheromone update, wherein the first pheromone is the pheromone concentration on the path where the first transfer task is transferred to the first host machine, the first transfer task is any one of the at least one transfer task, and the first host machine is any one of the at least one host machine;

[0094] Determining a probability of allocating the first transfer task to each of the at least one host machine based on the initialization heuristic, the second pheromone, the transfer coefficient, and the computing power factor;

[0095] Determine, according to the probability of the first transfer task being assigned to each of the at least one host machine, a second host machine corresponding to the first transfer task, the second host machine being a host machine with the highest probability of receiving the first transfer task assignment among the at least one host machine;

[0096] The data in the first transfer task is transferred to the second host machine.

[0097] It should be noted that the above initialization heuristic, i.e., the heuristic factor in the ant colony algorithm, can be determined by combining the average task transfer rate of multiple transfer tasks in the cloud database and the amount of storage to be transferred of multiple transfer tasks, and can be specifically expressed as the following formula:

[0098]

[0099] Among them, η rs (t) represents the heuristic at time t; V avg (t) represents the average task transfer speed of multiple transfer tasks at time t, d rs (t) represents the amount of transfer tasks to be transferred in the cloud database at time t.

[0100] In some embodiments, after all transfer tasks have completed one task assignment, the first pheromone may be updated according to the following formula:

[0101]

[0102] Among them, m represents the total number of ants, that is, the total number of transfer tasks; It represents the pheromone left by ant k from city i to city j, that is, the pheromone left by transfer task k from host i to host j, that is, the second pheromone. It is worth mentioning that the first pheromone can be the pheromone left by transfer task k from host i to host j. The above formula can be used to update the first pheromone and improve the matching rate of transfer task transfer scheduling.

[0103] Specifically, combined with the first transfer coefficient C rs , the second transfer coefficient W rs And the hash rate factor The probability calculation formula in the ant colony algorithm can be optimized, which is specifically expressed as:

[0104]

[0105] Among them, τ ij(t) represents the pheromone concentration on the path (i, j) at time t; η ij (t) represents the initialization heuristic of path (i, j) at time t; α is the heuristic factor, which can be taken as 1 / d ij ; represents the probability that the transfer task k is assigned from host machine i to host machine j at time t; β represents the expected heuristic factor (the larger β is, the closer the probability selection is to the greedy rule).

[0106] In this way, the probability of each transfer task being assigned to each host machine can be obtained through the above formula, and the allocation probabilities of different host machines corresponding to the same transfer task can be sorted according to size, so that the allocation probability sequence of the transfer task can be obtained, and the host machine with the highest probability in the probability sequence can be selected, and the transfer task can be assigned to the corresponding host machine with the highest probability, so as to meet the transfer task allocation requirements. It can be seen that the embodiment of the present application optimizes the probability formula of the ant colony algorithm through the transfer coefficient and the computing power factor, so that the time consumption of the ant colony algorithm task scheduling is reduced, the cluster load balancing is achieved, the matching rate and scheduling efficiency of the transfer task transfer scheduling are improved, and the utilization efficiency of resources in the cloud database is improved.

[0107] Optionally, the updating of the second pheromone based on the first pheromone includes:

[0108] determining a first pheromone according to a utilization rate of a central processing unit by the first host machine, a utilization rate of a memory by the first host machine, a task transfer rate of the first host machine, and a capacity of the first transfer task;

[0109] The first pheromone is updated based on a preset volatility factor to obtain the second pheromone.

[0110] In some specific embodiments, when an ant is assigned to a host in a cloud database as a transfer task, the original pheromone will evaporate as the number of iterations increases, and the pheromone remaining in the path needs to be updated in a timely manner. In this application, when the transfer task is scheduled to the host, the matching degree between the host and the transfer task will change, so the first pheromone needs to be updated in a timely manner. Among them, pheromone updates can be divided into local pheromone updates and global pheromone updates.

[0111] Specifically, when the transfer task is assigned to the host, the following formula can be used to update Node to Local pheromone of the node:

[0112] τ i (t1)=(1-ρ)×τ i (t)+ρ×τ0;

[0113] In the formula, τ i (t) means The pheromone concentration of the node at time t; τ i (t1) indicates that the new task arrives at time t1 The pheromone concentration of the node; ρ is the volatility factor of the pheromone, and ρ∈(0,1]; 1-ρ represents the residual factor of the pheromone; τ0 represents the initial pheromone; Indicates the current transfer task J i Assigned to the host Indicates the next transfer task J i+1 Assigned to the host

[0114] Furthermore, when all ants (i.e., all transfer tasks) complete a task assignment, the global pheromone can be updated according to the following formula:

[0115]

[0116] Among them, m represents the total number of ants, that is, the total number of transfer tasks; It represents the pheromone left by ant k from city i to city j, that is, the pheromone left by transfer task k from host machine i to host machine j, that is, the second pheromone.

[0117] Specifically, It can be determined by the following formula:

[0118]

[0119] Among them, U cpu Indicates the current CPU usage of the current host; U mem represents the current memory usage of the host machine; I(t) represents the transfer rate of each host machine; d represents the capacity of tasks to be transferred in the cloud database.

[0120] In this way, the above embodiment can realize updating of local pheromones according to load imbalance, and simultaneously updating of global pheromones to improve convergence speed, thereby improving transfer task allocation efficiency and matching rate of transfer task transfer scheduling.

[0121] like Figure 2 As shown, the embodiment of the present application further provides a data transfer device 200, the device comprising:

[0122] A first determination module 201 is used to determine a transfer coefficient of the cloud database according to the business information of the cloud database, wherein the transfer coefficient is related to the priority level of at least one transfer task in the cloud database and the average read and write rate of the at least one transfer task;

[0123] A second determination module 202 is used to determine a computing power factor according to the business information and resource utilization information of at least one host in the cloud database, wherein the computing power factor is used to measure a degree of computing power balance and a degree of computing power matching between the at least one transfer task and the at least one host;

[0124] The task transfer module 203 is used to transfer the data in each transfer task in the at least one transfer task to the corresponding host machine based on the transfer coefficient and the computing power factor.

[0125] Optionally, the transfer coefficient includes a first transfer coefficient, and the first determination module 201 includes:

[0126] A first acquisition unit is used to acquire the business information, wherein the business information includes user level information corresponding to each user to which each transfer task in the at least one transfer task belongs;

[0127] The level classification unit is used to classify the priority level of each of the at least one transfer task according to the user level information, and determine the first transfer coefficient, where the first transfer coefficient is related to the priority level of at least one transfer task in the cloud database.

[0128] Optionally, the transfer coefficient includes a second transfer coefficient, and the first determination module 201 includes:

[0129] A second acquisition unit is used to acquire the business information, wherein the business information includes first information corresponding to at least one historical transfer task in the cloud database, and a disk capacity required for each of the at least one transfer task to be stored in each of the at least one host machine; wherein the first information includes a first parameter, task duration, and an average read and write rate of the transfer task, and the first parameter is a parameter related to the size of a file generated by reading and writing the historical transfer task;

[0130] The first determination unit is used to determine the second transfer coefficient based on the first parameter, the task duration, the average read and write rate of the transfer task, and the disk capacity required for each transfer task in the at least one transfer task to be stored in each host machine in the at least one host machine, wherein the second transfer coefficient is related to the average read and write rate of the at least one transfer task.

[0131] Optionally, the second determining module 202 includes:

[0132] a third acquisition unit, configured to acquire resource utilization information of at least one host machine in the cloud database; wherein the resource utilization information includes a first utilization rate of each host machine in the at least one host machine and a second utilization rate corresponding to the at least one host machine, the first utilization rate being a utilization rate of at least one of a central processing unit, a memory, and a network bandwidth by the host machine, and the second utilization rate being an average utilization rate of at least one of a central processing unit, a memory, and a network bandwidth by the at least one host machine;

[0133] a fourth acquisition unit, configured to acquire the service information, wherein the service information includes the number of host machine types corresponding to the at least one host machine, the idle amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one host machine, and the demand amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one transfer task;

[0134] a second determining unit, configured to determine a first computing power factor according to the first utilization rate and the second utilization rate;

[0135] A third determining unit is used to determine a second computing power factor according to the number of host machine types corresponding to the at least one host machine, the idle amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one host machine, and the demand amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one transfer task.

[0136] A fourth determining unit is used to determine the computing power factor based on the first computing power factor and the second computing power factor.

[0137] Optionally, the task transfer module 203 includes:

[0138] A fifth determining unit, configured to determine an initialization heuristic according to an average task transfer rate of the at least one transfer task in the business information and a to-be-transferred amount of the at least one transfer task;

[0139] a pheromone updating unit, configured to obtain a second pheromone based on the first pheromone update when the first transfer task is assigned to the first host machine, wherein the first pheromone is the pheromone concentration on the path from the first transfer task to the first host machine, the first transfer task is any one of the at least one transfer task, and the first host machine is any one of the at least one host machine;

[0140] a sixth determining unit, configured to determine a probability of allocating the first transfer task to each of the at least one host machine based on the initialization heuristic, the second pheromone, the transfer coefficient, and the computing power factor;

[0141] a seventh determining unit, configured to determine a second host machine corresponding to the first transfer task according to a probability of the first transfer task being assigned to each of the at least one host machine, the second host machine being a host machine with the highest probability of receiving the first transfer task assigned to the at least one host machine;

[0142] The task transfer unit is used to transfer the data in the first transfer task to the second host machine.

[0143] Optionally, the pheromone updating unit is used for:

[0144] determining a first pheromone according to a utilization rate of a central processing unit by the first host machine, a utilization rate of a memory by the first host machine, a task transfer rate of the first host machine, and a capacity of the first transfer task;

[0145] The first pheromone is updated based on a preset volatility factor to obtain the second pheromone.

[0146] It should be noted that this embodiment is Figure 1 The specific implementation of the corresponding device side in the embodiment shown can be found in Figure 1 In order to avoid repeated description, the relevant description of the embodiment shown is not repeated in this embodiment.

[0147] See also Figure 3 , Figure 3 is a structural diagram of an electronic device provided in an embodiment of the present application, such as Figure 3 As shown, the electronic device includes: a processor 300, a memory 320, and a program or instruction stored in the memory 320 and executable on the processor 300, the processor 300 is used to read the program or instruction in the memory 320; the electronic device also includes a bus interface and a transceiver 310.

[0148] The transceiver 310 is configured to receive and send data under the control of the processor 300 .

[0149] Among them, Figure 3In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 300 and memory represented by memory 320. The bus architecture may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 310 may be a plurality of components, namely, a transmitter and a transceiver, providing a unit for communicating with various other devices on a transmission medium. The processor 300 is responsible for managing the bus architecture and general processing, and the memory 320 may store data used by the processor 300 when performing operations.

[0150] The processor 300 is used to read the program or instruction in the memory 320 and execute the following steps:

[0151] Determine a transfer coefficient of the cloud database according to the business information of the cloud database, wherein the transfer coefficient is related to the priority level of at least one transfer task in the cloud database and the average read and write rate of the at least one transfer task;

[0152] Determine a computing power factor according to the business information and resource utilization information of at least one host machine in the cloud database, where the computing power factor is used to measure a degree of computing power balance and a degree of computing power matching between the at least one transfer task and the at least one host machine;

[0153] Based on the transfer coefficient and the computing power factor, data in each transfer task in the at least one transfer task is transferred to the corresponding host machine.

[0154] Optionally, the transfer coefficient includes a first transfer coefficient, and the processor 300 is configured to read a program or instruction in the memory 320 and perform the following steps:

[0155] Acquire the business information, wherein the business information includes user level information corresponding to each user to which each transfer task in the at least one transfer task belongs;

[0156] According to the user level information, the priority level of each transfer task in the at least one transfer task is divided to determine the first transfer coefficient, where the first transfer coefficient is related to the priority level of at least one transfer task in the cloud database.

[0157] Optionally, the transfer coefficient includes a second transfer coefficient, and the processor 300 is configured to read a program or instruction in the memory 320 and perform the following steps:

[0158] Acquire the business information, the business information including first information corresponding to at least one historical transfer task in the cloud database, and disk capacity required for each of the at least one transfer task to be stored in each of the at least one host machine; wherein the first information includes a first parameter, task duration, and an average read and write rate of the transfer task, the first parameter being a parameter related to the size of a file generated by reading and writing the historical transfer task;

[0159] Based on the first parameter, the task duration, the average read and write rate of the transfer task, and the disk capacity required for each of the at least one transfer task to be stored in each of the at least one host machine, the second transfer coefficient is determined, and the second transfer coefficient is related to the average read and write rate of the at least one transfer task.

[0160] Optionally, the processor 300 is configured to read a program or instruction in the memory 320 and perform the following steps:

[0161] Obtain resource utilization information of at least one host machine in the cloud database; wherein the resource utilization information includes a first utilization rate of each host machine in the at least one host machine and a second utilization rate corresponding to the at least one host machine, the first utilization rate being a utilization rate of at least one of a central processing unit, a memory, and a network bandwidth by the host machine, and the second utilization rate being an average utilization rate of at least one of a central processing unit, a memory, and a network bandwidth by the at least one host machine;

[0162] Acquire the service information, wherein the service information includes the number of host machine types corresponding to the at least one host machine, the idle amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one host machine, and the demand amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one transfer task;

[0163] Determine a first computing power factor according to the first utilization rate and the second utilization rate;

[0164] Determine a second computing power factor according to the number of host machine types corresponding to the at least one host machine, the idle amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one host machine, and the demand amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one transfer task;

[0165] Determine the computing power factor based on the first computing power factor and the second computing power factor.

[0166] Optionally, the processor 300 is configured to read a program or instruction in the memory 320 and perform the following steps:

[0167] Determining an initialization heuristic according to an average task transfer rate of the at least one transfer task in the business information and a to-be-transferred amount of the at least one transfer task;

[0168] In the case where the first transfer task is assigned to the first host machine, a second pheromone is obtained based on the first pheromone update, wherein the first pheromone is the pheromone concentration on the path where the first transfer task is transferred to the first host machine, the first transfer task is any one of the at least one transfer task, and the first host machine is any one of the at least one host machine;

[0169] Determining a probability of allocating the first transfer task to each of the at least one host machine based on the initialization heuristic, the second pheromone, the transfer coefficient, and the computing power factor;

[0170] Determine, according to the probability of the first transfer task being assigned to each of the at least one host machine, a second host machine corresponding to the first transfer task, the second host machine being a host machine with the highest probability of receiving the first transfer task assignment among the at least one host machine;

[0171] The data in the first transfer task is transferred to the second host machine.

[0172] Optionally, the processor 300 is configured to read a program or instruction in the memory 320 and perform the following steps:

[0173] determining a first pheromone according to a utilization rate of a central processing unit by the first host machine, a utilization rate of a memory by the first host machine, a task transfer rate of the first host machine, and a capacity of the first transfer task;

[0174] The first pheromone is updated based on a preset volatility factor to obtain the second pheromone.

[0175] It should be noted that the electronic device in the embodiment of the present application can achieve Figure 1 The various processes implemented in the data transfer method embodiment shown can achieve the same beneficial effects, and will not be described again here to avoid repetition.

[0176] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned Figure 1The various processes of the data transfer method embodiment shown in the figure can achieve the same technical effect, and are not described here to avoid repetition. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0177] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0178] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0180] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A data transfer method, characterized in that: include: Determine a transfer coefficient of the cloud database according to the business information of the cloud database, wherein the transfer coefficient is related to the priority level of at least one transfer task in the cloud database and the average read and write rate of the at least one transfer task; Determine a computing power factor according to the business information and resource utilization information of at least one host machine in the cloud database, wherein the computing power factor is used to measure a degree of computing power balance and a degree of computing power matching between the at least one transfer task and the at least one host machine; Based on the transfer coefficient and the computing power factor, data in each transfer task in the at least one transfer task is transferred to the corresponding host machine.

2. The method according to claim 1, characterized in that: The transfer coefficient includes a first transfer coefficient, and determining the transfer coefficient of the cloud database according to the business information of the cloud database includes: Acquire the business information, wherein the business information includes user level information corresponding to each user to which each transfer task in the at least one transfer task belongs; According to the user level information, the priority level of each transfer task in the at least one transfer task is divided to determine the first transfer coefficient, where the first transfer coefficient is related to the priority level of at least one transfer task in the cloud database.

3. The method according to claim 1, characterized in that The transfer coefficient includes a second transfer coefficient, and determining the transfer coefficient of the cloud database according to the business information of the cloud database includes: Acquire the business information, the business information including first information corresponding to at least one historical transfer task in the cloud database, and disk capacity required for each of the at least one transfer task to be stored in each of the at least one host machine; wherein the first information includes a first parameter, task duration, and an average read and write rate of the transfer task, the first parameter being a parameter related to the size of a file generated by reading and writing the historical transfer task; Based on the first parameter, the task duration, the average read and write rate of the transfer task, and the disk capacity required for each of the at least one transfer task to be stored in each of the at least one host machine, the second transfer coefficient is determined, and the second transfer coefficient is related to the average read and write rate of the at least one transfer task.

4. The method according to claim 1, characterized in that The determining the computing power factor according to the business information and resource utilization information of at least one host machine in the cloud database includes: Obtain resource utilization information of at least one host machine in the cloud database; wherein the resource utilization information includes a first utilization rate of each host machine in the at least one host machine and a second utilization rate corresponding to the at least one host machine, the first utilization rate being a utilization rate of at least one of a central processing unit, a memory, and a network bandwidth by the host machine, and the second utilization rate being an average utilization rate of at least one of a central processing unit, a memory, and a network bandwidth by the at least one host machine; Acquire the service information, wherein the service information includes the number of host machine types corresponding to the at least one host machine, the idle amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one host machine, and the demand amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one transfer task; Determine a first computing power factor according to the first utilization rate and the second utilization rate; Determine a second computing power factor according to the number of host machine types corresponding to the at least one host machine, the idle amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one host machine, and the demand amount of at least one of a central processing unit, a memory, and a network bandwidth of each of the at least one transfer task; Determine the computing power factor based on the first computing power factor and the second computing power factor.

5. The method according to any one of claims 1 to 4, characterized in that The transferring of data in each of the at least one transfer task to the corresponding host machine based on the transfer coefficient and the computing power factor includes: Determining an initialization heuristic according to an average task transfer rate of the at least one transfer task in the business information and a to-be-transferred amount of the at least one transfer task; In the case where the first transfer task is assigned to the first host machine, a second pheromone is obtained based on the first pheromone update, wherein the first pheromone is the pheromone concentration on the path where the first transfer task is transferred to the first host machine, the first transfer task is any one of the at least one transfer task, and the first host machine is any one of the at least one host machine; Determining a probability of allocating the first transfer task to each of the at least one host machine based on the initialization heuristic, the second pheromone, the transfer coefficient, and the computing power factor; Determine, according to the probability of the first transfer task being assigned to each of the at least one host machine, a second host machine corresponding to the first transfer task, the second host machine being a host machine with the highest probability of receiving the first transfer task assignment among the at least one host machine; The data in the first transfer task is transferred to the second host machine.

6. The method according to claim 5, characterized in that The updating of the second pheromone based on the first pheromone includes: determining a first pheromone according to a utilization rate of a central processing unit by the first host machine, a utilization rate of a memory by the first host machine, a task transfer rate of the first host machine, and a capacity of the first transfer task; The first pheromone is updated based on a preset volatility factor to obtain the second pheromone.

7. A data transfer device, characterized in that: include: A first determination module is used to determine a transfer coefficient of the cloud database according to business information of the cloud database, wherein the transfer coefficient is related to a priority level of at least one transfer task in the cloud database and an average read and write rate of the at least one transfer task; A second determination module is used to determine a computing power factor according to the business information and resource utilization information of at least one host machine in the cloud database, wherein the computing power factor is used to measure a degree of computing power balance and a degree of computing power matching between the at least one transfer task and the at least one host machine; A task transfer module is used to transfer the data in each transfer task in the at least one transfer task to the corresponding host machine based on the transfer coefficient and the computing power factor.

8. An electronic device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data transfer method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data transfer method according to any one of claims 1 to 5 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the data transfer method as claimed in any one of claims 1 to 5.

Images