Data transmission method, system and electronic device

Abstract

The application provides a data transmission method, system and electronic device, relates to the technical field of data transmission, and the method comprises the steps of determining a target data transmission task to be executed; the target data transmission task indicates that target data is transmitted from a first data center to a second data center; a first bandwidth configured between the first data center and the second data center is acquired; a second bandwidth of the target data transmission task is determined according to the first bandwidth, and the second bandwidth is positively correlated with the first bandwidth; and the target data is transmitted from the first data center to the second data center according to the second bandwidth, so that the second bandwidth of the target data transmission task is dynamically adjusted through the first bandwidth, and the efficiency of transmitting the target data from the first data center to the second data center is improved.

Description

Technical Field

[0001] This application relates to the field of data transmission technology, and in particular to a data transmission method, system and electronic device. Background Technology

[0002] In scenarios with high data availability requirements, such as financial business scenarios, data clusters are typically deployed across multiple data centers for data storage. In this way, data is distributed across multiple data centers, and the data in these data centers may be independent of each other or may be transmitted to each other in real time. The data clusters collect business data, log data, etc., and then process and store the collected data to provide services to external parties.

[0003] Currently, when running data transmission tasks, the bandwidth of the data to be transmitted is fixed and cannot be dynamically adjusted, resulting in low data transmission efficiency. Summary of the Invention

[0004] This application provides a data transmission method, system, and electronic device to solve the problem of low efficiency in target data transmission.

[0005] The first aspect of this application provides a data transmission method applied to a data transmission system. The data transmission method includes: determining a target data transmission task to be run; the target data transmission task instructing the transmission of target data from a first data center to a second data center; obtaining a first bandwidth configured between the first data center and the second data center; determining a second bandwidth of the target data transmission task based on the first bandwidth, wherein the second bandwidth is positively correlated with the first bandwidth; and controlling the transmission of target data from the first data center to the second data center based on the second bandwidth.

[0006] In one embodiment of this application, determining a target data transmission task to be run includes: determining a target queue among multiple queues, wherein the queue stores at least one data transmission task; and determining the target data transmission task among the at least one data transmission task in the target queue.

[0007] In one embodiment of this application, determining the target queue among multiple queues includes:

[0008] Among multiple queues, the queue with the highest priority and / or the fewest number of data transfer tasks is determined as the target queue.

[0009] In one embodiment of this application, determining the target data transmission task in at least one data transmission task in the target queue includes: determining the data transmission task with the highest priority and / or the earliest submission time of the target queue as the target data transmission task in the at least one data transmission task.

[0010] In one embodiment of this application, determining the second bandwidth of a target data transmission task based on a first bandwidth includes: obtaining the bandwidth usage ratio of a target queue; determining the third bandwidth of the target queue based on the bandwidth usage ratio and the first bandwidth; obtaining the number of data transmission tasks in the target queue; and determining the second bandwidth based on the third bandwidth and the number of tasks.

[0011] In one embodiment of this application, the target data transmission task includes multiple subtasks, and the target data includes: transmission data corresponding to each subtask; controlling the first data center to transmit the target data to the second data center according to the second bandwidth includes: obtaining the number of threads running the target data transmission task; and determining the fourth bandwidth corresponding to each subtask according to the second bandwidth and the number of threads.

[0012] In one embodiment of this application, the method further includes: if the number of running subtasks is less than the number of threads, determining a fifth bandwidth for the running subtasks based on the second bandwidth and the number of running subtasks; and controlling the first data center to transmit the transmission data corresponding to the running subtasks to the second data center based on the fifth bandwidth.

[0013] In one embodiment of this application, the data transmission system includes a coordinator and a plurality of actuators, and further includes: the coordinator determining a target actuator based on the load of each of the plurality of actuators; and controlling a first data center to transmit target data to a second data center based on a second bandwidth, including: using the target actuator to control the first data center to transmit target data to the second data center based on the second bandwidth.

[0014] A second aspect of this application provides a data transmission system, comprising: a coordinator and a target executor, wherein:

[0015] The coordinator is used to: determine the target data transmission task to be run; the target data transmission task indicates the transmission of target data from the first data center to the second data center; obtain the first bandwidth configured between the first data center and the second data center; and determine the second bandwidth of the target data transmission task based on the first bandwidth, wherein the second bandwidth is positively correlated with the first bandwidth.

[0016] The target actuator is used to control the transmission of target data from the first data center to the second data center based on the second bandwidth.

[0017] In one embodiment of this application, when the coordinator determines the target data transmission task to be run, it is specifically used to: determine a target queue among multiple queues, wherein the queue stores at least one data transmission task; and determine the target data transmission task among the at least one data transmission task in the target queue.

[0018] In one embodiment of this application, when the coordinator determines the target queue among multiple queues, it is specifically used to: determine the queue with the highest priority and / or the smallest number of data transmission tasks as the target queue among multiple queues.

[0019] In one embodiment of this application, when the coordinator determines the target data transmission task in at least one data transmission task in the target queue, it specifically does so by: determining the data transmission task with the highest priority and / or the earliest submission time of the target queue as the target data transmission task in at least one data transmission task.

[0020] In one embodiment of this application, when the coordinator determines the second bandwidth of the target data transmission task based on the first bandwidth, it is specifically configured to: obtain the bandwidth usage ratio of the target queue; determine the third bandwidth of the target queue based on the bandwidth usage ratio and the first bandwidth; obtain the number of data transmission tasks in the target queue; and determine the second bandwidth based on the third bandwidth and the number.

[0021] In one embodiment of this application, the target data transmission task includes multiple subtasks, and the target data includes: transmission data corresponding to each subtask; when the target executor controls the first data center to transmit the target data to the second data center according to the second bandwidth, it is specifically used to: obtain the number of threads running the target data transmission task; and determine the fourth bandwidth corresponding to each subtask according to the second bandwidth and the number of threads.

[0022] In one embodiment of this application, the target executor is further configured to: if the number of running subtasks is less than the number of threads, determine the fifth bandwidth of the running subtasks based on the second bandwidth and the number of running subtasks; and control the first data center to transmit the transmission data corresponding to the running subtasks to the second data center based on the fifth bandwidth.

[0023] In one embodiment of this application, the coordinator is further configured to: determine a target executor based on the load of each of the plurality of executors; the target executor is specifically configured to: control the transmission of target data from the first data center to the second data center based on the second bandwidth.

[0024] As can be seen from the above technical solution, the embodiments of this application determine the target data transmission task to be run; the target data transmission task indicates the transmission of target data from the first data center to the second data center; obtain the first bandwidth configured between the first data center and the second data center; determine the second bandwidth of the target data transmission task based on the first bandwidth, the second bandwidth being positively correlated with the first bandwidth; and control the transmission of target data from the first data center to the second data center based on the second bandwidth, thereby realizing the dynamic adjustment of the second bandwidth of the target data transmission task through the first bandwidth and improving the efficiency of target data transmission from the first data center to the second data center. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram illustrating an application scenario of the data transmission method provided in this application;

[0027] Figure 2 This is a flowchart illustrating the steps of a data transmission method according to an embodiment of this application;

[0028] Figure 3 This is a flowchart of another data transmission method provided in an embodiment of this application;

[0029] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0031] The principles and spirit of this application will be explained in detail below with reference to several representative embodiments.

[0032] To meet the high availability requirements of the data cluster, the current data also needs to be stored in multiple data centers, which involves the following issues: (1) how to collect and aggregate data from multiple data centers, process and store the complete data; (2) how to store the processed data in multiple data centers to achieve high availability. In the current context of offline development, this involves cross-data center transmission and copying of data files. Furthermore, the network bandwidth within a single data center is large, while the bandwidth between data centers is limited, and the bandwidth between different data centers is different. Generally, the bandwidth between data centers in the same city is larger, while the bandwidth between data centers in different cities is smaller. When transmitting data, it is also necessary to ensure that the bandwidth of the transmitted data is lower than the actual bandwidth between data centers in order to reserve sufficient bandwidth for business.

[0033] Currently, there is a tool called Distcp (a copying tool) that can achieve distributed transmission of large amounts of data. However, this tool specifies the task bandwidth at task startup and does not support dynamic adjustment of the task bandwidth, resulting in the incomplete utilization of bandwidth resources. In data transmission scenarios, data transmission tasks have great uncertainty. A large number of data transmission tasks may be added simultaneously, high-priority data transmission tasks may be inserted, and a large number of low-priority data transmission tasks may be added, etc. If the task bandwidth cannot be dynamically adjusted, the data transmission efficiency will be low.

[0034] To address the aforementioned issues, this application provides a data transmission method that determines the second bandwidth of a target data transmission task by using a first bandwidth configured between a first data center and a second data center. This ensures that the bandwidth of the target data transmission task is not fixed and can be dynamically adjusted based on the bandwidth between the data centers, thereby improving the transmission efficiency of the target data.

[0035] refer to Figure 1 This is an application scenario diagram of the data transmission method provided in this application. Figure 1 In the data transmission system 10, there are: a coordinator, multiple actuators, and multiple data centers, such as... Figure 1 The system consists of executors a1, a2, a3, and a4, and data centers b1, b2, and b3. The coordinator receives data transmission tasks from the upper-layer application and stores them in corresponding queues, such as queues c1, c2, c3, and c4. The coordinator determines the target data transmission task from one of the queues and then issues the target data transmission task to one of the executors. The executor then executes the target data transmission task, transferring the target data from the first data center to the second data center, as described above. Figure 1 For example, the first data center is data center b1, and the second data center is data center b2.

[0036] After introducing the basic principles of this application, the various non-limiting embodiments of this application will be described in detail below.

[0037] First, the relevant terms appearing in this application are explained as follows:

[0038] Data centers are dedicated spaces for storing computer storage and computing devices (servers), communication equipment (switches, routers, firewalls), and supporting centralized data management (storage, computing, and switching). Data centers are interconnected via networks (public networks, leased lines, etc.), and the available bandwidth is limited.

[0039] Cross-data center data transfer: Data is transferred from one data center to another (the different data centers may be in the same city or in different cities).

[0040] Data transfer task: refers to the task of transferring data from one data center to another. The data can be file data, database data, etc.

[0041] Queues: Used to store multiple data transfer tasks, supporting operations such as adding and removing data transfer tasks, and removing tasks according to priority.

[0042] The technical solution of this application will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0043] Reference Figure 2 This is a flowchart illustrating the steps of a data transmission method provided in an embodiment of this application. This data transmission method is applied to, for example... Figure 1 The data transmission system shown specifically includes the following steps:

[0044] S201, Determine the target data transmission task to be run.

[0045] In the embodiments of this application, reference is made to Figure 1 Multiple data transfer tasks are stored in multiple queues, and a target data transfer task is determined from among these tasks. This target data transfer task is the data transfer task to be executed.

[0046] In this process, data transmission tasks are sent by upper-layer applications and then stored in corresponding queues based on the application from which the data transmission tasks originate. For example, refer to... Figure 1 Data transmission tasks sent by upper-layer application d1 are stored in queue c1, data transmission tasks sent by upper-layer application d2 are stored in queue c2, data transmission tasks sent by upper-layer application d3 are stored in queue c3, and data transmission tasks sent by upper-layer application d4 are stored in queue c4. In actual use, multiple upper-layer applications can send multiple data transmission tasks, and the corresponding queues will store multiple data transmission tasks.

[0047] Furthermore, the data transmission task indicates the data to be transmitted, as well as the source and destination data centers of the transmitted data. It can be understood that the target data transmission task indicates the transmission of target data from the first data center to the second data center; therefore, running the target data transmission task means transmitting the target data stored in the first data center to the second data center.

[0048] S202, Obtain the first bandwidth configured between the first data center and the second data center.

[0049] In this embodiment, the total bandwidth for data transmission between data centers is pre-configured. (Refer to...) Figure 1 For example, the total bandwidth configured for data transmission between data center b1 and data center b2 is 100G, the total bandwidth configured for data transmission between data center b1 and data center b2 is 100G / s, the total bandwidth configured for data transmission between data center b2 and data center b3 is 10T / s, and the total bandwidth configured for data transmission between data center b1 and data center b3 is 50G / s.

[0050] Furthermore, the total bandwidth for data transmission between data centers is adjustable. For example, a higher total bandwidth can be configured during off-peak hours and a lower total bandwidth can be configured during peak hours. The total bandwidth between data centers can be pre-configured as needed and is not limited thereto.

[0051] It can be understood that the first bandwidth is the total bandwidth for data transmission between the first data center and the second data center. In this embodiment, after determining the target data transmission task, the first data center and the second data center indicated by the target data transmission task are determined, and then the total bandwidth between the current first data center and the second data center is obtained as the first bandwidth.

[0052] S203, determine the second bandwidth of the target data transmission task based on the first bandwidth.

[0053] In this embodiment, the second bandwidth is positively correlated with the first bandwidth. Specifically, the second bandwidth of the target data transmission task is determined based on the first bandwidth; that is, when the first bandwidth is high, the second bandwidth is also high, and vice versa. This allows the second bandwidth to be dynamically adjusted according to the first bandwidth, ensuring that the transmission bandwidth of the target data transmission task is not fixed, thereby improving the transmission efficiency of the target data.

[0054] For example, during off-peak hours, if the primary bandwidth is higher, the secondary bandwidth will also be higher, allowing for faster transmission of target data. During peak hours, because services require a certain amount of bandwidth, the primary bandwidth is lower, and therefore the secondary bandwidth is also lower, allowing for the transmission of target data without affecting service processing.

[0055] It is understandable that S201 to S203 are Figure 1 The coordinator in the process performs the following: Furthermore, the coordinator is also used to determine the target actuator based on the load of each of the multiple actuators. Specifically, the actuator with the lowest load can be selected as the target actuator, or the actuator with a load less than a preset threshold can be selected as the target actuator based on the load and other factors. This application does not limit the other factors.

[0056] Furthermore, after determining the target data transmission task and the second bandwidth, the coordinator issues the target data transmission task and the second bandwidth to the selected target executor.

[0057] S204, based on the second bandwidth, controls the transmission of target data from the first data center to the second data center.

[0058] Specifically, controlling the transmission of target data from the first data center to the second data center based on the second bandwidth includes: using a target actuator to control the transmission of target data from the first data center to the second data center based on the second bandwidth.

[0059] In this embodiment of the application, the target executor is used to run the target data transmission task. Specifically, when running the target data transmission task, a second bandwidth is allocated to the target data transmission task so that the target data of the first data center is transmitted to the second data center according to the second bandwidth.

[0060] In summary, the embodiments of this application determine the second bandwidth of the target data transmission task by configuring the first bandwidth between the first data center and the second data center, so that the bandwidth of the target data transmission task is not fixed and can be dynamically adjusted according to the bandwidth between the data centers, thereby improving the transmission efficiency of the target data.

[0061] Furthermore, this application provides a flowchart of another data transmission method, referring to... Figure 3 Specifically, it includes the following steps:

[0062] S301, determine the target queue among multiple queues.

[0063] The queue stores at least one data transfer task. (See reference...) Figure 1 For example, multiple queues include: queue c1, queue c2, queue c3, and queue c4. Queues may or may not store data transfer tasks. For instance, queue c1 stores 3 data transfer tasks, queue c2 stores 4 data transfer tasks, queue c3 stores no data transfer tasks, and queue c4 stores 2 data transfer tasks.

[0064] In this embodiment of the application, determining the target queue among multiple queues includes: determining the queue with the highest priority and / or the smallest number of data transmission tasks as the target queue among multiple queues.

[0065] Priorities can be pre-set for each queue; for example, the priority of queue c3 > the priority of queue c2 > the priority of queue c1 > the priority of queue c4. Furthermore, the data transmission tasks within the queues are executable data transmission tasks.

[0066] In one embodiment, among multiple queues, the queue that stores the data transmission task and has the highest priority is determined as the target queue. For example, since queue c3 does not store data transmission tasks, the queue with the highest priority is queue c2, and queue c2 can be determined as the target queue.

[0067] In one embodiment, among multiple queues, the queue that stores data transmission tasks and has the smallest number of data transmission tasks is determined as the target queue. For example, referring to the above, if queue c4 is the queue that stores data transmission tasks and has the smallest number of data transmission tasks, then queue c4 is determined as the target queue.

[0068] In one embodiment, among multiple queues, the queue that stores data transmission tasks and has the highest priority is first queue. If there is only one first queue, then the first queue is determined as the target queue. If there are multiple first queues, then the first queue with the smallest number of data transmission tasks is determined as the target queue.

[0069] In the embodiments of this application, the target queue can be determined in a variety of ways, and no limitation is imposed on it.

[0070] In this embodiment, data transmission tasks in high-priority queues can be prioritized for processing. Alternatively, data transmission tasks in queues with fewer data transmission tasks can be prioritized, thus improving the processing efficiency of those queues.

[0071] S302, determine the target data transmission task in at least one data transmission task in the target queue.

[0072] Among at least one data transmission task in the target queue, determining the target data transmission task includes: determining the data transmission task with the highest priority and / or the earliest submission time of the target queue as the target data transmission task.

[0073] In one embodiment, the highest priority data transmission task in the target queue is determined as the target data transmission task. For example, queue c2 is the target queue, and queue c2 includes data transmission tasks e1, e2, e3, and e4. If the priority of data transmission task e2 > the priority of data transmission task e1 > the priority of data transmission task e3 > the priority of data transmission task e4, then data transmission task e2 can be determined as the target data transmission task. This method ensures that high-priority data transmission tasks are processed first.

[0074] In one embodiment, the data transmission task with the earliest submission time in the target queue is determined as the target data transmission task. For example, if the submission time of data transmission task e1 is earlier than the submission time of data transmission task e2, which is earlier than the submission time of data transmission task e3, which is earlier than the submission time of data transmission task e4, then data transmission task e1 is determined as the target data transmission task. This ensures that data transmission tasks with earlier submission times are processed first.

[0075] In one embodiment, in the target queue, the data transmission task with the highest priority is firstly determined as the first data transmission task. If there is only one first data transmission task, then the first data transmission task is determined as the target data transmission task. If there are multiple first data transmission tasks, then the data transmission task with the earliest submission time among the multiple first data transmission tasks is determined as the target data transmission task.

[0076] In summary, this application can prioritize the processing of higher-priority data transmission tasks when inserting them.

[0077] S303, obtain the bandwidth usage ratio of the target queue.

[0078] In this embodiment, the bandwidth usage ratio of each queue can be preset. For example, the bandwidth usage ratio of queue c1 is 20%, the bandwidth usage ratio of queue c2 is 30%, the bandwidth usage ratio of queue c3 is 40%, and the bandwidth usage ratio of queue c4 is 10%. If queue c2 is the target queue, then the bandwidth usage ratio of the target queue is 30%.

[0079] S304, determine the third bandwidth of the target queue based on the bandwidth usage ratio and the first bandwidth.

[0080] The third bandwidth is calculated as follows: third bandwidth = first bandwidth × bandwidth utilization rate. For example, if the first bandwidth is 100 G / s and the bandwidth utilization rate is 30%, then the third bandwidth is 30 G / s.

[0081] S305, Get the number of data transmission tasks in the target queue.

[0082] For example, if the target queue is queue c2, and queue c2 stores 4 data transmission tasks, then the number of data transmission tasks in the target queue is 4.

[0083] S306, determine the second bandwidth based on the third bandwidth and quantity.

[0084] The second bandwidth is equal to the third bandwidth divided by the number of items. For example, if the third bandwidth is 30 G / s and the number of items is 4, then the second bandwidth is 7.5 G / s.

[0085] S307, Get the number of threads running the target data transfer task.

[0086] In this embodiment of the application, the number of threads is fixed. For example, the current number of threads in the thread pool used to run the target data transmission task is 5.

[0087] S308 determines the fourth bandwidth corresponding to each subtask based on the second bandwidth and the number of threads.

[0088] The fourth bandwidth is calculated as the second bandwidth divided by the number of threads. For example, if the second bandwidth is 7.5 G / s and the number of threads is 5, then the fourth bandwidth is 1.5 G / s.

[0089] The target data transmission task includes multiple sub-tasks, and the target data includes the transmission data corresponding to each sub-task.

[0090] For example, if the target data transfer task is to transfer 1000 files (file data f1 to file data f1000) in directory 1, then the target data is these 1000 files. The target data transfer task includes 1000 subtasks (e.g., subtasks g1 to g1000), each subtask (gi) corresponds to one file (fi), where i ranges from 1 to 1000. Each subtask is run by one thread, and each thread corresponds to a fourth bandwidth (1.5G / s).

[0091] S309, based on the fourth bandwidth, controls the transmission of data corresponding to the subtask from the first data center to the second data center.

[0092] For example, if there are 5 threads, and the current 5 threads (x1 to x5) run 5 subtasks (g1 to g5) in parallel, then thread xj runs subtask gj based on the fourth bandwidth, realizing the transfer of file data fj from the first data center to the second data center.

[0093] S310, if the number of running subtasks is less than the number of threads, determine the fifth bandwidth of the running subtasks based on the second bandwidth and the number of running subtasks.

[0094] Wherein, the fifth bandwidth = the second bandwidth / the number of subtasks.

[0095] It's understandable that when multiple threads run subtasks, some subtasks may have already completed, resulting in a number of currently running subtasks that is less than the number of threads. Similarly, the number of threads currently working may be less than the total number of threads. For example, with 1000 subtasks, 5 threads might run 5 subtasks at a time. After these 5 subtasks finish, the same 5 threads run the next 5 subtasks, and so on, until all 1000 subtasks are completed. If 3 threads prioritize completing these 3 subtasks, then the number of currently running subtasks is 2, and the fifth bandwidth is 3.75 Gbps. Furthermore, the remaining subtasks can be run at 3.75 Gbps to improve subtask efficiency, thereby improving the overall efficiency of the target data transmission task.

[0096] S311, based on the fifth bandwidth, controls the transmission of data corresponding to the running sub-tasks from the first data center to the second data center.

[0097] It is understandable that allocating a fifth bandwidth to the running subtasks allows the data transmission corresponding to the running subtasks to be completed efficiently.

[0098] In this embodiment, the available bandwidth for each data transmission task is dynamically calculated based on changes in the number of data transmission tasks in the target queue, adjustments to bandwidth between data centers, and adjustments to queue priorities. The executor is then promptly notified to adjust the available bandwidth of the target data transmission task, ensuring the task adapts to the adjusted bandwidth. This fully utilizes available bandwidth resources between data centers while avoiding exceeding resource limits. Furthermore, dynamic bandwidth adjustment occurs between subtasks during the execution of the target data transmission task. Each subtask shares the target data transmission task's bandwidth. The available bandwidth of each subtask is dynamically calculated based on its completion status, and its availability is adjusted accordingly. This ensures that the bandwidth allocated to the target data transmission task is fully utilized, improving data transmission efficiency.

[0099] The following are system embodiments of this application, which can be used to execute the method embodiments of this application. For details not disclosed in the system embodiments of this application, please refer to the method embodiments of this application.

[0100] Reference Figure 1 This is a schematic diagram of a data transmission system provided in an embodiment of this application. The data transmission system includes: a coordinator and a target executor, wherein:

[0101] The coordinator is used to: determine the target data transmission task to be run; the target data transmission task indicates the transmission of target data from the first data center to the second data center; obtain the first bandwidth configured between the first data center and the second data center; and determine the second bandwidth of the target data transmission task based on the first bandwidth, wherein the second bandwidth is positively correlated with the first bandwidth.

[0102] The target actuator is used to control the transmission of target data from the first data center to the second data center based on the second bandwidth.

[0103] In one embodiment of this application, when the coordinator determines the target data transmission task to be run, it is specifically used to: determine a target queue among multiple queues, wherein the queue stores at least one data transmission task; and determine the target data transmission task among the at least one data transmission task in the target queue.

[0104] In one embodiment of this application, when the coordinator determines the target queue among multiple queues, it is specifically used to: determine the queue with the highest priority and / or the smallest number of data transmission tasks as the target queue among multiple queues.

[0105] In one embodiment of this application, when the coordinator determines the target data transmission task in at least one data transmission task in the target queue, it specifically does so by: determining the data transmission task with the highest priority and / or the earliest submission time of the target queue as the target data transmission task in at least one data transmission task.

[0106] In one embodiment of this application, when the coordinator determines the second bandwidth of the target data transmission task based on the first bandwidth, it is specifically configured to: obtain the bandwidth usage ratio of the target queue; determine the third bandwidth of the target queue based on the bandwidth usage ratio and the first bandwidth; obtain the number of data transmission tasks in the target queue; and determine the second bandwidth based on the third bandwidth and the number.

[0107] In one embodiment of this application, the target data transmission task includes multiple subtasks, and the target data includes: transmission data corresponding to each subtask; when the target executor controls the first data center to transmit the target data to the second data center according to the second bandwidth, it is specifically used to: obtain the number of threads running the target data transmission task; and determine the fourth bandwidth corresponding to each subtask according to the second bandwidth and the number of threads.

[0108] In one embodiment of this application, the target executor is further configured to: if the number of running subtasks is less than the number of threads, determine the fifth bandwidth of the running subtasks based on the second bandwidth and the number of running subtasks; and control the first data center to transmit the transmission data corresponding to the running subtasks to the second data center based on the fifth bandwidth.

[0109] In one embodiment of this application, the coordinator is further configured to: determine a target executor based on the load of each of the plurality of executors; the target executor is specifically configured to: control the transmission of target data from the first data center to the second data center based on the second bandwidth.

[0110] For the specific implementation process of the data transmission system in this application, please refer to the embodiments of the data transmission method described above, which will not be repeated here.

[0111] It should be noted that the division of the various modules in the above system is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software through processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, a processing module can be a separate processing element, or it can be integrated into a chip within the system. Alternatively, it can be stored as program code in the system's memory, and its functions can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. During implementation, the steps of the above method or the various modules can be completed through integrated logic circuits in the processor element or through software instructions.

[0112] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0113] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable system. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, central terminal, or data center to another website, computer, central terminal, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a central terminal or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0114] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device may include: a processor 41, a memory 42, a communication interface 43, and a system bus 44. The memory 42 and the communication interface 43 are connected to the processor 41 via the system bus 44 and communicate with each other. The memory 42 stores computer-executed instructions, and the communication interface 43 communicates with other devices. When the processor 41 executes the aforementioned computer-executed instructions, it implements the scheme described in the above embodiment.

[0115] Should Figure 4The system bus mentioned can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access system and other devices (e.g., user terminals, read-write databases, and read-only databases). Memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0116] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0117] Optionally, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes an electronic device to perform the method as described in the above embodiments.

[0118] Optionally, embodiments of this application also provide a computer program product having a computer program stored thereon, wherein when the computer program is executed by a processor, it causes an electronic device to perform the method described above.

[0119] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the preceding and following related objects; in formulas, the character " / " indicates a "division" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0120] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. In the embodiments of this application, the order of the above-mentioned process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data transmission method, characterized in that, The data transmission method, applied to a data transmission system, includes: Determine the target data transmission task to be executed; the target data transmission task indicates the transmission of target data from the first data center to the second data center; Obtain the first bandwidth configured between the first data center and the second data center; Based on the first bandwidth, a second bandwidth for the target data transmission task is determined, wherein the second bandwidth is positively correlated with the first bandwidth; Based on the second bandwidth, control the first data center to transmit the target data to the second data center.

2. The data transmission method according to claim 1, characterized in that, The process of determining the target data transmission task to be executed includes: A target queue is determined from multiple queues, wherein at least one data transmission task is stored in the queue; The target data transmission task is determined from at least one data transmission task in the target queue.

3. The data transmission method according to claim 2, characterized in that, The step of determining the target queue among multiple queues includes: Among the multiple queues, the queue with the highest priority and / or the smallest number of data transmission tasks is determined as the target queue.

4. The data transmission method according to claim 3, characterized in that, In at least one data transmission task in the target queue, determining the target data transmission task includes: In the at least one data transmission task, the data transmission task with the highest priority and / or the earliest submission time of the target data transmission task is determined as the target data transmission task.

5. The data transmission method according to any one of claims 2 to 4, characterized in that, Determining the second bandwidth of the target data transmission task based on the first bandwidth includes: Obtain the bandwidth usage ratio of the target queue; The third bandwidth of the target queue is determined based on the bandwidth usage ratio and the first bandwidth; Obtain the number of data transmission tasks in the target queue; The second bandwidth is determined based on the third bandwidth and the quantity.

6. The data transmission method according to claim 5, characterized in that, The target data transmission task includes multiple sub-tasks, and the target data includes: transmission data corresponding to each sub-task; controlling the first data center to transmit the target data to the second data center according to the second bandwidth includes: Obtain the number of threads running the target data transmission task; Based on the second bandwidth and the number of threads, determine the fourth bandwidth corresponding to each subtask; Based on the fourth bandwidth, control the first data center to transmit the transmission data corresponding to the subtask to the second data center.

7. The data transmission method according to claim 6, characterized in that, Also includes: If the number of running subtasks is less than the number of threads, the fifth bandwidth of the running subtasks is determined based on the second bandwidth and the number of running subtasks. Control the first data center to transmit the data corresponding to the running subtask to the second data center according to the fifth bandwidth.

8. The data transmission method according to any one of claims 1 to 4, characterized in that, The data transmission system includes a coordinator and multiple actuators, and the method further includes: the coordinator determining a target actuator based on the load of each of the multiple actuators; The step of controlling the first data center to transmit the target data to the second data center according to the second bandwidth includes: using the target actuator to control the first data center to transmit the target data to the second data center according to the second bandwidth.

9. A data transmission system, characterized in that, include: Coordinator and target executor, wherein: The coordinator is configured to: determine a target data transmission task to be run; the target data transmission task indicates the transmission of target data from a first data center to a second data center; obtain a first bandwidth configured between the first data center and the second data center; and determine a second bandwidth for the target data transmission task based on the first bandwidth, wherein the second bandwidth is positively correlated with the first bandwidth. The target actuator is used to control the first data center to transmit the target data to the second data center according to the second bandwidth.

10. An electronic device, characterized in that, Includes memory and processor; among which, The memory is used to store program code; The processor is used to call the program code to implement the data transmission method as described in any one of claims 1 to 8.

Images