A resource decoupled cross-cloud data transmission method and device

Abstract

The present application provides a resource-decoupled cross-cloud data transmission method and device, including: responding to a first data transmission instruction sent by a target user, determining, based on the first data transmission instruction, a first data storage address where the target data to be transmitted is located, the data volume of the target data to be transmitted, and the target transmission address, determining, based on the data transmission rate, data volume, and preset transmission duration between the first data storage address and the target transmission address, a first call quantity of a first data transmission function corresponding to the first data transmission instruction, calling each first data transmission function step by step, and counting the number of calls to the first data transmission function, and when the number of calls is greater than or equal to the first call quantity, sending the target data to be transmitted from the first data storage address to the target transmission address based on each first data transmission function, thereby improving the transmission efficiency of cross-cloud data transmission to a certain extent.

Description

Technical Field

[0001] The present application relates to the field of cloud storage technology, and more particularly to a resource-decoupled cross-cloud data transmission method and device. Background Art

[0002] With the development of Internet users, the amount of digital information created and shared has increased year by year, including photos, audio, video and other data information flooding the network and hard disk. In particular, the rapid development of mobile Internet has triggered a data revolution. In this data revolution, the storage space of a single storage hard disk cannot store massive amounts of data. Therefore, cloud storage technology that can combine various hard disks to form a cluster has become a reliable choice. However, with the continuous expansion of data centers, a single cloud storage method has been difficult to meet customer needs. How to stably transmit data from different regions has become an urgent problem to be solved.

[0003] In related technologies, various public cloud vendors provide their own cross-cloud data transmission solutions, such as Data Transmission Service (DTS), which implements cross-cloud data transmission under the same account by configuring source and destination data instances. Or some open source cross-cloud data transmission methods based on virtual machines, such as Skyplane.

[0004] However, in the above methods, cross-cloud data transmission tasks often consume a lot of time, the efficiency of cross-cloud data transmission is low, and due to the long task time, the transmission task will also have a greater transmission security risk. Summary of the invention

[0005] In view of the above problems, embodiments of the present application provide a resource-decoupled cross-cloud data transmission method, device, electronic device and readable storage medium to overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, an embodiment of the present application provides a resource decoupled cross-cloud data transmission method, the method comprising:

[0007] In response to a first data transmission instruction sent by a target user, determining a first data storage address where the target data to be transmitted is located, a data volume of the target data to be transmitted, and a target transmission address based on the first data transmission instruction; wherein the first data storage address and the target transmission address correspond to different cloud storage spaces;

[0008] Determine a first call quantity of a first data transmission function corresponding to the first data transmission instruction based on a data transmission rate between the first data storage address and the target transmission address, the data volume, and a preset transmission duration;

[0009] Calling each of the first data transmission functions step by step, and counting the number of calls of the first data transmission function;

[0010] When the number of calls is greater than or equal to the first number of calls, based on each of the first data transmission functions, the target data to be transmitted is sent from the first data storage address to the target transmission address to achieve cross-cloud data transmission between the first data storage address and the target transmission address.

[0011] Optionally, determining the first call quantity of the first data transmission function corresponding to the first data transmission instruction based on the data transmission rate between the first data storage address and the target transmission address, the data volume and a preset transmission duration includes:

[0012] Determine, based on the data transmission rate and the data volume between the first data storage address and the target transmission address, a first data transmission duration of the first data transmission function corresponding to the first data transmission instruction;

[0013] Based on the first data transmission duration and the preset transmission duration, a first call quantity of the first data transmission function is determined.

[0014] Optionally, the determining, based on the first data transmission duration and the preset transmission duration, a first call quantity of the first data transmission function includes:

[0015] Determining a maximum number of calls to the first data transmission function based on an upper limit of the startup duration of the data transmission function;

[0016] Based on the maximum call quantity, the first data transmission duration and the preset transmission duration, a first call quantity of the first data transmission function is determined.

[0017] Optionally, the data structure of the first data transmission function is a time-limited complete tree; the time-limited complete tree includes a main function node, a data transmission function node and a data transmission function call edge for connecting each of the data transmission function nodes and the main function node; the data transmission function node corresponds to the function startup delay duration of each of the first data transmission functions; the data transmission function call edge corresponds to the function call duration of each of the first data transmission functions; the startup duration of the first data transmission function is equal to the sum of all function startup delay durations and all function call durations corresponding to the connecting path between the data transmission function node corresponding to the first data transmission function and the main function node, and the startup duration is less than or equal to the startup duration upper limit.

[0018] Optionally, the sending the target data to be transmitted from the first data storage address to the target transmission address based on each of the first data transmission functions includes:

[0019] Adding a distributed lock to each of the first data transmission functions to obtain a second data transmission function;

[0020] Based on each of the second data transmission functions, the target data to be transmitted is sent from the first data storage address to the target transmission address.

[0021] Optionally, the method further comprises:

[0022] When the first data to be transmitted corresponding to the second data transmission function is transmitted completely, the distributed lock of the second data transmission function is released to obtain the first data transmission function.

[0023] Optionally, determining, based on the first data transmission instruction, a first data storage address of the target data to be transmitted, a data volume of the target data to be transmitted, and a target transmission address includes:

[0024] Parsing the first data transmission instruction to obtain a first hash value and a target transmission address;

[0025] Based on the first hash value, a matching query is performed in the first cloud storage space to obtain the target data to be transmitted and the first data storage address where the target data to be transmitted is located;

[0026] Based on the first data storage address, the data volume of the target data to be transmitted is determined.

[0027] Optionally, determining, based on the first data transmission instruction, a first data storage address of the target data to be transmitted, a data volume of the target data to be transmitted, and a target transmission address includes:

[0028] Based on a preset instruction data format, performing a data format detection on the first data transmission instruction to obtain a first detection result;

[0029] When the first detection result is that the detection is passed, based on the first data transmission instruction, a first data storage address of the target data to be transmitted, a data volume of the target data to be transmitted, and a target transmission address are determined.

[0030] Optionally, the method further comprises:

[0031] When the first detection result is detection failure, a prompt message indicating an instruction format error is sent to the target user.

[0032] In a second aspect, an embodiment of the present application provides a resource decoupled cross-cloud data transmission device, the device comprising:

[0033] A first determination module is used to respond to a first data transmission instruction sent by a target user, and determine a first data storage address where target data to be transmitted is located, a data volume of the target data to be transmitted, and a target transmission address based on the first data transmission instruction; wherein the first data storage address and the target transmission address correspond to different cloud storage spaces;

[0034] A second determining module, configured to determine a first call quantity of a first data transmission function corresponding to the first data transmission instruction based on a data transmission rate between the first data storage address and the target transmission address, the data volume and a preset transmission duration;

[0035] A call statistics module, used to call each of the first data transmission functions step by step, and count the number of calls of the first data transmission function;

[0036] A transmission module is used to send the target data to be transmitted from the first data storage address to the target transmission address based on each of the first data transmission functions when the number of calls is greater than or equal to the first call quantity, so as to realize cross-cloud data transmission between the first data storage address and the target transmission address.

[0037] Optionally, the second determining module includes:

[0038] A first determining submodule, configured to determine a first data transmission duration of a first data transmission function corresponding to the first data transmission instruction based on a data transmission rate and the data volume between the first data storage address and the target transmission address;

[0039] A second determining submodule is configured to determine a first calling quantity of the first data transmission function based on the first data transmission duration and the preset transmission duration.

[0040] Optionally, the second determining submodule includes:

[0041] A first determining unit, configured to determine a maximum number of calls to the first data transmission function based on an upper limit of a startup duration of the data transmission function;

[0042] The second determining unit is used to determine a first calling quantity of the first data transmission function based on the maximum calling quantity, the first data transmission duration and the preset transmission duration.

[0043] Optionally, the data structure of the first data transmission function is a time-limited complete tree; the time-limited complete tree includes a main function node, a data transmission function node and a data transmission function call edge for connecting each of the data transmission function nodes and the main function node; the data transmission function node corresponds to the function startup delay duration of each of the first data transmission functions; the data transmission function call edge corresponds to the function call duration of each of the first data transmission functions; the startup duration of the first data transmission function is equal to the sum of all function startup delay durations and all function call durations corresponding to the connecting path between the data transmission function node corresponding to the first data transmission function and the main function node, and the startup duration is less than or equal to the startup duration upper limit.

[0044] Optionally, the transmission module includes:

[0045] An adding submodule is used to add a distributed lock to each of the first data transmission functions to obtain a second data transmission function;

[0046] The transmission submodule is used to send the target data to be transmitted from the first data storage address to the target transmission address based on each of the second data transmission functions.

[0047] Optionally, the device further comprises:

[0048] A release module is used to release the distributed lock of the second data transmission function when the first data to be transmitted corresponding to the second data transmission function is transmitted, so as to obtain the first data transmission function.

[0049] Optionally, the first determining module includes:

[0050] A parsing submodule, used to parse the first data transmission instruction to obtain a first hash value and a target transmission address;

[0051] A matching submodule, configured to perform a matching query in a first cloud storage space based on the first hash value to obtain target data to be transmitted and a first data storage address where the target data to be transmitted is located;

[0052] The third determining submodule is configured to determine the amount of the target data to be transmitted based on the first data storage address.

[0053] Optionally, the first determining module includes:

[0054] A detection submodule, configured to perform a data format detection on the first data transmission instruction based on a preset instruction data format to obtain a first detection result;

[0055] The fourth determination submodule is used to determine, when the first detection result is detection passed, based on the first data transmission instruction, a first data storage address of the target data to be transmitted, a data volume of the target data to be transmitted, and a target transmission address.

[0056] Optionally, the device further comprises:

[0057] The prompt module is used to send a prompt message of an instruction format error to the target user when the first detection result is detection failure.

[0058] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the resource decoupled cross-cloud data transmission method as described in any one of the above.

[0059] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the resource-decoupled cross-cloud data transmission method as described in any one of the above is implemented.

[0060] The specific beneficial effects are:

[0061] The embodiment of the present application determines, in response to a first data transmission instruction sent by a target user, a first data storage address where the target data to be transmitted is located, the data volume of the target data to be transmitted, and the target transmission address based on the first data transmission instruction; wherein the first data storage address and the target transmission address correspond to different cloud storage spaces, and based on the data transmission rate, data volume, and preset transmission duration between the first data storage address and the target transmission address, a first call quantity of a first data transmission function corresponding to the first data transmission instruction is determined, each first data transmission function is called step by step, and the number of calls to the first data transmission function is counted, and when the number of calls is greater than or equal to the first call quantity, the target data to be transmitted is sent from the first data storage address to the target transmission address based on each first data transmission function to realize cross-cloud data transmission between the first data storage address and the target transmission address, and the first call quantity of the data transmission function can be obtained by the preset transmission duration, the data transmission rate, and the data volume of the target data to be transmitted, and the target data to be transmitted is sent to the target transmission address by the data transmission function of the above-mentioned first call quantity, so that the time consumption of the transmission task corresponding to the first data transmission instruction can be controlled within the preset transmission duration, thereby improving the transmission efficiency of cross-cloud data transmission to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0063] Figure 1 It is a flowchart of a resource decoupled cross-cloud data transmission method provided in an embodiment of the present application;

[0064] Figure 2 It is a flowchart of another resource decoupled cross-cloud data transmission method provided in an embodiment of the present application;

[0065] Figure 3 is a schematic diagram of a time-limited complete tree provided in an embodiment of the present application;

[0066] Figure 4 It is a flowchart of a specific implementation method of a resource decoupled cross-cloud data transmission method provided in an embodiment of the present application;

[0067] Figure 5 It is a logical block diagram of a resource decoupled cross-cloud data transmission device provided in an embodiment of the present application;

[0068] Figure 6 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The exemplary embodiments of the present application will be described in more detail below in conjunction with the accompanying drawings in the embodiments of the present application. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to enable the scope of the present application to be fully communicated to those skilled in the art.

[0070] Reference Figure 1 , Figure 1 A schematic diagram of a process flow of a resource decoupled cross-cloud data transmission method provided in an embodiment of the present application, the method comprising:

[0071] Step 101, in response to a first data transmission instruction sent by a target user, determine a first data storage address of the target data to be transmitted, the data volume of the target data to be transmitted, and the target transmission address based on the first data transmission instruction; wherein the first data storage address and the target transmission address correspond to different cloud storage spaces.

[0072] In an embodiment of the present application, cloud storage is a mode of online storage on the Internet, that is, data is stored on multiple virtual servers usually hosted by a third party, rather than on a dedicated server. Cloud storage space is the storage space of a virtual server, which can be used to store various data. The target user can send a first data transmission instruction to a terminal device connected to the cloud storage space to transfer the specified data in the cloud storage space to another cloud storage space. The terminal device can respond to the first data transmission instruction and determine the first data transmission address where the target data to be transmitted is located, the data volume of the target data to be transmitted, and the target transmission address according to the metadata information contained in the first data transmission instruction. The target transmission address is the storage address for receiving the target data to be transmitted. Resource decoupling in the present application means that the control terminal of the data transmission process and the storage terminal where the data to be transmitted is located are not the same terminal device, that is, the "computing and storage decoupling" method.

[0073] Step 102: Determine a first call quantity of a first data transmission function corresponding to the first data transmission instruction based on a data transmission rate between the first data storage address and the target transmission address, the data volume and a preset transmission duration.

[0074] In an embodiment of the present application, the first number of calls to the first data transmission function corresponding to the first data transmission instruction can be calculated based on the data transmission rate between the first data storage address of the target data to be transmitted and the target transmission address, the data volume of the target data to be transmitted, and the preset transmission duration. The first data transmission function is a code function that can read the target data to be transmitted from the first data storage address, and can send the target data to be transmitted to the target transmission address so that the target data to be transmitted can be stored in the cloud storage space corresponding to the target transmission address. The preset transmission duration can be set manually to control the task execution duration of the transmission task corresponding to the first data transmission instruction within the preset transmission duration.

[0075] Step 103: Call each of the first data transmission functions step by step, and count the number of times the first data transmission function is called.

[0076] In an embodiment of the present application, each first data transmission function can be called step by step so that the target data to be transmitted in the first data storage address can be read by the first data transmission function. When calling the first data transmission function, the number of calls of the first data transmission function can be counted at the same time. Among them, the step-by-step calling can refer to serial calling of each first data transmission function, or it can refer to calling step by step from top to bottom according to the nested relationship of each first data transmission function.

[0077] Step 104, when the number of calls is greater than or equal to the first number of calls, based on each of the first data transmission functions, the target data to be transmitted is sent from the first data storage address to the target transmission address to achieve cross-cloud data transmission between the first data storage address and the target transmission address.

[0078] In an embodiment of the present application, if the number of calls is greater than or equal to the first call number of the first data transmission function, it can be considered that the first data transmission function has been called successfully. At this time, the target data to be transmitted can be sent from the first data storage address to the target transmission address according to each first data transmission function, so that cross-cloud data transmission between the first data storage address and the target transmission address can be realized. Among them, according to the first call number, the target data to be transmitted can be dispersed into small portions of data to be transmitted with the same number and the same data volume, and allocated to each first data transmission function, so that the data transmission task corresponding to the first data transmission instruction can be executed in parallel through the first data transmission function.

[0079] In an embodiment of the present application, by responding to a first data transmission instruction sent by a target user, based on the first data transmission instruction, a first data storage address where the target data to be transmitted is located, a data volume of the target data to be transmitted, and a target transmission address are determined; wherein the first data storage address and the target transmission address correspond to different cloud storage spaces, based on the data transmission rate, data volume, and preset transmission duration between the first data storage address and the target transmission address, a first call quantity of a first data transmission function corresponding to the first data transmission instruction is determined, each first data transmission function is called step by step, and the number of calls of the first data transmission function is counted, and when the number of calls is greater than or equal to the first call quantity, based on each first data transmission function, the target data to be transmitted is sent from the first data storage address to the target transmission address to realize cross-cloud data transmission between the first data storage address and the target transmission address, the first call quantity of the data transmission function can be obtained by the preset transmission duration, the data transmission rate, and the data volume of the target data to be transmitted, and the target data to be transmitted is sent to the target transmission address by the data transmission function of the above-mentioned first call quantity, so that the time consumption of the transmission task corresponding to the first data transmission instruction can be controlled within the preset transmission duration, thereby improving the transmission efficiency of cross-cloud data transmission to a certain extent.

[0080] Reference Figure 2 , Figure 2 A flowchart of another resource decoupled cross-cloud data transmission method provided in an embodiment of the present application, the method may include:

[0081] Step 201, in response to a first data transmission instruction sent by a target user, based on the first data transmission instruction, determine a first data storage address where the target data to be transmitted is located, the data volume of the target data to be transmitted, and a target transmission address; wherein the first data storage address and the target transmission address correspond to different cloud storage spaces.

[0082] In the embodiments of the present application, the implementation content of this step can refer to the embodiment content of step 101, which will not be repeated here.

[0083] Optionally, step 201 may include the following sub-steps:

[0084] Sub-step 2011: Based on a preset instruction data format, perform a data format detection on the first data transmission instruction to obtain a first detection result.

[0085] In an embodiment of the present application, the first data transmission instruction for performing cross-cloud transmission may have a corresponding data format so that the terminal device can detect the transmission task information contained in the first data transmission instruction. Therefore, the first data transmission instruction may be subjected to data format detection according to a preset instruction data format set manually, thereby obtaining a first detection result. The first detection result may include a detection pass and a detection fail, wherein a detection pass may indicate that the data format of the first data transmission instruction is consistent with the preset instruction data format, and a detection fail may indicate that the data format of the first data transmission instruction is inconsistent with the preset instruction data format.

[0086] Sub-step 2012, when the first detection result is detection passed, determining a first data storage address of the target data to be transmitted, a data volume of the target data to be transmitted, and a target transmission address based on the first data transmission instruction.

[0087] In an embodiment of the present application, if the first detection result of the first data transmission instruction is detection pass, the first data storage address of the target data to be transmitted, the data volume of the target data to be transmitted and the target transmission address can be obtained according to the metadata information contained in the first data transmission instruction.

[0088] In an embodiment of the present application, a first data transmission instruction is subjected to a data format detection based on a preset instruction data format to obtain a first detection result. When the first detection result is a passed detection, the first data storage address of the target data to be transmitted, the data volume of the target data to be transmitted, and the target transmission address are determined based on the first data transmission instruction. The first data transmission instruction can be detected. When the detection passes, the first data storage address, the data volume of the target data to be transmitted, and the target transmission address are determined according to the first data transmission instruction. This can avoid program errors that may be caused by an error in the first data transmission instruction, thereby improving the stability of the cross-cloud data transmission method to a certain extent.

[0089] Sub-step 2013: When the first detection result is detection failure, send a prompt message indicating an instruction format error to the target user.

[0090] In an embodiment of the present application, if the first detection result is detection failure, the first data transmission instruction cannot trigger the data transmission task normally. At this time, an instruction format error prompt message can be sent to the target user so that the target user can correct the first data transmission instruction in time.

[0091] In an embodiment of the present application, by sending an instruction format error prompt message to the target user when the first detection result is a detection failure, an error prompt can be given to the target user when the first detection result is a detection failure, so that the user can correct the erroneous instruction in time, which can improve the accuracy of the first data transmission instruction to a certain extent, and at the same time avoid the transmission program from falling into a logical dead loop, which can improve the stability of the cross-cloud data transmission method to a certain extent.

[0092] Sub-step 2014: parsing the first data transmission instruction to obtain a first hash value and a target transmission address.

[0093] In an embodiment of the present application, the first data transmission instruction may be parsed to obtain metadata information contained in the first data transmission instruction. The metadata information may include a first hash value and a target transmission address.

[0094] Sub-step 2015: Based on the first hash value, a matching query is performed in the first cloud storage space to obtain the target data to be transmitted and the first data storage address where the target data to be transmitted is located.

[0095] In an embodiment of the present application, a matching query can be performed in the first cloud storage space according to the first hash value, so as to obtain the target data to be transmitted that matches the first hash value and the first data storage address where the target data to be transmitted is located.

[0096] Sub-step 2016: determining the amount of the target data to be transmitted based on the first data storage address.

[0097] In an embodiment of the present application, the storage space corresponding to the first data storage address may be determined first, and then the data volume of the target data to be transmitted may be determined according to the memory size of the storage space, wherein the data volume may be the same as the memory size of the storage space.

[0098] In an embodiment of the present application, a first hash value and a target transmission address are obtained by parsing a first data transmission instruction, and a matching query is performed in the first cloud storage space based on the first hash value to obtain the target data to be transmitted and the first data storage address where the target data to be transmitted is located, and based on the first data storage address, the data volume of the target data to be transmitted is determined. The target data to be transmitted, the first data storage address and the data volume of the target data to be transmitted can be determined in the first cloud storage space through the first hash value and the target transmission address contained in the first data transmission instruction, which improves the accuracy and reliability of the first data storage address, the data volume and the target transmission address to a certain extent.

[0099] Step 202: Determine a first data transmission duration of a first data transmission function corresponding to the first data transmission instruction based on the data transmission rate and the data volume between the first data storage address and the target transmission address.

[0100] In an embodiment of the present application, the first data transmission duration of the first data transmission function corresponding to the first data transmission instruction can be calculated based on the data transmission rate between the first data storage address and the target transmission address, and the data volume of the target data to be transmitted. Specifically, the data volume of the target data to be transmitted can be divided by the data transmission rate to obtain the quotient of the data volume of the target data to be transmitted and the data transmission rate, and the quotient is the first data transmission duration of the first data transmission function, that is, the transmission duration required when using a first data transmission function to transmit the target data to be transmitted.

[0101] Step 203: Determine a first call quantity of the first data transmission function based on the first data transmission duration and the preset transmission duration.

[0102] In an embodiment of the present application, the first call quantity of the first data transmission function can be calculated according to the first data transmission duration and the preset transmission duration. Specifically, the first data transmission duration can be divided by the preset transmission duration, and the obtained quotient is the first call quantity of the first data transmission function.

[0103] Optionally, step 203 may include the following sub-steps:

[0104] Sub-step 2031, determining the maximum number of calls to the first data transmission function based on the upper limit of the startup duration of the data transmission function.

[0105] In an embodiment of the present application, when the terminal device calls the data transmission function, in order to maintain the stability of the data transmission function, a certain startup delay time is required. In addition, the instruction to call the data transmission function itself also requires a certain calling time. At the same time, since each first data transmission function needs to be started step by step, the startup time upper limit of the data transmission function can be set in advance, so that the maximum number of calls to the first data transmission function can be calculated based on the startup time upper limit.

[0106] For example, if the upper limit of the startup duration is 10 time units, the startup delay duration of each first data transmission function is 3 time units, the call duration is 1 time unit, and there is no calling relationship between the first data transmission functions, for each first data transmission function, its own startup duration can be included in the sum of the call durations of all first data transmission functions called before it, as well as the sum of its own call duration and the startup delay duration. Based on the above principle, the maximum number of calls to the first data transmission function can be 7.

[0107] Optionally, the data structure of the first data transmission function is a time-limited complete tree; the time-limited complete tree includes a main function node, a data transmission function node and a data transmission function call edge for connecting each of the data transmission function nodes and the main function node; the data transmission function node corresponds to the function startup delay duration of each of the first data transmission functions; the data transmission function call edge corresponds to the function call duration of each of the first data transmission functions; the startup duration of the first data transmission function is equal to the sum of all function startup delay durations and all function call durations corresponding to the connecting path between the data transmission function node corresponding to the first data transmission function and the main function node, and the startup duration is less than or equal to the startup duration upper limit.

[0108] In an embodiment of the present application, the time-limited complete tree may be a data structure between each first data transmission function. In the time-limited complete tree, a main function node, a data transmission function node, and a data transmission function call edge may be included. Among them, the data transmission function call edge is used to connect each data transmission function node and the main function node. In the time-limited complete tree, the data transmission function node may represent the startup delay duration of the first data transmission function corresponding to the node, and the data transmission function call edge may represent the function call duration of the data transmission function connected to the data transmission function call edge. The startup duration of each first data transmission function may be equal to the sum of all function startup delay durations and all function call durations corresponding to the connection path between the data transmission function node corresponding to the first data transmission function and the main function node, and the startup duration may be less than or equal to the startup duration upper limit. By using the data structure of the time-limited complete tree, the upper limit of the number of first calls of the first data transmission function can be appropriately increased within the startup duration upper limit, which can improve the transmission efficiency of cross-cloud data to a certain extent.

[0109] For example, Figure 3 As shown, Figure 3 A schematic diagram of a time-limited complete tree provided for an embodiment of the present application. In the figure, "1" is the main function node, which is used to call each first data transmission function node. Node "2" can call node "7", node "9" and node "12", node "3" can call node "10" and node "13", and node 4 can call node "14". The startup delay time of each node is 3, and the first call time is 1. In this time-limited complete tree, the maximum startup duration of each node is 10. For example, for node "12", before it, it is necessary to call node "2", node "7" and node "9", then for node "12", it needs to go through 4 calling processes and 2 startup delay processes (respectively, the startup delay of node "2" and the startup delay of node "12" itself). Therefore, the startup delay duration of node "12" is For another example, for node "10", before it, it needs to call node "2" and node "3", then for node "10", it needs to go through 3 calling processes and 2 startup delay processes (the startup delay of node "3" and the startup delay of node "10" itself). Therefore, the startup delay of node "10" is 9.

[0110] Sub-step 2032: determining a first call quantity of the first data transmission function based on the maximum call quantity, the first data transmission duration and the preset transmission duration.

[0111] In an embodiment of the present application, the first call quantity of the first data transmission function can be calculated according to the maximum call quantity of the first data transmission function, the first data transmission duration, and the preset transmission duration. The quotient of the first data transmission duration and the preset transmission duration can be calculated, so that a numerical range can be constructed by the quotient value and the maximum call quantity, and the first call quantity of the first data transmission function can be greater than or equal to the quotient value, and can be less than or equal to the maximum call quantity.

[0112] In an embodiment of the present application, the maximum number of calls of the first data transmission function is determined based on the upper limit of the startup duration of the data transmission function, and the first number of calls of the first data transmission function is determined based on the maximum number of calls, the first data transmission duration and the preset transmission duration. The maximum number of calls of the first data function can be determined according to the preset upper limit of the startup duration, and the first number of calls of the first data transmission function can be further determined based on the maximum number of calls, combined with the first data transmission duration and the preset transmission duration. This improves the accuracy and reliability of the first number of calls to a certain extent, and can further improve the transmission efficiency of the data transmission process.

[0113] In an embodiment of the present application, the first data transmission duration of the first data transmission function corresponding to the first data transmission instruction is determined based on the data transmission rate and the data volume between the first data storage address and the target transmission address, and the first call quantity of the first data transmission function is determined based on the first data transmission duration and the preset transmission duration. The first call quantity of the first data transmission function can be calculated one by one, and the minimum number of first data transmission functions can be determined under the preset transmission duration, which can save data transmission pipeline resources to a certain extent and avoid waste of transmission resources.

[0114] Step 204: Call each of the first data transmission functions step by step, and count the number of times the first data transmission function is called.

[0115] In the embodiments of the present application, the implementation content of this step can refer to the implementation content of step 103, which will not be repeated here.

[0116] Step 205, when the number of calls is greater than or equal to the first number of calls, based on each of the first data transmission functions, the target data to be transmitted is sent from the first data storage address to the target transmission address to achieve cross-cloud data transmission between the first data storage address and the target transmission address.

[0117] In the embodiments of the present application, the implementation content of this step can refer to the implementation content of step 104, which will not be repeated here.

[0118] Optionally, step 205 may include the following sub-steps:

[0119] Sub-step 2051: Add a distributed lock to each of the first data transmission functions to obtain a second data transmission function.

[0120] In an embodiment of the present application, since each first data transmission function corresponds to the data transmission process of the target data to be transmitted, a distributed lock can be added to each first data transmission function, so that a second data transmission function with a distributed lock can be obtained. A distributed lock is a mechanism for controlling access to shared resources by multiple processes or threads in a distributed system. It ensures that only one client can execute a specific code segment at the same time, thereby avoiding data competition and inconsistency problems. A distributed lock can be added to the first data transmission function by a database-based method, a Zookeeper-based method, or a Redis-based method. The above-mentioned methods for adding distributed locks are all commonly used distributed lock methods, and are not the focus of this application, so they are not introduced in detail here.

[0121] Sub-step 2052: Based on each of the second data transmission functions, the target data to be transmitted is sent from the first data storage address to the target transmission address.

[0122] In the embodiment of the present application, the target data to be transmitted can be sent from the first data storage address to the target transmission address according to each second data transmission function. Since the second data transmission function only adds a distributed lock compared to the first data transmission function and does not change the data transmission method specified by the data transmission function, the specific data transmission implementation process can refer to the embodiment of step 104, which will not be repeated here.

[0123] In an embodiment of the present application, a second data transmission function is obtained by adding a distributed lock to each first data transmission function. Based on each second data transmission function, the target data to be transmitted is sent from the first data storage address to the target transmission address. The distributed lock can be used to lock the transmission process corresponding to the first data transmission function, which can avoid the target data to be transmitted from being modified during the data transmission process, thereby improving the data stability of the target data to be transmitted and the stability of the data transmission process to a certain extent.

[0124] Sub-step 2053, when the first data to be transmitted corresponding to the second data transmission function is transmitted, release the distributed lock of the second data transmission function to obtain the first data transmission function.

[0125] In an embodiment of the present application, when the first data to be transmitted corresponding to the second data transmission function is transmitted, the distributed lock of the second data transmission function can be released, so that the first data transmission function can be obtained.

[0126] In an embodiment of the present application, the first data transmission function is obtained by releasing the distributed lock of the second data transmission function when the first data to be transmitted corresponding to the second data transmission function is transmitted. The distributed lock of the second data transmission function can be released when the data transmission process corresponding to the second data transmission function is completed. This can improve the availability of the cloud storage space while maintaining the stability of the data transmission process. At the same time, data transmission resources can be released to avoid resource waste caused by locking data transmission resources.

[0127] Reference Figure 4 , Figure 4 It is a flowchart of a specific implementation method of a resource-decoupled cross-cloud data transmission method provided in an embodiment of the present application. In the figure, first, a data transmission instruction message can be received. After the instruction message arrives, the data transmission instruction message can be parsed to obtain the metadata information of the data transmission instruction. Then, the required number of data transmission functions can be calculated based on the above metadata information, and a calling scheme for the data transmission function can be generated. Afterwards, a distributed lock can be used to lock the data transmission function. After locking, the data transmission task corresponding to the data transmission instruction can be started, and each data transmission function can be called according to the above calling scheme. After the call is completed, data transmission can start. After the data transmission is completed, the transmission task ends, and the data transmission function can be unlocked at this time to release the data transmission pipeline resources.

[0128] Reference Figure 5 , Figure 5 A logical block diagram of a resource-decoupled cross-cloud data transmission device provided in an embodiment of the present application, the device 500 may include:

[0129] A first determination module 501 is used to respond to a first data transmission instruction sent by a target user, and determine a first data storage address where target data to be transmitted is located, a data volume of the target data to be transmitted, and a target transmission address based on the first data transmission instruction; wherein the first data storage address and the target transmission address correspond to different cloud storage spaces;

[0130] A second determining module 502 is used to determine a first call quantity of a first data transmission function corresponding to the first data transmission instruction based on a data transmission rate between the first data storage address and the target transmission address, the data volume and a preset transmission duration;

[0131] A call statistics module 503, used to call each of the first data transmission functions step by step, and count the number of calls of the first data transmission function;

[0132] The transmission module 504 is used to send the target data to be transmitted from the first data storage address to the target transmission address based on each of the first data transmission functions when the number of calls is greater than or equal to the first number of calls, so as to realize cross-cloud data transmission between the first data storage address and the target transmission address.

[0133] Optionally, the second determining module 502 includes:

[0134] A first determining submodule, configured to determine a first data transmission duration of a first data transmission function corresponding to the first data transmission instruction based on a data transmission rate and the data volume between the first data storage address and the target transmission address;

[0135] A second determining submodule is configured to determine a first calling quantity of the first data transmission function based on the first data transmission duration and the preset transmission duration.

[0136] Optionally, the second determining submodule includes:

[0137] A first determining unit, configured to determine a maximum number of calls to the first data transmission function based on an upper limit of a startup duration of the data transmission function;

[0138] The second determining unit is used to determine a first calling quantity of the first data transmission function based on the maximum calling quantity, the first data transmission duration and the preset transmission duration.

[0139] Optionally, the data structure of the first data transmission function is a time-limited complete tree; the time-limited complete tree includes a main function node, a data transmission function node and a data transmission function call edge for connecting each of the data transmission function nodes and the main function node; the data transmission function node corresponds to the function startup delay duration of each of the first data transmission functions; the data transmission function call edge corresponds to the function call duration of each of the first data transmission functions; the startup duration of the first data transmission function is equal to the sum of all function startup delay durations and all function call durations corresponding to the connecting path between the data transmission function node corresponding to the first data transmission function and the main function node, and the startup duration is less than or equal to the startup duration upper limit.

[0140] Optionally, the transmission module 504 includes:

[0141] An adding submodule is used to add a distributed lock to each of the first data transmission functions to obtain a second data transmission function;

[0142] The transmission submodule is used to send the target data to be transmitted from the first data storage address to the target transmission address based on each of the second data transmission functions.

[0143] Optionally, the device 500 further includes:

[0144] A release module is used to release the distributed lock of the second data transmission function when the first data to be transmitted corresponding to the second data transmission function is transmitted, so as to obtain the first data transmission function.

[0145] Optionally, the first determining module 501 includes:

[0146] A parsing submodule, used to parse the first data transmission instruction to obtain a first hash value and a target transmission address;

[0147] A matching submodule, configured to perform a matching query in a first cloud storage space based on the first hash value to obtain target data to be transmitted and a first data storage address where the target data to be transmitted is located;

[0148] The third determining submodule is configured to determine the amount of the target data to be transmitted based on the first data storage address.

[0149] Optionally, the first determining module 501 includes:

[0150] A detection submodule, configured to perform a data format detection on the first data transmission instruction based on a preset instruction data format to obtain a first detection result;

[0151] The fourth determination submodule is used to determine, when the first detection result is detection passed, based on the first data transmission instruction, a first data storage address of the target data to be transmitted, a data volume of the target data to be transmitted, and a target transmission address.

[0152] Optionally, the device 500 further includes:

[0153] The prompt module is used to send a prompt message of an instruction format error to the target user when the first detection result is detection failure.

[0154] The resource decoupled cross-cloud data transmission device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a device other than a terminal. Exemplarily, the electronic device can be a GPUBOX, a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0155] The resource decoupled cross-cloud data transmission device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, a Linux operating system, a Windows operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0156] The cross-cloud data transmission device for resource decoupling provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0157] The present application embodiment provides an electronic device, see Figure 6 The electronic device 60 includes: a processor 601, a memory 602, and a computer program 6021 stored in the memory 602 and executable on the processor 601. When the processor 601 executes the program, the resource decoupled cross-cloud data transmission method of the aforementioned embodiment is implemented.

[0158] The embodiment of the present application also provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps in the resource-decoupled cross-cloud data transmission method disclosed in the embodiment of the present application.

[0159] The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device, the processor is enabled to implement the steps in the resource decoupled cross-cloud data transmission method disclosed in the embodiment of the present application.

[0160] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0161] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0164] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0165] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0166] The above is a detailed introduction to a resource-decoupled cross-cloud data transmission method provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A resource decoupled cross-cloud data transmission method, characterized in that: The method comprises: In response to a first data transmission instruction sent by a target user, determining a first data storage address where the target data to be transmitted is located, a data volume of the target data to be transmitted, and a target transmission address based on the first data transmission instruction; wherein the first data storage address and the target transmission address correspond to different cloud storage spaces; Determine a first call quantity of a first data transmission function corresponding to the first data transmission instruction based on a data transmission rate between the first data storage address and the target transmission address, the data volume, and a preset transmission duration; Calling each of the first data transmission functions step by step, and counting the number of calls of the first data transmission function; When the number of calls is greater than or equal to the first number of calls, based on each of the first data transmission functions, the target data to be transmitted is sent from the first data storage address to the target transmission address, so as to realize cross-cloud data transmission between the first data storage address and the target transmission address; The determining, based on the data transmission rate between the first data storage address and the target transmission address, the data volume and the preset transmission duration, of the first call quantity of the first data transmission function corresponding to the first data transmission instruction comprises: Determine, based on the data transmission rate and the data volume between the first data storage address and the target transmission address, a first data transmission duration of the first data transmission function corresponding to the first data transmission instruction; Determining a first call quantity of the first data transmission function based on the first data transmission duration and the preset transmission duration; The sending the target data to be transmitted from the first data storage address to the target transmission address based on each of the first data transmission functions includes: Adding a distributed lock to each of the first data transmission functions to obtain a second data transmission function; Based on each of the second data transmission functions, sending the target data to be transmitted from the first data storage address to the target transmission address; The data structure of the first data transmission function is a time-limited complete tree; the time-limited complete tree includes a main function node, a data transmission function node and a data transmission function call edge for connecting each of the data transmission function nodes and the main function node; the data transmission function node corresponds to the function startup delay duration of each of the first data transmission functions; the data transmission function call edge corresponds to the function call duration of each of the first data transmission functions; the startup duration of the first data transmission function is equal to the sum of all function startup delay durations and all function call durations corresponding to the connecting path between the data transmission function node corresponding to the first data transmission function and the main function node, and the startup durations are all less than or equal to the startup duration upper limit.

2. The method according to claim 1, characterized in that The determining, based on the first data transmission duration and the preset transmission duration, a first call quantity of the first data transmission function includes: Determining a maximum number of calls to the first data transmission function based on an upper limit of the startup duration of the data transmission function; Based on the maximum call quantity, the first data transmission duration and the preset transmission duration, a first call quantity of the first data transmission function is determined.

3. The method according to claim 1, characterized in that: The method further comprises: When the first data to be transmitted corresponding to the second data transmission function is transmitted completely, the distributed lock of the second data transmission function is released to obtain the first data transmission function.

4. The method according to claim 1, characterized in that The determining, based on the first data transmission instruction, a first data storage address where the target data to be transmitted is located, a data volume of the target data to be transmitted, and a target transmission address includes: Parsing the first data transmission instruction to obtain a first hash value and a target transmission address; Based on the first hash value, a matching query is performed in the first cloud storage space to obtain the target data to be transmitted and the first data storage address where the target data to be transmitted is located; Based on the first data storage address, the data volume of the target data to be transmitted is determined.

5. The method according to claim 1, characterized in that The determining, based on the first data transmission instruction, a first data storage address where the target data to be transmitted is located, a data volume of the target data to be transmitted, and a target transmission address includes: Based on a preset instruction data format, performing a data format detection on the first data transmission instruction to obtain a first detection result; When the first detection result is that the detection is passed, based on the first data transmission instruction, a first data storage address of the target data to be transmitted, a data volume of the target data to be transmitted, and a target transmission address are determined.

6. The method according to claim 5, characterized in that The method further comprises: When the first detection result is detection failure, a prompt message indicating an instruction format error is sent to the target user.

7. A resource decoupled cross-cloud data transmission device, characterized in that: The device comprises: A first determination module is used to respond to a first data transmission instruction sent by a target user, and determine a first data storage address where target data to be transmitted is located, a data volume of the target data to be transmitted, and a target transmission address based on the first data transmission instruction; wherein the first data storage address and the target transmission address correspond to different cloud storage spaces; A second determining module, configured to determine a first call quantity of a first data transmission function corresponding to the first data transmission instruction based on a data transmission rate between the first data storage address and the target transmission address, the data volume and a preset transmission duration; A call statistics module, used to call each of the first data transmission functions step by step, and count the number of calls of the first data transmission function; a transmission module, configured to, when the number of calls is greater than or equal to the first number of calls, send the target data to be transmitted from the first data storage address to the target transmission address based on each of the first data transmission functions, so as to realize cross-cloud data transmission between the first data storage address and the target transmission address; Wherein, the second determining module includes: A first determining submodule, configured to determine a first data transmission duration of a first data transmission function corresponding to the first data transmission instruction based on a data transmission rate and the data volume between the first data storage address and the target transmission address; A second determining submodule, configured to determine a first call quantity of the first data transmission function based on the first data transmission duration and the preset transmission duration; The transmission module comprises: An adding submodule is used to add a distributed lock to each of the first data transmission functions to obtain a second data transmission function; A transmission submodule, configured to send the target data to be transmitted from the first data storage address to the target transmission address based on each of the second data transmission functions; The data structure of the first data transmission function is a time-limited complete tree; the time-limited complete tree includes a main function node, a data transmission function node and a data transmission function call edge for connecting each of the data transmission function nodes and the main function node; the data transmission function node corresponds to the function startup delay duration of each of the first data transmission functions; the data transmission function call edge corresponds to the function call duration of each of the first data transmission functions; the startup duration of the first data transmission function is equal to the sum of all function startup delay durations and all function call durations corresponding to the connecting path between the data transmission function node corresponding to the first data transmission function and the main function node, and the startup durations are all less than or equal to the startup duration upper limit.

Images