On-demand network-based replication agent

Abstract

A computer-implemented method for on-demand data replication in a network architecture includes activating an on-demand program to pair a network-based replication agent with a data source. Based on one or more hints received from the data source, computing resources for the replication agent are allocated. The one or more hints indicate at least a size of an upcoming data traffic transmission from the data source. Upon receiving data from the data traffic transmission, the replication agent is activated by the on-demand program to replicate the received data using the allocated replication resources. Upon completion of data replication of the received data, the allocated computing resources are released. The on-demand program is a serverless function for activating the replication agent on demand upon receiving the data from the data traffic transmission.

Description

[0001] Cross-application of related applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 777,658, filed on December 10, 2018, entitled “On-demand Network-based Replication Agent,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to data replication technology. Some aspects relate to an on-demand network-based replication agent. Background Art

[0004] Data replication is the process of creating a backup (or copy) of data from one network environment to another, or within the same network environment. Data replication technology is a crucial tool in many disaster recovery scenarios. Following a catastrophic event within a network-based infrastructure, it is crucial to recover the technical environment, including its software and other data assets, and reestablish network survivability. Conventional replication agents, such as virtual machines, run continuously. Therefore, the operating costs of conventional replication agents are long-term, necessitating a cost-effective replication agent to improve data replication efficiency. Summary of the Invention

[0005] Various examples are now described to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0006] According to a first aspect of the present disclosure, a computer-implemented method for on-demand data replication in a network architecture is provided. The method includes activating an on-demand program to pair a network-based replication agent with a data source. Based on a prompt received from the data source, computing resources of the network architecture are allocated to the replication agent. The prompt indicates at least the size of an upcoming data service transmission from the data source. Upon receiving data from the data service transmission, the network-based replication agent is activated by the on-demand program to replicate the received data using the allocated computing resources. Upon completion of data replication of the received data, the allocated computing resources are released.

[0007] In a first implementation form of the method according to this first aspect, the on-demand program is a serverless function for activating the network-based replication agent on demand after receiving data from the data traffic transmission.

[0008] In a second implementation form of the method according to the first aspect or any previous implementation form of the first aspect, the prompt indicates at least one of: a data period start indicator, a data period end indicator, and a number of bytes in the upcoming data traffic transmission.

[0009] In a third implementation form of the method according to the first aspect or any previous implementation form of the first aspect, computing resources of the network fabric are allocated based on one or both of: historical resource usage of network-based replication agents, and pricing information for usage of computing resources.

[0010] In a fourth implementation form of the method according to the first aspect or any previous implementation form of the first aspect, the prompt comprises a cycle start prompt comprising a start cycle indicator of an upcoming data traffic transmission.

[0011] In a fifth implementation form of the method according to the first aspect or any previous implementation form of the first aspect, a period end notification is received from a data source. The period end notification includes an end period indicator for a data period ending data service transmission. Based on the end period indicator, allocated computing resources are released.

[0012] In a sixth embodiment of the method according to the first aspect or any previous embodiment of the first aspect, allocating computing resources further comprises allocating one or more of central processing unit (CPU) resources, disk space resources, and memory resources available within the network architecture to the network-based replication agent based on the one or more hints.

[0013] In a seventh implementation form of the method according to the first aspect or any previous implementation form of the first aspect, a further prompt is received after receiving the data of the first portion of the data traffic transmission, the further prompt indicating the data size of the upcoming second portion of the data traffic transmission.

[0014] In an eighth implementation form of the method according to the first aspect or any previous implementation form of the first aspect, one or more of the allocated CPU resources, disk space resources, and memory resources are dynamically adjusted based on another indication of the data size of the upcoming second part of the data service transmission.

[0015] According to a second aspect of the present disclosure, a system is provided, comprising a memory storing instructions and one or more processors in communication with the memory. The one or more processors execute instructions to activate an on-demand program to pair a network-based replication agent with a data source. The one or more processors execute instructions to allocate computing resources to the network-based replication agent based on a hint received from the data source. The hint at least indicates the size of an upcoming data service transmission from the data source. The one or more processors execute instructions to activate the network-based replication agent through the on-demand program after receiving data from the data service transmission to use the allocated computing resources to replicate the received data. The one or more processors execute instructions to release the allocated computing resources after completing the data replication of the received data.

[0016] In a first implementation form of the system according to the second aspect, the on-demand program is a serverless function for activating the cloud-based replication agent on demand after receiving data from the data traffic transmission.

[0017] In a second implementation form of the system according to the second aspect or any previous implementation form of the second aspect, the prompt indicates at least one of: a data period start indicator, a data period end indicator, and a number of bytes in the upcoming data traffic transmission.

[0018] In a third implementation form of the system according to the second aspect or any previous implementation form of the second aspect, computing resources are allocated further based on one or both of: historical resource usage by network-based replication agents, and pricing information for usage of computing resources.

[0019] In a fourth implementation form of the system according to the second aspect or any previous implementation form of the second aspect, the prompt comprises a cycle start prompt comprising a start cycle indicator of an upcoming data traffic transmission.

[0020] In a fifth implementation form of the system according to the second aspect or any previous implementation form of the second aspect, the one or more processors execute instructions to: receive a cycle end prompt from a data source, the cycle end prompt including an end cycle indicator of a data cycle that ends data service transmission; and release allocated computing resources based on the end cycle indicator.

[0021] In a sixth embodiment of the system according to the second aspect or any previous embodiment of the second aspect, to allocate computing resources, the one or more processors execute instructions to allocate one or more of central processing unit (CPU) resources, disk space resources, and memory resources to the network-based replication agent based on one or more hints.

[0022] In a seventh embodiment of the system according to the second aspect or any previous embodiment of the second aspect, the one or more processors execute instructions to receive another prompt after receiving data of the first part of the data service transmission, the other prompt indicating the data size of the upcoming second part of the data service transmission.

[0023] In an eighth embodiment of the system according to the second aspect or any previous embodiment of the second aspect, the one or more processors execute instructions to dynamically adjust one or more of the allocated CPU resources, disk space resources, and memory resources based on another indication of the data size of the upcoming second portion of the data service transmission.

[0024] According to a third aspect of the present disclosure, a non-transitory computer-readable medium storing instructions for on-demand data replication in a network architecture is provided, which, when executed by one or more processors, causes the one or more processors to perform operations. The operations include activating an on-demand program to pair a network-based replication agent with a data source. The operations also include allocating computing resources to the network-based replication agent based on a prompt received from the data source, the prompt at least indicating the size of an upcoming data service transmission from the data source. The operations also include activating the network-based replication agent through the on-demand program after receiving data from the data service transmission to use the allocated computing resources to replicate the received data. The operations also include releasing the allocated computing resources after completing the data replication of the received data.

[0025] In a first implementation form of the non-transitory computer-readable medium according to the third aspect, when executed, the instructions further cause the one or more processors to perform operations comprising allocating one or more of central processing unit (CPU) resources, disk space resources, and memory resources to the network-based replication agent based on the one or more hints. After receiving data of a first portion of a data traffic transmission, receiving another hint. The another hint indicates a size of data of an upcoming second portion of the data traffic transmission. Dynamically adjusting the allocated one or more of the CPU resources, disk space resources, and memory resources based on at least one other hint indicating the size of data of the upcoming second portion of the data traffic transmission.

[0026] Any of the preceding examples may be combined with any one or more of the other preceding examples to create new embodiments within the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views.The drawings illustrate generally, by way of example, and not limitation, various embodiments discussed in this document.

[0028] Figure 1 Provides a high-level system overview of a network architecture using conventional replication agents.

[0029] Figure 2 is a high-level system overview of a network architecture using replication agents running on dynamically allocated resources within a network-based service infrastructure, according to some example embodiments.

[0030] Figure 3 is a block diagram illustrating a process flow for establishing an on-demand replication agent, according to some example embodiments.

[0031] Figure 4 is a high-level system overview of a network architecture using replication agents with a resource management engine running within a network-based service infrastructure, according to some example embodiments.

[0032] Figure 5 FIG. 1 is a diagram illustrating methods for determining resource allocation according to some example embodiments. Figure 4 A block diagram of the functionality of the resource management engine.

[0033] Figure 6 is a diagram illustrating the use of Figure 4 Block diagram of an example on-demand data replication cycle of a resource management engine.

[0034] Figure 7 is a block diagram illustrating the process flow between a sender replication agent, a receiver replication agent, and a receiver resource management engine during on-demand data replication, according to some example embodiments.

[0035] Figure 8 is a flow chart of a method suitable for on-demand data replication in a network-based service infrastructure, according to some example embodiments.

[0036] Figure 9 is a block diagram illustrating a representative software architecture that may be used in conjunction with the various device hardware described herein, according to some example embodiments.

[0037] Figure 10 is a block diagram illustrating circuitry of an apparatus for implementing an algorithm and performing a method according to some example embodiments. DETAILED DESCRIPTION

[0038] It should be understood at the outset that although the following provides illustrative implementations of one or more embodiments, Figures 1 to 10The disclosed systems and / or methods described can be implemented using any number of techniques currently known or not yet in existence. The disclosure should in no way be limited to the illustrative embodiments, drawings, and techniques shown below, including the exemplary designs and embodiments shown and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0039] In the following description, reference is made to the accompanying drawings which form a part of the present invention, and in which specific embodiments that can be practiced are shown by way of illustration. These embodiments are described in sufficient detail to enable those skilled in the art to practice the subject matter of the present invention, and it is understood that other embodiments may be utilized and that structural, logical, and electrical changes may be made without departing from the scope of the present disclosure. Therefore, the following description of the example embodiments should not be understood in a restrictive sense, and the scope of the present disclosure is defined by the appended claims.

[0040] As used herein, the term "network-based service infrastructure" includes a plurality of network devices that provide on-demand computing capacity and storage capacity as a service to a group of end recipients (e.g., customers of the service infrastructure) (e.g., via one or more virtual machines or other virtual resources running on the network devices), where the end recipients are communicatively coupled to the network devices within the service infrastructure via a network. Customers of the service infrastructure can use one or more computing devices (or customer devices) to access and manage the services provided by the service infrastructure via the network. The customer devices, network, and network-based service infrastructure may be collectively referred to as the "network architecture."

[0041] A replication technology solution may include a storage array using a software-based replication agent that can run as a virtual machine (VM) or as a container in a network-based service infrastructure and is used to store data in a network storage device when the data is received. The technology disclosed herein for on-demand data replication in a network-based service infrastructure may include an initial phase in which the replication agent is booted by an on-demand program (e.g., a serverless function). The bootstrapping of the replication agent may occur after receiving a pairing request from a data source device (which may be a computing device operated by a client of the network-based service infrastructure). By using an on-demand program, the replication agent can be triggered on demand rather than running continuously (e.g., on a virtual machine) to perform replication, which improves data synchronization / replication efficiency and reduces replication costs associated with data replication services provided by the network-based service infrastructure.

[0042] In a subsequent phase, the data source device may send one or more hints (e.g., information within the transferred data replication packet) to indicate at least the size of the data to be replicated by the replication agent for the upcoming transfer. In some aspects, the network-based service infrastructure may use a resource management engine to allocate and release computing resources for use by the replication agent based on the received one or more hints, historical resource usage data, and / or current pricing of computing resources provided to customers of the network-based service infrastructure. In some aspects, computing resources used by the replication agent may be reallocated periodically, for example, whenever a new hint is received indicating a new upcoming data transfer or the termination of a current data transfer.

[0043] Unlike prior art replication solutions (which use replication agents that are always running), the technology disclosed in this article has the following cost-effectiveness: (a) the replication agent does not incur any cost before replication is established because the sender-receiver relationship is triggered and established on demand; (b) after the relationship is established, the cost of using computing resources for each replication period is evaluated and resources are allocated on demand; and (c) the current scheme monitors costs based on the cost function and the cloud provider's charging model, and then dynamically allocates computing resources on demand taking into account the costs.

[0044] Figure 1 is a high-level system overview of a network architecture 100 using conventional replication agents. Figure 1 , the network architecture 100 may include a plurality of data senders 114 communicatively coupled to a network-based service infrastructure 102 via a network 124. The data senders 114 may include a plurality of computing devices (or senders) 116, ..., 118, each of which is coupled to a corresponding database 120, ..., 122 including data to be replicated. In some aspects, the devices 116, ..., 118 may be devices associated with customers of the network-based service infrastructure 102, associated with customers using services provided by the network-based service infrastructure 102 (e.g., replication services).

[0045] The network-based service infrastructure 102 may include a plurality of computing devices 108, ..., 110 and a network storage device 112. At least one computing device within the infrastructure 102 (e.g., device 108) may include a replication agent 106 coupled to the network storage device 112. In some aspects, each computing device 116, ..., 118 is operable to communicate with the replication agent 106 over a network 124 to perform data replication, such as asynchronous data replication of data received from databases 120, ..., 122. More specifically, each computing device 116, ..., 118 may include a software-based replication agent or a storage array that may be communicatively coupled to the replication agent 106 within the network-based service infrastructure 102 over the network 124.

[0046] exist Figure 1 Any device or database shown in may be implemented in a general-purpose computer that is modified (e.g., configured or programmed) by software to be a special-purpose computer to perform the functions described herein with respect to that machine, database, or device. As used herein, a "database" is a data storage resource that stores data structured as a text file, table, spreadsheet, relational database (e.g., object-relational database, NoSQL database, network or graph database), triple store, hierarchical data store, or any suitable combination thereof. Additionally, data accessed (or stored) via an application programming interface (API) or remote procedure call (RPC) may be considered to be accessed (or stored) in a database. Furthermore, Figure 1 Any two or more devices or databases shown in the may be combined into a single machine, database, or device, and the functionality described herein with respect to any single machine, database, or device may be subdivided among multiple machines, databases, or devices.

[0047] The network 124 can be any network that enables communication between or among machines, databases, and devices (e.g., devices 116, ..., 118, and devices within the network-based service infrastructure 102). Thus, the network 124 can be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. The network 124 can include one or more portions that constitute a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0048] In some aspects, the replica agent 106 can be constantly running, for example, as a virtual machine (VM) 104 or as a software container running within the computing device 108. Figures 2 to 10 The disclosed technology can be used to implement replication agents within a network-based service infrastructure by dynamically allocating resources, thereby allowing replication agents to be used on demand with minimal operating costs (e.g., the operating costs are triggered only when the replication agents are used) and with computing resources dynamically allocated for use by the replication agents.

[0049] Figure 2 is a high-level system overview of a network architecture 200 using replication agents 206 running on dynamically allocated resources 204 within a network-based service infrastructure 202, according to some example embodiments. Figure 2, the network architecture 200 may include a plurality of data senders 214 communicatively coupled to a network-based service infrastructure 202 via a network 224. The data senders 214 may include a plurality of computing devices (or senders) 216, ..., 218, each of which is coupled to a corresponding database 220, ..., 222 that includes data to be replicated.

[0050] The network-based service infrastructure 202 may be similar to Figure 1 , 222. In some aspects, each computing device 216, ..., 218 is operable to communicate with the replication agent 206 via a network 224 to perform data replication, such as asynchronous data replication of data received from the databases 220, ..., 222. In some aspects, each computing device 216, ..., 218 may comprise a software-based replication agent or storage array that may be communicatively coupled to the replication agent 206 within the network-based service infrastructure 202 via the network 224. The network 224 may be similar to Figure 1 Network 124.

[0051] In some aspects, computing device 208 may use dynamically allocated resources 204 to run replication agent 206 to replicate data received from sender 214 to network storage 212. Dynamically allocated resources 204 may include serverless functionality, server-based functionality, or other cost-effective mechanisms provided to clients of network-based service infrastructure 202 that may be executed on-demand after data sender 210 communicates data for replication. Each computing device 216, ..., 218 is operable to communicate with replication agent 206 to perform data replication, such as asynchronous data replication of data from databases 220, ..., 222. More specifically, each computing device 216, ..., 218 may include a software-based replication agent or storage array that may be communicatively coupled to replication agent 206 within network-based service infrastructure 202 after communicating a pairing request 226. The initiation process for establishing replication agent 206 as an on-demand replication agent is described in detail in the accompanying drawings. Figure 3 Shown in more detail in .

[0052] Figure 3 is a block diagram illustrating a process flow 300 for establishing an on-demand replication agent according to some example embodiments. Figure 3Process flow 300 may be initiated when one or more of computing devices 216, ..., 218 communicate a pairing request 226 to pair replication agent 206 with the device that sent the pairing request 226. Upon receiving the pairing request 226, replication agent 206 may be booted using dynamically allocated resources 204, which may include on-demand programs, such as serverless functions. At operation 302, serverless function 204 may be executed to launch replication agent 206. The cost to a customer of network-based service infrastructure 202 for launching replication agent 206 may be minimal and near zero, as the customer only incurs the cost of running the serverless function.

[0053] After the replication agent 206 is started at operation 302, an initial running state 304 of the replication agent 206 is established. In the initial running state 304, the dynamically allocated resources 204, such as serverless functions, are running and waiting, while the replication agent 206 is not running and incurring no costs. The replication agent 206 may transition to the running state 306 only when data traffic for replication is communicated from the data sender 214 that communicated the pairing request 226 to the network-based service infrastructure 202 by the paired computing device. In this regard, the replication agent 206 may run on demand, performing data replication only when there is data traffic. In addition, in conjunction with the replication agent 206, a client of the network-based service infrastructure 202 may select a computing device 208 (or another predetermined / selected computing resource within the network-based service infrastructure 202) to host the dynamically allocated resources 204 for running the replication agent 206 on demand.

[0054] When data is communicated for replication by the replication agent 206, various metadata may be collected from the communicated data at operation 308. For example, the metadata may include the size of the replicated data (e.g., the total number of bytes transferred), the total time the replication agent 206 was used, the type of data communicated for replication, the CPU processing resources used to replicate the data, the storage space used to replicate the data, etc. The collected metadata may be communicated to a resource management agent (e.g., a resource manager) at operation 310. Figure 4 426), for example, to dynamically allocate or reallocate computing resources to run the replica agent 206.

[0055] Figure 4 is a high-level system overview of a network architecture 400 using a replication agent 406 with a resource management engine 426 running within a network-based service infrastructure 402, according to some example embodiments. Figure 4 , the network architecture 400 may be similar to Figure 2, and may include a plurality of data senders 414 communicatively coupled to a network-based service infrastructure 402 via a network 424. The data senders 414 may include a plurality of computing devices (or senders) 416, ..., 418, each of which is coupled to a corresponding database 420, ..., 422 that includes data to be replicated.

[0056] The network-based service infrastructure 402 includes a plurality of computing devices 408, ..., 410, a network storage device 412, and a resource management engine 426. The plurality of computing devices 408, ..., 410 and the network storage device 412 may be similar to Figure 2 , 210 and network storage 212. One or more computing devices (e.g., device 408) of the plurality of computing devices 408, ..., 410 may be configured with dynamically allocated resources (e.g., serverless functions) 404 for executing replication agent 406 on demand. In some aspects, after a data sender (e.g., one of devices 416, ..., 418) establishes a connection with the network-based service infrastructure 402 and replication agent 406 has been booted through serverless functions 404, the data sender (e.g., device 416) may communicate one or more hints, such as hints 430, ..., 432, within a data packet 428 that is communicated to the network-based service infrastructure 402 via the network 424 for replication within the network storage 412.

[0057] As used herein, the term "hint" may refer to information conveyed within data sent to a network-based service infrastructure for replication, wherein the information also describes the data being communicated. For example, a hint may include information indicating the size of an upcoming data transfer (e.g., a cycle start indication, a cycle end indication, a number of bytes to synchronize / replicate, etc.).

[0058] The resource management engine 426 may comprise suitable circuitry, logic, interfaces, and / or code and may be operable to manage computing resources used by the replica agent 406 in connection with data replication. For example, the resource management engine 426 may utilize information from one or more of the hints 430, ..., 432 to proactively adjust resource allocations associated with the replica agent 406.

[0059] Figure 5 FIG. 1 is a diagram illustrating methods for determining resource allocation according to some example embodiments. Figure 4 A block diagram of the functionality of the resource management engine 426. Figure 5, the resource management engine 426 may use the input information 502, 504, and 506 to generate a resource allocation determination 508. In some aspects, the input information 502 may include hints (e.g., hints 430, ..., 432) conveyed via data packets (e.g., data packet 428) originating from one or more of the data transmitters 414. The hints may provide additional information about an upcoming data traffic transmission (e.g., a cycle start indicator identifying a communication cycle including an initial packet of an upcoming data transmission) or information about a current data traffic transmission (e.g., a cycle end indicator identifying a communication cycle including a final packet of a current data transmission).

[0060] In some aspects, information 504 may include historical usage data associated with previously received data that has been replicated by replication agent 406. For example, information 504 may include Figure 3 Metadata collected at operation 308 in or other statistical or historical data associated with previous resource usage related to a particular type of data or a particular amount of data that has been replicated, as well as other statistical or historical data.

[0061] In some aspects, information 506 may include cost information associated with different computing resources available for use by replication agents 406 within the network-based service infrastructure 402. For example, information 506 may indicate the cost of different types of CPU processing power configurations, disk space configurations, memory configurations, etc. The resource management engine 426 is operable to generate a resource allocation determination 508 based on information 502, 504, and / or 506. Resource allocation determination 508 may include a CPU processing power allocation 510, a disk space allocation 512, and a memory allocation 514 that may be used to run the dynamically allocated resources 404 and replication agents 406.

[0062] Figure 6 is a diagram illustrating the use of Figure 4 A block diagram of an example on-demand data replication cycle 600 of the resource management engine 426 is shown. Figure 6, a replication cycle 600 may begin at operation 602 when the resource management engine 426 receives a cycle start hint 612. The resource management engine 426 may use the cycle start hint 612 (and any of the additional information 504 and 506) to generate a resource allocation determination 614 to allocate resources for the replication agent 406 at operation 604. At operation 606, data may be received and replicated by the replication agent 406. At operation 608, the resource management engine 426 may receive a cycle end hint 618. During operations 606 and 608, additional metadata 616 related to the data communicated for replication may also be obtained. Based on the cycle end hint 618, the resource management engine 426 may generate a resource deallocation determination 620 to release computing resources used by the replication agent 406 at operation 610.

[0063] Figure 7 is a block diagram illustrating a process flow 700 between a sender replication agent 702, a receiver replication agent 704, and a receiver resource management engine 706 during on-demand data replication according to some example embodiments. Figure 7 , the sender replication agent 702 may be a replication agent used by one of the data sender devices 416, ..., 418 (e.g., device 416). The receiver replication agent 704 may be the same as the replication agent 406 within the network-based service infrastructure 402, and the resource management engine 706 may be the same as the replication agent 406 within the network-based service infrastructure 402. Figure 4 The resource management engine 426 is the same.

[0064] At operation 708, the sender replica agent 702 may receive a backup notification, which may include a start time for initiating a backup of data available to the sender replica agent 702. At operation 712, a backup notification 710 (which may be based on notification 708) is communicated to the receiver replica agent 704. In some aspects, the backup notification 710 may include a pairing request for initializing the replica agent 704. At operation 714, the receiver replica agent 704 may be directed to perform on-demand replication using dynamically allocated resources, such as serverless functions. At operation 718, a hint 716 may be communicated from the sender replica agent 702 to the receiver replica agent 704. The hint 716 may include, for example, a period start indicator or other information associated with the upcoming data transfer. The hint 716 may be communicated to the resource management engine 706 at operation 720.

[0065] The resource management engine 706 may perform a computing resource assessment at operation 722 using information from the prompt 716, along with historical usage data and computing resource costs. As a result of the resource assessment 722, a resource allocation determination 724 may be generated and communicated to the receiver replication agent 704 at operation 726, so that the receiver replication agent 704 can use the allocated resources during data replication. At operation 730, data 728 may be communicated to the receiver replication agent 704 for replication. At operation 732, upon receipt of the data 728, the receiver replication agent 704 is activated on demand via a serverless function. At operation 734, the receiver replication agent 704 may use network storage (e.g., network storage 412 within the network-based service infrastructure 402) to replicate the data. At operation 738, a period end prompt 736 may be communicated to the receiver replication agent 704, indicating that the transmission of the data 728 is about to terminate. The received hint 736 may also be communicated to the resource management engine 706, which may generate a resource deallocation determination 742 and communicate the deallocation determination 742 to the receiver replica agent 704 at operation 744. At operation 746, the replica agent resources used by the replica agent 704 may be released based on the resource deallocation determination 742.

[0066] Figure 8 800 is a flow chart of a method 800 suitable for on-demand data replication in a network-based service infrastructure according to some example embodiments. The method 800 includes operations 802, 804, 806, and 808. By way of example and not limitation, the method 800 is described as being performed by device 408 using Figure 9 Modules 960 and 962 (or Figure 10Modules 1060 and 1065 of ) are executed. At operation 802, an on-demand program is activated to pair a network-based replication agent with a data source. For example, dynamically allocated resources 404, which may be a serverless function, may be activated such that based on a pairing request (e.g., 226), a replication agent 406 is paired with a data source (e.g., one or more of data senders 414). At operation 804, computing resources for the network-based replication agent are allocated based on at least one hint received from the data source. For example, the resource management engine 426 may generate a resource allocation determination 508 based on the at least one hint (e.g., 430, ..., 432, or 502), which may indicate at least a size of an upcoming data traffic transmission from the data source. At operation 806, upon receiving data from the data traffic transmission, the network-based replication agent is activated via the on-demand program to replicate the received data using the allocated computing resources. For example, whenever data is communicated from a data sender (e.g., device 416), the serverless function 404 may be activated, which may activate a replication agent (e.g., 406) to perform data replication using a network storage device (e.g., 412). At operation 808, after completing the data replication of the received data, the allocated computing resources may be released. For example, Figure 6 As shown in , after receiving the period end prompt 618 at operation 608 , the resource management engine 426 may generate a resource deallocation determination 620 to release computing resources of the replica agent 406 at operation 610 .

[0067] In some aspects, the replication agent 406 can be implemented by always running the dynamically allocated resources 404 as a smaller container or VM, which can incur fixed costs when the system is idle. In this regard, when the replication agent 406 needs more resources, the additional resources can be dynamically allocated to the replication agent container.

[0068] In some aspects, hints (e.g., 416, ..., 418) may be optional, as long as the synchronization period of data replication does not change so that the receiver (e.g., replication agent 406) can predict when the next round of workload will be received. Additional input (e.g., heuristic information) may be used to reduce costs (e.g., scaling back replication activities at certain times when cloud computing resources are expensive).

[0069] In some aspects, the entire replication agent 406 can be run as a serverless function all the time.Depending on the price plan offered by the cloud network provider, this technique can save more money by providing serverless functions at a low cost.

[0070] Figure 9 is a block diagram illustrating a representative software architecture 900 that may be used in conjunction with the various device hardware described herein, according to some example embodiments. Figure 9This is merely a non-limiting example of a software architecture 902, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. The software architecture 902 may be implemented in, for example, Figure 10 1000, which includes, among other things, a processor 1005, a memory 1010, storage devices 1015 and 1020, and I / O components 1025 and 1030. A representative hardware layer 904 is shown, which may represent, for example, Figure 10 The device 1000 is a device 1000. The representative hardware layer 904 includes one or more processing units 906 with associated executable instructions 908. The executable instructions 908 represent executable instructions of the software architecture 902, including Figures 1 to 8 The hardware layer 904 also includes a memory and / or storage module 910 that also has executable instructions 908. The hardware layer 904 may also include other hardware 912, which represents any other hardware of the hardware layer 904, such as other hardware shown as part of the device 1000.

[0071] exist Figure 9 In the example architecture of FIG, software architecture 902 can be conceptualized as a stack of layers, where each layer provides specific functionality. For example, software architecture 902 may include layers such as operating system 914, library 916, framework / middleware 918, application 920, and presentation layer 944. In operation, application 920 and / or other components within a layer may call application programming interface (API) calls 924 through the software stack and receive responses, return values, etc., shown as messages 926 after API calls 924. Figure 9 The layers shown in are representative in nature, and not all software architectures 902 have all layers. For example, some mobile or proprietary operating systems may not provide framework / middleware 918, while other mobile or proprietary operating systems may provide such layers. Other software architectures may include additional or different layers.

[0072] The operating system 914 may manage hardware resources and provide common services. The operating system 914 may include, for example, a kernel 928, services 930, drivers 932, a copy agent 960, and a resource management engine 962. The kernel 928 may act as an abstraction layer between the hardware and other software layers. For example, the kernel 928 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The services 930 may provide other common services for other software layers. The drivers 932 may be responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, the drivers 932 may include a display driver, a camera driver, drivers, flash drives, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), drivers, audio drivers, power management drivers, and more.

[0073] In some aspects, the replica agent 960 may be combined with Figure 4 The resource management engine 962 may be used in conjunction with the replica agent 406 discussed herein. Figures 4 to 6 The resource management engine 426 discussed is the same.

[0074] The library 916 may provide a common infrastructure that can be utilized by the application 920 and / or other components and / or layers. The functionality typically provided by the library 916 allows other software modules to perform tasks in an easier manner than directly interfacing with the underlying operating system 914 functions (e.g., kernel 928, services 930, drivers 932, and / or modules 960-962). The library 916 may include a system library 934 (e.g., a C standard library) that can provide functions such as memory allocation functions, string manipulation functions, and mathematical functions. In addition, the library 916 may include an API library 936, such as a media library (e.g., a library that supports the presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG), a graphics library (e.g., an OpenGL framework that can be used to render 2D and 3D graphics content on a display), a database (e.g., SQLite that can provide various relational database functions), a web library (e.g., WebKit that can provide web browsing functions), etc. The library 916 may also include a wide variety of other libraries 938 to provide numerous other APIs to the applications 920 and other software components / modules.

[0075] The framework / middleware 918 (sometimes also referred to as middleware) can provide a high-level common infrastructure that can be utilized by the applications 920 and / or other software components / modules. For example, the framework / middleware 918 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. The framework / middleware 918 can provide a wide range of other APIs that can be utilized by the applications 920 and / or other software components / modules, some of which can be specific to a particular operating system 914 or platform.

[0076] Applications 920 include built-in applications 940 and / or third-party applications 942. Examples of representative built-in applications 940 may include, but are not limited to, a contacts application, a browser application, a book reader application, a location application, a media application, a messaging application, and / or a game application. Third-party applications 942 may include any of the built-in applications 940 as well as a variety of other applications. In a specific example, a third-party application 942 (e.g., an application created by an entity other than the vendor of a particular platform using Android TMor iOS TM Software Development Kit (SDK) applications can be developed on iOS TM 、Android TM 、 Mobile software running on a mobile operating system such as a iPhone or other mobile operating system. In this example, third-party applications 942 can call API calls 924 provided by a mobile operating system such as operating system 914 to facilitate the functions described herein.

[0077] Applications 920 may utilize built-in operating system functionality (e.g., kernel 928, services 930, drivers 932, and / or modules 960-962), libraries (e.g., system libraries 934, API libraries 936, and other libraries 938), and framework / middleware 918 to create a user interface to interact with users of the system. Alternatively or in addition, in some systems, interaction with the user may occur in a presentation layer, such as presentation layer 944. In these systems, the application / module "logic" may be separated from the aspects of the application / module that interact with the user.

[0078] Some software architectures utilize virtual machines. Figure 9 In the example of , this is illustrated by virtual machine 948. A virtual machine creates a software environment in which applications / modules can run as if they were on a hardware machine (e.g., Figure 10 The virtual machine 948 is executed by the host operating system ( Figure 9 The virtual machine 948 is hosted by an operating system 914 in a virtual machine and typically, but not always, has a virtual machine monitor 946 that manages the operation of the virtual machine 948 and interfaces with the host operating system (i.e., the operating system 914). The software architecture 902 executes, for example, an operating system 950, libraries 952, framework / middleware 954, applications 956, and / or a presentation layer 958 within the virtual machine 948. These software architecture layers executing within the virtual machine 948 may be the same as the corresponding layers previously described, or may be different.

[0079] Figure 10 is a block diagram illustrating circuitry of an apparatus for implementing an algorithm and performing a method according to some example embodiments. Not all components need be used in various embodiments. For example, a client, a server, and a cloud-based network device may each use a different set of components or, in the case of a server, a larger storage device, for example.

[0080] One example computing device in the form of a computer 1000 (also referred to as computing device 1000, computer system 1000, or computer 1000) may include a processor 1005, memory 1010, removable storage 1015, non-removable storage 1020, an input interface 1025, an output interface 1030, and a communication interface 1035, all connected by a bus 1040. Although the example computing device is illustrated and described as a computer 1000, the computing device may take different forms in different embodiments.

[0081] Memory 1010 may include volatile memory 1045 and nonvolatile memory 1050, and may store programs 1055. Computer 1000 may include, or have access to, a variety of computer-readable media, such as volatile memory 1045, nonvolatile memory 1050, removable storage 1015, and non-removable storage 1020. Computer storage includes random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) and electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

[0082] Computer-readable instructions stored on a computer-readable medium (e.g., program 1055 stored in memory 1010) can be executed by processor 1005 of computer 1000. Hard drives, CD-ROMs, and RAM are some examples of articles of manufacture that include non-transitory computer-readable media, such as storage devices. The terms "computer-readable medium" and "storage device" do not include carrier waves, as carrier waves are considered too transient. "Computer-readable non-transitory medium" includes all types of computer-readable media, including magnetic storage media, optical storage media, flash media, and solid-state storage media. It should be understood that the software can be installed in a computer and sold with the computer. Alternatively, the software can be obtained and loaded into the computer, including obtaining the software through physical media or a distribution system, including, for example, obtaining the software from a server owned by the software creator or from a server not owned by the software creator but used by the software creator. For example, the software can be stored on a server for distribution over the Internet. As used herein, the terms "computer-readable medium" and "machine-readable medium" are interchangeable.

[0083] The program 1055 can utilize the modules discussed herein, such as the replication agent 1060 and the resource management engine 1065, to utilize the user preference structure. The replication agent 1060 and the resource management engine 1065 can each be combined with at least Figure 4 The replica agent 406 and resource management engine 426 discussed are the same.

[0084] Any one or more modules described herein may be implemented using hardware (e.g., a processor of a machine, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any suitable combination thereof). In addition, any two or more of these modules may be combined into a single module, and the functionality described herein for a single module may be subdivided among multiple modules. Furthermore, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0085] In some aspects, one or more of modules 1060 through 1065 may be integrated into a single module, thereby performing the corresponding functionality of the integrated module.

[0086] Although several embodiments have been described in detail above, other modifications are possible. For example, the logic flows depicted in the figures do not require the particular order shown or sequential order to achieve desired results. Other steps may be provided or steps may be eliminated from the described flows, and other components may be added to or removed from the described systems. Other embodiments may be within the scope of the following claims.

[0087] It should also be understood that software can be installed in and sold with one or more computing devices consistent with the present disclosure, the software including one or more computer-executable instructions that facilitate the processes and operations described above with reference to any or all of the steps of the present disclosure. Alternatively, the software can be obtained and loaded into one or more computing devices, including obtaining the software via physical media or a distribution system, including, for example, from a server owned by the creator of the software or from a server not owned by the creator of the software but used by the creator of the software. For example, the software can be stored on a server for distribution over the Internet.

[0088] In addition, it will be appreciated by those skilled in the art that the present disclosure is not limited in its application to the arrangements of the construction details and components set forth in the specification or shown in the accompanying drawings. The embodiments herein allow for other embodiments and can be practiced or implemented in various ways. In addition, it will be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The use of "including" or "having" and their variations herein is intended to cover the items listed thereafter and their equivalents as well as additional items. Unless otherwise limited, the terms "connect", "couple" and "install" and their variations herein are used in a broad sense and cover direct and indirect connections, couplings and installations. In addition, the terms "connect" and "couple" and their variations are not limited to physical or mechanical connections or couplings. In addition, terms such as upper, lower, bottom and top are relative and are used to assist in explanation and are not restrictive.

[0089] The components of the illustrative apparatus, systems, and methods used in accordance with the illustrated embodiments may be implemented at least partially in digital electronic circuitry, analog electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. These components may be implemented as a computer program product, such as a computer program, program code, or computer instructions, tangibly embodied in an information carrier or machine-readable storage device, for execution by a data processing device, such as a programmable processor, a computer, or multiple computers, or to control the operation of the data processing device.

[0090] The computer program can be written in any form of programming language, including compiled or interpreted languages, and can be deployed in any form, including in the form of a stand-alone program or in the form of a module, component, subroutine, or other unit suitable for a computing environment. The computer program can be deployed to execute on one computer, or on multiple computers distributed in one place or across multiple places and interconnected by a communication network. In addition, programmers proficient in the fields related to the technology described herein can easily understand the functional programs, codes, and code segments that implement the technology described herein as being within the scope of the claims. The method steps associated with the illustrative embodiments can be performed by one or more programmable processors that execute computer programs, codes, or instructions to perform functions (e.g., by operating on input data and / or generating output). The method steps can also be performed by dedicated logic circuits, and the apparatus for performing the method can be implemented as dedicated logic circuits, for example, field programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs).

[0091] The various illustrative logical blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed by a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device designed to perform the functions described herein, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0092] Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Generally speaking, the processor will receive instructions and data from read-only memory or random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Generally speaking, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to receive data from or transfer data to the one or more mass storage devices, or to perform both operations. Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, such as semiconductor memory devices, such as electrically programmable read-only memory or ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory devices, and data storage disks (e.g., magnetic disks, internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks). The processor and memory can be supplemented with or incorporated into dedicated logic circuits.

[0093] Those skilled in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0094] As used herein, a "machine-readable medium" (or "computer-readable medium") means a device capable of storing instructions and data, either temporarily or permanently, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database or associated caches and servers) that can store processor instructions. The term "machine-readable medium" should also be understood to include any medium or combination of multiple media that can store instructions for execution by one or more processors 1005, such that when the instructions are executed by one or more processors 1005, the one or more processors 1005 perform any one or more of the methods described herein. Thus, a "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" as used herein does not itself include signals.

[0095] In addition, the techniques, systems, subsystems, and methods described and illustrated as independent or separate in various embodiments may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled or directly coupled or communicating with each other may also be indirectly coupled or communicating electrically, mechanically, or otherwise through an interface, device, or intermediate component. Other variations, substitutions, and modifications will be apparent to those skilled in the art and do not depart from the spirit and scope of the present disclosure.

[0096] Although the present disclosure has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations may be made thereto without departing from the scope of the present disclosure. For example, other components may be added to or removed from the described system. Accordingly, the specification and drawings should be considered merely as illustrative of the present disclosure as defined by the appended claims, and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of the present disclosure. Other aspects may be within the scope of the appended claims.

Claims

1. A computer-implemented method for on-demand data replication in a network architecture, characterized in that The method comprises: Activate an on-demand program to pair a network-based replication agent with a data source; allocating computing resources of the network fabric to replica agents based on a hint received from the data source, the hint indicating at least a size of an upcoming data traffic transmission from the data source; Upon receiving data transmitted by the data service, activating the network-based replication agent through the on-demand program to replicate the received data using the allocated computing resources; and After completing the data copying of the received data, releasing the allocated computing resources.

2. The computer-implemented method of claim 1, wherein: The on-demand program is a server-less function for activating the network-based replication agent on demand after receiving the data transmitted by the data service.

3. The computer-implemented method of claim 1, wherein: The prompt indicates at least one of the following: A data cycle start indicator, a data cycle end indicator, and the number of bytes in the upcoming data service transmission.

4. The computer-implemented method of claim 1, wherein: The allocating computing resources of the network architecture is further based on one or both of the following: historical resources used by the network-based replication agent, and price information of the used computing resources.

5. The computer-implemented method of claim 1, wherein: The prompts include a cycle start prompt, The cycle start prompt includes a start cycle indicator of the upcoming data service transmission.

6. The computer-implemented method of claim 5, wherein: The method further comprises: receiving a cycle end prompt from the data source, the cycle end prompt comprising an end cycle indicator for ending a data cycle of the data service transmission; and Based on the end cycle indicator, the allocated computing resources are released.

7. The computer-implemented method of claim 1, wherein: The allocating computing resources further comprises: Based on the one or more hints, one or more of central processing unit (CPU) resources, disk space resources, and memory resources available within the network architecture are allocated to the network-based replication agent.

8. The computer-implemented method of claim 7, wherein: The method further comprises: Another prompt is received after receiving the data of the first portion of the data service transmission, the other prompt indicating the size of the second portion of the data of the data service transmission to be transmitted.

9. The computer-implemented method of claim 8, wherein: The method further comprises: Based on the other prompt indicating the data size of the second portion of the data service transmission to be transmitted, one or more of the allocated CPU resources, disk space resources, and memory resources are dynamically adjusted.

10. A system, characterized in that: The system comprises: a memory for storing instructions; and One or more processors in communication with the memory, wherein the one or more processors execute the instructions to: Activate an on-demand program to pair a network-based replication agent with a data source; allocating computing resources to the network-based replication agent based on a hint received from the data source, the hint indicating at least a size of an upcoming data traffic transmission from the data source; Upon receiving data transmitted by the data service, activating the network-based replication agent through the on-demand program to replicate the received data using the allocated computing resources; and After completing the data copying of the received data, releasing the allocated computing resources.

11. The system according to claim 10, characterized in that The on-demand program is a server-less function for activating the network-based replication agent on demand after receiving the data transmitted by the data service.

12. The system according to claim 10, characterized in that The prompt indicates at least one of: a data period start indicator, a data period end indicator, and a number of bytes within the upcoming data service transmission.

13. The system according to claim 10, characterized in that The allocated computing resources are further based on one or both of the following: historical resources used by the network-based replication agent, and price information of the used computing resources.

14. The system according to claim 10, characterized in that The prompt includes a cycle start prompt, and the cycle start prompt includes a start cycle indicator of the upcoming data service transmission.

15. The system according to claim 14, characterized in that The one or more processors execute the instructions to perform the following operations: receiving a cycle end prompt from the data source, the cycle end prompt comprising an end cycle indicator for ending a data cycle of the data service transmission; as well as Based on the end cycle indicator, the allocated computing resources are released.

16. The system according to claim 10, characterized in that To allocate the computing resources, the one or more processors execute the instructions to perform the following operations: Based on the one or more hints, one or more of central processing unit (CPU) resources, disk space resources, and memory resources are allocated to the network-based replication agent.

17. The system according to claim 16, characterized in that The one or more processors execute the instructions to perform the following operations: Another prompt is received after receiving the data of the first portion of the data service transmission, the other prompt indicating the size of the second portion of the data of the data service transmission to be transmitted.

18. The system according to claim 17, characterized in that The one or more processors execute the instructions to perform the following operations: Based on the other prompt indicating the size of the second portion of the data service transmission to be transmitted, one or more of the allocated CPU resources, disk space resources, and memory resources are dynamically adjusted.

19. A non-transitory computer-readable medium storing instructions for on-demand data replication in a network architecture, characterized in that: When one or more processors execute the instructions, the one or more processors perform operations including: Activate an on-demand program to pair a network-based replication agent with a data source; allocating computing resources to the network-based replication agent based on a hint received from the data source, the hint indicating at least a size of an upcoming data traffic transmission from the data source; Upon receiving data transmitted by the data service, activating the network-based replication agent through the on-demand program to replicate the received data using the allocated computing resources; as well as After completing the data copying of the received data, releasing the allocated computing resources.

20. The non-transitory computer readable medium of claim 19, wherein: When executed, the instructions further cause the one or more processors to perform operations including: allocating one or more of central processing unit (CPU) resources, disk space resources, and memory resources to the network-based replication agent based on the one or more prompts; receiving another prompt after receiving data of the first portion of the data traffic transmission, the another prompt indicating a data size of an upcoming second portion of the data traffic transmission; as well as Based on the at least one other hint indicating a data size of an upcoming second portion of the data traffic transmission, one or more of the allocated CPU resources, disk space resources, and memory resources are dynamically adjusted.

Images