Method, apparatus and computer program product for concurrent data transmission

Abstract

The invention relates to a method, a device and a computer program product for concurrent data transmission. The method includes obtaining a processor utilization rate and a bandwidth utilization rate of a data link associated with transmitting one or more historical data packets within a target time. The method also includes determining a time consumption for transmitting the target data packet, a splitting processing resource associated with the splitting operation, and a packing processing resource associated with the packing operation. The method further includes determining a splitting number for the target data packet for concurrent transmission of the target data packet based on the processor utilization rate, the bandwidth utilization rate, the time consumption, the splitting processing resources, and the packing processing resources. In this way, the data transmission speed can be increased, the time consumed for transmitting the target data packet can be shortened, and meanwhile the problems that due to concurrent transmission, processing resources are consumed too much, and a processor operates in an overload mode can be solved.

Description

Technical Field

[0001] The present disclosure relates to the field of data storage, and more particularly, to methods, devices, and computer program products for concurrent data transmission. Background Art

[0002] Recovery Point Objective (RPO) refers to the degree of data loss that an organization or system can tolerate in the event of a failure. RPO is usually measured in time units, indicating the time of the last valid data backup before the failure. A lower RPO means that data is replicated and backed up at a higher frequency, and when a failure occurs, the organization or system will suffer data loss in a shorter period of time. A higher RPO means that the organization or system can tolerate data loss for a longer period of time.

[0003] Replication snapshot is a method of backing up and restoring data in a computer system. Replication snapshot is to take a snapshot of data at a certain point in time and then copy the snapshot to another location or storage system to provide data redundancy and fault recovery capabilities. Summary of the invention

[0004] The embodiment of the present disclosure proposes a method, device and computer program product for concurrent data transmission. In the embodiment of the present disclosure, statistical data such as processor utilization and bandwidth utilization of data links associated with transmission history data packets in the past target time can be obtained. The time consumption of transmitting the target data packet can also be calculated, and the processing resources required to split the data packet in the source storage system and the processing resources required to pack multiple data packets in the destination storage system can be determined. Then, the optimal number of splits can be calculated based on these data. In this way, when transmitting the target data packet, the source storage system can split the target data packet into multiple data packets based on the calculated number of splits and transmit them to the destination storage system concurrently, and then pack the multiple data packets into the target data packet at the destination storage system. In this way, the target data packet can be transmitted concurrently, so that the data transmission speed can be improved and the time consumed for transmitting the target data packet can be reduced. At the same time, the problems such as excessive consumption of processing resources and processor overload caused by concurrent transmission can be reduced, so as to ensure that the processor performance of the source storage system and the destination storage system is acceptable while improving the data transmission speed.

[0005] In a first aspect of an embodiment of the present disclosure, a method for concurrent data transmission is provided. The method includes obtaining a processor utilization and a bandwidth utilization of a data link associated with transmitting one or more historical data packets within a target time. The method also includes determining a time consumption for transmitting a target data packet, a splitting processing resource associated with a splitting operation, and a packing processing resource associated with a packing operation, and the method also includes determining a number of splits for a target data packet for concurrent transmission of the target data packet based on the processor utilization, the bandwidth utilization, the time consumption, the splitting processing resource, and the packing processing resource.

[0006] In a second aspect of an embodiment of the present disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement a method for concurrent data transmission, the method comprising obtaining the processor utilization associated with the transmission of one or more historical data packets and the bandwidth utilization of the data link within the target time. The method also includes determining the time consumption of transmitting the target data packet, the splitting processing resources associated with the splitting operation, and the packing processing resources associated with the packing operation, and the method also includes determining the number of splits for the target data packet for concurrent transmission of the target data packet based on the processor utilization, bandwidth utilization, time consumption, splitting processing resources, and packing processing resources.

[0007] In a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, a method for concurrent data transmission is implemented, the method comprising obtaining a processor utilization and a bandwidth utilization of a data link associated with transmitting one or more historical data packets within a target time. The method also includes determining the time consumption of transmitting a target data packet, the splitting processing resources associated with a splitting operation, and the packing processing resources associated with a packing operation, and the method also includes determining the number of splits for a target data packet for concurrent transmission of the target data packet based on the processor utilization, the bandwidth utilization, the time consumption, the splitting processing resources, and the packing processing resources.

[0008] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram illustrating an example environment in which various embodiments of the present disclosure may be implemented;

[0011] Figure 2 A schematic diagram showing a process of replicating incremental data based on RPO asynchronous transmission according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A flowchart of a method for concurrent data transmission according to some embodiments of the present disclosure is shown;

[0013] Figures 4A to 4B A schematic diagram showing a process of splitting a replication snapshot at a source storage system according to some embodiments of the present disclosure, and a schematic diagram showing a process of packing the split multiple data packets at a destination storage system according to some embodiments of the present disclosure;

[0014] Figure 5 A flowchart showing a process of determining whether to concurrently transmit a replication snapshot and determining the number of splits according to some embodiments of the present disclosure is shown;

[0015] Figure 6 A schematic diagram showing a process of determining a performance benefit score according to some embodiments of the present disclosure;

[0016] Figure 7 A schematic diagram showing an example of determining a maximum performance gain score according to some embodiments of the present disclosure; and

[0017] Figure 8 A block diagram of a device in which various embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] Data replication is the process of creating a copy or backup of data from a source storage system in a destination storage system, and data replication includes synchronous replication and asynchronous replication. Synchronous replication can create data copies in real time, but creating data copies in real time will have an adverse effect on host writes, and real-time creation of data copies can only be achieved in scenarios with short transmission distances. Asynchronous replication is designed for data replication in scenarios with long transmission distances, and asynchronous replication can perform periodic replication in the background, which does not negatively affect the performance of the host. However, since the replication process is asynchronous rather than real-time, there may be a certain degree of data loss. The recovery point objective (RPO) refers to the maximum acceptable amount of data loss in the event of a catastrophic event or data loss, and the RPO is expressed in terms of time, for example, the RPO can be 30 minutes, 1 hour, 2 hours, etc. If the RPO of a system is 1 hour, it means that the time interval between the time when the disaster occurred and the last time the system replicated data does not exceed 1 hour.

[0021] In order to reduce the amount of data loss, users (e.g., system administrators) can select a smaller RPO value (e.g., 5 minutes) and hope that only a small amount of data will be lost once a failure occurs. However, when the I / O task is heavy and there are a large number of storage objects involved in asynchronous replication sessions on the source storage system, it is possible that when the second RPO is about to be reached, the incremental data (i.e., replication snapshot) to be replicated in the first RPO has not been completed from the source storage system to the destination storage system. In this case, if a disaster occurs during the second RPO, the data during the two RPO periods may be lost. Therefore, it is necessary to improve the data transmission performance to ensure that all incremental data is transmitted within the time window required by the RPO.

[0022] To this end, an embodiment of the present disclosure proposes a scheme for concurrently transmitting data. In an embodiment of the present disclosure, statistical data such as processor utilization and bandwidth utilization of data links used to transmit historical replication snapshots (also referred to as historical data packets) within the past RPO (also referred to as target time) can be obtained. The time consumption for transmitting the target data packet can also be calculated, and the processing resources required to split the data packet in the source storage system can be determined, as well as the processing resources required to package multiple data packets in the destination storage system. Then, the optimal number of splits can be calculated based on these data. In this way, when transmitting the target replication snapshot (also referred to as the target data packet), the source storage system can split the target replication snapshot into multiple data packets based on the calculated number of splits and transmit them concurrently to the destination storage system, and then package the multiple data packets into the target replication snapshot at the destination storage system.

[0023] In this way, the target replication snapshots can be transmitted concurrently, thereby increasing the data transmission speed and the bandwidth utilization of the data link, reducing the number of cases where the replication snapshot transmission is not completed within the RPO time, and reducing the problems of excessive processing resource consumption and processor overload caused by concurrent transmission. This ensures that the processor performance of the source storage system and the destination storage system is acceptable while increasing the data transmission speed.

[0024] Figure 1 1 is a schematic diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented. Figure 1 As shown, the environment 100 includes a source storage system 102 and a destination storage system 112. The source storage system 102 includes storage objects 104-1, 104-2, ..., 104-N (collectively referred to as storage objects 104), which are a way of organizing and managing data in the storage system, and can be data blocks of any size, and storage objects can be different types of data such as files, database records, images, videos, etc. In the environment 100, the storage objects 104 are replicated or backed up through asynchronous replication sessions. Figure 1 As shown, each storage object 104 has a replication snapshot associated therewith, namely, replication snapshots 106 - 1 , 106 - 2 , . . . , 106 -N (collectively, replication snapshots 106 ).

[0025] like Figure 1 As shown, in the environment 100, a data link 120 can be used to transmit data between a source storage system 102 and a destination storage system 112. For example, the data link 120 can be used to transmit the replication snapshot 106 from the source storage system 102 to the destination storage system 112, so that replication snapshots 116-1, 116-2, ..., 116-N (collectively referred to as replication snapshots 116) can be created in the destination storage system as a result of the transmission. The data link 120 has a theoretical bandwidth, which refers to the maximum transmission speed supported by the data link 120, which indicates the maximum amount of data that can be transmitted by the data link 120 under ideal conditions, and is usually measured in bits per second (bps). For example, if the theoretical bandwidth of the data link 120 is 100Mbps, it means that under ideal conditions, it can transmit one hundred megabits of data per second. However, in reality, due to other factors that may exist in the network (for example, signal interference, bandwidth sharing, transmission delay, etc.), the actual transmission speed of 100Mbps may not be achieved.

[0026] like Figure 1As shown, in the environment 100, the source storage system 102 includes a processor 108, and the destination storage system 112 includes a processor 118. The processors 108 and 118 can perform operations related to storage management, such as data access, data compression and decompression, data replication, and some computing tasks. The capabilities and performance of the processors 108 and 118 can directly affect indicators such as the throughput and response time of the source storage system 102 and the destination storage system 112.

[0027] Figure 2 FIG. 2 is a schematic diagram showing a process 200 of replicating incremental data based on RPO asynchronous transmission according to some embodiments of the present disclosure. Figure 2 As shown, process 200 first transfers a source storage system (eg, Figure 1 The data to be replicated (ie, the replication snapshot 212) in the source storage system 102 in the destination storage system (eg, Figure 1 The first replication is also referred to as the initial replication in the destination storage system 112 in the process 200. After the initial replication is completed, the process 200 will replicate the incremental data according to the RPO-based plan, and these incremental data will monitor the changes since the last replication to achieve faster replication speed and lower storage requirements. When a disaster occurs in the system, the administrator can choose to restore the system to the time point of the most recent incremental replication, thereby minimizing data loss.

[0028] like Figure 2 As shown, the process 200 starts at time point 2 to transfer replication incremental data from the source storage system 102 to the destination storage system 112 at each RPO time interval. For example, at time point 2, the process 200 transfers a replication snapshot 214 from the source storage system 102 to the destination storage system 112. After the RPO time has passed, at time point 3, the process 200 transfers a replication snapshot 216 from the source storage system 102 to the destination storage system 112. After the RPO time has passed again, at time point 4, the process 200 transfers a replication snapshot 218 from the source storage system 102 to the destination storage system 112. It should be noted that the replication snapshots 214, 216, and 218 may include one or more replication snapshots, and the number of replication snapshots depends on the newly added or changed data that has occurred since the last replication.

[0029] Figure 3 FIG. 3 is a flow chart showing a method 300 for concurrently transmitting data according to some embodiments of the present disclosure. Figure 3 As shown, at block 302, method 300 obtains processor utilization and bandwidth utilization of a data link associated with transmitting one or more historical data packets within a target time. Figure 2In the process 200 shown, the method 300 can obtain the processor utilization associated with the transport replication snapshot 214 within the RPO time between the time point 2 and the time point 3 at the time point 3 (for example, Figure 1 processor utilization of the processor 108 of the source memory 102 and the processor 118 of the destination storage system 112) and data links (e.g., Figure 1 Bandwidth utilization of data link 120 in.

[0030] At block 304, method 300 determines the time consumption for transmitting the target data packet, the splitting processing resources associated with the splitting operation, and the packing processing resources associated with the packing operation. Figure 1 In the environment 100 shown, the time consumption of transmitting the copy snapshot 106 can be calculated based on the data size of the copy snapshot 106 and the average transmission speed of the data link 120. In addition, the processor 108 of the source storage system 102 can perform a split operation to split a data packet into multiple data packets. In this document, the operation of splitting a data packet into two data packets is referred to as a split operation. The method 300 can determine the processor consumption required for the processor 108 of the source storage system 102 to perform a split operation. In addition, the processor 118 of the destination storage system 112 can perform a packing operation to pack multiple data packets into one data packet. In this document, the operation of packing two data packets into one data packet is referred to as a packing operation. The method 300 can determine the processor consumption required for the processor 118 of the destination storage system 112 to perform a packing operation.

[0031] At block 306, the method 300 determines the number of splits for the target data packet for concurrent transmission of the target data packet based on the processor utilization, time consumption, bandwidth utilization, split processing resources, and packet processing resources. Figure 2 In the process 200 shown, the method 300 can determine the number of splits for the replication snapshot 216 based on the processor utilization, time consumption, bandwidth utilization, processor resources required for the processor 108 of the source storage system 102 to perform a split operation, and processor resources required for the processor 118 of the destination storage system 112 to perform a packing operation during the RPO time between time point 2 and time point 3 for transmitting the replication snapshot 214. After determining the number of splits, the source storage system 102 can split the replication snapshot 216 into the number of split sub-data packets before transmitting the replication snapshot 216, and then transmit these data packets concurrently. Then, the destination storage system 112 can pack the number of split sub-data packets into a complete replication snapshot for storage based on the identifiers in the sub-data packets.

[0032] In this way, the method 300 can concurrently transmit the replication snapshot 216, thereby increasing the data transmission speed and improving the bandwidth utilization of the data link 120, thereby reducing the Figure 2 The possibility that the transmission of the replication snapshot 216 cannot be completed within the RPO time between the time point 3 and the time point 4 shown can also reduce the problems of excessive resource consumption and processor overload of the source storage system 102 and the destination storage system 112 caused by concurrent transmission, thereby ensuring that the performance of the processors 108 and 118 of the source storage system 102 and the destination storage system 112 is acceptable while improving the data transmission speed.

[0033] In some embodiments, in order to speed up the transmission speed, the target data packet to be transmitted can be split into multiple sub-data packets at the source storage system, and then the multiple sub-data packets can be concurrently transmitted to the destination storage system using the data link. After receiving the multiple sub-data packets, the destination storage system can package the multiple sub-data packets to obtain a restored target data packet, thereby completing the process of copying the target data packet from the source storage system to the destination storage system.

[0034] Figures 4A to 4B FIG. 4 is a schematic diagram of a process 400 of splitting a replication snapshot at a source storage system according to some embodiments of the present disclosure, and a schematic diagram of a process 420 of packaging the split multiple data packets at a destination storage system according to some embodiments of the present disclosure. Figure 4A As shown, source storage system 402 (eg, Figure 1 The source storage system 102 in the source storage system 402 stores the replication snapshot 404. The processor 408 of the source storage system 402 can split the replication snapshot 404 into multiple data packets, that is, data packets 406-1, 406-2, ..., 406-N (collectively referred to as data packets 406), and then concurrently transmit the multiple data packets 406 to the destination storage system 412 using the data link 410. It should be noted that when the replication snapshot is split into multiple data packets 406 using the processor 408, corresponding processor resources for splitting the data packets need to be consumed, and the consumed processor resources depend on the number of times the data is split. The more the number of splits, the more the number of split data packets, the higher the efficiency of concurrent transmission, and the more processor resources are consumed accordingly.

[0035] like Figure 4BAs shown, after the destination storage system 412 receives the multiple data packets 406 transmitted from the source storage system 402 through the data link 410, the processor 418 of the destination storage system 412 may package the multiple data packets 406 to obtain the restored replication snapshot 414, thereby completing the process of replicating the replication snapshot 404 from the source storage system 402 to the destination storage system 412. It should be noted that when the processor 418 is used to package the multiple data packets 406, corresponding processor resources for packaging the data packets need to be consumed, and the consumed processor resources depend on the number of sub-data packets after the replication snapshot 404 is split into sub-data packets at the source storage system 402. The more the number of split sub-data packets is, the more times the packaging is required, and the more processor resources are consumed accordingly.

[0036] In this way, the replication snapshot 404 can be split into multiple data packets 406 at the source storage system 402, and then the multiple data packets 406 can be transmitted in parallel through the data link 410, thereby improving the bandwidth utilization of the data link 410, accelerating the speed of copying the replication snapshot 404 from the source storage system 402 to the destination storage system 412, reducing the transmission time, and increasing the possibility of completing the replication incremental data transmission within the RPO time.

[0037] Although the more data packets are split, the higher the efficiency of parallel transmission and the faster the transmission speed, as mentioned above, the more split times are, the more processor resources are consumed for splitting operations and for packaging operations. When the number of split data packets is too large, the processor resources will become a bottleneck and even have an adverse effect on the processor's execution of other tasks. Therefore, in order to complete the transmission of the replicated incremental data within the RPO time and limit the consumed processor resources to an acceptable range, it is necessary to determine the optimal number of splits for concurrent transmission of replicated incremental data.

[0038] In some embodiments, before determining the optimal number of splits, it may be determined whether to concurrently transmit the target data packet (e.g., the target replication snapshot). In some embodiments, the average transmission speed of the data link may be determined, and the ratio of the average transmission speed to the theoretical bandwidth of the data link may be determined. The ratio may then be compared with a predetermined threshold, and in response to the ratio being greater than or equal to the predetermined threshold, the target data packet may be directly transmitted without being split.

[0039] In some embodiments, before determining the optimal number of splits, the average transmission speed of the data link can be determined, and the data size of the target data packet can be determined. Then, based on the average transmission speed and the data size of the target data packet, the transmission time for transmitting the target data packet without splitting the target data packet can be determined. In response to the transmission time being less than or equal to the target time, the target data packet is directly transmitted without splitting.

[0040] Figure 5 FIG. 5 is a flowchart of a process 500 for determining whether to concurrently transmit a replication snapshot and determining the number of splits according to some embodiments of the present disclosure. Figure 5 As shown, at block 502, process 500 collects information. For example, the collected information may include data links (e.g., Figure 1 The actual average transmission speed and theoretical bandwidth of the data link 120 in the replication session may also include the RPO value of the replication session and the transmission target replication snapshot (for example, Figure 2 In some embodiments, the processor utilization when transmitting the historical replication snapshots within a certain RPO time in the past and the average transmission speed and bandwidth utilization of the data link can be determined. In some embodiments, the processor utilization can be determined by determining the average value of multiple processor utilizations for transmitting multiple historical replication snapshots within multiple previous RPO times, and the average transmission speed and bandwidth utilization of the data link can be determined by determining the average value of multiple bandwidth utilizations for transmitting multiple historical replication snapshots.

[0041] In block 504, the process 500 determines whether the transmission speed of the data link can be improved based on the average transmission speed and theoretical bandwidth of the data link. If the bandwidth of the data link has been fully utilized, the transmission speed cannot be improved even if the replication snapshot is split for concurrent transmission. In some embodiments, the ratio of the average transmission speed of the data link to the theoretical bandwidth can be determined. If the ratio is greater than or equal to a predetermined threshold, it means that the bandwidth of the data link has been fully utilized and there is no room for improvement, and then the process 500 proceeds to block 508. In block 508, the process 500 directly transmits the replication snapshot from the source storage system to the destination storage system without splitting it.

[0042] Returning to box 504, if the ratio of the average transmission speed of the data link to the theoretical bandwidth is less than a predetermined threshold, it means that the transmission speed still has room for improvement, and process 500 proceeds to box 506. In box 506, process 500 determines whether the target replication snapshot needs to be split based on the RPO time and the transmission time required to transmit the target replication snapshot. In some embodiments, the transmission time for transmitting the replication snapshot can be determined based on the average transmission speed of the data link and the total data size of the replication snapshot to be transmitted. If the transmission time is less than or equal to the RPO time, it means that the replication snapshot can be transmitted within the RPO time, and process 500 proceeds to box 508. If the transmission time is greater than the RPO time, it means that the transmission of the replication snapshot cannot be completed within the RPO time, and the replication snapshot needs to be split to transmit the split multiple data packets concurrently, so as to improve the transmission speed so that the replication snapshot can be transmitted within the RPO time.

[0043] In box 510, process 500 calculates the performance benefit score when the replication snapshot is not split. The performance benefit score indicates the overall performance of the system under the current split mode (e.g., not split, split into two data packets, split into three data packets, etc.), and the higher the performance benefit score, the better the overall performance of the system or the more it meets the user's requirements. In some embodiments, the performance benefit score can be determined by comprehensively considering the processor consumption, transmission time consumption, and bandwidth utilization under the current split mode. In some embodiments, the processor consumption score, transmission time consumption score, and bandwidth utilization score of transmitting the target data packet in a specific split mode can be determined based on the processor utilization associated with the transmission history replication snapshot and the bandwidth utilization of the data link, the time consumption of transmitting the target replication snapshot, the split processor resources associated with the split operation, and the packaging processor resources associated with the packaging operation. Then, the performance benefit score can be determined based on the processor consumption score, the transmission time consumption score, and the bandwidth utilization score.

[0044] Combine the following Figure 6 The composition of the performance benefit score according to some embodiments of the present disclosure is described. Figure 6 FIG. 6 is a schematic diagram of a process 600 for determining a performance benefit score according to some embodiments of the present disclosure. Figure 6As shown, the performance benefit score 602 includes a processor consumption score 604, a transmission time consumption score 606, and a bandwidth consumption score 608. The processor consumption score 604 can be calculated based on the processor utilization 610, the split processor resources 612, and the packaged processor resources 614. In some embodiments, the processor utilization 610 can be calculated by counting the processor utilization of the source storage system and the processor utilization of the target storage system when the historical replication snapshot is transmitted. In some embodiments, the transmission time consumption score 606 can be calculated based on the RPO time 616 (i.e., the RPO value of the replication session) and the transmission time consumption 618 for transmitting the replication snapshot from the source storage system to the destination storage system, wherein the measurement unit of the RPO time 616 and the transmission time consumption 618 can be seconds. For example, the RPO time 616 can be 600 seconds, and the transmission time consumption 618 can be 240 seconds. In some embodiments, the average transmission speed 622 of the data link when transmitting the historical replication snapshot can be counted, and then the bandwidth utilization score 608 can be calculated based on the average transmission speed 622 and the theoretical bandwidth 620 of the data link, where the measurement unit of the theoretical bandwidth 620 and the average transmission speed 622 can be megabits per second (Mbps).

[0045] In some embodiments, different weights may be assigned to the processor consumption score 604, the transmission time consumption score 606, and the bandwidth utilization score 608 based on the user's requirements, and then the performance benefit score 602 may be determined by weighting these scores. For example, if the processor configuration of the storage system is low, the user may be more sensitive to the processor consumption, and thus the weight of the processor consumption score 604 may be increased so that the change in the processor resource consumption may more significantly affect the performance benefit score 602. On the contrary, if the processor configuration of the storage system is high, the user may not be sensitive to the processor resource consumption, and may hope that the source storage system can split the replication snapshot into a larger number of data packets to increase the speed of concurrent transmission, and thus the weight of the processor consumption score 604 may be reduced. In some embodiments, the sum of the weight of the processor consumption score 604, the weight of the transmission time consumption score 606, and the weight of the bandwidth utilization score 608 is one.

[0046] return Figure 5, at box 512, process 500 can determine multiple candidate split numbers and calculate the candidate split number that maximizes the performance benefit score. For example, process 500 can calculate multiple performance benefit scores for transferring the target replication snapshot based on different candidate split numbers, and then determine the candidate split number that maximizes the performance benefit score by comparing the multiple performance benefit scores. At box 514, process 500 determines whether the maximum performance benefit score is greater than the performance benefit score when not splitting. If the performance benefit score when not splitting is greater than or equal to the maximum performance benefit score, process 500 proceeds to box 508. If the maximum performance benefit score is greater than the performance benefit score when not splitting, process 500 determines the candidate split number as the target split number and proceeds to box 516.

[0047] Combine the following Figure 7 The process of determining the optimal split number according to some embodiments of the present disclosure is described. Figure 7 FIG. 7 is a schematic diagram showing an example 700 of determining an optimal number of splits according to some embodiments of the present disclosure. Figure 7 As shown, the replication snapshot 702 is a target replication snapshot to be transferred from the source storage system to the destination storage system. If the replication snapshot 702 is not split, the score associated with transferring the replication snapshot 702 includes a processor consumption score 703, a transfer time consumption score 704, and a bandwidth utilization score 705. In example 700, since the replication snapshot 702 is not split, there is no need to consume split processor resources and packaging processor resources, so the processor consumption score 703 is high, but the transfer time consumption score 704 and the bandwidth utilization score 705 are low. After weighted summing the processor consumption score 703, the transfer time consumption score 704, and the bandwidth utilization score 705, a performance benefit score 706 is obtained.

[0048] In example 700, if the replication snapshot 702 is split into two packets (i.e., the number of splits is 2) for parallel transmission, the scores associated with transmitting the replication snapshot 702 include a processor consumption score 713, a transmission time consumption score 714, and a bandwidth utilization score 715. Figure 7 As shown, due to the use of parallel transmission, the transmission time consumption score 714 is higher than the transmission time consumption score 704 when not split, and the bandwidth utilization score 715 is higher than the bandwidth utilization score 705 when not split. However, since splitting the replication snapshot 702 requires the consumption of splitting processor resources and packaging processor resources, the processor consumption score 713 is lower than the processor consumption score 703 when not split. After weighted summing the processor consumption score 713, the transmission time consumption score 714, and the bandwidth utilization score 715, a performance benefit score 716 is obtained. In the example 700, the performance benefit score 716 is higher than the performance benefit score 706 when not split.

[0049] In example 700, if the replication snapshot 702 is split into three packets (i.e., the number of splits is 3) for parallel transmission, the scores associated with transmitting the replication snapshot 702 include a processor consumption score 723, a transmission time consumption score 724, and a bandwidth utilization score 725. Figure 7 As shown, since the replication snapshot 702 is split into more data packets for parallel transmission, the transmission time consumption score 724 is higher than the transmission time consumption score 704 when there is no splitting and the transmission time consumption score 714 when the number of splits is 2, and the bandwidth utilization score 725 is higher than the bandwidth utilization score 705 when there is no splitting and the bandwidth utilization score 705 when the number of splits is 2. However, since more data packet splitting and packaging operations need to be performed, the processor consumption score 723 is lower than the processor consumption score 703 when there is no splitting and the processor consumption score 713 when the number of splits is 2. After weighted summing of the processor consumption score 723, the transmission time consumption score 724, and the bandwidth utilization score 725, a performance benefit score 726 is obtained. In example 700, the performance benefit score 726 is higher than the performance benefit score 706 when there is no splitting, but lower than the performance benefit score 716 when the number of splits is 2. In Figure 7 Among the candidate split numbers shown, the performance benefit score can be maximized when the split number is 2, and the performance benefit score 716 is greater than the performance benefit score 706 when there is no split, so the target split number can be determined to be 2.

[0050] return Figure 5 In block 516, process 500 splits the target replication snapshot into a plurality of data packets, the number of which is the target number of splits, and then concurrently transmits the data packets from the source storage system to the destination storage system, and packages the data packets at the destination storage system to obtain the restored replication snapshot. In some embodiments, process 500 may return from block 508 or block 516 to block 502 to recollect information, thereby re-determining the optimal number of splits based on updated data.

[0051] In this way, the number of splits that maximize the performance benefit score can be calculated based on the degree of importance that the user places on each factor (processor factor, time factor, and bandwidth factor), so that the processor resource consumption can be ensured to be within the user's acceptable range while improving bandwidth utilization and reducing transmission time. In this way, it is possible to avoid splitting the target replication snapshot into too many data packets just for the sake of reducing transmission time, which can cause the processor resource to become a bottleneck and reduce the overall performance of the system.

[0052] In some embodiments, the following formulas can be used to implement Figure 5The process 500 shown in FIG. 5 is a process 500 shown in FIG. 5 . In this document, As is used to represent a source storage system (eg, Figure 1 denoted by source storage system 102 in FIG. 1 , and Ad denoted by destination storage system (eg, Figure 1 The destination storage system 112 in the source storage system includes a storage object having an asynchronous replication session. i (For example, Figure 1 The number of storage objects 104 in the storage object is represented by I. In addition, R i Represents and stores object o i The corresponding replication snapshot is D i Represents the replication snapshot R i The data size of all replicated snapshots D can be calculated by the following equation (1): t Total data size:

[0053]

[0054] In this paper, f B_r Indicates the average transmission speed of the data link, denoted by f B_t represents the theoretical bandwidth of the data link. Due to some factors in the network (such as signal interference, bandwidth sharing, transmission delay, etc.), the actual bandwidth of the data link cannot reach the theoretical bandwidth. Therefore, a threshold ratio w can be set. When f B_r ≥w*f B_t When f B_r <w*f B_t , it means that the current data transmission has not reached the theoretical bandwidth of the data link, and there is still room for the data link to be used.

[0055] In this paper, T RPO The RPO value of the replication session can be calculated by the following equation (2): t The time consumption T for transferring from the source storage system to the destination storage system d :

[0056]

[0057] If T d >T RPO , it means that the replication snapshot cannot be transferred within the RPO time and the transmission speed of the replication snapshot needs to be increased. B_r <w*f B_t And T d >T RPO In this case, you can split the replication snapshot into multiple data packets, and then transmit these data packets in parallel to increase the transmission speed and reduce the transmission time.

[0058] The following describes a method for calculating a performance benefit score for transferring a replication snapshot from a source storage system to a destination storage system without splitting the replication snapshot according to some embodiments. Assume that the processor utilization of the source storage system is C s , the processor utilization of the destination storage system is C d , then the overall processor utilization C of the system can be calculated by the following equation (3): wo , and the processor utilization C wo No units of measurement:

[0059] C wo =C s +C d (3)

[0060] Due to C wo The lower the score, the higher the processor consumption score, so you can use C wo The processor consumption score is represented by the inverse of . In addition, the time consumption T for transferring the replication snapshot from the source storage system to the destination storage system can be calculated by the following equation (4): wo :

[0061]

[0062] Due to the time consumption T wo The unit of measurement is time (e.g., seconds), so it needs to be standardized. For example, T can be normalized by the following equation (5): wo With T RPO Compare and get the time consumption score RT wo :

[0063]

[0064] In addition, the bandwidth utilization B of the data link can be calculated by the following equation (6): wo (i.e., bandwidth utilization score):

[0065]

[0066] In some embodiments, a weight w may be assigned to the processor consumption score according to the user's requirements. C , assign weight w to the time consumption score T , and assign weights w to the bandwidth utilization scores B , and make w C +w T +w B =1, so the performance benefit score S when not split can be calculated by the following equation (7) wo :

[0067]

[0068] The following describes a method for calculating a performance benefit score for concurrently transmitting multiple data packets to a destination storage system when a replication snapshot is split into multiple data packets according to some embodiments. c represents the maximum number of splits into which the replication snapshot can be split. Then N can be calculated by the following equation (8): c :

[0069]

[0070] Where int represents the rounding function.

[0071] If N c = 1, it means that the bandwidth of the data link has been fully utilized, and the replication snapshot will not be split. c >1, it means that the bandwidth of the data link still has room for utilization, and the data transmission speed can also be improved accordingly. k Data packets, using C u It represents the processor resource consumption of the source storage system to perform a split operation, and is expressed as C p represents the processor resource consumption of the destination storage system to perform a packaging operation. The overall processor utilization C of the system can be calculated by the following equation (9): w :

[0072] C w =C s +C d +C u *(N k -1)+C p *(N k -1) (9)

[0073] Due to C w The lower the score, the higher the processor consumption score, so you can use C w In addition, since the replication snapshot is split into N k data packets for concurrent transmission, so the data transmission speed can be increased to f B_r *N k , where f B_r *N k ≤f B_t , and 1≤N k ≤N c Then, the total replication snapshot D can be calculated by the following equation (10): tThe time consumption T for transferring from the source storage system to the destination storage system w :

[0074] Then, T can be calculated by the following equation (11): w With T RPO Compare and get the time consumption score R Tw :

[0075]

[0076] In addition, the bandwidth utilization score B can be calculated by the following equation (12): w :

[0077]

[0078] Combining the above equations (9), (11) and (12), the following equation (13) can be used to calculate the number of replicated snapshots to be split into N k The performance benefit score S when multiple packets are transmitted concurrently w :

[0079]

[0080] Performance Benefit Score w The larger the value, the better the data transmission performance. w Depends on N k The value of S can be calculated w Maximized N k , and N k The following equation (14) must be satisfied:

[0081] Then, the performance benefit score S when not splitting can be wo The maximum performance benefit score when splitting (S w ) for comparison, if S wo ≥max(S w ), the replication snapshot can be directly transferred without alignment for splitting. wo <max(S w ), the replication snapshot is split into N k The concurrent transmission of multiple data packets can improve the data transmission performance.

[0082] In this way, the performance benefit of splitting the replication snapshot into different numbers of data packets for concurrent transmission can be quantified, so that the optimal number of splits can be found by comparing the performance benefit scores corresponding to different numbers of splits, thereby improving the performance of the process of transmitting the replication snapshot from the source storage system to the destination storage system. In this way, it is possible to avoid splitting the target replication snapshot into too many data packets just for the sake of reducing the transmission time, which will cause the processor resources to become a bottleneck and reduce the overall performance of the system.

[0083] Figure 8 A schematic block diagram of an example device 800 that can be used to implement an embodiment of the present disclosure is shown. As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 802 or loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0084] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0085] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as method 300. For example, in some embodiments, the method 300 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the method 300 described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the method 300 in any other appropriate manner (e.g., by means of firmware).

[0086] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0087] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0088] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In addition, although each operation is depicted in a specific order, this should be understood as requiring such operations to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of separate embodiments can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination.

[0089] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for concurrent data transmission, comprising: obtaining a processor utilization and a bandwidth utilization of a data link associated with transmitting one or more historical data packets within a target time; Determining a time consumption for transmitting a target data packet, a first processing resource associated with a splitting operation, and a second processing resource associated with a packing operation; as well as A number of splits for the target data packet is determined based on the processor utilization, the bandwidth utilization, the time consumption, the first processing resource, and the second processing resource for concurrently transmitting the target data packet.

2. The method of claim 1 , wherein the number of splits is a target number of splits, and determining the target number of splits comprises: Determine a first candidate split number and a second candidate split number; Determine, based on the processor utilization, the bandwidth utilization, the time consumption, the first processing resource, and the second processing resource, a first performance benefit score for transmitting the target data packet based on the first candidate split number and a second performance benefit score for transmitting the target data packet based on the second candidate split number; as well as In response to the first performance benefit score being greater than the second performance benefit score and the first candidate split number being less than or equal to a theoretical maximum split number, the first candidate split number is determined as the target split number.

3. The method according to claim 2, wherein determining a first performance benefit score for transmitting the target data packet based on the first candidate split number comprises: Determine, based on the processor utilization, the bandwidth utilization, the time consumption, the first processing resource, and the second processing resource, a processor consumption score, a transmission time consumption score, and a bandwidth utilization score for transmitting the target data packet based on the first candidate split number; as well as The first performance benefit score is determined based on the processor consumption score, the transmission time consumption score, and the bandwidth utilization score.

4. The method of claim 3, wherein determining the processor consumption score comprises: The processor consumption score is determined based on the processor utilization, the second processing resource, the first processing resource, and the first candidate split number.

5. The method of claim 3, wherein determining the transmission time consumption score comprises: The transmission time consumption score is determined based on an average transmission speed of the data link, a size of the target data packet, the first candidate split number, and the target time.

6. The method of claim 3, wherein determining the bandwidth utilization score comprises: The bandwidth utilization score is determined based on an average transmission speed of the data link, a theoretical bandwidth of the data link, and the first candidate split number.

7. The method of claim 3, wherein determining the first performance benefit score further comprises: determining a first weight for the processor consumption score, a second weight for the transmission time consumption score, and a third weight for the bandwidth utilization score; as well as The first performance benefit score is determined based on the processor consumption score, the first weight, the transmission time consumption score, the second weight, the bandwidth utilization score, and the third weight.

8. The method according to claim 1, further comprising: determining an average transmission speed of the data link; determining a ratio of the average transmission speed to a theoretical bandwidth of the data link; comparing the ratio to a predetermined threshold; as well as In response to the ratio being greater than or equal to the predetermined threshold, directly transmitting the target data packet without splitting it.

9. The method according to claim 1, further comprising: determining an average transmission speed of the data link; Determining the data size of the target data packet; determining, based on the average transmission speed and the data size, a transmission time for transmitting the target data packet without splitting the target data packet; comparing the transmission time with the target time; as well as In response to the transmission time being less than or equal to the target time, directly transmitting the target data packet without splitting it.

10. The method of claim 1, wherein determining the processor utilization, the time consumption, and the bandwidth utilization comprises: determining the processor utilization by determining an average of a plurality of processor utilizations for transmitting a plurality of historical data packets within a plurality of previous target times; as well as The bandwidth utilization is determined by determining an average of a plurality of bandwidth utilizations used to transmit the plurality of historical data packets.

11. The method of claim 1, wherein the target time is a recovery point objective, the one or more historical data packages are one or more historical replication snapshots, and the target data package is a target replication snapshot.

12. An electronic device comprising: at least one processor; as well as coupled to the at least one processor and having instructions stored thereon, the instructions, when executed by the at least one processor, causing the electronic device to perform actions, the actions comprising: obtaining a processor utilization and a bandwidth utilization of a data link associated with transmitting one or more historical data packets within a target time; Determining a time consumption for transmitting a target data packet, a first processing resource associated with a splitting operation, and a second processing resource associated with a packing operation; and A number of splits for the target data packet is determined based on the processor utilization, the bandwidth utilization, the time consumption, the first processing resource, and the second processing resource for concurrently transmitting the target data packet.

13. The apparatus of claim 12, wherein the number of splits is a target number of splits, and determining the target number of splits comprises: Determine a first candidate split number and a second candidate split number; Determine, based on the processor utilization, the bandwidth utilization, the time consumption, the first processing resource, and the second processing resource, a first performance benefit score for transmitting the target data packet based on the first candidate split number and a second performance benefit score for transmitting the target data packet based on the second candidate split number; as well as In response to the first performance benefit score being greater than the second performance benefit score and the first candidate split number being less than or equal to a theoretical maximum split number, the first candidate split number is determined as the target split number.

14. The apparatus of claim 13, wherein determining a first performance benefit score for transmitting the target data packet based on the first candidate split number comprises: Determine, based on the processor utilization, the bandwidth utilization, the time consumption, the first processing resource, and the second processing resource, a processor consumption score, a transmission time consumption score, and a bandwidth utilization score for transmitting the target data packet based on the first candidate split number; as well as The first performance benefit score is determined based on the processor consumption score, the transmission time consumption score, and the bandwidth utilization score.

15. The apparatus of claim 14, wherein determining the processor consumption score comprises: The processor consumption score is determined based on the processor utilization, the second processing resource, the first processing resource, and the first candidate split number.

16. The apparatus of claim 14, wherein determining the transmission time consumption score comprises: The transmission time consumption score is determined based on an average transmission speed of the data link, a size of the target data packet, the first candidate split number, and the target time.

17. The apparatus of claim 14, wherein determining the bandwidth utilization score comprises: The bandwidth utilization score is determined based on an average transmission speed of the data link, a theoretical bandwidth of the data link, and the first candidate split number.

18. The apparatus of claim 14, wherein determining the first performance benefit score further comprises: determining a first weight for the processor consumption score, a second weight for the transmission time consumption score, and a third weight for the bandwidth utilization score; as well as The first performance benefit score is determined based on the processor consumption score, the first weight, the transmission time consumption score, the second weight, the bandwidth utilization score, and the third weight.

19. The apparatus of claim 12, further comprising: determining an average transmission speed of the data link; determining a ratio of the average transmission speed to a theoretical bandwidth of the data link; comparing the ratio to a predetermined threshold; as well as In response to the ratio being greater than or equal to the predetermined threshold, directly transmitting the target data packet without splitting it.

20. A computer program product tangibly stored on a non-transitory computer readable medium and comprising machine executable instructions that when executed cause a machine to perform actions comprising: obtaining a processor utilization and a bandwidth utilization of a data link associated with transmitting one or more historical data packets within a target time; Determining a time consumption for transmitting a target data packet, a first processing resource associated with a splitting operation, and a second processing resource associated with a packing operation; as well as A number of splits for the target data packet is determined based on the processor utilization, the bandwidth utilization, the time consumption, the first processing resource, and the second processing resource for concurrently transmitting the target data packet.

Images