Object migration method and device, equipment, storage medium and program product

Abstract

The invention relates to an object migration method and device, equipment, a storage medium and a program product. The method comprises the steps that in response to a data migration instruction, object information of a to-be-migrated object list in a migration window in a host process is obtained, the to-be-migrated object list comprises all data objects in the same migration batch, according to the object information and the current available bandwidth of a storage system, the total required bandwidth of a sub-process associated with the host process is determined, and the total required bandwidth of the sub-process associated with the host process is obtained. And allocating the total required bandwidth to the sub-process, so that the sub-process performs migration processing on the data object in the to-be-migrated object list under the total required bandwidth. By adopting the method, the flexibility of resource use can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an object migration method, apparatus, device, storage medium and program product. Background Art

[0002] To meet the growing business needs for unstructured data storage, major cloud vendors have provided their own object storage services. In order to migrate data objects in object storage in a timely manner when business changes, object migration methods have emerged. Existing object migration methods migrate objects based on resources pre-allocated to object migration tasks.

[0003] However, using the existing object migration method to directly migrate objects based on fixed resources will cause a waste of resources when there are fewer migration tasks, or reduce the migration efficiency when there are more migration tasks, thereby reducing the flexibility of resource use. Summary of the invention

[0004] Based on this, it is necessary to provide an object migration method, device, equipment, storage medium and program product that can improve the flexibility of resource utilization in response to the above technical problems.

[0005] In a first aspect, the present application provides an object migration method, comprising:

[0006] In response to the data migration instruction, obtaining object information of a list of objects to be migrated in the migration window in the main process; wherein the list of objects to be migrated includes data objects in the same migration batch;

[0007] Determine the total required bandwidth of the child processes associated with the main process based on the object information and the current available bandwidth of the storage system;

[0008] The total required bandwidth is allocated to the child process, so that the child process performs migration processing on the data objects in the to-be-migrated object list within the total required bandwidth.

[0009] In one embodiment, obtaining object information of a list of objects to be migrated in a migration window in a main process includes:

[0010] The window size of the migration window is determined according to the number of child processes associated with the main process; the window position of the migration window is determined according to the object migration progress of the main process; the list of objects to be migrated in the migration window of the main process is determined according to the window size and the window position, and the object information of the list of objects to be migrated is obtained.

[0011] In one embodiment, when the number of the list of objects to be migrated in the migration window is at least two, determining the total required bandwidth of the sub-processes associated with the main process according to the object information and the current available bandwidth of the storage system includes:

[0012] The bandwidth weight of each object list to be migrated is determined according to the object information of each object list to be migrated; and the total required bandwidth of the sub-processes associated with the main process is determined according to the bandwidth weight of each object list to be migrated and the current available bandwidth of the storage system.

[0013] In one embodiment, the object information includes the object quantity and total data volume of the data objects in the to-be-migrated object list; and determining the bandwidth weight of each to-be-migrated object list according to the object information of each to-be-migrated object list includes:

[0014] For each list of objects to be migrated, the ratio between the total data volume and the number of objects in the list of objects to be migrated is used as the first bandwidth parameter of the list of objects to be migrated; the number of sub-processes associated with the main process is used as the second bandwidth parameter of the list of objects to be migrated; and the product of the first bandwidth parameter and the second bandwidth parameter is used as the bandwidth weight of the list of objects to be migrated.

[0015] In one embodiment, determining the total required bandwidth of the sub-processes associated with the main process according to the bandwidth weights of the lists of objects to be migrated and the current available bandwidth of the storage system includes:

[0016] According to the bandwidth weights of the lists of objects to be migrated, the total bandwidth weights corresponding to the sub-processes are determined; the product of the total bandwidth weights and the current available bandwidth of the storage system is used as the total required bandwidth of the sub-processes.

[0017] In one embodiment, allocating total required bandwidth to subprocesses includes:

[0018] The total number of virtual machines allocated to each subprocess is determined according to the total bandwidth weight; the total number of virtual machines is used to allocate the total required bandwidth to each subprocess.

[0019] In a second aspect, the present application further provides an object migration device, comprising:

[0020] An information acquisition module, used to respond to the data migration instruction and acquire object information of a list of objects to be migrated in the migration window of the main process; wherein the list of objects to be migrated includes data objects in the same migration batch;

[0021] A bandwidth determination module, used to determine the total required bandwidth of the sub-processes associated with the main process according to the object information and the current available bandwidth of the storage system;

[0022] The object migration module is used to allocate the total required bandwidth to the child process so that the child process can perform migration processing on the data objects in the to-be-migrated object list under the total required bandwidth.

[0023] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0024] In response to the data migration instruction, obtaining object information of a list of objects to be migrated in a migration window in a main process; wherein the list of objects to be migrated includes data objects in the same migration batch;

[0025] Determine the total required bandwidth of the child processes associated with the main process based on the object information and the current available bandwidth of the storage system;

[0026] The total required bandwidth is allocated to the child process, so that the child process performs migration processing on the data objects in the to-be-migrated object list within the total required bandwidth.

[0027] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0028] In response to the data migration instruction, obtaining object information of a list of objects to be migrated in a migration window in a main process; wherein the list of objects to be migrated includes data objects in the same migration batch;

[0029] Determine the total required bandwidth of the child processes associated with the main process based on the object information and the current available bandwidth of the storage system;

[0030] The total required bandwidth is allocated to the child process, so that the child process performs migration processing on the data objects in the to-be-migrated object list within the total required bandwidth.

[0031] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0032] In response to the data migration instruction, obtaining object information of a list of objects to be migrated in a migration window in a main process; wherein the list of objects to be migrated includes data objects in the same migration batch;

[0033] Determine the total required bandwidth of the child processes associated with the main process based on the object information and the current available bandwidth of the storage system;

[0034] The total required bandwidth is allocated to the child process, so that the child process performs migration processing on the data objects in the to-be-migrated object list within the total required bandwidth.

[0035] The object migration method, device, equipment, storage medium and program product described above obtain object information of the list of objects to be migrated of each data object under the same migration batch in the migration window of the main process by responding to the data migration instruction, and determine the total required bandwidth of the sub-process associated with the main process based on the object information and the current available bandwidth of the storage system, and allocate the total required bandwidth to the sub-process so that the sub-process can perform migration processing on the data objects in the list of objects to be migrated under the total required bandwidth. Compared with the related art, which directly migrates objects based on fixed resources, the above method can improve the flexibility of resource use by dynamically determining the bandwidth to be allocated to the sub-process based on the object information and the current available bandwidth of the storage system, while ensuring the efficiency of object migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 is a flow chart of an object migration method in one embodiment;

[0038] Figure 2A A schematic diagram of a process for obtaining object information in one embodiment;

[0039] Figure 2B A schematic diagram of the main process in one embodiment;

[0040] Figure 3 A schematic diagram of a process for determining a total required bandwidth in one embodiment;

[0041] Figure 4 A diagram of a storage system environment architecture in one embodiment;

[0042] Figure 5 is a flow chart of an object migration method in another embodiment;

[0043] Figure 6 is a structural block diagram of an object migration device in one embodiment;

[0044] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] To meet the growing business needs for unstructured data storage, major cloud vendors have provided their own object storage services. In order to migrate data objects in object storage in a timely manner when business changes, object migration methods have emerged. Existing object migration methods migrate objects based on resources pre-allocated to object migration tasks.

[0047] However, using the existing object migration method to directly migrate objects based on fixed resources will cause a waste of resources when there are fewer migration tasks, or reduce the migration efficiency when there are more migration tasks, thereby reducing the flexibility of resource use.

[0048] Based on this, in an exemplary embodiment, Figure 1 As shown, an object migration method is provided, which is described by taking the method applied to an object migration device deployed in an object storage system as an example, and specifically includes the following steps:

[0049] S101, in response to a data migration instruction, obtaining object information of a list of objects to be migrated in a migration window in a main process.

[0050] The data migration instruction is used to represent an instruction that can instruct the object storage system to perform data migration; the main process is the main process of the migration service. In the embodiment of the present application, multiple main processes for object migration can be constructed simultaneously based on conditions such as data object type.

[0051] The migration window is used to select the list of objects to be migrated in the main process for object migration. The list of objects to be migrated contains the data objects in the same migration batch. The object information is the relevant information of each data object in the list of objects to be migrated, for example, including but not limited to the data object name (object_names) and the size of each data object (object_size). In addition, in order to perform the object migration task more accurately, the object information can also store information such as the export end mark (is_truncated) that can indicate whether the list of objects to be migrated is the last object list in the main process, and the start mark (next_key_marker) that can indicate whether the list of objects to be migrated is the start object list of the next export.

[0052] In one possible implementation, the main process of the migration service can use the interface provided by the object storage system to cyclically export (list) data objects to be migrated from the source object storage, and treat the data objects under each exported batch as a whole to generate a list of objects to be migrated; then, collect object information of the data objects in the list of objects to be migrated, and store the object information at the end of the corresponding list of objects to be migrated.

[0053] After the data migration instruction is detected, the object information of the list of objects to be migrated in the migration window may be obtained on the main process to perform subsequent migration processing.

[0054] S102: Determine the total required bandwidth of the sub-processes associated with the main process according to the object information and the current available bandwidth of the storage system.

[0055] The current available bandwidth is the available bandwidth that the storage system can provide for the migration task at the current moment; the sub-process is used to represent the migration process that executes the data migration task; and the total required bandwidth is the bandwidth required by all sub-processes associated with the main process.

[0056] It is understandable that in a storage system, there may be multiple main processes for data migration at the same time, and each main process is associated with a corresponding sub-process for process migration processing. Therefore, in order to ensure the rationality of the current available bandwidth allocation, the total required bandwidth to be allocated to the sub-processes associated with the main process can be determined based on the object information obtained from the main process. Among them, the number of sub-processes associated with each main process can be configured one by one by the technician based on experience, or can be uniformly set to the same number, and this application does not limit this.

[0057] In one possible implementation, for each main process, the object information obtained from the main process can be input into a bandwidth determination model being trained, and the bandwidth determination model outputs the total required bandwidth to be allocated to the sub-processes associated with the main process based on the object information and model parameters.

[0058] In another possible implementation, for each main process, the bandwidth allocation weight corresponding to the main process can be determined based on the object information obtained from the main process; then, the product of the bandwidth allocation weight corresponding to the main process and the current available bandwidth is used as the total required bandwidth to be allocated to the child processes associated with the main process.

[0059] S103: Allocate a total required bandwidth to the child process, so that the child process performs migration processing on the data objects in the to-be-migrated object list within the total required bandwidth.

[0060] In one possible implementation, the determined total required bandwidth can be allocated to the child process so that the child process can download each data object in the migration object list from the source object storage to the memory according to the list of objects to be migrated under the total required bandwidth, and call the object storage interface to upload the data objects in the memory to the destination object storage to complete the object migration process.

[0061] In the above object migration method, by responding to the data migration instruction, the object information of the list of objects to be migrated of each data object under the same migration batch in the migration window of the main process is obtained, and the total required bandwidth of the sub-process associated with the main process is determined according to the object information and the current available bandwidth of the storage system, and the total required bandwidth is allocated to the sub-process, so that the sub-process can perform migration processing on the data objects in the list of objects to be migrated under the total required bandwidth. Compared with the related art, in which objects are directly migrated based on fixed resources, the above method can improve the flexibility of resource use by dynamically determining the bandwidth to be allocated to the sub-process according to the object information and the current available bandwidth of the storage system, while ensuring the efficiency of object migration.

[0062] In order to ensure the reliability of object information acquisition, based on the above embodiments, an optional method for acquiring object information is provided in the embodiments of the present application, such as Figure 2A As shown, the specific steps include:

[0063] S201, determining the window size of the migration window according to the number of child processes associated with the main process.

[0064] The number of processes refers to the number of child processes associated with the main process; the window size refers to the number of lists of objects to be migrated that can be included in the migration window.

[0065] In one possible implementation, the number of lists of objects to be migrated that can be migrated at the same time can be determined according to the number of sub-processes associated with the main process, and the number of lists of objects to be migrated can be used as the window size of the migration window. Exemplarily, the number of sub-processes associated with the main process can be directly used as the window size of the migration window.

[0066] For example, refer to Figure 2B The main process diagram shown in the figure, where obj_list i is the i-th list of objects to be migrated. When it is determined that the number of child processes (MigProcess) associated with the main process is 3, the size of the migration window is determined to be able to accommodate three lists of objects to be migrated.

[0067] S202: Determine the window position of the migration window according to the object migration progress of the main process.

[0068] The object migration progress is used to represent the migration status of each list of objects to be migrated on the main process; the window position is the position of the migration window on the main process.

[0069] In one embodiment, the window position of the migration window can be determined based on the object migration progress of each object list to be migrated based on the main process. Exemplarily, when the migration process of any object list to be migrated in the migration window is detected to be completed, the next object list to be migrated of the object list to be migrated can be migrated into the migration window.

[0070] For example, refer to Figure 2B As shown in the main process schematic diagram, after the migration processing of obj_list0 is completed, the window position of the migration window can be determined to be on obj_list1-obj_list3.

[0071] S203: Determine a list of objects to be migrated in the migration window in the main process according to the window size and the window position, and obtain object information of the list of objects to be migrated.

[0072] In one possible implementation, the lists of objects to be migrated contained in the migration window in the main process may be determined according to the window size and the window position; then, the object information of the list of objects to be migrated may be obtained at the end of the list of objects to be migrated.

[0073] In the embodiment of the present application, by determining the list of objects to be migrated in the migration window in the main process according to the window size and the window position, and obtaining the object information of the list of objects to be migrated, the accuracy of the object information acquisition can be ensured.

[0074] In order to ensure the accuracy of the total required bandwidth, based on the above embodiment, when the number of the list of objects to be migrated in the migration window is at least two, in the embodiment of the present application, an optional method for determining the total required bandwidth is provided, such as Figure 3 As shown, the specific steps include:

[0075] S301: Determine the bandwidth weight of each list of objects to be migrated according to the object information of each list of objects to be migrated.

[0076] The bandwidth weight is the weight of the bandwidth required to migrate the data objects in the list of objects to be migrated.

[0077] In one possible implementation, for each list of objects to be migrated, the object information of the list of objects to be migrated may be input into a trained weight determination model, and the weight determination model outputs the bandwidth weight of the list of objects to be migrated according to the object information and model parameters.

[0078] In another possible implementation, for each list of objects to be migrated, the number of data objects and the total data volume can be obtained from the object information of the list of objects to be migrated; then, the bandwidth weight of the list of objects to be migrated is calculated according to the number of data objects and the total data volume.

[0079] S302: Determine the total required bandwidth of the sub-processes associated with the main process according to the bandwidth weights of the lists of objects to be migrated and the current available bandwidth of the storage system.

[0080] The total required bandwidth is the total bandwidth required by each sub-process that executes the migration task. For example, the main process A is associated with sub-processes A1, A2, and A3. The total required bandwidth of the sub-processes associated with the main process A is the sum of the bandwidths required by sub-processes A1, A2, and A3.

[0081] In one possible implementation, for each list of objects to be migrated, the product of the bandwidth weight of the list of objects to be migrated and the current available bandwidth of the storage system can be used as the required bandwidth of the sub-process processing the list of objects to be migrated; then, the total required bandwidth of each sub-process is determined based on the required bandwidth corresponding to each sub-process.

[0082] In another possible implementation, the total bandwidth weight corresponding to each sub-process is determined according to the bandwidth weight of each list of objects to be migrated; the product of the total bandwidth weight and the current available bandwidth of the storage system is used as the total required bandwidth of each sub-process. The total bandwidth weight is the sum of the bandwidth weights corresponding to each sub-process.

[0083] Exemplarily, the sum of the bandwidth weights corresponding to each list of objects to be migrated can be used as the total bandwidth weight corresponding to each sub-process; then, the product of the total bandwidth weight and the current available bandwidth of the storage system can be directly used as the total required bandwidth of each sub-process.

[0084] It is understandable that, since the position of the migration window will move according to the migration status of each list of objects to be migrated, the total required bandwidth of each sub-process will also change according to the change of each list of objects to be migrated in the migration window.

[0085] For example, at the current moment, the migration window contains the to-be-migrated object lists E, F, and G, where the bandwidth weight of the to-be-migrated object list E is W. E , the bandwidth weight of the list of objects to be migrated F is W F , the bandwidth weight of the list of objects to be migrated G is W G , at this time, the total bandwidth weight corresponding to each sub-process is W E +W F +W G .

[0086] After the migration of each data object in the list of objects to be migrated E is completed, the list of objects to be migrated E is removed from the migration window, and the bandwidth weight is W. H The list of objects to be migrated H is migrated into the migration window. At this time, the total bandwidth weight corresponding to each sub-process is W F +W G +W H .

[0087] In the embodiment of the present application, by determining the bandwidth weight of each list of objects to be migrated and determining the total required bandwidth according to the bandwidth weight of each list of objects to be migrated and the current available bandwidth, the accuracy of the total required bandwidth can be ensured.

[0088] In order to ensure the accuracy of the bandwidth weight, on the basis of the above embodiment, the object information includes the object quantity and total data volume of the data objects in the list of objects to be migrated; further, in an embodiment of the present application, an optional method for determining the bandwidth weight is provided, specifically: for each list of objects to be migrated, the ratio between the total data volume and the object quantity of the list of objects to be migrated is used as the first bandwidth parameter of the list of objects to be migrated; the process number of sub-processes associated with the main process is used as the second bandwidth parameter of the list of objects to be migrated; the product of the first bandwidth parameter and the second bandwidth parameter is used as the bandwidth weight of the list of objects to be migrated.

[0089] Among them, the first bandwidth parameter is used to represent the value determined based on the total data volume and the number of objects, which can calculate the bandwidth weight; the second bandwidth parameter is used to represent the value determined based on the number of processes, which can calculate the bandwidth weight.

[0090] In one possible implementation, referring to formula (1), for each list of objects to be migrated, the ratio between the total amount of data in the list of objects to be migrated and the number of objects in the list of objects to be migrated may be used as the first bandwidth parameter of the list of objects to be migrated.

[0091] (1)

[0092] Among them, w1(obj_list i ) is the first bandwidth parameter of the i-th list of objects to be migrated; total_size(obj_list i ) is the total amount of data in the i-th object list to be migrated; object_nums(obj_list i ) is the number of objects in the i-th list of objects to be migrated.

[0093] Furthermore, the number of sub-processes associated with the main process can be directly used as the second bandwidth parameter of the list of objects to be migrated; then, referring to the formula W(obj_list i )=w1(obj_listi )*T, multiply the first bandwidth parameter by the second bandwidth parameter to obtain the bandwidth weight of the object list to be migrated. Where T is the number of subprocesses associated with the main process; W(obj_list i ) is the bandwidth weight of the i-th list of objects to be migrated.

[0094] In an embodiment of the present application, by determining the bandwidth weight of the object list to be migrated based on the ratio between the total data volume and the number of objects in the object list to be migrated and the number of sub-processes associated with the main process, the accuracy of the bandwidth weight determination can be ensured.

[0095] In order to ensure the rationality of bandwidth allocation, based on the above embodiments, in an embodiment of the present application, an optional method for allocating the total required bandwidth is provided, specifically, according to the total bandwidth weight, the total number of virtual machines allocated to each sub-process is determined; the total number of virtual machines is used to allocate the total required bandwidth to each sub-process.

[0096] References Figure 4 In the storage system environment architecture diagram shown, the top layer of the storage system is the application service layer, which is where the migration service is located in this scenario. The middle layer is the virtual resource layer, which provides a running carrier for the application service layer. The bottom layer is the physical resource layer, which provides hardware support for the virtual resource layer. It is mainly composed of physical machines equipped with migration services, each of which can be equipped with multiple virtual machines according to its configuration.

[0097] It is understandable that a virtual machine manager is maintained in the virtual resource layer, which quickly creates and modifies the configuration of virtual hosts through virtualization technology and virtual device technology, and provides support for the migration process of each application service layer to meet the different bandwidth requirements of different batches of object migration. The virtual machine manager can dynamically adjust the number of virtual machines used by the object migration service. When the task is reduced, the virtual machine can be scheduled to other services to reduce resource waste.

[0098] In one possible implementation, for each main process, the total number of virtual machines allocated to each sub-process associated with the main process can be determined based on the ratio of the total bandwidth weight required by each sub-process associated with the main process to the total available bandwidth in the storage system and the current number of virtual machines; then, the total number of virtual machines is used to allocate the total required bandwidth to each sub-process associated with the main process.

[0099] In another possible implementation, the number of virtual machines allocated to each subprocess may be determined based on the bandwidth weight corresponding to each subprocess and the current number of virtual machines, and the number of virtual machines corresponding to each subprocess may be used to allocate the required bandwidth to the subprocess.

[0100] In the embodiment of the present application, by using a virtual machine to dynamically provide virtual resources for each sub-process according to the bandwidth occupancy of each sub-process, the rationality of bandwidth allocation can be ensured.

[0101] Figure 5 FIG. 1 is a flow chart of an object migration method in another embodiment. Based on the above embodiment, this embodiment provides an optional example of an object migration method. Figure 5 The specific implementation process is as follows:

[0102] S501 , in response to a data migration instruction, determining a window size of a migration window according to the number of child processes associated with a main process.

[0103] S502: Determine the window position of the migration window according to the object migration progress of the main process.

[0104] S503: Determine each list of objects to be migrated in the migration window in the main process according to the window size and the window position, and obtain object information of each list of objects to be migrated.

[0105] The list of objects to be migrated includes all data objects in the same migration batch.

[0106] S504: Determine the bandwidth weight of each list of objects to be migrated according to the object information of each list of objects to be migrated.

[0107] Optionally, for each list of objects to be migrated, the ratio between the total data volume and the number of objects in the list of objects to be migrated is used as the first bandwidth parameter of the list of objects to be migrated; the number of sub-processes associated with the main process is used as the second bandwidth parameter of the list of objects to be migrated; and the product of the first bandwidth parameter and the second bandwidth parameter is used as the bandwidth weight of the list of objects to be migrated.

[0108] S505: Determine the total bandwidth weight corresponding to each sub-process according to the bandwidth weight of each to-be-migrated object list.

[0109] S506: The product of the total bandwidth weight and the current available bandwidth of the storage system is used as the total required bandwidth of each sub-process.

[0110] S507: Determine the total number of virtual machines allocated to each sub-process according to the total bandwidth weight.

[0111] S508: Use the total number of virtual machines to allocate a total required bandwidth to each child process, so that the child process performs migration processing on the data objects in the to-be-migrated object list under the total required bandwidth.

[0112] The specific process of the above S501-S508 can refer to the description of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0113] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0114] Based on the same inventive concept, the embodiment of the present application also provides an object migration device for implementing the object migration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more object migration device embodiments provided below can refer to the limitations on the object migration method above, and will not be repeated here.

[0115] In an exemplary embodiment, Figure 6 As shown, an object migration device 1 is provided, comprising: an information acquisition module 10, a bandwidth determination module 20 and an object migration module 30, wherein:

[0116] The information acquisition module 10 is used to obtain object information of a list of objects to be migrated in the migration window in the main process in response to the data migration instruction; wherein the list of objects to be migrated includes data objects in the same migration batch;

[0117] The bandwidth determination module 20 is used to determine the total required bandwidth of the sub-processes associated with the main process according to the object information and the current available bandwidth of the storage system;

[0118] The object migration module 30 is used to allocate a total required bandwidth to the child process, so that the child process performs migration processing on the data objects in the to-be-migrated object list within the total required bandwidth.

[0119] In an exemplary embodiment, the information acquisition module 10 is specifically used for:

[0120] The window size of the migration window is determined according to the number of child processes associated with the main process; the window position of the migration window is determined according to the object migration progress of the main process; the list of objects to be migrated in the migration window of the main process is determined according to the window size and the window position, and the object information of the list of objects to be migrated is obtained.

[0121] In an exemplary embodiment, when the number of the to-be-migrated object lists in the migration window is at least two, the bandwidth determination module 20 includes:

[0122] A first determining unit, configured to determine a bandwidth weight of each list of objects to be migrated according to object information of each list of objects to be migrated;

[0123] The second determining unit is configured to determine the total required bandwidth of the sub-processes associated with the main process according to the bandwidth weights of the lists of objects to be migrated and the current available bandwidth of the storage system.

[0124] In an exemplary embodiment, the object information includes the object quantity and total data volume of the data objects in the to-be-migrated object list; the first determining unit is specifically configured to:

[0125] For each list of objects to be migrated, the ratio between the total data volume and the number of objects in the list of objects to be migrated is used as the first bandwidth parameter of the list of objects to be migrated; the number of sub-processes associated with the main process is used as the second bandwidth parameter of the list of objects to be migrated; and the product of the first bandwidth parameter and the second bandwidth parameter is used as the bandwidth weight of the list of objects to be migrated.

[0126] In an exemplary embodiment, the second determining unit is specifically configured to:

[0127] According to the bandwidth weights of the lists of objects to be migrated, the total bandwidth weights corresponding to the sub-processes are determined; the product of the total bandwidth weights and the current available bandwidth of the storage system is used as the total required bandwidth of the sub-processes.

[0128] In an exemplary embodiment, the object migration module 30 is specifically used to:

[0129] The total number of virtual machines allocated to each subprocess is determined according to the total bandwidth weight; the total number of virtual machines is used to allocate the total required bandwidth to each subprocess.

[0130] Each module in the above-mentioned object migration device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0131] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be migrated. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an object migration method is implemented.

[0132] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0133] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0134] In response to the data migration instruction, obtaining object information of a list of objects to be migrated in the migration window in the main process; wherein the list of objects to be migrated includes data objects in the same migration batch;

[0135] Determine the total required bandwidth of the child processes associated with the main process based on the object information and the current available bandwidth of the storage system;

[0136] The total required bandwidth is allocated to the child process, so that the child process performs migration processing on the data objects in the to-be-migrated object list within the total required bandwidth.

[0137] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0138] The window size of the migration window is determined according to the number of child processes associated with the main process; the window position of the migration window is determined according to the object migration progress of the main process; the list of objects to be migrated in the migration window of the main process is determined according to the window size and the window position, and the object information of the list of objects to be migrated is obtained.

[0139] In one embodiment, when the number of the list of objects to be migrated in the migration window is at least two, the processor further implements the following steps when executing the computer program:

[0140] The bandwidth weight of each object list to be migrated is determined according to the object information of each object list to be migrated; and the total required bandwidth of the sub-processes associated with the main process is determined according to the bandwidth weight of each object list to be migrated and the current available bandwidth of the storage system.

[0141] In one embodiment, the object information includes the object quantity and total data volume of the data objects in the list of objects to be migrated; when the processor executes the computer program, the following steps are also implemented:

[0142] For each list of objects to be migrated, the ratio between the total data volume and the number of objects in the list of objects to be migrated is used as the first bandwidth parameter of the list of objects to be migrated; the number of sub-processes associated with the main process is used as the second bandwidth parameter of the list of objects to be migrated; and the product of the first bandwidth parameter and the second bandwidth parameter is used as the bandwidth weight of the list of objects to be migrated.

[0143] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0144] According to the bandwidth weights of the lists of objects to be migrated, the total bandwidth weights corresponding to the sub-processes are determined; the product of the total bandwidth weights and the current available bandwidth of the storage system is used as the total required bandwidth of the sub-processes.

[0145] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0146] The total number of virtual machines allocated to each subprocess is determined according to the total bandwidth weight; the total number of virtual machines is used to allocate the total required bandwidth to each subprocess.

[0147] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0148] In response to the data migration instruction, obtaining object information of a list of objects to be migrated in the migration window in the main process; wherein the list of objects to be migrated includes data objects in the same migration batch;

[0149] Determine the total required bandwidth of the child processes associated with the main process based on the object information and the current available bandwidth of the storage system;

[0150] The total required bandwidth is allocated to the child process, so that the child process performs migration processing on the data objects in the to-be-migrated object list within the total required bandwidth.

[0151] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0152] The window size of the migration window is determined according to the number of child processes associated with the main process; the window position of the migration window is determined according to the object migration progress of the main process; the list of objects to be migrated in the migration window of the main process is determined according to the window size and the window position, and the object information of the list of objects to be migrated is obtained.

[0153] In one embodiment, when the number of the list of objects to be migrated in the migration window is at least two, the computer program further implements the following steps when executed by the processor:

[0154] The bandwidth weight of each object list to be migrated is determined according to the object information of each object list to be migrated; and the total required bandwidth of the sub-processes associated with the main process is determined according to the bandwidth weight of each object list to be migrated and the current available bandwidth of the storage system.

[0155] In one embodiment, the object information includes the object quantity and total data volume of the data objects in the list of objects to be migrated; when the computer program is executed by the processor, the following steps are also implemented:

[0156] For each list of objects to be migrated, the ratio between the total data volume and the number of objects in the list of objects to be migrated is used as the first bandwidth parameter of the list of objects to be migrated; the number of sub-processes associated with the main process is used as the second bandwidth parameter of the list of objects to be migrated; and the product of the first bandwidth parameter and the second bandwidth parameter is used as the bandwidth weight of the list of objects to be migrated.

[0157] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0158] According to the bandwidth weights of the lists of objects to be migrated, the total bandwidth weights corresponding to the sub-processes are determined; the product of the total bandwidth weights and the current available bandwidth of the storage system is used as the total required bandwidth of the sub-processes.

[0159] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0160] The total number of virtual machines allocated to each subprocess is determined according to the total bandwidth weight; the total number of virtual machines is used to allocate the total required bandwidth to each subprocess.

[0161] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0162] In response to the data migration instruction, obtaining object information of a list of objects to be migrated in the migration window in the main process; wherein the list of objects to be migrated includes data objects in the same migration batch;

[0163] Determine the total required bandwidth of the child processes associated with the main process based on the object information and the current available bandwidth of the storage system;

[0164] The total required bandwidth is allocated to the child process, so that the child process performs migration processing on the data objects in the to-be-migrated object list within the total required bandwidth.

[0165] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0166] The window size of the migration window is determined according to the number of child processes associated with the main process; the window position of the migration window is determined according to the object migration progress of the main process; the list of objects to be migrated in the migration window of the main process is determined according to the window size and the window position, and the object information of the list of objects to be migrated is obtained.

[0167] In one embodiment, when the number of the list of objects to be migrated in the migration window is at least two, the computer program further implements the following steps when executed by the processor:

[0168] The bandwidth weight of each object list to be migrated is determined according to the object information of each object list to be migrated; and the total required bandwidth of the sub-processes associated with the main process is determined according to the bandwidth weight of each object list to be migrated and the current available bandwidth of the storage system.

[0169] In one embodiment, the object information includes the object quantity and total data volume of the data objects in the list of objects to be migrated; when the computer program is executed by the processor, the following steps are also implemented:

[0170] For each list of objects to be migrated, the ratio between the total data volume and the number of objects in the list of objects to be migrated is used as the first bandwidth parameter of the list of objects to be migrated; the number of sub-processes associated with the main process is used as the second bandwidth parameter of the list of objects to be migrated; and the product of the first bandwidth parameter and the second bandwidth parameter is used as the bandwidth weight of the list of objects to be migrated.

[0171] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0172] According to the bandwidth weights of the lists of objects to be migrated, the total bandwidth weights corresponding to the sub-processes are determined; the product of the total bandwidth weights and the current available bandwidth of the storage system is used as the total required bandwidth of the sub-processes.

[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0174] The total number of virtual machines allocated to each subprocess is determined according to the total bandwidth weight; the total number of virtual machines is used to allocate the total required bandwidth to each subprocess.

[0175] It should be noted that the data involved in this application (including but not limited to data to be migrated, etc.) are all data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0176] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0177] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0178] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An object migration method, characterized in that: The method comprises: In response to the data migration instruction, obtaining object information of a list of objects to be migrated in the migration window in the main process; wherein the list of objects to be migrated includes data objects in the same migration batch; Determine the total required bandwidth of the sub-processes associated with the main process according to the object information and the current available bandwidth of the storage system; The total required bandwidth is allocated to the sub-process, so that the sub-process performs migration processing on the data objects in the list of objects to be migrated under the total required bandwidth.

2. The method according to claim 1, characterized in that The step of obtaining object information of a list of objects to be migrated in a migration window in the main process includes: Determining the window size of the migration window according to the number of child processes associated with the main process; Determining a window position of the migration window according to the object migration progress of the main process; According to the window size and the window position, a list of objects to be migrated in the migration window in the main process is determined, and object information of the list of objects to be migrated is obtained.

3. The method according to claim 1, characterized in that: When the number of the list of objects to be migrated within the migration window is at least two, determining the total required bandwidth of the sub-processes associated with the main process according to the object information and the current available bandwidth of the storage system includes: Determine the bandwidth weight of each list of objects to be migrated according to the object information of each list of objects to be migrated; The total required bandwidth of the sub-processes associated with the main process is determined according to the bandwidth weights of the lists of objects to be migrated and the current available bandwidth of the storage system.

4. The method according to claim 3, characterized in that The object information includes the object quantity and total data volume of the data objects in the list of objects to be migrated; The step of determining the bandwidth weight of each list of objects to be migrated according to the object information of each list of objects to be migrated includes: For each list of objects to be migrated, a ratio between a total amount of data in the list of objects to be migrated and the number of objects in the list of objects to be migrated is used as a first bandwidth parameter of the list of objects to be migrated; Using the number of sub-processes associated with the main process as a second bandwidth parameter of the list of objects to be migrated; The product of the first bandwidth parameter and the second bandwidth parameter is used as the bandwidth weight of the list of objects to be migrated.

5. The method according to claim 3, characterized in that: The determining, according to the bandwidth weights of the lists of objects to be migrated and the current available bandwidth of the storage system, the total required bandwidth of the sub-processes associated with the main process includes: According to the bandwidth weight of each list of objects to be migrated, determine the total bandwidth weight corresponding to each sub-process; The product of the total bandwidth weight and the current available bandwidth of the storage system is used as the total required bandwidth of each sub-process.

6. The method according to claim 5, characterized in that The allocating the total required bandwidth to the sub-process comprises: Determining the total number of virtual machines allocated to each of the sub-processes according to the total bandwidth weight; The total number of virtual machines is used to allocate the total required bandwidth to each of the sub-processes.

7. An object migration device, characterized in that: The device comprises: An information acquisition module, configured to obtain object information of a list of objects to be migrated in a migration window in a main process in response to a data migration instruction; wherein the list of objects to be migrated includes data objects in the same migration batch; A bandwidth determination module, configured to determine a total required bandwidth of a sub-process associated with the main process according to the object information and a current available bandwidth of the storage system; The object migration module is used to allocate the total required bandwidth to the sub-process, so that the sub-process performs migration processing on the data objects in the list of objects to be migrated under the total required bandwidth.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images