Data storage control method and device

Abstract

The present disclosure provides a data storage control method and device, which relates to the field of distributed cluster technology. The method includes: constructing a computer room allocation model of the distributed storage system according to the dependency relationship between each storage object in the distributed storage system, the computer room allocation model is used to output the corresponding migration evaluation result according to the computer room allocation strategy, the computer room allocation strategy is used to indicate the target computer room allocated for the storage object in the distributed storage system, and the migration evaluation result is used to indicate the overhead caused by the data migration processing from the original computer room to the target computer room; based on the computer room allocation model, determining the computer room allocation strategy corresponding to the minimum migration evaluation result as the target computer room allocation strategy; and performing data migration on the distributed storage system according to the target computer room allocation strategy. The present disclosure can minimize the overhead introduced by data migration.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of distributed cluster technology, and more particularly to a data storage control method and device. Background Art

[0002] In a distributed storage system, multiple storage devices are set up, and the storage objects of each storage device may include data and tasks. The expansion of a distributed storage system includes vertical expansion and horizontal expansion. Vertical expansion is to migrate the data in all storage devices of the distributed storage system to a computer room with sufficient capacity. Horizontal expansion is based on a multi-computer room deployment solution, migrating part of the data from the original computer room to a newly added computer room.

[0003] In the prior art, in the above horizontal expansion solution, a migration strategy needs to be determined to establish a correspondence between storage objects and computer rooms, so that the storage objects can be migrated to the corresponding computer rooms according to the correspondence. The migration strategy can be flexibly set according to the computer room capacity and business needs.

[0004] However, the above technical solution has the problem of high overhead. Summary of the invention

[0005] The embodiments of the present disclosure provide a data storage control method and device, which can reduce the overhead introduced by data migration.

[0006] In a first aspect, an embodiment of the present disclosure provides a data storage control method, the method comprising:

[0007] According to the dependency relationship between the storage objects in the distributed storage system, a computer room allocation model of the distributed storage system is constructed, wherein the computer room allocation model is used to output a corresponding migration evaluation result according to a computer room allocation strategy, wherein the computer room allocation strategy is used to indicate a target computer room allocated to the storage objects in the distributed storage system, and the migration evaluation result is used to indicate the overhead caused by data migration processing from an original computer room to the target computer room;

[0008] Calling the computer room allocation model, based on the migration evaluation results corresponding to each computer room allocation strategy, determining the computer room allocation strategy corresponding to the minimum migration evaluation result as the target computer room allocation strategy;

[0009] According to the target computer room allocation strategy, data migration is performed on the distributed storage system.

[0010] In a second aspect, an embodiment of the present disclosure provides a data storage control device, the device comprising:

[0011] A function construction module, used for constructing a computer room allocation model of the distributed storage system according to the dependency relationship between the storage objects in the distributed storage system, the computer room allocation model is used for outputting a corresponding migration evaluation result according to a computer room allocation strategy, the computer room allocation strategy is used for indicating a target computer room allocated to the storage objects in the distributed storage system, and the migration evaluation result is used for indicating the overhead caused by data migration processing from an original computer room to the target computer room;

[0012] A target strategy determination module is used to call the computer room allocation model, and based on the migration evaluation results corresponding to each computer room allocation strategy, determine the computer room allocation strategy corresponding to the minimum migration evaluation result as the target computer room allocation strategy;

[0013] The data migration module is used to migrate data of the distributed storage system according to the target computer room allocation strategy.

[0014] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0015] The memory stores computer-executable instructions;

[0016] The at least one processor executes the computer-executable instructions stored in the memory, so that the electronic device implements the method as described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, a computing device implements the method described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program, wherein the computer program is used to implement the method described in the first aspect.

[0019] The disclosed embodiment provides a data storage control method and device, the method comprising: constructing a computer room allocation model of the distributed storage system according to the dependency relationship between each storage object in the distributed storage system, the computer room allocation model is used to output the corresponding migration evaluation result according to the computer room allocation strategy, the computer room allocation strategy is used to indicate the target computer room allocated for the storage object in the distributed storage system, and the migration evaluation result is used to indicate the overhead caused by the data migration processing from the original computer room to the target computer room; calling the computer room allocation model, based on the migration evaluation results corresponding to each computer room allocation strategy, determining the computer room allocation strategy corresponding to the minimum migration evaluation result as the target computer room allocation strategy; and performing data migration on the distributed storage system according to the target computer room allocation strategy. Before migration, a computer room allocation model is constructed for the distributed storage system, and the computer room allocation model is used to output the corresponding migration evaluation result according to the computer room allocation strategy. Therefore, optimizing and solving the computer room allocation model can obtain the target computer room allocation strategy corresponding to the minimum overhead corresponding to the data migration. In this way, when performing data migration by comparing the target computer room allocation strategy with the original computer room, the overhead can be minimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 is a structural diagram of a distributed storage system provided by an embodiment of the present disclosure;

[0022] Figure 2 is a flowchart of a data storage control method provided by an embodiment of the present disclosure;

[0023] Figure 3 It is a schematic diagram of a representation method of a computer room allocation strategy provided by an embodiment of the present disclosure;

[0024] Figure 4 is a structural diagram of a DAG provided by an embodiment of the present disclosure;

[0025] Figure 5 is a schematic diagram of a structure of a directed graph to be migrated provided by an embodiment of the present disclosure;

[0026] Figure 6 is a flowchart of another data storage control method provided by an embodiment of the present disclosure;

[0027] Figure 7is a structural block diagram of a data storage control device provided by an embodiment of the present disclosure;

[0028] Figure 8 It is a structural block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] The embodiments of the present disclosure are applicable to a capacity expansion scenario of a distributed storage system, so as to migrate data from an original computer room to a newly added computer room after capacity expansion. Figure 1 Schematic diagram of a distributed storage system provided by an embodiment of the present disclosure. Figure 1 As shown, the distributed storage system can be composed of three computer rooms A1, A2 and A3. Computer rooms A1 and A2 are the original computer rooms before horizontal expansion, and computer room A3 is the newly added computer room after horizontal expansion. Therefore, part of the data in the original computer rooms A1 and A2 may need to be migrated to the newly added computer room A3.

[0031] However, in the prior art, data migration is usually performed flexibly according to business requirements and the capacity of the original computer room and the newly added computer room, which can solve the problem of insufficient computer room capacity. However, it may lead to a large cost for data migration.

[0032] In order to solve the above technical problems, the embodiment of the present disclosure constructs a computer room allocation model for the distributed storage system before migration. The computer room allocation model is used to output the corresponding migration evaluation result according to the computer room allocation strategy. The migration evaluation result indicates the overhead caused by the data migration processing from the original computer room to the target computer room. Therefore, based on the computer room allocation model, the target computer room allocation strategy corresponding to the minimum migration evaluation result can be determined. In this way, when the target computer room allocated by the target computer room allocation strategy is compared with the original computer room for data migration, the overhead during data migration can be minimized.

[0033] It should be noted that the distributed storage system of the embodiment of the present disclosure may be any system with a storage function, and the storage object of the distributed storage system may be of any type. For example, when the storage object is a file, the distributed storage system may be HDFS (Hadoop distributed file system), CephFS (CephFile System, Ceph file system), GlusterFS (Gluster file system, Gluster file system), etc. The embodiment of the present disclosure may be described later using HDFS as an example, and the principles of other distributed storage systems are similar.

[0034] The following specific embodiments are used to describe in detail the technical solutions of the embodiments of the present disclosure and how the technical solutions of the present disclosure solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present disclosure will be described below in conjunction with the accompanying drawings.

[0035] Figure 2 is a flowchart of a data storage control method provided by an embodiment of the present disclosure. Figure 2 As shown, the data storage control method includes:

[0036] S201: Based on the dependency relationship between the storage objects in the distributed storage system, a computer room allocation model of the distributed storage system is constructed. The computer room allocation model is used to output the corresponding migration evaluation result according to the computer room allocation strategy. The computer room allocation strategy is used to indicate the target computer room allocated to the storage objects in the distributed storage system. The migration evaluation result is used to indicate the overhead caused by the data migration processing of migrating the storage from the original computer room to the target computer room.

[0037] Among them, the migration evaluation result may include the overhead introduced by the data change processing in any dimension, including at least one of the following: the migration cost corresponding to the distributed storage system, and the cross-computer room access volume corresponding to the computer room allocation according to the computer room allocation strategy. The migration cost includes: the total cost required to migrate the storage object in the distributed storage system from the original computer room corresponding to the storage object to the target computer room corresponding to the storage object in the computer room allocation strategy. In this way, minimizing the migration evaluation result means minimizing the migration cost and the cross-computer room access volume. Therefore, the computer room allocation strategy corresponding to the minimized migration evaluation result can ensure that the interdependent storage objects are in one computer room as much as possible, avoid traffic forwarding between different computer rooms, that is, reduce cross-computer room access. It helps to reduce access time, improve access efficiency, and reduce migration costs.

[0038] The above-mentioned interdependent storage objects mainly refer to the computing tasks that depend on the data path and the data path. This means that the computing task needs to read data from the data path or write data to the data path. When the data path and the computing tasks that depend on the data path are not in the same computer room, the computing tasks will call the data path across computer rooms, resulting in a cross-computer room process.

[0039] The above migration cost can also be understood as the migration cost required when switching the original computer room allocation strategy to the computer room allocation strategy. The cross-computer room access volume includes the access volume between at least two storage objects located in different computer rooms but interdependent. The original computer room allocation strategy can be understood as the original computer room used to indicate the corresponding storage objects. The above target computer room can be understood as the computer room corresponding to each storage object indicated in the computer room allocation strategy. The original computer room is the current computer room corresponding to the storage object when constructing the above computer room allocation model.

[0040] The storage objects may include but are not limited to: data paths and computing tasks. A data path is used to represent a path for storing data and uniquely represents a data table, for example, a Hive data table or an HDFS path.

[0041] The computer room allocation strategy can be used to indicate the target computer room assigned to each storage object in the distributed storage system. For example, the distributed storage system includes three computer rooms A1, A2 and A3, which are used to store five storage objects: B1, B2, B3, B4 and B5. In this case, the computer room allocation strategy can be: the target computer rooms corresponding to the storage objects B1, B2, B3, B4 and B5 are A1, A2, A3, A1 and A2 respectively.

[0042] The above-mentioned computer room allocation strategy can be represented by a set of numerical values. Figure 3 Schematic diagram of a representation method of a computer room allocation strategy provided by an embodiment of the present disclosure. Figure 3 As shown in the figure, a set of values ​​in 3 rows and 5 columns is used to represent the correspondence between 3 storage objects and 5 computer rooms. The computer room corresponding to each storage object is the target computer room assigned to the storage object according to the computer room allocation strategy. The i-th row and j-th column are used to indicate whether the i-th storage object is assigned to the j-th computer room, that is, whether the j-th computer room is the target computer room of the i-th storage object. Figure 3It can be seen that only the first column in the first row is 1, which means that the first storage object B1 corresponds to the first computer room A1, only the second column in the second row is 1, which means that the second storage object B2 corresponds to the second computer room A2, only the third column in the third row is 1, which means that the third storage object B3 corresponds to the third computer room A3, only the first column in the fourth row is 1, which means that the fourth storage object B4 corresponds to the first computer room A1, and only the second column in the fifth row is 1, which means that the fifth storage object B5 corresponds to the second computer room A2.

[0043] It is understandable that the above-mentioned computer room allocation strategy can be expressed in the form of an array or a matrix, and the embodiments of the present disclosure do not limit it.

[0044] The embodiment of the present disclosure can construct a computer room allocation model of the distributed storage system through the structure of the distributed storage system and the historical access situation. There are two implementation methods for constructing the computer room allocation model.

[0045] In the first implementation of constructing a computer room allocation model, the relevant information of the distributed storage system can be first determined; then, some relevant information of the above distributed storage system can be used as parameters of the migration cost calculation model and the parameters of the cross-computer room traffic calculation model respectively; finally, the migration cost calculation model and the cross-computer room traffic calculation model are merged into a computer room allocation model.

[0046] In the second implementation of constructing the computer room allocation model, the construction of the computer room allocation model can be achieved by constructing a directed acyclic graph. Specifically, first, according to the dependency relationship between the data path and the computing task, a DAG (Directed Acyclic Graph) is constructed, and each node of the DAG includes a data path and a computing task. When the data path and the computing task are connected by an edge, the first edge weight of the edge is used to represent the historical access volume of the computing task to the data path; then, the computer room allocation model is constructed through the DAG.

[0047] Each node in the DAG represents a storage object in the distributed storage system, that is, a node is a data path or a computing task, so the number of nodes in the DAG is consistent with the number of storage objects in the distributed storage system. When a computing task accesses a data path, the computing task and the data path are connected by an edge between the corresponding nodes in the DAG, and the first edge weight of the edge represents the historical access volume of the computing task to the data path.

[0048] Figure 4 is a schematic diagram of a DAG structure provided by an embodiment of the present disclosure. Figure 4As shown, there are 5 data path nodes: C1, C2, C3, C4 and C5, and 5 computing task nodes: D1, D2, D3, D4, D5 and D6. The historical access volume of computing task D1 to data path C1 is E1, the historical access volume of computing task D2 to data paths C1 and C2 is E2 and E3 respectively, the historical access volume of computing task D3 to data paths C2 and C3 is E4 and E6 respectively, the historical access volume of computing task D4 to data paths C2 and C4 is E5 and E7 respectively, the historical access volume of computing task D5 to data paths C3 and C5 is E8 and E10 respectively, and the historical access volume of computing task D6 to data paths C3 and C5 is E9 and E11 respectively.

[0049] It should be noted that the computing task is used to read data from the data path in the upstream node to perform calculations on it, and write the calculated result data to the data path in the downstream node. Therefore, the amount of access to the data path in the upstream node by the computing task can be understood as the amount of data read from the data path by the computing task, and the amount of access to the data path in the downstream node by the computing task can be understood as the amount of data written to the data path by the computing task.

[0050] From the above Figure 4 As can be seen from the DAG shown, DAG can clearly represent the dependencies between various data paths and computing tasks in the distributed storage system, thereby facilitating a faster and more accurate construction of the computer room allocation model.

[0051] In the process of constructing the computer room allocation model through the above DAG, the number of nodes where the data paths in the DAG are located, the number of nodes where the computing tasks are located, the first edge weight of each edge, etc. can be used as parameters of the computer room allocation model.

[0052] In the disclosed embodiment, the parameters required by the computer room allocation model may include both constants and variables. The constants may include: the index set Njob of the computing task, the index set Ndata of the data path, the node combination V, the edge set F, the edge weight set E, the number of computer rooms K, the balance control constant, the original computer room index Xi(init) of the i-th storage object, the size Si of the i-th storage object, the capacity Cj of the j-th computer room, and the migration cost of a single storage object. The migration cost of a single storage object includes the migration cost Pjob of a single computing task and the migration cost Pdata of a single data path. The above variables include: computer room allocation strategy variables Xij, storage object migration variables MGRi, and computer room similarities and differences variables Yii'.

[0053] The index set Njob of computing tasks includes the indexes of all computing tasks in the distributed storage system, and the index i of any computing task point belongs to Njob.

[0054] The index set Ndata of the data paths includes the indexes of all the data paths in the distributed storage system, and the index i' of any data path belongs to Ndata.

[0055] The node set V includes all nodes in the DAG, corresponding to the storage objects in the distributed storage system. The node set V includes the nodes where the computing tasks are located and the nodes where the data paths are located. The node set V is the union of the index set Njob of the computing tasks and the index set Ndata of the data paths.

[0056] The edge set F includes all the edges in the DAG. The edges are used to represent the dependency between computing tasks and data paths in the DAG. When a computing task has accessed a data path in the past, there is an edge connecting the computing task and the data path.<i,i’> , where i belongs to the index set Njob of computing tasks, i' belongs to the index set Ndata of data paths, and is used to represent the edge between the i-th computing task and the i'th data path.

[0057] The edge weight set E includes the first edge weights of all edges in the DAG. The first edge weight between the i-th computing task and the i'th data path is denoted by Eii'.

[0058] The number of computer rooms K is the total number of all computer rooms after the expansion, including the original computer rooms before the expansion and the newly added computer rooms during the expansion.

[0059] The balance control constant is used to control the data distribution of all computer rooms after migration to be relatively balanced, to avoid excessive or insufficient data in some computer rooms. The balance control constant can include two βmax and βmin, which are used to control the data volume of each computer room after migration to satisfy the following formula (1):

[0060]

[0061] It can be understood that the above Num is the number of storage objects in the distributed storage system. It can be understood as the number of storage objects corresponding to each computer room when all storage objects in the distributed storage system are evenly distributed to K computer rooms. Of course, since the storage space occupied by computing tasks is usually small, we can ignore whether the computing tasks are balanced in each computer room and only focus on whether the data paths are balanced. At this time, the above Num is set to the number of data paths in the distributed storage system.

[0062] The original computer room index Xi(init) of the i-th storage object is used to represent the index of the original computer room of the i-th storage object before this data migration.

[0063] The size Si of the i-th storage object is used to represent the data amount of the i-th storage object.

[0064] The capacity Cj of the jth computer room is used to represent the maximum amount of data that can be stored in the jth computer room.

[0065] The migration cost of a single storage object is used to represent the cost required to migrate a storage object, which can be measured from multiple aspects such as time, computational complexity, and occupied resources. Considering the different migration costs of data paths and computing tasks, the migration cost of a single storage object can be divided into the migration cost of a single data path Pdata and the migration cost of a single computing task Pjob.

[0066] The computer room allocation strategy variable Xij is used to indicate whether the i-th storage object is allocated to the j-th computer room. It is a Boolean variable with a value of 1 or 0. When the computer room allocation strategy variable Xij=1, it means that the i-th storage object is allocated to the j-th computer room. When the computer room allocation strategy variable Xij=0, it means that the i-th storage object is not allocated to the j-th computer room. Therefore, the computer room allocation strategy can be represented by multiple computer room allocation strategy variables Xij. The number of computer room allocation strategy variables Xij included in the storage computer room is the product of the number of storage objects in the distributed storage system and the number of computer rooms K after expansion.

[0067] The storage object migration variable MGRi is used to indicate whether the i-th storage object is migrated. It is a Boolean variable with a value of 1 or 0. When the storage object migration variable MGRi is 1, it means that the i-th storage object is migrated. When the storage object migration variable MGRi is 1, it means that the i-th storage object is not migrated.

[0068] The computer room difference variable Yii' is used to indicate whether the i-th storage object and the i'th storage object are assigned to the same target computer room. It is a Boolean variable with a value of 1 or 0. When the computer room difference variable Yii' is 1, it means that the i-th storage object and the i'th storage object are not assigned to the same target computer room. When the computer room difference variable Yii' is 0, it means that the i-th storage object and the i'th storage object are assigned to the same target computer room.

[0069] After setting the above variables and constants, the above constants and variables can be used as parameters of the computer room allocation model to obtain the computer room allocation model. The computer room allocation model includes migration cost and cross-computer room access volume.

[0070] Among them, the migration cost included in the migration assessment result is related to the migration cost of a single storage object and the number of storage objects to be migrated. The number of storage objects to be migrated is related to the original computer room and the computer room allocation strategy. It can be understood that the computer room allocation strategy indicates the target computer room assigned to each storage object, so that the same storage object in the original computer room and the target computer room corresponding to the computer room allocation strategy can be compared. When the two are different, it can be determined that the storage object is a storage object to be migrated. Conversely, when the two are the same, it can be determined that the storage object is not a storage object to be migrated.

[0071] For example, refer to Figure 3 In the computer room allocation strategy shown, the target computer rooms corresponding to the first storage objects B1, B2, B3, B4 and B5 are: the first computer room A1, the second computer room A2, the third computer room A3, the first computer room A1 and the second computer room A2. When the original computer rooms corresponding to the first storage objects B1, B2, B3, B4 and B5 are: the first computer room A1, A2, A2, A1 and A2, the storage objects to be migrated include B3.

[0072] It should be noted that the migration cost included in the migration assessment result is positively correlated with the number of storage objects to be migrated and the migration cost of a single storage object. That is, as the number of storage objects to be migrated increases and / or the migration cost of a single storage object increases, the migration cost included in the computer room allocation model also increases.

[0073] The cross-computer room access volume included in the migration assessment result is related to the first edge weight corresponding to the cross-computer room node group in the directed acyclic graph. The cross-computer room node group includes the nodes where the computing tasks and data paths with edge connections are located in the directed acyclic graph, and the computing tasks and data paths of the cross-computer room node group correspond to different target computer rooms in the computer room allocation strategy.

[0074] The cross-computer room access volume included in the migration evaluation result is positively correlated with the first edge weight, that is, as the first edge weight increases and / or the number of cross-computer room node groups increases, the cross-computer room access volume increases.

[0075] The above cross-computer room node group is used to indicate that computing tasks and data paths with dependencies are located in different target computer rooms, so there is cross-computer room access. Specifically, you can first obtain any edge from the DAG to determine the computing tasks and data paths connected by the edge, and then determine the target computer rooms corresponding to the computing tasks and data paths in the computer room allocation strategy. For example, when the target computer rooms of the computing tasks and data paths are computer rooms A1 and A2 respectively, then the computing tasks and data paths constitute cross-computer room access.

[0076] Based on the above analysis process, the specific formula of the above computer room allocation model is given below. The computer room allocation model Func can be expressed as the following formula (2):

[0077] Func=Func1+Func2 (2)

[0078] Among them, Func1 and Func2 are the cross-computer room traffic, and Func2 is the migration cost. Func1 can refer to the following formula (3), and Func2 can refer to the following formula (4).

[0079]

[0080]

[0081] For the above formula (3), since Eii' represents the first edge weight between the ith computing task and the ith data path, Yii' is used to represent whether the ith computing task and the ith data path are assigned to the same target computer room, so the product of the two represents the cross-computer room access volume between the ith computing task and the ith data path when they are not assigned to the same target computer room. Furthermore, Func1 is used to represent the cross-computer room access volume corresponding to the computer room allocation according to a computer room allocation strategy.

[0082] The above Yii' can be determined according to Xij in the computer room allocation strategy. When Xij and Xi'j, both of which have values ​​of 1, have different j, Yii' is 1, otherwise, Yii' is 0. Therefore, the above Func1 can be understood as a function of the computer room allocation strategy and the first edge weight.

[0083] For the above formula (4), since MGRi is used to indicate whether the i-th storage object is migrated, It is used to indicate the number of computing tasks that are migrated among all computing tasks. It is used to indicate the number of data paths that are migrated among all storage paths. Used to represent the migration cost required for the migration computing task. Used to represent the migration cost required to migrate the data path.

[0084] It is understandable that the above MGRi can be calculated by the following formula:

[0085] MGRi=1-X i,Xi(init) (5)

[0086] For the above formula (5), since Xi(init) is the original room index of the i-th storage object, X i,Xi(init)It is used to indicate whether the i-th storage object is allocated to the original computer room of the storage object. It can be seen that MGRi can be determined according to the original computer room and the computer room allocation strategy, so Func2 is a function of the original computer room and the computer room allocation strategy.

[0087] From the above description, we can know that Func1 and Func2 are functions of the computer room allocation strategy variable Xij, and then the computer room allocation model is a function of Xij. Therefore, after optimizing the computer room allocation model, we can get Xij, which indicates whether each storage object is allocated to each computer room, to form the target computer room allocation strategy. In other words, when the output of the computer room allocation model is the minimum, the corresponding computer room allocation strategy Xij is the target computer room allocation strategy.

[0088] S202: calling the computer room allocation model, and based on the migration evaluation results corresponding to each computer room allocation strategy, determining the computer room allocation strategy corresponding to the minimum migration evaluation result as the target computer room allocation strategy.

[0089] Specifically, the computer room allocation model can be optimized and solved to obtain the target computer room allocation strategy, and the optimization goal is to minimize the above migration evaluation result.

[0090] The target computer room allocation strategy is the computer room allocation strategy corresponding to the computer room allocation model output when the computer room allocation model output is optimal. The optimal here may include but is not limited to: the migration evaluation result output by the computer room allocation model is the smallest, and / or is less than or equal to a preset threshold. In other words, the target computer room allocation strategy is the computer room allocation strategy corresponding to the cross-computer room traffic and migration cost when they are as small or minimum as possible.

[0091] Similar to the aforementioned computer room allocation strategy, the target computer room allocation strategy here is used to indicate the optimal target computer room corresponding to each storage object.

[0092] In the embodiment of the present disclosure, the mixed-integer linear programming (MILP) technology is used to optimize and solve the above-mentioned computer room allocation model. Specifically, a mathematical programming solver can be called to optimize and solve the computer room allocation model, for example, Gurobi, Cplex, etc. The embodiment of the present disclosure does not limit the solver.

[0093] It can be understood that since the migration assessment results output by the computer room allocation model include cross-computer room traffic, the process of optimizing and solving the above-mentioned computer room allocation model includes the process of minimizing the cross-computer room traffic. Therefore, it is equivalent to the process of clustering storage objects according to dependencies, so as to allocate interdependent computing tasks and storage paths to the same computer room as much as possible, so as to reduce cross-computer room access.

[0094] In some implementations, in order to improve the effect of computer room allocation, the computer room allocation strategy corresponding to the minimum migration evaluation result can be determined based on the computer room allocation model under preset constraints as the target computer room allocation strategy. The preset constraints here include: each storage object is allocated to a target computer room, the total amount of data of the storage objects allocated to each target computer room is less than or equal to the capacity of the target computer room, and the number of storage objects allocated to each target computer room is within a preset number range, and the preset number range includes the value range of the ratio of the number of storage objects to the number of computer rooms in the distributed storage system.

[0095] The constraint condition that each storage object is assigned to a computer room can be understood as follows: for any i-th storage object, there is only one j with Xij=1, which means that the i-th storage object is assigned to the j-th computer room; and the remaining j all have Xij=0. The above logic can also be expressed by the following formula:

[0096]

[0097] Among them, Xij is whether the i-th storage object is allocated to the j-th computer room. When Xij=1, it means that the i-th storage object is allocated to the j-th computer room, and the j-th computer room is the target computer room of the i-th storage object with the computer room allocation strategy.

[0098] The constraint that the total amount of data of the storage objects allocated to each computer room is less than or equal to the capacity of the computer room can be understood as follows: for any j-th computer room, the sum of the amount of data of the storage objects allocated to the computer room is less than or equal to the capacity Cj of the computer room, which can be specifically expressed by the following formula:

[0099]

[0100] Where V is the node set in DAG, and Si is used to represent the size of the i-th storage object.

[0101] The constraint condition that the number of storage objects allocated to each computer room is within a preset number range can be understood as: for any j-th computer room, the number of storage objects allocated to the computer room is within a preset number range. Specifically, it can be expressed by the above formula (1).

[0102] In summary, the disclosed embodiments can control each storage object to be stored in one computer room, avoid data of the same storage object being distributed in different computer rooms, and improve query efficiency and management efficiency. It is also possible to control the total amount of data of storage objects in each computer room to be less than its capacity to avoid data loss. It is also possible to control all storage objects of the distributed storage system to be evenly distributed in different computer rooms, which helps to improve balance.

[0103] In the embodiment of the present disclosure, when Xij=1 represents that the i-th storage object is assigned to the j-th computer room, the j-th computer room is the target computer room of the i-th storage object. At this time, the following constraints are also met:

[0104] Yii'≥Xij-Xi'j (8)

[0105] The above constraint condition (8) can be understood in two cases: Yii'=1 and Yii'=0.

[0106] When Yii'=1, the i-th and i'-th storage objects are assigned to different target computer rooms. At this time, for the target computer room of the i-th storage object, Xij=1, Xi'j=0, so Yii'=Xij-Xi'j; for the target computer room of the i'-th storage object, Xij=0, Xi'j=1, so Yii'>Xij-Xi'j.

[0107] When Yii'=0, the i-th and i'-th storage objects are assigned to the same target computer room. At this time, for the same target computer room, Xij=Xi'j=1, Yii'=Xij-Xi'j; for computer rooms other than the same target computer room, Xij=Xi'j=0, so Yii'=Xij-Xi'j.

[0108] S203: Migrate data in the distributed storage system according to the target computer room allocation strategy.

[0109] Specifically, the target computer room of each storage object in the target computer room allocation strategy can be compared with the original computer room corresponding to each storage object indicated by the aforementioned original computer room allocation strategy to obtain different storage objects between the original computer room and the target computer room as storage objects to be migrated, so as to migrate them from the original computer room to the target computer room.

[0110] In some implementations, after obtaining one or more storage objects to be migrated, the storage objects can be migrated in any order. However, this approach does not take into account the dependencies between the storage objects, which may result in access errors during the migration process. Therefore, the migration order can be determined according to the dependencies between the storage objects to perform data migration. Specifically, the migration order can be determined based on a directed acyclic graph, and then data migration can be performed on the distributed storage system based on the target computer room allocation strategy, the original computer room, and the migration order.

[0111] The migration order is determined according to the dependency relationship in the directed acyclic graph. Specifically, the migration order of the dependent storage objects is first, and the migration order of the storage objects that depend on other storage objects is last. Figure 4As shown, storage object D3 depends on storage object C2, so the migration order of storage object D3 is after the migration order of storage object C2. Of course, if there is no edge connection between two storage objects, their migration order is not restricted.

[0112] After determining the migration order, the storage objects to be migrated can be determined according to the target computer room allocation strategy and the original computer room, so as to perform data migration according to the migration order. In this way, it is possible to avoid the dependent storage objects being migrated first, which may lead to access errors during the migration process, and help ensure correct access during the migration process.

[0113] In the disclosed embodiment, the target computer room corresponding to each storage object can be determined from the target computer room allocation strategy to migrate all storage objects that are different between the original computer room and the target computer room. However, considering some special scenarios, some storage objects that are different between the original computer room and the target computer room can also be migrated. Specifically, the upstream node of the target node in the DAG and the downstream node of the target node are connected to delete the target node from the DAG to obtain a directed graph to be migrated, so as to determine the migration order according to the directed graph to be migrated.

[0114] The target node may be a node where a storage object that does not need to be migrated is located according to the actual scenario. It may include one of the following: a node where a computing task is located, a node where a data path is located. For example, in actual applications, considering that computing tasks occupy a small space and the migration process is more complicated, it may be considered not to migrate computing tasks but only to migrate data paths. Thus, the node where the computing task is located may be used as the target node. Of course, there are some other special scenarios where the node where the data path is located may be used as the target node so as not to migrate the data path.

[0115] The upstream node of the target node is the node that the target node depends on, and the downstream node of the target node is the node that depends on the target node. When the target node has both upstream nodes and downstream nodes, after deleting the target node, its upstream nodes and downstream nodes are connected. When the target node does not have upstream nodes and / or downstream nodes, after deleting the target node, the above connection does not need to be performed.

[0116] Figure 5 is a schematic diagram of a structure of a directed graph to be migrated provided by an embodiment of the present disclosure, Figure 5 Yes Figure 4 The directed graph to be migrated is obtained by deleting the nodes where the computing tasks are located in the DAG shown in FIG. Since there are no nodes that depend on the nodes where the computing tasks D1 and D2 are located, Figure 4 The nodes where the computing tasks D1 and D2 are located are directly deleted, and the following is obtained: Figure 5The node where the isolated data path C1 is located. Since there is a node that depends on the node where the computing task D3 is located, Figure 4 After deleting the node where computing task D3 is located, it is necessary to connect data path C3 that depends on computing task D3 and data path C2 that computing task D3 depends on. Similarly, after deleting computing task D4, connect data path C2 and data path C4, and after deleting computing tasks D5 and D6, connect data paths C3 and C5.

[0117] After obtaining the directed graph to be migrated, the migration order can be determined according to the directed graph to be migrated. Similarly, the migration order is determined by the dependency relationship between the nodes in the directed graph to be migrated, which will not be described in detail here.

[0118] The disclosed embodiment can flexibly perform data migration by deleting the target node, thereby improving the flexibility of migration to meet specific requirements of specific scenarios.

[0119] It can be seen from the generation process of the directed graph to be migrated that after deleting the target node, there may or may not be a ring structure. When there is no ring structure in the directed graph to be migrated, the migration order can be determined according to the dependency relationship between the nodes in the directed graph to be migrated to perform data migration. However, when there is a ring structure in the directed graph to be migrated, it is necessary to open the ring structure. In other words, one of the edges in the ring structure can be disconnected, and the disconnected edge can be any edge. However, when the disconnected edge is an edge with a relatively strong dependency relationship, it may cause the dependent storage object to migrate later, resulting in more access errors during the data migration process. Therefore, the embodiment of the present disclosure considers disconnecting the edges with relatively weak associations in the ring structure to avoid the impact on access during the data migration process as much as possible.

[0120] Specifically, if the directed graph to be migrated includes a ring structure, the second edge weights of the edges of the upstream node and the downstream node in the directed graph to be migrated are determined according to the first edge weights between the target node and the upstream node and the downstream node in the directed acyclic graph respectively; then the edge with the smallest second edge weight in the ring structure is disconnected to obtain the directed acyclic graph to be migrated; finally, the migration order is determined according to the dependency relationship between the nodes in the directed acyclic graph to be migrated.

[0121] Among them, the second edge weight of the edge between the upstream node and the downstream node in the directed graph to be migrated is positively correlated with the first edge weight between the target node and the upstream node and the downstream node in the directed acyclic graph, including but not limited to: sum, or weighted sum, or calculated in any other way.

[0122] For example, when deleting Figure 4When the target node D3 is shown, the first edge weight E4 between the upstream node C2 and the target node D3, and the first edge weight E6 between the target node D3 and the downstream node C3 are added together to obtain Figure 5 The second edge weight between midpoints C2 and C3 is E4+E6.

[0123] Similarly, when deleting Figure 4 When the target node D4 is shown, the first edge weight E5 between the upstream node C2 and the target node D4, and the first edge weight E7 between the target node D4 and the downstream node C4 are added together to obtain Figure 5 The second edge weight between midpoints C2 and C4 is E5+E7.

[0124] On Deletion Figure 4 When the target node D5 is shown, the first edge weight E8 between the upstream node C3 and the target node D5, and the first edge weight E10 between the target node D5 and the downstream node C5 are added to obtain a partial second edge weight E8+E10 between the nodes C3 and C5.

[0125] On Deletion Figure 4 When the target node D6 is shown, the first edge weight E9 between the upstream node C3 and the target node D6, and the first edge weight E11 between the target node D6 and the downstream node C5 are added to obtain another part of the second edge weight E9+E11 between the nodes C3 and C5.

[0126] Finally, after deleting D5 and D6, the weight of the second edge between nodes C3 and C5 is Figure 5 E8+E9+E10+E11 shown.

[0127] Reference Figure 4 As shown in Figure 2, if computing task D6 depends on data path C4, and the historical number of accesses to the data path by the computing task is E12, then when the target node D6 is deleted, we can get Figure 5 The connecting edge between C4 and C5 shown in the figure corresponds to the second edge weight E11 + E12. At this time, the directed graph to be migrated has a ring structure, and it is necessary to disconnect the edge with the smallest second edge weight from the ring structure according to the second edge weights E4 + E6, E5 + E7, E + E9 + E10 + E11 and E11 + E12. For example, when the second edge weight E11 + E12 is the smallest, disconnect its corresponding edge.

[0128] The directed graph to be migrated can be understood as a maximum weight spanning tree, in which the weights between connected storage objects are relatively large. After determining the migration order according to the directed graph to be migrated, data migration can be performed. The data migration process can include three processes: data preparation, copy migration, and effect follow-up.

[0129] Data preparation mainly involves obtaining the original computer room of the storage object, the storage space occupied by the storage object, cross-computer room access conditions, etc., in order to evaluate the amount of resources expected to be consumed by data migration.

[0130] The data migration process is mainly based on the data room migration capability of the data copy. The implementation principle is to scan the copy distribution of all file blocks in all directories of the distributed storage system to determine whether each copy complies with the target data room allocation policy. If the copy complies with the target data room allocation policy, then there is no need to copy and delete the copy. Otherwise, the missing copy is copied through the copy repair process, and the redundant copy that does not comply with the target data room allocation policy is deleted using the corresponding copy deletion mechanism. In this way, the purpose of directory-level data room redistribution can be achieved.

[0131] The above basic process is used to evaluate the overall data migration effect by continuously monitoring parameters such as cross-data center bandwidth consumption and migration speed during the data migration process.

[0132] Figure 6 is a flowchart of another data storage control method provided by an embodiment of the present disclosure. Figure 6 As shown, the above data storage control method may include:

[0133] S301: Construct a directed acyclic graph based on the dependency between the data path and the computing task. Each node of the directed acyclic graph includes a data path and a computing task. When the data path and the computing task are connected by an edge, the first edge weight of the edge is used to represent the historical access volume of the computing task to the data path.

[0134] S302: constructing a computer room allocation model through a directed acyclic graph, the computer room allocation model is used to output corresponding migration evaluation results according to a computer room allocation strategy, and the computer room allocation strategy is used to indicate the overhead caused by data migration processing from the original computer room to the target computer room.

[0135] S303: Under preset constraints, the computer room allocation model is optimized and solved to obtain a target computer room allocation strategy. The preset constraints include: each storage object is allocated to a target computer room, the total data volume of the storage objects allocated to each target computer room is less than or equal to the capacity of the target computer room, and the number of storage objects allocated to each target computer room is within a preset number range, and the preset number range includes the value range of the ratio of the number of storage objects to the number of computer rooms in the distributed storage system.

[0136] S304: Connecting the upstream node of the target node and the downstream node of the target node in the directed acyclic graph to delete the target node from the directed acyclic graph, thereby obtaining a directed graph to be migrated.

[0137] S305: If the directed graph to be migrated includes a ring structure, determine the second edge weights of the edges between the upstream node and the downstream node in the directed graph to be migrated according to the first edge weights between the target node and the upstream node and the downstream node in the directed acyclic graph.

[0138] S306: Disconnect the edge with the smallest second edge weight in the ring structure to obtain a directed acyclic graph to be migrated.

[0139] S307: Determine a migration order according to the dependency relationship between the nodes in the directed acyclic graph to be migrated.

[0140] S308: Migrate data in the distributed storage system according to the target computer room allocation strategy, the original computer room, and the migration sequence.

[0141] The order of the above steps S301 to S308 can be flexibly adjusted on a mutually independent basis, and the corresponding descriptions can refer to the above Figure 2 The embodiments of the method shown will not be described in detail here.

[0142] Corresponding to the data storage control method of the above embodiment, Figure 7 is a structural block diagram of a data storage control device provided by an embodiment of the present disclosure. For the convenience of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 7 , the data storage control device 400 comprises:

[0143] Function construction module 401 is used to construct a computer room allocation model of the distributed storage system according to the dependency relationship between the storage objects in the distributed storage system. The computer room allocation model is used to output the corresponding migration evaluation result according to the computer room allocation strategy. The computer room allocation strategy is used to indicate the target computer room allocated to the storage objects in the distributed storage system. The migration evaluation result is used to indicate the overhead caused by the data migration processing of migrating the storage from the original computer room to the target computer room.

[0144] The target strategy determination module 402 is used to call the computer room allocation model and determine the computer room allocation strategy corresponding to the minimum migration evaluation result as the target computer room allocation strategy based on the migration evaluation results corresponding to each computer room allocation strategy.

[0145] The data migration module 403 is used to migrate data in the distributed storage system according to the target computer room allocation strategy.

[0146] Optionally, the migration evaluation result includes: the migration cost corresponding to the distributed storage system, and the cross-computer room access volume corresponding to the computer room allocation according to the computer room allocation strategy. The migration cost includes: the total cost required to migrate the storage object in the distributed storage system from the original computer room corresponding to the storage object to the target computer room corresponding to the storage object in the computer room allocation strategy.

[0147] Optionally, the storage object includes a data path and a computing task, and the function construction module 401 is further used to:

[0148] According to the dependency relationship between the data path and the computing task, a directed acyclic graph is constructed, wherein each node of the directed acyclic graph includes the data path and the computing task, and when the data path and the computing task are connected by an edge, a first edge weight of the edge is used to represent a historical access volume of the computing task to the data path;

[0149] The computer room allocation model is constructed through the directed acyclic graph.

[0150] Optionally, the migration cost is related to the migration cost of a single storage object and the number of storage objects to be migrated, and the number of storage objects to be migrated is related to the original computer room and the computer room allocation strategy;

[0151] The cross-computer room access volume is related to the first edge weight corresponding to the cross-computer room node group in the directed acyclic graph, the cross-computer room node group includes the nodes where the computing tasks and the nodes where the data paths are located, which are connected by edges in the directed acyclic graph, and the computing tasks and the data paths of the cross-computer room node group correspond to different target computer rooms in the computer room allocation strategy.

[0152] Optionally, the target strategy determination module 402 is further configured to:

[0153] Under preset constraints, the computer room allocation strategy corresponding to the minimum migration evaluation result is determined based on the computer room allocation model as the target computer room allocation strategy, and the preset constraints include: each storage object is allocated to a target computer room, the total data volume of the storage objects allocated to each target computer room is less than or equal to the capacity of the target computer room, and the number of storage objects allocated to each target computer room is within a preset number range, and the preset number range includes the value range of the ratio of the number of storage objects to the number of computer rooms in the distributed storage system.

[0154] Optionally, the data migration module 403 is further used to:

[0155] Determining a migration order according to the directed acyclic graph;

[0156] Data migration is performed on the distributed storage system according to the target computer room allocation strategy, the original computer room and the migration sequence.

[0157] Optionally, the data migration module 403 is further used to:

[0158] Connecting an upstream node of a target node in the directed acyclic graph and a downstream node of the target node to delete the target node from the directed acyclic graph, thereby obtaining a directed graph to be migrated;

[0159] A migration order is determined according to the directed graph to be migrated.

[0160] Optionally, the data migration module 403 is further used to:

[0161] If the directed graph to be migrated includes a ring structure, determining the second edge weights of the edges of the upstream node and the downstream node in the directed graph to be migrated according to the first edge weights between the target node and the upstream node and the downstream node in the directed acyclic graph respectively;

[0162] Disconnect the second edge with the smallest weight in the ring structure to obtain a directed acyclic graph to be migrated;

[0163] The migration order is determined according to the dependency relationship between the nodes in the directed acyclic graph to be migrated.

[0164] Optionally, the target node includes one of the following: a node where the computing task is located, and a node where the data path is located.

[0165] The data storage control device provided in this embodiment can be used to perform the above Figure 2 The technical solution of the method embodiment shown has similar implementation principles and technical effects, and will not be repeated here in this embodiment.

[0166] Figure 8 6 is a block diagram of an electronic device provided by an embodiment of the present disclosure. The electronic device 600 includes a memory 602 and at least one processor 601 .

[0167] The memory 602 stores computer-executable instructions.

[0168] At least one processor 601 executes the computer execution instructions stored in the memory 602, so that the electronic device 600 implements the aforementioned Figure 2 The method in .

[0169] In addition, the electronic device may further include a receiver 603 and a transmitter 604, wherein the receiver 603 is used to receive information from other devices or equipment and forward it to the processor 601, and the transmitter 604 is used to send the information to other devices or equipment.

[0170] In a first example of the first aspect, an embodiment of the present disclosure provides a data storage control method, the method including:

[0171] According to the dependency relationship between the storage objects in the distributed storage system, a computer room allocation model of the distributed storage system is constructed, wherein the computer room allocation model is used to output a corresponding migration evaluation result according to a computer room allocation strategy, wherein the computer room allocation strategy is used to indicate a target computer room allocated to the storage objects in the distributed storage system, and the migration evaluation result is used to indicate the overhead caused by data migration processing from an original computer room to the target computer room;

[0172] Calling the computer room allocation model, based on the migration evaluation results corresponding to each computer room allocation strategy, determining the computer room allocation strategy corresponding to the minimum migration evaluation result as the target computer room allocation strategy;

[0173] According to the target computer room allocation strategy, data migration is performed on the distributed storage system.

[0174] Based on the first example of the first aspect, in the second example of the first aspect, the migration evaluation result includes: the migration cost corresponding to the distributed storage system, and the cross-computer room access volume corresponding to the computer room allocation according to the computer room allocation strategy, and the migration cost includes: the total cost required to migrate the storage object in the distributed storage system from the original computer room corresponding to the storage object to the target computer room corresponding to the storage object in the computer room allocation strategy.

[0175] Based on the second example of the first aspect, in a third example of the first aspect, the storage object includes a data path and a computing task, and constructing a computer room allocation model of the distributed storage system according to the dependency relationship between the storage objects in the distributed storage system includes:

[0176] According to the dependency relationship between the data path and the computing task, a directed acyclic graph is constructed, wherein each node of the directed acyclic graph includes the data path and the computing task, and when the data path and the computing task are connected by an edge, a first edge weight of the edge is used to represent a historical access volume of the computing task to the data path;

[0177] The computer room allocation model is constructed through the directed acyclic graph.

[0178] Based on the third example of the first aspect, in the fourth example of the first aspect, the migration cost is related to the migration cost of a single storage object and the number of storage objects to be migrated, and the number of storage objects to be migrated is related to the original computer room and the computer room allocation strategy;

[0179] The cross-computer room access volume is related to the first edge weight corresponding to the cross-computer room node group in the directed acyclic graph, the cross-computer room node group includes the nodes where the computing tasks and the nodes where the data paths are located, which are connected by edges in the directed acyclic graph, and the computing tasks and the data paths of the cross-computer room node group correspond to different target computer rooms in the computer room allocation strategy.

[0180] Based on the first to fourth examples of the first aspect, in a fifth example of the first aspect, calling the computer room allocation model, and determining, based on the migration evaluation results corresponding to each computer room allocation strategy, the computer room allocation strategy corresponding to the minimum migration evaluation result as the target computer room allocation strategy, includes:

[0181] Under preset constraints, the computer room allocation strategy corresponding to the minimum migration evaluation result is determined based on the computer room allocation model as the target computer room allocation strategy, and the preset constraints include: each storage object is allocated to a target computer room, the total data volume of the storage objects allocated to each target computer room is less than or equal to the capacity of the target computer room, and the number of storage objects allocated to each target computer room is within a preset number range, and the preset number range includes the value range of the ratio of the number of storage objects to the number of computer rooms in the distributed storage system.

[0182] Based on the fourth example of the first aspect, in a sixth example of the first aspect, migrating data on the distributed storage system according to the target computer room allocation strategy includes:

[0183] Determining a migration order according to the directed acyclic graph;

[0184] Data migration is performed on the distributed storage system according to the target computer room allocation strategy, the original computer room and the migration sequence.

[0185] Based on the sixth example of the first aspect, in a seventh example of the first aspect, determining the migration order according to the directed acyclic graph includes:

[0186] Connecting an upstream node of a target node in the directed acyclic graph and a downstream node of the target node to delete the target node from the directed acyclic graph, thereby obtaining a directed graph to be migrated;

[0187] A migration order is determined according to the directed graph to be migrated.

[0188] Based on the seventh example of the first aspect, in an eighth example of the first aspect, determining the migration order according to the directed graph to be migrated includes:

[0189] If the directed graph to be migrated includes a ring structure, determining the second edge weights of the edges of the upstream node and the downstream node in the directed graph to be migrated according to the first edge weights between the target node and the upstream node and the downstream node in the directed acyclic graph respectively;

[0190] Disconnect the second edge with the smallest weight in the ring structure to obtain a directed acyclic graph to be migrated;

[0191] The migration order is determined according to the dependency relationship between the nodes in the directed acyclic graph to be migrated.

[0192] Based on the seventh example of the first aspect, in a ninth example of the first aspect, the target node includes one of the following: a node where the computing task is located, and a node where the data path is located.

[0193] In a first example of the second aspect, a data storage control device is provided, the device comprising:

[0194] A function construction module, used for constructing a computer room allocation model of the distributed storage system according to the dependency relationship between the storage objects in the distributed storage system, the computer room allocation model is used for outputting a corresponding migration evaluation result according to a computer room allocation strategy, the computer room allocation strategy is used for indicating a target computer room allocated to the storage objects in the distributed storage system, and the migration evaluation result is used for indicating the overhead caused by data migration processing from an original computer room to the target computer room;

[0195] A target strategy determination module is used to call the computer room allocation model, and based on the migration evaluation results corresponding to each computer room allocation strategy, determine the computer room allocation strategy corresponding to the minimum migration evaluation result as the target computer room allocation strategy;

[0196] The data migration module is used to migrate data of the distributed storage system according to the target computer room allocation strategy.

[0197] Based on the first example of the second aspect, in the second example of the second aspect, the migration evaluation result includes: the migration cost corresponding to the distributed storage system, and the cross-computer room access volume corresponding to the computer room allocation according to the computer room allocation strategy, and the migration cost includes: the total cost required to migrate the storage object in the distributed storage system from the original computer room corresponding to the storage object to the target computer room corresponding to the storage object in the computer room allocation strategy.

[0198] Based on the second example of the second aspect, in a third example of the second aspect, the storage object includes a data path and a computing task, and the function building module is further used to:

[0199] According to the dependency relationship between the data path and the computing task, a directed acyclic graph is constructed, wherein each node of the directed acyclic graph includes the data path and the computing task, and when the data path and the computing task are connected by an edge, a first edge weight of the edge is used to represent a historical access volume of the computing task to the data path;

[0200] The computer room allocation model is constructed through the directed acyclic graph.

[0201] Based on the third example of the second aspect, in the fourth example of the second aspect, the migration cost is related to the migration cost of a single storage object and the number of storage objects to be migrated, and the number of storage objects to be migrated is related to the original computer room and the computer room allocation strategy;

[0202] The cross-computer room access volume is related to the first edge weight corresponding to the cross-computer room node group in the directed acyclic graph, the cross-computer room node group includes the nodes where the computing tasks and the nodes where the data paths are located, which are connected by edges in the directed acyclic graph, and the computing tasks and the data paths of the cross-computer room node group correspond to different target computer rooms in the computer room allocation strategy.

[0203] Based on the first to fourth examples of the second aspect, in a fifth example of the second aspect, the target strategy determination module is further used to:

[0204] Under preset constraints, the computer room allocation strategy corresponding to the minimum migration evaluation result is determined based on the computer room allocation model as the target computer room allocation strategy, and the preset constraints include: each storage object is allocated to a target computer room, the total data volume of the storage objects allocated to each target computer room is less than or equal to the capacity of the target computer room, and the number of storage objects allocated to each target computer room is within a preset number range, and the preset number range includes the value range of the ratio of the number of storage objects to the number of computer rooms in the distributed storage system.

[0205] Based on the fourth example of the second aspect, in a sixth example of the second aspect, the data migration module is further used to:

[0206] Determining a migration order according to the directed acyclic graph;

[0207] Data migration is performed on the distributed storage system according to the target computer room allocation strategy, the original computer room and the migration sequence.

[0208] Based on the sixth example of the second aspect, in a seventh example of the second aspect, the data migration module is further used to:

[0209] Connecting an upstream node of a target node in the directed acyclic graph and a downstream node of the target node to delete the target node from the directed acyclic graph, thereby obtaining a directed graph to be migrated;

[0210] A migration order is determined according to the directed graph to be migrated.

[0211] Based on the seventh example of the second aspect, in an eighth example of the second aspect, the data migration module is further used to:

[0212] If the directed graph to be migrated includes a ring structure, determining the second edge weights of the edges of the upstream node and the downstream node in the directed graph to be migrated according to the first edge weights between the target node and the upstream node and the downstream node in the directed acyclic graph respectively;

[0213] Disconnect the second edge with the smallest weight in the ring structure to obtain a directed acyclic graph to be migrated;

[0214] The migration order is determined according to the dependency relationship between the nodes in the directed acyclic graph to be migrated.

[0215] Based on the seventh example of the second aspect, in a ninth example of the second aspect, the target node includes one of the following: a node where the computing task is located, and a node where the data path is located.

[0216] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory.

[0217] The memory stores computer executable instructions.

[0218] The at least one processor executes the computer-executable instructions stored in the memory, so that the electronic device implements the method described in any one of the first aspects.

[0219] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions, and when a processor executes the computer execution instructions, a computing device implements any method described in the first aspect.

[0220] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program is provided, wherein the computer program is used to implement the method described in any one of the first aspects.

[0221] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0222] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0223] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A data storage control method, characterized in that: The method comprises: According to the dependency relationship between each storage object in the distributed storage system, a computer room allocation model of the distributed storage system is constructed, the computer room allocation model is used to output the corresponding migration evaluation result according to the computer room allocation strategy, the computer room allocation strategy is used to indicate the target computer room allocated for the storage object in the distributed storage system, and the migration evaluation result is used to indicate the overhead caused by the data migration processing from the original computer room to the target computer room; wherein the dependency relationship includes the dependency relationship between the data path and the computing task, the data path is used to characterize the path for storing data, and the computing task is used to read data from the data path of the upstream node to perform calculation, and write the calculation result data into the data path of the downstream node; Calling the computer room allocation model, based on the migration evaluation results corresponding to each computer room allocation strategy, determining the computer room allocation strategy corresponding to the minimum migration evaluation result as the target computer room allocation strategy; According to the target computer room allocation strategy, data migration is performed on the distributed storage system.

2. The method according to claim 1, characterized in that: The migration evaluation result includes: the migration cost corresponding to the distributed storage system, and the cross-computer room access volume corresponding to the computer room allocation according to the computer room allocation strategy. The migration cost includes: the total cost required to migrate the storage object in the distributed storage system from the original computer room corresponding to the storage object to the target computer room corresponding to the storage object in the computer room allocation strategy.

3. The method according to claim 2, characterized in that The storage objects include data paths and computing tasks. The computer room allocation model of the distributed storage system is constructed according to the dependency relationship between the storage objects in the distributed storage system, including: According to the dependency relationship between the data path and the computing task, a directed acyclic graph is constructed, wherein each node of the directed acyclic graph includes the data path and the computing task, and when the data path and the computing task are connected by an edge, a first edge weight of the edge is used to represent a historical access amount of the computing task to the data path; The computer room allocation model is constructed through the directed acyclic graph.

4. The method according to claim 3, characterized in that: The migration cost is related to the migration cost of a single storage object and the number of storage objects to be migrated, and the number of storage objects to be migrated is related to the original computer room and the computer room allocation strategy; The cross-computer room access volume is related to the first edge weight corresponding to the cross-computer room node group in the directed acyclic graph, the cross-computer room node group includes the nodes where the computing tasks and the nodes where the data paths are located, which are connected by edges in the directed acyclic graph, and the computing tasks and the data paths of the cross-computer room node group correspond to different target computer rooms in the computer room allocation strategy.

5. The method according to any one of claims 1 to 4, characterized in that: The calling of the computer room allocation model, based on the migration evaluation results corresponding to each computer room allocation strategy, determines the computer room allocation strategy corresponding to the minimum migration evaluation result as the target computer room allocation strategy, including: Under preset constraints, the computer room allocation strategy corresponding to the minimum migration evaluation result is determined based on the computer room allocation model as the target computer room allocation strategy, and the preset constraints include: each storage object is allocated to a target computer room, the total data volume of the storage objects allocated to each target computer room is less than or equal to the capacity of the target computer room, and the number of storage objects allocated to each target computer room is within a preset number range, and the preset number range includes the value range of the ratio of the number of storage objects to the number of computer rooms in the distributed storage system.

6. The method according to claim 4, characterized in that The step of migrating data from the distributed storage system according to the target computer room allocation strategy includes: Determining a migration order according to the directed acyclic graph; Data migration is performed on the distributed storage system according to the target computer room allocation strategy, the original computer room and the migration sequence.

7. The method according to claim 6, characterized in that Determining the migration order according to the directed acyclic graph includes: Connecting an upstream node of a target node in the directed acyclic graph and a downstream node of the target node to delete the target node from the directed acyclic graph, thereby obtaining a directed graph to be migrated; A migration order is determined according to the directed graph to be migrated.

8. The method according to claim 7, characterized in that The step of determining the migration order according to the directed graph to be migrated includes: If the directed graph to be migrated includes a ring structure, determining the second edge weights of the edges of the upstream node and the downstream node in the directed graph to be migrated according to the first edge weights between the target node and the upstream node and the downstream node in the directed acyclic graph respectively; Disconnect the second edge with the smallest weight in the ring structure to obtain a directed acyclic graph to be migrated; The migration order is determined according to the dependency relationship between the nodes in the directed acyclic graph to be migrated.

9. The method according to claim 7, characterized in that: The target node includes one of the following: the node where the computing task is located, and the node where the data path is located.

10. A data storage control device using the data storage control method according to any one of claims 1 to 9, characterized in that: include: A function construction module, used to construct a computer room allocation model of the distributed storage system according to the dependency relationship between the storage objects in the distributed storage system, the computer room allocation model is used to output the corresponding migration evaluation result according to the computer room allocation strategy, the computer room allocation strategy is used to indicate the target computer room allocated for the storage objects in the distributed storage system, and the migration evaluation result is used to indicate the overhead caused by the data migration processing from the original computer room to the target computer room; wherein the dependency relationship includes the dependency relationship between the data path and the computing task, the data path is used to characterize the path for storing data, and the computing task is used to read data from the data path of the upstream node to perform calculations, and write the calculation result data into the data path of the downstream node; A target strategy determination module is used to call the computer room allocation model, and based on the migration evaluation results corresponding to each computer room allocation strategy, determine the computer room allocation strategy corresponding to the minimum migration evaluation result as the target computer room allocation strategy; The data migration module is used to migrate data of the distributed storage system according to the target computer room allocation strategy.

11. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the electronic device implements the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, a computing device implements the method according to any one of claims 1 to 9.

13. A computer program, characterized in that The computer program is used to implement the method according to any one of claims 1 to 9.

Images