Method and system for managing and scheduling cross-domain heterogeneous storage cluster

Abstract

Provided in the present invention are a method and system for managing and scheduling a cross-domain heterogeneous storage cluster. The lowest total storage cost, the maximum total throughput and the nearest location of a data set to be stored with respect to a storage system in which the data set is about to be stored are taken as optimization targets, and a non-dominated sorting genetic algorithm is used to solve a constraint condition, so as to select an optimal storage location, so that the data access performance is improved, delay is reduced, and unnecessary consumption of a network bandwidth and storage resources is also reduced.

Description

A management and scheduling method and system for a cross-domain heterogeneous storage cluster TECHNICAL FIELD

[0001] The present application belongs to the technical field of distributed virtual storage, and particularly relates to a management and scheduling method and system for a cross-domain heterogeneous storage cluster. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] With the development of big data, cloud computing and Internet of Things technology, data in various industries is growing explosively, and mining and effective utilization of massive data information has become the strategic core of enterprise development and technological progress. However, these massive data often exist in different storage systems, use different access protocols and interfaces, and cannot be directly accessed by each other, making it difficult to achieve unified management and sharing. When multiple data centers need to concurrently process massive data, data migration or repeated storage is generally used, which seriously affects the efficiency of data processing. Therefore, it is urgent to seamlessly integrate storage systems in different regions and build a storage base based on data-intensive applications to achieve unified management and scheduling of heterogeneous storage systems.

[0004] Under this background, the emergence of distributed virtual storage systems provides a solution. Distributed virtual storage systems virtualize storage systems scattered in different regions through a distributed architecture, build a unified storage pool, cross regional restrictions, and achieve data sharing between different storage systems, such as HDFS, Ceph, Alluxio, etc. Alluxio is an open-source distributed virtual storage system that builds a unified namespace on heterogeneous physical storage resources, abstracts storage media into a shared file system, and realizes centralized storage of scattered data. Existing distributed file systems follow the traditional local file system data storage mechanism, that is, first use hard disk storage to persistently save data, and when receiving an access request for a file data, the corresponding data of the file data needs to be found in the hard disk first, and then the data is placed in the memory before it can be operated. Data in memory is usually moved out of memory after a period of time after access is completed. Each data access process is similar, which is tedious and time-consuming.

[0005] Therefore, how to improve the data access performance of the distributed virtual storage system, reduce the delay, and reduce unnecessary consumption of network bandwidth and storage resources is a technical problem to be solved at present.

[0006] SUMMARY

[0007] In order to overcome the above-mentioned deficiencies of the prior art, the present application provides a management and scheduling method and system for a cross-domain heterogeneous storage cluster, a non-dominated sorting genetic algorithm is used to solve the constraint conditions of a constructed multi-objective optimization model, and the optimal storage location is selected, so that the data access performance is improved, the delay is reduced, and unnecessary consumption of network bandwidth and storage resources is reduced.

[0008] In order to achieve the above-mentioned purpose, a first aspect of the present application provides a management and scheduling method for a cross-domain heterogeneous storage cluster, comprising:

[0009] According to the storage cost, the throughput of different to-be-stored data sets stored on different storage systems, and the relative position of the to-be-stored data sets and the different storage systems, a multi-objective optimization model for storage scheduling is constructed;

[0010] Constraint conditions are constructed, with the goal of minimizing the total storage cost of all to-be-stored data sets, maximizing the total throughput, and the to-be-stored data sets being closest to the storage system to be stored, a non-dominated sorting genetic algorithm is used to optimize and solve the multi-objective optimization problem, a Pareto optimal solution set of the multi-objective optimization model satisfying the constraint conditions is obtained, and a storage scheduling result of the to-be-stored data sets is obtained according to the Pareto optimal solution set.

[0011] A second aspect of the present application provides a management and scheduling system for a cross-domain heterogeneous storage cluster, comprising:

[0012] A multi-objective model construction module: according to the storage cost, the throughput of different to-be-stored data sets stored on different storage systems, and the relative position of the to-be-stored data sets and the different storage systems, a multi-objective optimization model for storage scheduling is constructed;

[0013] A multi-objective model solving module: constraint conditions are constructed, with the goal of minimizing the total storage cost of all to-be-stored data sets, maximizing the total throughput, and the to-be-stored data sets being closest to the storage system to be stored, a non-dominated sorting genetic algorithm is used to optimize and solve the multi-objective optimization problem, a Pareto optimal solution set of the multi-objective optimization model satisfying the constraint conditions is obtained, and a storage scheduling result of the to-be-stored data sets is obtained according to the Pareto optimal solution set.

[0014] A third aspect of the present application provides a computer device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the computer device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform a management and scheduling method for a cross-domain heterogeneous storage cluster.

[0015] The fourth aspect of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to perform a management and scheduling method of a cross-domain heterogeneous storage cluster.

[0016] The above one or more technical solutions have the following beneficial effects:

[0017] In the present application, the optimization objectives are the minimum total storage cost, the maximum total throughput and the nearest position of the data set to be stored and the storage system to be stored, the non-dominated sorting genetic algorithm is used to solve the constraint conditions, and the optimal storage position is selected, so that the data access performance is improved, the delay is reduced, and unnecessary consumption of network bandwidth and storage resources is reduced.

[0018] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be known by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] The drawings accompanying the specification of the present application form a part thereof and serve to provide further understanding of the present application, the illustrative embodiments of the present application and its description serve to explain the present application, and do not constitute improper limitations on the present application.

[0020] Fig. 1 is a whole framework diagram of a cross-domain heterogeneous storage system in embodiment one of the present application;

[0021] Fig. 2 is a distribution diagram of different storage systems in embodiment one of the present application;

[0022] Fig. 3 is an example diagram of encoding and decoding mode in embodiment one of the present application;

[0023] Fig. 4 is a flowchart of solving based on non-dominated sorting genetic algorithm in embodiment one of the present application. DETAILED DESCRIPTION

[0024] It should be noted that the following detailed description is exemplary, and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0025] It should be noted that the terms used herein are only for the purpose of describing the specific embodiments, and are not intended to limit the exemplary embodiments according to the present application.

[0026] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0027] Embodiment one

[0028] The embodiment discloses a management and scheduling method of a cross-domain heterogeneous storage cluster, comprising:

[0029] According to the storage cost, throughput of different to-be-stored data sets stored on different storage systems, and the relative position of the to-be-stored data sets and the different storage systems, a multi-objective optimization model of storage scheduling is constructed;

[0030] The constraint condition is constructed to minimize the total storage cost of all to-be-stored data sets, maximize the total throughput, and minimize the position of the to-be-stored data sets and the storage system to be stored, a non-dominated sorting genetic algorithm is used to optimize and solve the multi-objective optimization problem, a Pareto optimal solution set of the multi-objective optimization model satisfying the constraint condition is obtained, and a storage scheduling result of the to-be-stored data sets is obtained according to the Pareto optimal solution set.

[0031] The embodiment takes open source Alluxio as an example to illustrate the storage layer abstraction of the cross-domain heterogeneous storage system, Alluxio is an open source distributed virtual storage system, which builds a unified namespace on heterogeneous physical storage resources, abstracts storage media into a shared file system, and realizes centralized storage of scattered data, the system builds a high-availability (HA) Alluxio cluster, and mounts multiple storage systems of data centers, adopts a naming rule of place name-storage system name+number to establish a global namespace tree for convenient management and access, creates a data storage adapter in the upper layer, extracts metadata information, provides a data storage optimization strategy, and provides storage system management capability and data space management capability for the upper layer.

[0032] The overall framework of the cross-domain heterogeneous storage system includes a unified storage adapter, a database and Alluxio data orchestration.

[0033] The unified storage adapter: creates a data logical space according to input parameters, a unified storage view and the like; extracts metadata information, and internally builds a multi-objective optimization strategy; automatically selects an optimal storage scheme for a user through the data logical space, the metadata information, the unified storage view and the like when the user performs a data operation; stores the metadata information, storage location information and the like into the database; and provides an interface for storage service calling of an upper layer storage and computing task.

[0034] Database: Mainly includes storage system information table, storage device information table, data space information table, user storage table, operation log table and other information. The storage system information table records the relevant information of the storage system, including storage system ID, address, name, type, model, mounting state and other information. The storage device table records the relevant information of the storage device, including storage device ID, model, manufacturer, hard disk information, etc. The data space information table records the relevant information of the user storage space, including data space ID, name, quota, used capacity, creator, user ID, etc. The user storage table records the relevant information of the user storage, including user storage ID, name, user ID, storage system ID, storage capacity, used capacity, etc. The operation log table records all operation logs, such as data read / write, system maintenance and other operations.

[0035] Alluxio data orchestration: Mount HDFS, NFS, Lustre, minio, QingStor, etc. as the underlying storage system to form an intermediate layer. The intermediate layer uniformly manages data in these different storage systems and provides a unified client API for the upper layer application, which can seamlessly access and operate data between different storage systems.

[0036] When storing data, in order to improve the efficiency of data management, the global namespace tree is designed, and the tree structure is used to effectively organize the files and directories of the storage system. In this structure, the root directory is located at the highest level, and the storage systems of each data center are mounted under the root directory. No matter where the data is stored in the storage system in the data center, users can access these data in the storage system by accessing the path in the global namespace tree. In order to distinguish different storage systems, the naming rule of place name-storage system name+number is adopted, for example, Qingdao-nfs1, which represents the first NFS storage system located in Qingdao.

[0037] When accessing the underlying storage system, the appropriate storage solution is selected through intelligent algorithm, which can effectively reduce the data access time, improve the response speed of the storage system, optimize the data throughput, and thus improve the overall system performance. This automatic decision-making process considers multiple factors, including storage cost, data access speed, location information, etc., to comprehensively optimize various performance indicators and provide efficient and reliable data access services for users.

[0038] In solving the problem of automatic selection of the optimal solution of the underlying storage system, NSGA-III algorithm is used to solve the multi-objective optimization problem. First, the three objective functions of total cost, total throughput and location information are modeled. In the initialization stage of the population, a screening mechanism is introduced to retain individuals that meet the requirements, improve the quality of the initial population, and promote the algorithm to explore the search space of excellent solutions more specifically. The adaptive algorithm introduced in the crossover and mutation stage dynamically adjusts the crossover probability and mutation probability, thereby balancing the needs of global search and local search in the search space. The introduction of reference points generates a diverse and uniformly distributed solution through the reference point-based niche preservation strategy, which adaptively adjusts the reference points. This method of generating reference points improves the algorithm's exploration ability in the multi-objective storage system selection problem and helps search for potential Pareto optimal solution sets.

[0039] Suppose there are n data sets to be stored, which need to be stored and accessed on m storage systems. The storage and access requirements of each data set are different; each storage system has different costs, throughputs, and location information, and different storage solutions may result in different performance. The goal of the multi-objective storage system selection problem is to select a storage system for each data set to minimize the total cost, maximize the total throughput, and fully consider the location information of the storage system.

[0040] In order to select the appropriate storage system, its cost, throughput, and location information need to be understood. Obtain quotes from different vendors and compare and analyze the costs of hardware, software, and operation and maintenance. Use performance monitoring tools to monitor the performance indicators of each storage system in real time. Record the geographical location distribution of the storage system, such as whether it is in the same data center, different data centers, etc.

[0041] Three objective functions are defined for this problem, namely cost minimization, throughput maximization, and nearest storage location.

[0042] 1. Total cost

[0043] Including the cost of transmission and the cost of storage.

[0044] Where n represents the number of data sets to be stored, m is the number of storage systems, x ij is a binary variable indicating whether the ith data set is stored on the jth storage system, C1 ij is the transmission cost of storing the ith data set on the jth storage system, C2 ij is the storage cost of storing the ith data set on the jth storage system.

[0045] Where the cost of storage can be obtained from different vendors and calculated based on hardware, software, operation, etc.

[0046] 2. Total throughput

[0047] Where T ij is the throughput of storing dataset i on storage system j, x ij is a binary variable indicating whether the ith dataset is stored on the jth storage system.

[0048] 3. Location objective function

[0049] Where score ij is the score considering the impact of the location information of the storage system on dataset i, for example, the score is higher if the storage system and dataset are in the same data center, and the score is lower if the storage system and dataset are in different data centers.

[0050] Constraints:

[0051] 1. Each dataset can only be stored on one storage system:

[0052] 2. The size of the dataset cannot exceed the remaining capacity of the storage system:

[0053] Where datasize i is the size of the ith dataset, and remaining_capacity j is the remaining capacity of the storage system j.

[0054] 3. Binary variable constraints:

[0055] In the evolutionary algorithm for solving the multi-objective storage system selection problem, first generate an initial population P t with a size of N and a set of random reference points, filter the initial population, and then obtain a descendant population Q t with a size of N through adaptive crossover and mutation operations. Combine the initial population P t and the descendant population Q t to obtain a population R t with a size of 2N. Use non-dominated sorting to divide R t into different non-dominated levels; then use the niche preservation strategy based on reference points to adaptively adjust the reference points so that exactly N reference points satisfy ρ j= 1, finally make the N as the parent population of the next iteration, through continuous iteration, until the predetermined number of iterations, the algorithm eventually produces a set of optimized solution set. From which the largest remaining storage capacity of the storage system as the optimal solution, to meet the demand for storage systems in practical applications, to ensure the quality of the solution and the efficiency of the storage system selection.

[0056] The solving process is as follows:

[0057] (1) Initialization of population

[0058] In the initialization of the population stage, in addition to using the method of randomly generating initial chromosomes, a screening mechanism is introduced. This mechanism limits the remaining capacity and throughput of the storage system of the initial population, and sets the threshold value of the remaining capacity and throughput of the storage system. For each initially generated chromosome, it is judged whether the corresponding remaining capacity and throughput of the storage system meet the requirements, and those chromosomes that cannot meet the performance requirements are eliminated in the initial stage, thereby significantly reducing the similarity between chromosomes, so that the algorithm can more targetedly explore the excellent solution space.

[0059] In this embodiment, each chromosome represents a possible solution, i.e. a scheme in which all data sets to be stored are allocated to storage systems.

[0060] (2) Encoding

[0061] Since the encoding method adopted by the standard NSGA-III algorithm is not suitable for the problem proposed in this embodiment, the real number encoding method is used to determine the storage system selection scheme of each data set. The length of the chromosome is n, indicating n data sets to be stored, and the length of the array is m, indicating m storage systems that can be selected. The storage system number is assigned to the corresponding position of the gene string, indicating that the data set is allocated to the corresponding storage system.

[0062] (3) Decoding

[0063] Each encoded part in the chromosome is traversed, and each part represents the storage system selection of a data set. The integer encoding of each data set is mapped to a specific storage system number. According to the mapping result, each data set is allocated to the corresponding storage system, and the scheduling scheme list is updated.

[0064] As shown in FIG. 3, how to encode and decode in an example of 5 storage systems and 9 data sets is shown. The scheme represented by this example is: data sets N1, N2 and N6 are allocated to storage system M1, data sets N4 and N8 are allocated to storage system M2, data set N3 is allocated to storage system M3, data set N7 is allocated to storage system M4, and data sets N5 and N9 are allocated to storage system M5.

[0065] (4) Crossover, Mutation

[0066] SBX (Simulated Binary Crossover) and polynomial mutation are used. SBX uses a uniform distribution to perform gene crossover, which can uniformly mix genes and avoid the situation that the solution of the multi-objective storage system selection falls into a local optimal solution and cannot further search for a better solution. Polynomial mutation introduces a certain degree of random disturbance, which helps to introduce diversity in the search space of the problem, thereby better exploring possible solutions. In the solving process, an adaptive algorithm is introduced to dynamically adjust the crossover probability (Pc) and mutation probability (Pm). The update range of the population individuals is dynamically adjusted according to the evolution number and the distance between individuals, and the evolution direction of the crossover and mutation probabilities is determined. When the distance between an individual and its nearest individual is less than the average distance between individuals, the crossover and mutation probabilities are increased with the increase of the iteration number, so as to enhance the global search ability and ensure that the algorithm searches the solution space more extensively; when the distance between an individual and its nearest individual is greater than the average distance between individuals, the crossover and mutation probabilities are reduced with the increase of the iteration number, so as to refine the local search and focus more on the detailed adjustment of the local optimal solution.

[0067] wherein the crossover and mutation probability calculation formulae are:

[0068] wherein Pc1 and Pc2 are the maximum and minimum values of the initial set crossover probability, Pm1 and Pm2 are the maximum and minimum values of the initial set mutation probability, T is the maximum number of iterations of the population, t is the current iteration number, a is a constant, d(t) is the distance between an individual and the nearest individual, and dagv(t) is the average distance between individuals.

[0069] (5) Selection mechanism based on reference points

[0070] After the population obtained by crossover and mutation is combined with the parent population, non-dominated sorting is performed to divide it into multiple non-dominated layers (F1, F2,...). Then a new population is constructed from F1, until its size is N or it exceeds N for the first time. The last layer is the lth layer. Solutions above the (l+1)th layer will be eliminated. In most cases, only part of the individuals in the lth layer are accepted. NSGA-III uses a reference point-based method to select individuals in the lth layer.

[0071] Due to the existence of constraint conditions, the feasible region of the algorithm can only occupy a small part of the entire region, however, the reference points are uniformly distributed in the target space, which leads to the fact that part of the reference points are not utilized, and those utilized reference points can establish connections with multiple individuals, which cannot achieve good distribution, thereby affecting the convergence of the algorithm. Therefore, the adaptive method proposed by Deb can be used to identify the reference points that do not meet the uniform distribution of the Pareto optimal point set. After generating N P t+1 points, two new operations are introduced: introducing new reference points and eliminating useless reference points, so as to always keep the number of populations and selected reference points equal.

[0072] Introducing new reference points: after generating P t+1 individuals, update the niche number of each reference point ρ j . When all reference points have non-dominated individuals, set ρ j = 1. If some reference points ρ j ≥ 2, it will lead to some reference points ρ k = 0, at this time, if the k reference point is valid, it needs to find an associated individual, replace the k reference point with a new reference point close to the j reference point, and add a set of reference points around the j reference point. If there are multiple reference points ρ j ≥ 2, repeat the above steps for each reference point.

[0073] Eliminate useless reference points: update the niche number of all reference points, if there are exactly N reference points satisfying ρ j = 1, that is, N reference points have an associated individual in P t+1 , which indicates that the distribution between the reference points is uniform, remove the reference points with increased ρ j = 0 (not including the original reference points).

[0074] The adaptive operation described above is embedded in the present scheme, in this way, the reference points are dynamically adjusted, the uniform distribution is maintained, and the search effect of the algorithm on the Pareto front of the multi-objective storage system selection problem is improved.

[0075] The storage location adaptation algorithm of the present scheme uses an intelligent decision strategy to automatically select the optimal storage location according to the performance indicators of the storage resources. This algorithm helps to improve data access performance, reduce latency, and reduce unnecessary consumption of network bandwidth and storage resources. Users do not need to manually select storage locations, and the system automatically provides the best storage experience.

[0076] Embodiment two

[0077] The purpose of the present embodiment is to provide a management and scheduling system for a cross-domain heterogeneous storage cluster, which comprises:

[0078] The multi-objective model construction module: according to the storage cost, the throughput of different storage systems on which different to-be-stored data sets are stored, and the relative positions of the to-be-stored data sets and the different storage systems, a multi-objective optimization model of storage scheduling is constructed;

[0079] The multi-objective model solution module: constraint conditions are constructed, with the total storage cost of all to-be-stored data sets being minimized, the total throughput being maximized, and the to-be-stored data sets being closest to the storage system to be stored as the target, a non-dominated sorting genetic algorithm is used to optimize and solve the multi-objective optimization problem, a Pareto optimal solution set of the multi-objective optimization model satisfying the constraint conditions is obtained, and a storage scheduling result of the to-be-stored data sets is obtained according to the Pareto optimal solution set.

[0080] Embodiment three

[0081] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0082] Embodiment four

[0083] The purpose of this embodiment is to provide a computer-readable storage medium.

[0084] A computer-readable storage medium having a computer program stored thereon, the program being executable by a processor to perform the steps of the above method.

[0085] The steps and methods involved in the above embodiments two, three and four correspond to embodiment one, and the specific embodiments can be referred to the relevant description part of embodiment one. The term "computer-readable storage medium" should be understood to include a single medium or multiple media of one or more instruction sets; it should also be understood to include any medium capable of storing, encoding or carrying instruction sets for execution by a processor and causing the processor to perform any of the methods in the present application.

[0086] Those skilled in the art should understand that the above modules or steps of the present application can be realized by a general computer device, alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module. The present application is not limited to any specific combination of hardware and software.

[0087] The above describes the specific embodiments of the present application in combination with the drawings, but is not a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A management and scheduling method for a cross-domain heterogeneous storage cluster, characterized in that: include: A multi-objective optimization model for storage scheduling is constructed based on the storage cost and throughput of different data sets to be stored on different storage systems, as well as the relative positions of the data sets to be stored and the different storage systems. Constraints are constructed, with the goals of minimizing the total storage cost of all data sets to be stored, maximizing the total throughput, and ensuring that the data sets to be stored are closest to the storage system in which they will be stored. A non-dominated sorting genetic algorithm is used to optimize and solve the multi-objective optimization problem, and a Pareto optimal solution set of the multi-objective optimization model that meets the constraints is obtained. Based on the Pareto optimal solution set, a storage scheduling result for the data sets to be stored is obtained.

2. A cross-domain heterogeneous storage cluster management and scheduling method as claimed in claim 1, characterized in that: The constraints include: each data set to be stored can only be stored in one storage system; the size of the data set to be stored is not greater than the remaining capacity of the storage system in which it will be stored.

3. A cross-domain heterogeneous storage cluster management and scheduling method as claimed in claim 1, characterized in that: The multi-objective optimization problem is optimized and solved using a non-dominated sorting genetic algorithm, specifically: Initialize the population; The binary crossover operator and polynomial mutation are used to perform crossover and mutation on individuals in the population, and the offspring population is generated by adaptively adjusting the crossover and mutation rates; Merge the initial population and the offspring population, and use the non-dominated sorting method to divide the merged population into different non-dominated levels; Using the reference point-based niche preservation strategy, the reference point is adaptively adjusted to generate the next iterative parent population; Continuously iterate and update until the preset iteration point is reached to obtain a set of optimized solutions.

4. A cross-domain heterogeneous storage cluster management and scheduling method as claimed in claim 3, characterized in that: Using the reference point-based niche preservation strategy, the reference points are adaptively adjusted so that there are exactly N reference points that satisfy ρ j =1, then remove the added ρ j = 0, the removed reference points do not include the original reference points; where N is the size of the individuals in the offspring population, ρ j is the number of microhabitats at the reference point.

5. A cross-domain heterogeneous storage cluster management and scheduling method as claimed in claim 3, characterized in that: Binary crossover operator and polynomial mutation are used to perform crossover and mutation on individuals in the population. The offspring population is generated by adaptively adjusting the crossover and mutation rates. Specifically, when the distance between an individual and its nearest individual is less than the average distance between individuals, the crossover and mutation probabilities increase with the increase of the number of iterations; when the distance between an individual and its nearest individual is greater than the average distance between individuals, the crossover and mutation probabilities decrease with the increase of the number of iterations.

6. A cross-domain heterogeneous storage cluster management and scheduling method as claimed in claim 3, characterized in that: During the population initialization phase, each initially generated chromosome is judged to see whether it meets the threshold requirements of the remaining capacity and throughput of the storage system, and chromosomes that do not meet the requirements are screened out.

7. A cross-domain heterogeneous storage cluster management and scheduling method as claimed in claim 3, characterized in that: It also includes encoding the data sets to be stored and the storage system. Specifically, the chromosome length is n, indicating n data sets to be stored; the storage system number is assigned to the position corresponding to the gene string of the chromosome, indicating that the corresponding data set to be stored is allocated to the corresponding storage system.

8. A management and scheduling system for a cross-domain heterogeneous storage cluster, characterized in that: include: Multi-objective model building module: This module builds a multi-objective optimization model for storage scheduling based on the storage cost and throughput of different data sets stored on different storage systems, as well as the relative positions of the data sets to be stored and the different storage systems. Multi-objective model solving module: Constructs constraints, takes minimizing the total storage cost of all data sets to be stored, maximizing the total throughput, and ensuring that the data sets to be stored are closest to the storage system where they will be stored as the goal, and uses a non-dominated sorting genetic algorithm to optimize and solve the multi-objective optimization problem, obtaining the Pareto optimal solution set of the multi-objective optimization model that meets the constraints, and obtains the storage scheduling result of the data sets to be stored based on the Pareto optimal solution set.

9. A computer device, characterized in that: include: A processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, a management and scheduling method for a cross-domain heterogeneous storage cluster as described in any one of claims 1 to 7 is performed.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for managing and scheduling a cross-domain heterogeneous storage cluster according to any one of claims 1 to 7 is executed.

Images