A management and scheduling method and system for cross-domain heterogeneous storage clusters

Abstract

The present invention proposes a management and scheduling method and system for a cross-domain heterogeneous storage cluster, which takes minimizing the total storage cost, maximizing the total throughput, and making the location of the data set to be stored closest to the storage system to be stored as the optimization goal, and adopts a non-dominated sorting genetic algorithm to solve the constraints and select the optimal storage location, which not only improves the data access performance and reduces the delay, but also reduces the unnecessary consumption of network bandwidth and storage resources.

Description

Technical Field

[0001] The present invention belongs to the technical field related to distributed virtual storage, and in particular relates to a management and scheduling method and system for a cross-domain heterogeneous storage cluster. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the development of big data, cloud computing, and the Internet of Things, data in all walks of life have exploded. Mining and effectively utilizing massive data information has become the strategic core of enterprise development and scientific and technological progress. However, these massive data often exist in different storage systems, using different access protocols and interfaces. They cannot directly access each other, making it difficult to achieve unified management and sharing. When multiple data centers need to process massive data concurrently, data migration or duplicate storage is generally used, which seriously affects the efficiency of data processing. Therefore, it is urgent to seamlessly integrate storage systems in different regions, build a storage base based on data-intensive applications, and achieve unified management and scheduling of heterogeneous storage systems.

[0004] In this context, the emergence of distributed virtual storage systems provides a solution. Distributed virtual storage systems virtualize storage systems scattered in different regions through a distributed architecture, build a unified storage pool, transcend geographical restrictions, and realize data sharing among different storage systems, such as HDFS, Ceph, Alluxio, etc. Alluxio is an open source distributed virtual storage system that builds a unified namespace on heterogeneous physical storage resources, abstracts the storage medium into a shared file system, and realizes centralized storage of scattered data. The existing distributed file system uses the traditional local file system data storage mechanism, that is, first use the hard disk storage to persist the data. When receiving a request to access file data, it is necessary to first find the data corresponding to the file data in the hard disk, and then put it into the memory before various operations can be performed on it. The data in the memory is usually moved out of the memory after a period of time after the access is completed. The access process of each data is similar, the steps are cumbersome, and it takes a long time.

[0005] Therefore, how to improve the data access performance of distributed virtual storage systems, reduce latency, and reduce unnecessary consumption of network bandwidth and storage resources is a technical problem that needs to be solved at present. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a management and scheduling method and system for a cross-domain heterogeneous storage cluster, which adopts a non-dominated sorting genetic algorithm to solve the constraints of the constructed multi-objective optimization model and select the optimal storage location, which not only improves data access performance and reduces latency, but also reduces unnecessary consumption of network bandwidth and storage resources.

[0007] To achieve the above object, a first aspect of the present invention provides a management and scheduling method for a cross-domain heterogeneous storage cluster, comprising:

[0008] A multi-objective optimization model for storage scheduling is constructed based on the storage cost and throughput of different data sets to be stored on different storage systems, as well as the relative positions of the data sets to be stored and different storage systems.

[0009] Constraints are constructed, with the goal of minimizing the total storage cost of all data sets to be stored, maximizing the total throughput, and ensuring that the data sets to be stored are located closest to the storage system to be stored. A non-dominated sorting genetic algorithm is used to optimize and solve the multi-objective optimization problem, and a Pareto optimal solution set of the multi-objective optimization model that meets the constraints is obtained. According to the Pareto optimal solution set, a storage scheduling result of the data sets to be stored is obtained.

[0010] A second aspect of the present invention provides a management and scheduling system for a cross-domain heterogeneous storage cluster, comprising:

[0011] Multi-objective model building module: builds a multi-objective optimization model for storage scheduling based on the storage cost and throughput of different data sets to be stored on different storage systems, as well as the relative positions of the data sets to be stored and different storage systems;

[0012] Multi-objective model solving module: Construct constraints, take the minimum total storage cost of all data sets to be stored, the maximum total throughput, and the closest location of the data sets to be stored to the storage system to be stored as the goal, use the non-dominated sorting genetic algorithm to optimize and solve the multi-objective optimization problem, obtain the Pareto optimal solution set of the multi-objective optimization model that meets the constraints, and obtain the storage scheduling result of the data sets to be stored based on the Pareto optimal solution set.

[0013] The third aspect of the present invention provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, a management and scheduling method for a cross-domain heterogeneous storage cluster is performed.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, a management and scheduling method for a cross-domain heterogeneous storage cluster is executed.

[0015] One or more of the above technical solutions have the following beneficial effects:

[0016] In the present invention, the total storage cost is minimized, the total throughput is maximized, and the location of the data set to be stored and the storage system to be stored is closest as the optimization goal. A non-dominated sorting genetic algorithm is used to solve the constraints and select the optimal storage location, which not only improves data access performance and reduces latency, but also reduces unnecessary consumption of network bandwidth and storage resources.

[0017] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0019] Figure 1 This is an overall framework diagram of a cross-domain heterogeneous storage system in Embodiment 1 of the present invention;

[0020] Figure 2 This is a distribution diagram of different storage systems in Embodiment 1 of the present invention;

[0021] Figure 3 This is an example diagram of the encoding and decoding method in the first embodiment of the present invention;

[0022] Figure 4 This is a flowchart of a genetic algorithm solution based on non-dominated sorting in Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0023] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0024] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0025] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0026] Embodiment 1

[0027] This embodiment discloses a management and scheduling method for a cross-domain heterogeneous storage cluster, including:

[0028] A multi-objective optimization model for storage scheduling is constructed based on the storage cost and throughput of different data sets to be stored on different storage systems, as well as the relative positions of the data sets to be stored and different storage systems.

[0029] Constraints are constructed, with the goal of minimizing the total storage cost of all data sets to be stored, maximizing the total throughput, and ensuring that the data sets to be stored are located closest to the storage system to be stored. A non-dominated sorting genetic algorithm is used to optimize and solve the multi-objective optimization problem, and a Pareto optimal solution set of the multi-objective optimization model that meets the constraints is obtained. According to the Pareto optimal solution set, a storage scheduling result of the data sets to be stored is obtained.

[0030] This embodiment uses the open source Alluxio as an example to illustrate the storage layer abstraction of the cross-domain heterogeneous storage system. Alluxio is an open source distributed virtual storage system that builds a unified namespace on heterogeneous physical storage resources, abstracts the storage medium into a shared file system, and centrally stores the scattered data. This system builds a high-availability (HA) Alluxio cluster and mounts the storage systems of multiple data centers. For easy management and access, the naming rule of place name-storage system name+number is adopted to establish a global namespace tree. Create a data storage adapter at the upper layer, extract metadata information, provide a data storage optimization strategy, and provide storage system management capabilities and data space management capabilities for the upper layer.

[0031] The overall framework of the cross-domain heterogeneous storage system in this embodiment includes: a unified storage adapter, a database, and Alluxio data orchestration.

[0032] Among them, the unified storage adapter: creates a data logical space based on input parameters, unified storage views and other information; extracts metadata information, and has a built-in multi-objective optimization strategy. When users perform data operations, it automatically selects the optimal storage solution for users through data logical space, metadata information, unified storage views and other information; stores metadata information, storage location information, etc. in the database; and provides an interface for storage service calls for upper-level storage and computing tasks.

[0033] Database: mainly includes storage system information table, storage device information table, data space information table, user storage table, operation log table and other information. The storage system information table records the relevant information of the storage system, including storage system ID, address, name, type, model, mounting status and other information. The storage device table records the relevant information of the storage device, including storage device ID, model, manufacturer, hard disk information and other information. The data space information table records the relevant information of the user storage space, including data space ID, name, quota, used capacity, creator, user ID and other information. The user storage table records the relevant information of the user storage, including user storage ID, name, user ID, storage system ID, storage capacity, used capacity and other information. The operation log table records all operation logs, such as data reading and writing, system maintenance and other operations.

[0034] Alluxio data orchestration: Mount HDFS, NFS, Lustre, minio, QingStor, etc. as the underlying storage system to form an intermediate layer. The intermediate layer uniformly manages the data in these different storage systems and provides a unified client API for its upper-layer applications. The upper-layer applications can seamlessly access and operate data between different storage systems.

[0035] In order to improve the efficiency of data management during data storage, a global namespace tree is designed. The files and directories of the storage system are effectively organized using a tree structure. In this structure, the root directory is at the highest level, and the storage systems of each data center are mounted under the root directory. No matter which data center the data is stored in, users can access the data in the storage system by accessing the path in the global namespace tree. In order to distinguish different storage systems, the naming rule of place name-storage system name+number is adopted, such as Qingdao-nfs1, which means the first NFS storage system located in Qingdao.

[0036] When accessing the underlying storage system, the intelligent algorithm selects the appropriate storage solution, which can effectively reduce data access time, improve the response speed of the storage system, optimize data throughput, and thus improve the performance of the overall system. This automated decision-making process takes into account multiple factors, including storage cost, data access speed, location information, etc., to comprehensively optimize various performance indicators and provide users with efficient and reliable data access services.

[0037] In solving the problem of automatically selecting the optimal solution for the underlying storage system, the NSGA-III algorithm is used to solve the multi-objective optimization problem. First, the three objective functions of total cost, total throughput and location information are modeled. In the population initialization stage, a screening mechanism is introduced to retain individuals that meet the requirements, improve the quality of the initial population, and promote the algorithm to explore the search space of excellent solutions more targeted. The adaptive algorithm introduced in the crossover and mutation stage dynamically adjusts the crossover probability and mutation probability, thereby balancing the needs of global search and local search in the search space. The introduction of reference points adaptively adjusts the reference points through a microhabitat preservation strategy based on the reference points, thereby generating diverse and evenly distributed solutions. This method of generating reference points improves the algorithm's exploration ability in the multi-objective storage system selection problem and helps to search for potential Pareto optimal solution sets.

[0038] Assume that there are n data sets to be stored, which need to be stored and accessed on m storage systems. The storage and access requirements of each data set are different. Each storage system has different cost, throughput and location information. Different storage solutions may lead to different performance. The goal of the multi-objective storage system selection problem is to reasonably select a storage system for each data set, so that the total cost is minimized, the total throughput is maximized, and the location information of the storage system is fully considered.

[0039] In order to select the right storage system, you need to understand its cost, throughput, and location information. Get quotes from different vendors and compare and analyze them based on the cost of hardware, software, and operation and maintenance. Use performance monitoring tools to monitor the performance indicators of each storage system in real time. Record the geographical distribution of the storage system, such as whether it is in the same data center, different data centers, etc.

[0040] Three objective functions are defined for this problem, namely cost minimization, throughput maximization, and nearest storage location.

[0041] 1. Total cost

[0042] Including transmission costs and storage costs.

[0043]

[0044] Where n is the number of data sets to be stored, m is the number of storage systems, and x is the number of ij is a binary variable, indicating whether the i-th data set is stored on the j-th storage system. ij is the transmission cost of the i-th data set stored in the j-th storage system, C2 ij is the storage cost of storing the i-th data set in the j-th storage system.

[0045] Among them, you can get quotations from different suppliers and calculate the storage cost based on hardware, software, operation and maintenance, etc.

[0046] 2. Total throughput

[0047]

[0048] Among them, T ij is the throughput of storing dataset i on storage system j, x ij is a binary variable indicating whether the i-th dataset is stored on the j-th storage system.

[0049] 3. Position objective function

[0050]

[0051] Among them, score ij is a score that considers the impact of the storage system’s location information on dataset i. For example, if the storage system and the dataset are in the same data center, the score is high, and if the storage system and the dataset are in different data centers, the score is low.

[0052] Constraints:

[0053] 1. Each data set can only be stored on one storage system:

[0054]

[0055] 2. The size of the data set cannot exceed the remaining capacity of the storage system:

[0056]

[0057] Among them, datasize i is the size of the i-th data set, remaining_capacity j is the remaining capacity of storage system j.

[0058] 3. Binary variable constraints:

[0059]

[0060] In the evolutionary algorithm for solving the multi-objective storage system selection problem, an initial population P with a population size of N is first generated. t and a set of random reference points, screen the initial population, and then obtain a descendant population Q of size N through adaptive crossover and mutation operations t . The initial population P t and the offspring population Q t Merge to get a population R of size 2N t. Use the non-dominated sorting method to sort R t The reference points are divided into different non-dominated levels. Then, the reference point-based niche preservation strategy is used to adaptively adjust the reference points so that there are exactly N reference points that satisfy ρ j = 1, and finally let these N as the parent population of the next iteration. Through continuous iteration until the predetermined number of iterations is reached, the algorithm finally generates a set of optimized solutions. The storage system with the largest remaining storage capacity is selected as the optimal solution to meet the needs of the storage system in practical applications, thereby ensuring the quality of the solution and the efficiency of the storage system selection.

[0061] The solution process is as follows:

[0062] (1) Initialize the population

[0063] In the stage of initializing the population, in addition to the method of randomly generating the initial chromosomes, a screening mechanism is introduced. This mechanism limits the remaining capacity and throughput of the storage system of the initial population, sets the thresholds for the remaining capacity and throughput of the storage system, and for each initially generated chromosome, determines whether the corresponding remaining capacity and throughput of the storage system meet the requirements. In the initial stage, chromosomes that cannot meet the performance requirements are eliminated, thereby significantly reducing the similarity between chromosomes, allowing the algorithm to explore the space of excellent solutions more targeted.

[0064] In this implementation, each chromosome represents a possible solution, that is, a solution for allocating all data sets to be stored to the storage system.

[0065] (2) Coding

[0066] Since the encoding method used by the standard NSGA-III algorithm is not compatible with the problem raised in this embodiment, this embodiment uses a real number encoding method to determine the storage system selection scheme for each data set. The length of the chromosome is n, representing n data sets to be stored, and an array of length m represents m selectable storage systems. The storage system number is assigned to the position corresponding to the gene string, indicating that the data set is allocated to the corresponding storage system.

[0067] (3) Decoding

[0068] Traverse each encoding part in the chromosome, each part represents the storage system selection of a data set. Map the integer encoding of each data set to a specific storage system number. According to the mapping result, assign the corresponding storage system to each data set and update the scheduling plan list.

[0069] like Figure 3The example shows how encoding and decoding are performed in an example consisting of 5 storage systems and 9 data sets. The example shows that data sets N1, N2, and N6 are assigned to storage system M1, data sets N4 and N8 are assigned to storage system M2, data set N3 is assigned to storage system M3, data set N7 is assigned to storage system M4, and data sets N5 and N9 are assigned to storage system M5.

[0070] (4) Crossover and mutation

[0071] The simulated binary crossover operator (SBX) and polynomial mutation are used. SBX uses a uniform distribution to perform gene crossover, which can evenly mix genes and avoid falling into the local optimal solution when solving the multi-objective storage system selection and being unable to further search for a better solution. Polynomial mutation introduces a certain degree of random perturbation, which helps to introduce diversity in the search space of this problem, so as to better explore possible solutions. In the solution process, an adaptive algorithm is introduced to dynamically adjust the crossover probability (Pc) and mutation probability (Pm), dynamically adjust the update range of population individuals according to the evolutionary generation and the distance between individuals, and determine the evolution direction of the crossover and mutation probabilities. When the distance between an individual and its nearest individual is less than the average distance between individuals, as the number of iterations increases, the crossover and mutation probabilities are increased to enhance the global search capability and ensure that the algorithm searches the solution space more widely; when the distance between an individual and its nearest individual is greater than the average distance between individuals, as the number of iterations increases, the crossover and mutation probabilities are reduced, the local search is refined, and more focused on the detailed adjustment of the local optimal solution.

[0072] Among them, the calculation formula for crossover and mutation probability is:

[0073]

[0074]

[0075] Among them, Pc1 and Pc2 are the maximum and minimum values ​​of the initially set crossover probability, Pm1 and Pm2 are the maximum and minimum values ​​of the initially set mutation probability, T is the maximum number of iterations of the population, t is the current number of iterations, ɑ is a constant, d(t) is the distance between an individual and the nearest individual, and dagv(t) is the average distance between individuals.

[0076] (5) Reference point-based selection mechanism

[0077] The population obtained by crossover mutation is merged with the parent population and then non-dominated sorting is performed to divide it into multiple non-dominated layers (F1, F2, ...), and then a new population is constructed starting from F1 until its size reaches N or exceeds N for the first time, and the last layer is the lth layer. Solutions above the l+1th layer will be eliminated. In most cases, only some individuals in the lth layer are accepted. NSGA-III uses a reference point-based approach to select individuals in the lth layer.

[0078] Due to the existence of constraints, the feasible area of ​​the algorithm may only occupy a small part of the entire area. However, the reference points are evenly distributed in the target space, which results in some reference points not being used. Those reference points that are used may establish connections with multiple individuals and fail to achieve good distribution, thus affecting the convergence of the algorithm. Therefore, this algorithm can identify reference points that do not conform to the uniformly distributed Pareto optimal point set by adopting the adaptive method proposed by Deb. t+1 Finally, two new operations are introduced: introducing new reference points and eliminating useless reference points, always keeping the population size equal to the number of selected reference points.

[0079] Introducing a new reference point: When generating P t+1 After each individual, update the number of microhabitats at each reference point ρ j When all reference points have non-dominated individuals, let ρ j = 1. If some reference point ρ j ≥2, which will result in some reference points ρ k = 0, if the k reference point is valid, we need to find an associated individual for it, replace the k reference point with a new reference point close to the j reference point, and add a set of reference points around the j reference point. j ≥2, repeat the above steps for each reference point.

[0080] Eliminate useless reference points: Update the number of niches for all reference points. If there are exactly N reference points that satisfy ρ j =1, that is, one of the N reference points is consistent with P t+1 This shows that the distribution between reference points is uniform. j =0 reference point (excluding the original reference point).

[0081] The above adaptive operation is embedded in this scheme. In this way, the reference point is dynamically adjusted to maintain a uniform distribution, thereby improving the algorithm's search effect on the Pareto frontier of the multi-objective storage system selection problem.

[0082] The storage location adaptation algorithm of this solution uses an intelligent decision-making strategy to automatically select the optimal storage location based on the performance indicators of storage resources. This algorithm helps improve data access performance, reduce latency, and reduce unnecessary consumption of network bandwidth and storage resources. Users do not need to manually select storage locations, and the system automatically provides them with the best storage experience.

[0083] Embodiment 2

[0084] The purpose of this embodiment is to provide a management and scheduling system for a cross-domain heterogeneous storage cluster, including:

[0085] Multi-objective model building module: builds a multi-objective optimization model for storage scheduling based on the storage cost and throughput of different data sets to be stored on different storage systems, as well as the relative positions of the data sets to be stored and different storage systems;

[0086] Multi-objective model solving module: Construct constraints, take the minimum total storage cost of all data sets to be stored, the maximum total throughput, and the closest location of the data sets to be stored to the storage system to be stored as the goal, use the non-dominated sorting genetic algorithm to optimize and solve the multi-objective optimization problem, obtain the Pareto optimal solution set of the multi-objective optimization model that meets the constraints, and obtain the storage scheduling result of the data sets to be stored based on the Pareto optimal solution set.

[0087] Embodiment 3

[0088] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0089] Embodiment 4

[0090] The purpose of this embodiment is to provide a computer-readable storage medium.

[0091] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0092] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0093] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0094] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A management and scheduling method for a cross-domain heterogeneous storage cluster, characterized in that: include: A multi-objective optimization model for storage scheduling is constructed based on the storage cost and throughput of different data sets to be stored on different storage systems, as well as the relative positions of the data sets to be stored and different storage systems. Constructing constraint conditions, taking the minimum total storage cost of all data sets to be stored, the maximum total throughput, and the location of the data sets to be stored closest to the storage system to be stored as the goal, using a non-dominated sorting genetic algorithm to optimize and solve the multi-objective optimization problem, and obtaining a Pareto optimal solution set of the multi-objective optimization model that meets the constraint conditions, and obtaining a storage scheduling result of the data sets to be stored according to the Pareto optimal solution set; The non-dominated sorting genetic algorithm is used to optimize and solve the multi-objective optimization problem, specifically: Initialize the population; The binary crossover operator and polynomial mutation are used to perform crossover and mutation on individuals in the population, and the offspring population is generated by adaptively adjusting the crossover and mutation rates; The initial population and the offspring population are merged, and the merged population is divided into different non-dominated levels using the non-dominated sorting method; Using the microhabitat preservation strategy based on the reference point, the reference point is adaptively adjusted to generate the next iterative parent population; Continuously iterate and update until the preset iteration point is reached to obtain a set of optimized solutions; The reference point-based niche preservation strategy is used to adaptively adjust the reference points so that there are exactly N reference points that satisfy ρ j =1, then remove the added ρ j = 0, the removed reference points do not include the original reference points; where N is the size of the individuals in the offspring population, ρ j is the number of microhabitats at the reference point; The binary crossover operator and polynomial mutation are used to perform crossover and mutation on individuals in the population, and the crossover and mutation rates are adjusted adaptively to generate the offspring population, specifically: when the distance between an individual and its nearest individual is less than the average distance between individuals, the crossover and mutation probability increases with the increase of the number of iterations; when the distance between an individual and its nearest individual is greater than the average distance between individuals, the crossover and mutation probability decreases with the increase of the number of iterations; The formula for calculating crossover and mutation probability is: Among them, Pc1 and Pc2 are the maximum and minimum values ​​of the initially set crossover probability, Pm1 and Pm2 are the maximum and minimum values ​​of the initially set mutation probability, T is the maximum number of iterations of the population, t is the current number of iterations, ɑ is a constant, d(t) is the distance between an individual and the nearest individual, and dagv(t) is the average distance between individuals.

2. A management and scheduling method for a cross-domain heterogeneous storage cluster as claimed in claim 1, characterized in that: The constraint conditions include: each data set to be stored can only be stored in one storage system; the size of the data set to be stored is not greater than the remaining capacity of the storage system to be stored.

3. A management and scheduling method for a cross-domain heterogeneous storage cluster as claimed in claim 1, characterized in that: In the population initialization stage, for each initially generated chromosome, it is determined whether it meets the threshold requirements of the remaining capacity and throughput of the storage system, and the chromosomes that do not meet the requirements are screened out.

4. A management and scheduling method for a cross-domain heterogeneous storage cluster as claimed in claim 1, characterized in that: It also includes encoding the data sets to be stored and the storage system, specifically: the chromosome length is n, indicating n data sets to be stored; the storage system number is assigned to the position corresponding to the gene string of the chromosome, indicating that the corresponding data set to be stored is allocated to the corresponding storage system.

5. A management and scheduling system for a cross-domain heterogeneous storage cluster, characterized in that: include: Multi-objective model building module: builds a multi-objective optimization model for storage scheduling based on the storage cost and throughput of different data sets to be stored on different storage systems, as well as the relative positions of the data sets to be stored and different storage systems; Multi-objective model solving module: constructs constraint conditions, takes the minimum total storage cost of all data sets to be stored, the maximum total throughput, and the location of the data sets to be stored closest to the storage system to be stored as the goal, uses a non-dominated sorting genetic algorithm to optimize and solve the multi-objective optimization problem, obtains the Pareto optimal solution set of the multi-objective optimization model that meets the constraint conditions, and obtains the storage scheduling result of the data sets to be stored based on the Pareto optimal solution set; The non-dominated sorting genetic algorithm is used to optimize and solve the multi-objective optimization problem, specifically: Initialize the population; The binary crossover operator and polynomial mutation are used to perform crossover and mutation on individuals in the population, and the offspring population is generated by adaptively adjusting the crossover and mutation rates; The initial population and the offspring population are merged, and the merged population is divided into different non-dominated levels using the non-dominated sorting method; Using the microhabitat preservation strategy based on the reference point, the reference point is adaptively adjusted to generate the next iterative parent population; Continuously iterate and update until the preset iteration point is reached to obtain a set of optimized solutions; The reference point-based niche preservation strategy is used to adaptively adjust the reference points so that there are exactly N reference points that satisfy ρ j =1, then remove the added ρ j = 0, the removed reference points do not include the original reference points; where N is the size of the individuals in the offspring population, ρ j is the number of microhabitats at the reference point; The binary crossover operator and polynomial mutation are used to perform crossover and mutation on individuals in the population, and the crossover and mutation rates are adjusted adaptively to generate the offspring population, specifically: when the distance between an individual and its nearest individual is less than the average distance between individuals, the crossover and mutation probability increases with the increase of the number of iterations; when the distance between an individual and its nearest individual is greater than the average distance between individuals, the crossover and mutation probability decreases with the increase of the number of iterations; The formula for calculating crossover and mutation probability is: Among them, Pc1 and Pc2 are the maximum and minimum values ​​of the initially set crossover probability, Pm1 and Pm2 are the maximum and minimum values ​​of the initially set mutation probability, T is the maximum number of iterations of the population, t is the current number of iterations, ɑ is a constant, d(t) is the distance between an individual and the nearest individual, and dagv(t) is the average distance between individuals.

6. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, a management and scheduling method for a cross-domain heterogeneous storage cluster as described in any one of claims 1 to 4 is performed.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for managing and scheduling a cross-domain heterogeneous storage cluster as claimed in any one of claims 1 to 4 is executed.

Images