A data migration method and system for cross-domain heterogeneous storage systems

Abstract

The present invention proposes a data migration method and system for a cross-domain heterogeneous storage system, which solves the data migration path optimization problem by using an improved ant colony algorithm, dynamically adjusts the pheromone concentration according to the transmission success rate of the path, so that it is proportional to the transmission success rate, that is, the higher the transmission success rate of the path, the higher the corresponding pheromone concentration, thereby increasing the probability of selecting the path; dynamically adjusts the heuristic function value according to the amount of transmitted data and the load of the target node, so that it is inversely proportional to the load of the target node, the heavier the load of the node, the lower the heuristic function value of the corresponding path, thereby reducing the probability of selecting the path, thereby achieving load balancing of the storage system and improving the efficiency and quality of data migration.

Description

Technical Field

[0001] The present invention belongs to the technical field of data migration, and in particular, relates to a data migration method and system for a cross-domain heterogeneous storage system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of the Internet and the Internet of Things, traditional computing-intensive applications are transforming into data-intensive applications. The data in each data center is growing exponentially and is stored in different types of storage systems in different regions, such as distributed file systems, object storage systems, etc. As the scale of data continues to expand, more and more business scenarios require joint processing of multi-source data. The scattered data has to be migrated from the storage system of one data center to the storage system of another data center for data fusion, analysis, sharing and use. This also leads to duplicate storage and reduced data access capabilities in each data center, seriously affecting data processing efficiency. Therefore, how to achieve efficient migration and scheduling of data between different storage systems in a cross-domain environment and overcome redundant and inefficient data transmission has become an urgent problem to be solved.

[0004] At present, the data migration methods between storage systems include command line tools, cloud migration services, third-party migration tools, etc., and the solutions between heterogeneous storage systems for multiple data centers include Antep, Cloud Data Migration (CDM), and Storage Migration Service (SMS). These data migration solutions between heterogeneous storage systems for multiple data centers can usually effectively meet the data migration needs of different users, but there are also some problems and challenges. For example, Antep may have problems such as slow data migration speed, unstable performance, data loss or damage during migration, and it needs to be optimized and improved according to the characteristics and needs of different storage systems; in addition, for CDM, there may be problems such as high data migration cost, dependence on cloud service providers, privacy and security risks, and the cost and security of data migration need to be considered. When using SMS for data migration, there may be performance bottlenecks of storage virtualization platforms, dependence on suppliers, and data consistency during migration. In view of the problems of the above heterogeneous storage systems in data migration performance, cost control, security, and reliability, it is necessary to study a new data migration method and system for cross-domain heterogeneous storage systems to optimize the performance and efficiency of data migration and improve the speed and stability of data migration. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a data migration method and system for a cross-domain heterogeneous storage system, which introduces the transmission success rate to update the pheromone concentration, and introduces the transmission data volume and the load information of the target node to update the heuristic function value, thereby improving the efficiency and quality of data migration on the basis of achieving load balancing of the storage system.

[0006] To achieve the above object, a first aspect of the present invention provides a data migration method for a cross-domain heterogeneous storage system, comprising:

[0007] Build a data migration path optimization model;

[0008] An improved ant colony algorithm is used to solve the constructed data migration path optimization model to obtain the migration result; wherein the improved ant colony algorithm is: according to the data to be migrated, the data transmission success rate between different destination address node paths to be migrated, the local pheromone concentration and the global pheromone in the cycle are updated; based on the amount of data to be transmitted, the current storage status of the destination address node to be migrated, and the real-time network status, the node load function is calculated, and the value of the heuristic function is updated based on the inverse of the node load function.

[0009] A second aspect of the present invention provides a data migration system for a cross-domain heterogeneous storage system, comprising:

[0010] Building module: building a data migration path optimization model;

[0011] Solution module: Use an improved ant colony algorithm to solve the constructed data migration path optimization model to obtain the migration result; wherein, the improved ant colony algorithm is: according to the data to be migrated, the data transmission success rate between different destination address node paths to be migrated, update the local pheromone concentration and the global pheromone in the cycle process; calculate the node load function based on the amount of data to be transmitted, the current storage status of the destination address node to be migrated, and the real-time network status, and update the value of the heuristic function based on the inverse of the node load function.

[0012] The third aspect of the present invention provides a computer device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, a data migration method for a cross-domain heterogeneous storage system is performed.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a method for migrating data across heterogeneous storage systems is executed.

[0014] One or more of the above technical solutions have the following beneficial effects:

[0015] In the present invention, an improved ant colony algorithm is used to solve the data migration path optimization problem. The pheromone concentration is dynamically adjusted according to the transmission success rate of the path to make it proportional to the transmission success rate, that is, the higher the transmission success rate of the path, the higher the corresponding pheromone concentration, thereby increasing the probability of selecting the path; according to the load of the target node (the amount of data transmitted by the migration task, the node storage and network conditions), the heuristic function value is dynamically adjusted to make it inversely proportional to the node load. The heavier the load of the node, the lower the heuristic function value of the corresponding path, so as to reduce the probability of selecting the path, thereby achieving load balancing of the storage system and improving the efficiency and quality of data migration.

[0016] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0018] Figure 1 This is a schematic diagram of the system architecture in Embodiment 1 of the present invention;

[0019] Figure 2 This is a schematic diagram of a data intelligent flow controller in Embodiment 1 of the present invention;

[0020] Figure 3 This is the ant colony algorithm process in the first embodiment of the present invention;

[0021] Figure 4 This is the overall architecture of the improved ant colony algorithm in the first embodiment of the present invention. DETAILED DESCRIPTION

[0022] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0023] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0024] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0025] Embodiment 1

[0026] This embodiment is based on the distributed virtual storage system Alluxio, which integrates multiple data centers based on different types of storage systems to achieve efficient data migration.

[0027] Combination Figure 1 The cross-domain heterogeneous system built based on the distributed virtual storage system Alluxio in this embodiment is described as follows:

[0028] 1. Data lineage database

[0029] In order to trace the data flow path of the entire data migration process, a data lineage information library is established to record the source, flow and processing path of the data during the migration process, which can help users trace the flow path and processing history of the data, understand the source and destination of the data, and promptly discover and handle problems and anomalies in the data migration to ensure the smooth progress of the entire data migration process. The data lineage information library includes two parts: traceability information and metadata information. The traceability information records the source and flow of the data during the migration process, including information such as the source, transmission path and destination of the data, which can help users trace the flow path and processing history of the data, understand the source and destination of the data, and monitor the status and progress of the data during the migration process; metadata information includes the basic attributes and characteristics of the data, such as data type, size, format, structure, owner and other information, which can help users understand the structure and content of the data, analyze the characteristics and quality of the data, and optimize the strategies and plans for data migration.

[0030] The data tables in the data lineage database mainly include migration task evaluation table, migration strategy information table, migration template information table, data address table, migration task information table, etc. The migration task evaluation table mainly records information including task name, whether to evaluate the migration task, data source address, data destination address and creator; the migration strategy information table mainly records information including task id, template id, migration mode status, number of parallel jobs, migration encryption method, etc.; the migration template information table records information including template number, template name, data source storage area, data source address, etc.; the data address table mainly records information including data address, storage primary key, creator, status, etc.; the migration task information table mainly records information including migration task id, migration job id, user id, task name, etc.

[0031] 2. Intelligent data flow controller

[0032] The data intelligent flow controller is a plug-in developed based on the data migration system. It is written in Java and implemented based on the MyBatis framework. It aims to improve the efficiency, reliability and security of data flow. The controller is mainly used to manage and schedule the data flow process, including data migration, replication, synchronization and other operations.

[0033] As an important part of the entire data migration system, the execution process of the data intelligent flow controller is as follows: the Controller layer receives the user's request and uses the developed Restful interface to call the Service layer to process the request. The Service layer processes the business logic and calls the Mapper layer to access the database. The Mapper layer performs database operations and returns the results to the Service layer. Finally, the Service layer returns the processing results to the Controller layer to complete the entire data migration process. By using the Java language and the MyBatis framework, the efficient, stable and reliable operation of the data intelligent flow controller can be achieved.

[0034] The data intelligent flow controller provides a wealth of configuration options and interfaces, which can flexibly configure and manage data flow tasks. Through intelligent algorithms and strategies, data flow tasks can be managed and scheduled to achieve intelligent and automated data flow. Through the application of the data intelligent flow controller, users can monitor and manage the data migration process, improve the efficiency, reliability and security of data migration, and meet users' various needs and requirements for data migration. At the same time, as a plug-in, the controller has good flexibility and scalability, and can be customized and expanded according to the needs of different users to meet the needs of different scenarios and applications.

[0035] 3. Data security control

[0036] During the data migration process, data security control is very important, mainly including two aspects: data encryption transmission and consistency check.

[0037] In the process of data migration, the data needs to be encrypted to prevent the data from being stolen or tampered with during transmission. The encryption algorithms we use here include MD5, SHA-256, SHA-512 and other hash algorithms. Users can choose the specific encryption algorithm. After encryption, the data is transmitted from the source node to the destination node.

[0038] After the data migration is completed, the transmitted data needs to be checked and calculated to ensure the integrity and consistency of the data. The specific process is to perform hash algorithm calculations such as MD5 and SHA on the transmitted data, compare the calculated information with the original data, and check whether the data has been lost, damaged, or tampered with. If the checksum of the data is inconsistent with the expected checksum, it means that there may be problems with the data. At this time, the data can be retransmitted or other measures can be taken to ensure the security and integrity of the data.

[0039] Data migration systems often need to transmit data under different network environments and load conditions. By combining the improved ant colony algorithm with the automatic flow in the current computing network operating system, it has real-time adaptability and can dynamically adjust the data transmission path according to the real-time load of the target node and the working status of the node to ensure the stability of data migration.

[0040] A data lineage database is added to the system to comprehensively record and manage the source and flow of data during the migration process, providing reliable monitoring and management support for the data migration process; a data security control module is added to ensure the confidentiality and integrity of the data and prevent data leakage and tampering.

[0041] like Figure 4 As shown, this embodiment proposes a data migration method for a cross-domain heterogeneous storage system. The cross-domain heterogeneous system built based on the distributed virtual storage system Alluxio is taken as an example for explanation. The data migration method specifically includes:

[0042] Build a data migration path optimization model;

[0043] An improved ant colony algorithm is used to solve the constructed data migration path optimization model to obtain the migration result; wherein the improved ant colony algorithm is: according to the data to be migrated, the data transmission success rate between different destination address node paths to be migrated, the local pheromone concentration and the global pheromone in the cycle are updated; based on the amount of data to be transmitted and the current storage status and real-time network status of the destination address node to be migrated, the node load function is calculated, and the value of the heuristic function is updated based on the inverse of the node load function.

[0044] Taking into account the characteristics of cross-domain heterogeneous storage systems, the data transmission success rate and node workload (amount of data to be transmitted, storage status, network status) are introduced to improve the ant colony algorithm. The improved algorithm is used to achieve path optimization, rationally utilize storage system and network resources, reduce the cost of data migration, and improve resource utilization.

[0045] like Figure 3 As shown, before introducing the improved version of the algorithm, the process of the existing ant colony algorithm is briefly introduced through the following flowchart, as shown below:

[0046] Assume that there are n nodes to be selected for the destination address to be migrated, the node set is represented by N, and P is represented by ij represents the probability of an ant (representing the data to be migrated) moving from node i to node j. According to the principle of the ant colony algorithm, the moving probability P ij The calculation formula is as follows:

[0047]

[0048] Among them, τ(i,j) represents the concentration of pheromone from node i to node j, and η(i,j) is a heuristic information. α represents the importance of pheromone. The larger its value, the greater the probability that the ant will choose the path it has passed before; β represents the importance of the heuristic function. The larger its value, the greater the possibility that the ant will choose the route it has passed before; using the data lineage information library mentioned in the previous part, a migration database is established for the data migration process, and the data table T is used to record the node paths that the ants have passed during this migration process.

[0049] 1. Adaptive adjustment of parameters based on data transmission delay

[0050] Adaptive parameter adjustment is to dynamically adjust the key parameters in the ant colony algorithm in order to maintain the optimal performance of the algorithm under different environments and load conditions. This adjustment can improve the efficiency, reliability and adaptability of data migration. When the ant colony algorithm is applied to data migration tasks in different situations, its performance depends largely on the two key parameters α and β mentioned above, which are the parameters that determine the importance of pheromones and heuristic functions respectively, and these two parameters further determine the weight of pheromones and heuristic functions in path selection. In the existing applications of ant colony algorithms in various fields, these parameters are usually set statically, that is, they are set before the algorithm runs and remain unchanged throughout the process. However, the specific environment of the data migration system for different types of data migration tasks is dynamically changing, so static parameter settings may not be able to adapt to all situations.

[0051] Here we intend to adopt the idea of ​​adaptive parameter adjustment, dynamically adjust the above parameters according to the real-time monitored data transmission delay, so as to optimize the data transmission performance in different environments. By dynamically adjusting these parameters, we can balance the dependence on pheromones and heuristic functions, optimize the path selection strategy according to the real-time environment, and thus improve the efficiency and reliability of data migration. The specific implementation steps of adaptive parameter adjustment are as follows:

[0052] First, we need to define the parameter adjustment rules. The specific rules are as follows: when the node load is high, increase the value of β to increase the weight of the reliance on the load factor in the heuristic function and avoid selecting high-load nodes; when the data transmission success rate is low, increase the value of α to increase the weight of the reliance on the success rate and select a more reliable path.

[0053] The dynamic adjustment formula is based on the feedback control system (PID controller is proposed to be used) for parameter adjustment. The PID controller adjusts the parameters through three parts: proportional (P), integral (I) and differential (D) to achieve the purpose of adaptive optimization. The PID controller dynamically adjusts the parameters through the following formula:

[0054]

[0055] Among them, K(t) is the parameter value at the current moment (here used to calculate α, β), K(t-1) is the parameter value at the previous moment, Kp is the proportional coefficient, K i is the integration coefficient, K d is the differential coefficient, e(t) is the error at the current moment, e(t-1) is the error at the previous moment, and the error e(t) reflects the difference between the current system state and the target state.

[0056] For the current data migration system, the error is defined as the difference between the actual data transmission delay and the expected value, that is:

[0057] e(t)=DC (3)

[0058] In the above formula, D represents the expected transmission delay, and C represents the transmission delay corresponding to the actual data migration task. The system monitors the actual data transmission delay in real time and calculates the error e(t) based on the monitored current value and the expected value. Formula (2) is used to dynamically adjust parameters α and β.

[0059] The specific adjustment examples of each parameter are as follows:

[0060] (1) Adjust the pheromone importance factor (α):

[0061]

[0062] (2) Adjust the importance factor of the heuristic function (β):

[0063]

[0064] The above method dynamically adjusts the two importance factors α and β of the ant colony algorithm, so that the algorithm can better adapt to complex and changing environments and improve the efficiency and reliability of data migration. This method uses a feedback control mechanism to continuously optimize parameter settings, which can effectively improve the performance of the ant colony algorithm.

[0065] The following is an introduction to the calculation of pheromone in the ant colony algorithm: At the beginning of the (t+1)th migration task, the global pheromone τ (t+1) The update process formula of (i,j) in each iteration is as follows:

[0066]

[0067] Among them, ρ is the volatility coefficient of pheromone. The specific value is specified according to the different migration tasks. It controls the volatility of global pheromone and usually takes a value between (0, 1); m represents the number of ants in each iteration. In the data migration task, the number of ants is abstracted into the number of data migration tasks; τ t (i,j) represents the pheromone concentration corresponding to the tth migration task, It is the increment of pheromone left on the path i→j during the kth migration; the global pheromone is updated by the increment of pheromone left by the ants on the path.

[0068] From the above analysis, we can see that there are two factors that affect the final result of the ant colony algorithm, namely pheromone concentration and heuristic function. Combined with the characteristics of cross-domain heterogeneous storage systems, in the storage system, the success rate of data transmission directly affects the efficiency and quality of data migration.

[0069] First, the pheromone concentration is updated by introducing the transmission success rate; the pheromone concentration is dynamically adjusted according to the transmission success rate of the path, so that it is proportional to the transmission success rate, that is, the higher the transmission success rate of the path, the higher the corresponding pheromone concentration, thereby increasing the probability of selecting the path; on the other hand, in the storage system, the load of the node is dynamically changing, and excessive load will affect the efficiency and quality of data migration; the amount of data involved in the migration task will also affect the efficiency of data transmission, so the heuristic function value can be updated by introducing the amount of transmitted data and node load information. That is, the heuristic function value can be dynamically adjusted according to the calculated node load function, so that it is inversely proportional to the node load. The heavier the load of the node, the lower the heuristic function value of the corresponding path, so as to reduce the probability of selecting the path, thereby achieving load balancing of the storage system.

[0070] Therefore, it is proposed to update the pheromone concentration based on the transmission success rate and to update the heuristic function based on the amount of transmitted data and the workload of the target node, so as to better adapt to the characteristics of cross-domain heterogeneous storage systems and improve the efficiency and quality of data migration.

[0071] The data transmission success rate is calculated by the ratio of the amount of data successfully transmitted to the total amount of data transmitted. First, a data table is established to record whether each data transmission is successful. The database fields include transmission success, transmission failure, and transmission success rate. When each migration task is completed, each field in the database is updated and the total transmission success rate is calculated. Assuming that the amount of data successfully transmitted is S and the total amount of data transmitted is T, the data transmission success rate P can be expressed as:

[0072]

[0073] ∈ is used to represent a threshold of the transmission success rate. When the pheromone concentration is updated, if it is higher than the specified threshold, the pheromone concentration needs to be updated, otherwise no adjustment is made. The specific update rule is expressed by the following formula:

[0074] Local pheromone Δτ k (i,j) Update:

[0075]

[0076] Global pheromone Δτ(i,j) update:

[0077] Δτ(i,j)=p max (9)

[0078] Substituting equations (8) and (9) into equation (1) respectively, we can obtain the local pheromone concentration and global pheromone concentration corresponding to this migration task (the specific calculation rules are in p max ), establish a pheromone concentration data table, and record the values ​​of the two pheromone concentrations obtained each time, so as to serve as the probability of migrating data from node i to node j in subsequent calculations.

[0079] Δτ k (i,j) represents the local pheromone concentration for migrating data from node i to node j in the kth cycle. When performing a specific data migration task, when selecting the next target node, assume that there are k nodes under the target address for selection, calculate the transmission success rate of the historical transmission tasks corresponding to these k nodes as the local pheromone concentration corresponding to the node, and store the calculated results in the order of transmission success rate from high to low in the pheromone concentration data table established in advance, and take the highest value of the transmission success rate corresponding to each target node as the global pheromone concentration value p corresponding to this transmission task. max .

[0080] First, considering that the amount of migration data involved in individual data migration tasks is large, it is necessary to split this part of the migration task and then transmit it. Therefore, the amount of transmission data is included in the consideration of the improvement of the heuristic function of the ant colony algorithm; based on the workload of the target node, the current operating status of the node and resource utilization can be considered. This covers multiple aspects of indicators that can help evaluate the performance and stability of the node. Based on the characteristics of data migration tasks in cross-domain heterogeneous storage systems for multiple data sources, it is planned to specifically evaluate the workload of the target storage node based on the storage status of the target address storage device and the real-time network status; in addition, the specific calculation methods of the three indicators are introduced below:

[0081] (1) Amount of data transmitted

[0082] When performing a data migration task with a large amount of data, transmitting all the data at once may cause the destination node to be overloaded. In this case, consider splitting large files or large data sets into multiple blocks and transmitting these blocks in parallel. This can make full use of network bandwidth and system resources and improve the efficiency and speed of data transmission. The specific transmission rules are as follows: When the amount of data transmitted in a single time reaches 10GB or exceeds the limit of the target storage system, consider transmitting the data in blocks. According to the specific amount of data to be transmitted and the actual situation of the specific migration task, the overall data is divided into n parts for parallel transmission of data. Each data block after segmentation will be assigned a unique number for identification and tracking during the transmission process. After the data is segmented, a check code is generated for each data block for data verification. Here, MD5 hash check is proposed. The generation of the check code usually considers the content and size of the data block, as well as the stability of the transmission environment, and then the segmented data blocks are transmitted to the target node one by one. During the transmission process, each data block will carry its corresponding number and check code. After receiving the data, the transmission node will verify the check code. After the data transmission is completed, the target node will merge the received data blocks in the order of numbering to restore the original large data file. During the merging process, verification will be performed again to ensure the integrity and correctness of all data blocks.

[0083] When block transmission is used, the proportion R of the current transmitted data volume in this migration task can be expressed as follows:

[0084]

[0085] Among them, V represents the amount of data transmitted this time, and T represents the total amount of data transmitted.

[0086] (2) Storage status of the target address

[0087] The storage status of the target address is usually expressed by the storage capacity utilization rate, which can be calculated by the ratio of the current storage usage of the target node to the total storage capacity. The specific formula is as follows:

[0088]

[0089] Among them, U storage represents the storage capacity utilization, C used Indicates the current storage usage, C total Indicates total storage capacity.

[0090] (3) Real-time network status

[0091] The real-time network status can be described by available bandwidth. Available bandwidth refers to the bandwidth resources actually available for use in the network. The higher the available bandwidth, the better the network status, the faster the data transmission speed, and the lower the network delay. bandwidth Calculated by the following formula:

[0092]

[0093] Among them, u bandwidth Indicates the proportion of available bandwidth, B available Indicates the actual available bandwidth, B total Indicates the total network bandwidth.

[0094] Next, based on the characteristics of multiple storage systems of different types at different locations, the load function of the target node to be migrated is defined as the weighted sum of the above three factors (including the amount of data transferred in this migration task and the workload of the target node), calculated as follows:

[0095] L=γ1R+γ2Ustorage+γ3u bandwidth (13)

[0096] γ1, γ2, and γ3 represent the weights of the amount of data transmitted, the storage status of the target address, and the real-time network status in the node load function. When the data is not transmitted in blocks, γ3 is 0. In other cases, the specific weights of the three variables are adjusted according to the actual situation of the migration task.

[0097] In order to make the calculation of the load function more accurate, the average value of the load of each target migration node is calculated as the overall load function. Assuming that there are n migration target nodes to choose from, the specific calculation is as follows:

[0098]

[0099] The inverse of the averaged load function is further taken as the value of the improved heuristic function, as follows:

[0100]

[0101] In summary, after the above improved ant colony algorithm, the moving probability P ij The improvements are as follows:

[0102]

[0103] After the improvement is completed, the improved ant colony algorithm will calculate the probability of each target path corresponding to the migration task path being selected, sort this part of the calculated probability information and store it in the data lineage information database, and use the greedy algorithm to select the target path with the highest probability for data migration. By continuously iterating the migration probability in the information database, the improved ant colony algorithm can intelligently select the migration task plan that has the highest transmission success rate for the migration task and can effectively take into account the working status of the target node.

[0104] This example adds an improved ant colony algorithm to the data migration system to help the system select the optimal data transmission path. By dynamically adjusting the path selection probability and pheromone update strategy, the transmission success rate is improved and the efficiency of data migration is accelerated.

[0105] Embodiment 2

[0106] The purpose of this embodiment is to provide a data migration system for a cross-domain heterogeneous storage system, including:

[0107] Building module: building a data migration path optimization model;

[0108] Solution module: Use an improved ant colony algorithm to solve the constructed data migration path optimization model to obtain the migration result; wherein, the improved ant colony algorithm is: according to the data to be migrated, the data transmission success rate between different destination address node paths to be migrated, update the local pheromone concentration and the global pheromone in the cycle process; calculate the node load function based on the current storage status, real-time network status and amount of data to be transmitted of the destination address node to be migrated, and update the value of the heuristic function based on the inverse of the node load function.

[0109] Embodiment 3

[0110] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0111] Embodiment 4

[0112] The purpose of this embodiment is to provide a computer-readable storage medium.

[0113] A computer-readable storage medium stores a computer program, which executes the steps of the above method when executed by a processor.

[0114] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0115] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0116] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A data migration method for a cross-domain heterogeneous storage system, characterized in that: include: Build a data migration path optimization model; The constructed data migration path optimization model is solved by using an improved ant colony algorithm to obtain a migration result; wherein the improved ant colony algorithm is as follows: according to the data to be migrated, the data transmission success rate between different destination address node paths to be migrated, the local pheromone concentration and the global pheromone in the cycle are updated; based on the amount of data to be transmitted, the current storage status of the destination address node to be migrated, and the real-time network status, the node load function is calculated, and the value of the heuristic function is updated based on the inverse of the node load function; Determine the data transmission success rate based on the ratio between the amount of successfully transmitted data and the total amount of transmitted data; According to the data to be migrated, the data transmission success rate between different destination address node paths to be migrated is used to update the local pheromone concentration in the cycle process, specifically: If the data transmission success rate is greater than a set threshold, the local pheromone is updated based on the data transmission success rate, otherwise no adjustment is made; According to the data to be migrated and the data transmission success rate between different destination address node paths to be migrated, the global pheromone in the cycle is updated, specifically: In each cycle, the transmission success rate of all historical transmission tasks corresponding to the next target node of data migration is calculated respectively, and each calculated result is used as the local pheromone concentration of the node; The calculated local pheromone concentrations of the next target node of data migration are sorted in descending order, and the highest transmission success rate corresponding to each next target node of data migration is used as the global pheromone value in this cycle; Based on the actual data transmission delay and the expected transmission delay, the PID controller is used to adaptively adjust the local pheromone concentration and the weight parameters of the global pheromone respectively; the PID controller adjusts the parameters through three parts: proportional P, integral I and differential D to achieve the purpose of adaptive optimization; The PID controller dynamically adjusts the parameters using the following formula: ； in, K ( t ) is the parameter value at the current moment, K ( t -1) is the parameter value at the previous moment, K p is the proportionality coefficient, K i is the integration coefficient, K d is the differential coefficient, e ( t ) is the error at the current moment, e ( t -1) is the error at the previous moment, the error e ( t ) reflects the difference between the current system state and the target state; For the current data migration system, the error is defined as the difference between the actual data transmission delay and the expected value, that is: ； In the above formula, D represents the expected transmission delay, C Indicates the transmission delay corresponding to the actual data migration task; the system monitors the actual data transmission delay in real time and calculates the error based on the monitored current value and expected value e ( t ); Adjust pheromone importance factor a : 。 2. The data migration method of a cross-domain heterogeneous storage system according to claim 1, characterized in that: Determine the storage status of the node based on the ratio of current storage usage to total storage capacity; The real-time network status of the node is determined based on the ratio of the actual available bandwidth to the total network bandwidth; the amount of data to be transmitted is determined based on the proportion of the current amount of data transmitted in this migration.

3. The data migration method of a cross-domain heterogeneous storage system according to claim 1, characterized in that: The calculated average value of the load of each destination address node to be migrated is used as the overall load function, and the inverse of the overall load function is taken as the value of the heuristic function.

4. A data migration system for a cross-domain heterogeneous storage system, characterized in that: include: Building module: building a data migration path optimization model; Solution module: using an improved ant colony algorithm to solve the constructed data migration path optimization model to obtain a migration result; wherein the improved ant colony algorithm is: according to the data to be migrated, the data transmission success rate between different destination address node paths to be migrated, the local pheromone concentration and the global pheromone in the cycle are updated; based on the amount of data to be transmitted, the current storage status, and the real-time network status of the destination address node to be migrated, the node load function is calculated, and the value of the heuristic function is updated based on the inverse of the node load function; Determine the data transmission success rate based on the ratio between the amount of successfully transmitted data and the total amount of transmitted data; According to the data to be migrated, the data transmission success rate between different destination address node paths to be migrated is used to update the local pheromone concentration in the cycle process, specifically: If the data transmission success rate is greater than a set threshold, the local pheromone is updated based on the data transmission success rate, otherwise no adjustment is made; According to the data to be migrated and the data transmission success rate between different destination address node paths to be migrated, the global pheromone in the cycle is updated, specifically: In each cycle, the transmission success rate of all historical transmission tasks corresponding to the next target node of data migration is calculated respectively, and each calculated result is used as the local pheromone concentration of the node; The calculated local pheromone concentrations of the next target node of data migration are sorted in descending order, and the highest transmission success rate corresponding to each next target node of data migration is used as the global pheromone value in this cycle; Based on the actual data transmission delay and the expected transmission delay, the PID controller is used to adaptively adjust the local pheromone concentration and the weight parameters of the global pheromone respectively; the PID controller adjusts the parameters through three parts: proportional P, integral I and differential D to achieve the purpose of adaptive optimization; The PID controller dynamically adjusts the parameters using the following formula: ； in, K ( t ) is the parameter value at the current moment, K ( t -1) is the parameter value at the previous moment, K p is the proportionality coefficient, K i is the integration coefficient, K d is the differential coefficient, e ( t ) is the error at the current moment, e ( t -1) is the error at the previous moment, the error e ( t ) reflects the difference between the current system state and the target state; For the current data migration system, the error is defined as the difference between the actual data transmission delay and the expected value, that is: ； In the above formula, D represents the expected transmission delay, C Indicates the transmission delay corresponding to the actual data migration task; the system monitors the actual data transmission delay in real time and calculates the error based on the monitored current value and expected value e ( t ); Adjust pheromone importance factor a : 。 5. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, a data migration method for a cross-domain heterogeneous storage system as described in any one of claims 1 to 3 is performed.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for migrating data in a cross-domain heterogeneous storage system according to any one of claims 1 to 3 is executed.

Images