A data migration method, product, device and medium

By using deep learning model evaluation and consistent hashing algorithm in distributed storage systems, the problem of data migration relying on client requests is solved, flexible and efficient data migration is achieved, network load and latency are reduced, and data management is improved efficiency.

CN118672517BActive Publication Date: 2024-11-08INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411149583.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-08
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

Data migration in distributed storage systems relies on client requests and is not flexible enough, which may lead to additional network load and latency.

Method used

Use preset cycles to obtain the basic attributes of the target data in the node, evaluate the importance, access speed requirements and backup requirements of the data based on the deep learning model, formulate migration strategies and use a consistent hashing algorithm to migrate data to ensure that the data is in the target node. equilibrium distribution between.

Benefits of technology

It realizes flexible data migration without relying on client requests, reduces network load and delay, and improves the data management efficiency of distributed storage systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118672517B_ABST
    Figure CN118672517B_ABST
Patent Text Reader

Abstract

The present invention discloses a data migration method, product, device and medium, and relates to the field of distributed storage systems. The solution does not need to rely on the data migration request of the client, actively monitors and obtains the basic attributes of the target data in the node according to the preset period, and determines the evaluation result of the target data according to the basic attributes of the target data based on the data migration evaluation model, so as to judge whether the target data needs to be migrated according to the evaluation result. When it is confirmed that the target data needs to be migrated, a migration strategy for the target data is formulated according to the evaluation result, and the migration of the target data to the target node is realized in combination with the migration strategy. The data migration has the advantage of being more flexible, and will not generate additional network load and delay, thereby realizing efficient management of data in the distributed storage system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of distributed storage systems, and in particular to a data migration method, product, device and medium. Background Art

[0002] Distributed storage systems are crucial in modern information technology, processing massive amounts of data from applications such as cloud computing, big data analytics, and the Internet of Things. These systems have excellent scalability, flexibility, and fault tolerance. As the amount of data increases and application requirements diversify, how to efficiently manage and migrate distributed stored data becomes a key issue.

[0003] Some existing data migration solutions still have some limitations and shortcomings in their implementation. For example, the migration trigger method mainly relies on client requests, which may lead to failure to respond to storage status changes in a timely manner; the lack of intelligent decision-making capabilities may make data migration strategies inflexible; in large-scale environments, broadcasting messages to obtain target server disk information may increase additional network load and latency. Therefore, solving these challenges and limitations is the key to further improving the performance, efficiency and reliability of distributed storage systems.

[0004] In view of the above problems, how to solve the problem that data migration in the current distributed storage system relies on client requests, is not flexible enough, and may generate additional network load and delay is an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0005] The purpose of the present invention is to provide a data migration method, product, device and medium to solve the problem that data migration in the current distributed storage system depends on client requests, is not flexible enough, and may generate additional network load and delay.

[0006] In order to solve the above technical problems, the present invention provides a data migration method, which is applied to a distributed storage system; the method comprises:

[0007] Acquire basic attributes of target data in the node according to a preset period; wherein the basic attributes at least include data type and data size;

[0008] Based on the data migration evaluation model, determining the evaluation result of the target data according to the basic attributes of the target data; wherein the data migration evaluation model is a pre-created deep learning model for evaluating the importance, access speed requirements and backup requirements of the data;

[0009] Determining whether the target data needs to be migrated according to the evaluation result;

[0010] If yes, a migration strategy for the target data is formulated according to the evaluation result; wherein the migration strategy at least includes a target node for migration, a migration time, and a migration method;

[0011] The target data is migrated to the target node according to the migration strategy.

[0012] On the one hand, the data migration evaluation model is based on which the evaluation result of the target data is determined according to the basic attributes of the target data, including:

[0013] Extracting feature vectors according to basic attributes of the target data; wherein the feature vectors include data type, data size, user usage frequency and modification records;

[0014] The feature vector is input into the data migration assessment model to output the importance label, access speed requirement and backup requirement corresponding to the target data.

[0015] On the other hand, the inputting of the feature vector into the data migration assessment model to output the importance label, access speed requirement and backup requirement corresponding to the target data includes:

[0016] Mapping the feature vector into a vector space to obtain the mapped feature vector;

[0017] Calculate an attention score of the data transfer evaluation model based on the query matrix, the key matrix, the value matrix, the vector dimension, and the activation function;

[0018] Performing a dot multiplication operation on the data feature vector and the node feature vector in the mapped feature vector to obtain a feature similarity score;

[0019] Determine the transition probability between the target data and the node according to the path between the target data and the node and the occurrence probability of the edge in the path;

[0020] Determining a local topological structure similarity score based on the node feature vector and the feature vectors of neighboring nodes;

[0021] Determining a credibility score according to the feature similarity score, the transition probability and the local topological structure similarity score;

[0022] Retain the neighbor nodes whose credibility scores are not less than a threshold to obtain a neighbor node set;

[0023] Determine a weight value of the neighbor node under the data migration evaluation model according to the attention score;

[0024] Determining a final vector representation of the target data according to the weight value and the node feature vector of the neighbor node in the neighbor node set;

[0025] The final vector representation is input into a fully connected layer to output the importance label, access speed requirement and backup requirement corresponding to the target data.

[0026] On the other hand, judging whether the target data needs to be migrated according to the evaluation result includes:

[0027] Obtaining an initial importance label, an initial access speed requirement, and an initial backup requirement of the target data in the node; wherein the initial importance label, the initial access speed requirement, and the initial backup requirement are generated by the data migration evaluation model when the target data is first stored in the node;

[0028] Determine whether the importance label of the target data is the same as the initial importance label, the access speed requirement is the same as the initial access speed requirement, and the backup requirement is the same as the initial backup requirement;

[0029] If so, confirming that the target data does not need to be migrated;

[0030] If not, it is confirmed that the target data needs to be migrated.

[0031] On the other hand, formulating a migration strategy for the target data according to the evaluation result includes:

[0032] When the importance label of the target data is different from the initial importance label, and / or the access speed requirement is different from the initial access speed requirement, and / or the backup requirement is different from the initial backup requirement, determining the target node and target storage medium for migration according to the importance label of the target data and / or the access speed requirement and / or the backup requirement;

[0033] Set the migration time and migration method for the target data.

[0034] On the other hand, migrating the target data to the target node according to the migration strategy includes:

[0035] Determine the data type of the target data;

[0036] Based on the migration time and the migration method, the target data is migrated to the target storage medium of the target node using a consistent hashing algorithm.

[0037] On the other hand, the step of migrating the target data to the target storage medium of the target node by using a consistent hashing algorithm includes:

[0038] Calculating a hash value of the target data according to a hash function;

[0039] Determine the target node for storing the target data corresponding to the hash value according to the routing algorithm;

[0040] The target data is migrated to the target storage medium of the corresponding target node.

[0041] On the other hand, after migrating the target data to the target node according to the migration strategy, the method further includes:

[0042] Generate a checksum of the target data after migration according to a preset checksum algorithm;

[0043] Determining whether a checksum of the target data before migration and a checksum of the target data after migration are the same;

[0044] If yes, confirm that the target data after migration is correct;

[0045] If not, it is confirmed that the target data after migration is incorrect.

[0046] On the other hand, if it is confirmed that the target data after migration is incorrect, the method further includes:

[0047] Obtaining an original image file of the target data;

[0048] The original image file of the target data is re-migrated to the target node.

[0049] On the other hand, after migrating the target data to the target node according to the migration strategy, the method further includes:

[0050] Monitoring the load status of each of the target nodes; wherein the load status at least includes capacity, data access frequency and processing performance;

[0051] The target data is controlled to be evenly distributed among the target nodes according to the load status.

[0052] On the other hand, it also includes:

[0053] When a new node is added to the distributed storage system, the load status of the new node is obtained;

[0054] According to the load status of the new node and the load status of the target node, controlling the target data to be evenly distributed between the new node and the target node;

[0055] When it is detected that the target node exits in the distributed storage system, the target data is controlled to be evenly distributed among the remaining target nodes.

[0056] On the other hand, before acquiring the basic attributes of the target data in the node according to the preset period, the method further includes:

[0057] generating a unique identifier for the target data;

[0058] The location information and processing status of the target data are tracked according to the unique identifier of the target data.

[0059] In order to solve the above technical problem, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the above data migration method when executed by a processor.

[0060] In order to solve the above technical problems, the present invention further provides a data migration device, comprising:

[0061] Memory for storing computer programs;

[0062] A processor is used to implement the steps of the above-mentioned data migration method when executing the computer program.

[0063] In order to solve the above technical problem, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above data migration method are implemented.

[0064] The data migration method provided by the present invention is applied to a distributed storage system; basic attributes of target data in a node are obtained according to a preset period; wherein the basic attributes include at least a data type and a data size; based on a data migration evaluation model, an evaluation result of the target data is determined according to the basic attributes of the target data; wherein the data migration evaluation model is a pre-created deep learning model for evaluating the importance, access speed requirements and backup requirements of the data; whether the target data needs to be migrated is determined according to the evaluation result; if so, a migration strategy for the target data is formulated according to the evaluation result; wherein the migration strategy includes at least a target node, a migration time and a migration method for migration; and the target data is migrated to the target node according to the migration strategy.

[0065] The beneficial effect of the present invention is that the solution does not need to rely on the data migration request of the client, but actively monitors and obtains the basic attributes of the target data in the node according to the preset period, and determines the evaluation result of the target data based on the basic attributes of the target data based on the data migration evaluation model, so as to judge whether the target data needs to be migrated according to the evaluation result. When it is confirmed that the target data needs to be migrated, a migration strategy for the target data is formulated according to the evaluation result, and the migration of the target data to the target node is realized in combination with the migration strategy. The data migration has the advantage of being more flexible, and will not generate additional network load and delay, thereby realizing efficient management of data in the distributed storage system.

[0066] Another scheme, the present invention specifically obtains the initial importance label, initial access speed requirement and initial backup requirement of the target data in the node; determines whether the importance label of the target data is the same as the initial importance label, and the access speed requirement is the same as the initial access speed requirement, and the backup requirement is the same as the initial backup requirement; if so, it is confirmed that the target data does not need to be migrated; otherwise, it is confirmed that the target data needs to be migrated. Accurate judgment on whether the target data needs to be migrated is achieved. According to the importance label and / or access speed requirement and / or backup requirement of the target data, the target node and target storage medium for migration are determined, and the migration time and migration method for migrating the target data are set, thereby achieving the setting of the migration strategy, so as to migrate the target data according to the migration strategy. By determining the data type of the target data; based on the migration time and migration method, the consistent hashing algorithm is used to migrate the target data to the target storage medium of the target node, thereby ensuring the balanced distribution of the target data among the target nodes.

[0067] In addition, the present invention also provides a computer program product, a data migration device and a medium, which have the same effects as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0069] Figure 1 A flowchart of a data migration method provided by an embodiment of the present invention;

[0070] Figure 2 A schematic diagram of a data migration device provided by an embodiment of the present invention;

[0071] Figure 3 A schematic diagram of a data migration device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] The core of the present invention is to provide a data migration method, product, device and medium to solve the problem that data migration in the current distributed storage system depends on client requests, is not flexible enough, and may generate additional network load and delay.

[0074] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0075] Distributed storage systems are crucial in modern information technology, processing massive amounts of data from applications such as cloud computing, big data analytics, and the Internet of Things. These systems offer excellent scalability, flexibility, and fault tolerance. However, as data volumes increase and application requirements diversify, how to efficiently manage and migrate distributed stored data becomes a critical issue. The main challenges facing current distributed storage systems include: data migration and access efficiency. Designing an efficient distributed storage system requires solving the problem of how to quickly migrate data while minimizing interruptions during the migration process. Data consistency and reliability. In a distributed environment, it is critical to ensure that all users obtain the latest data. Data migration operations must consider consistency and reliability to avoid data loss or corruption. Automated and intelligent management. As the size of data centers grows, the cost and complexity of manual management increase, so automated and intelligent data management strategies are needed.

[0076] Some existing data migration solutions still have some limitations and shortcomings, such as the migration trigger method mainly relies on client requests, which may lead to the inability to respond to storage status changes in a timely manner; the lack of intelligent decision-making capabilities may make the data migration strategy inflexible; in a large-scale environment, broadcasting messages to obtain target server disk information may increase additional network load and delay. Therefore, in order to solve the above problems, the present invention provides a data migration method.

[0077] Figure 1 A flow chart of a data migration method provided by an embodiment of the present invention. The method is applied to a distributed storage system; Figure 1 As shown, the method includes:

[0078] S10: Obtain basic attributes of the target data in the node according to a preset period.

[0079] The basic attributes at least include data type and data size.

[0080] In order to maintain optimized data storage and efficient operation of the system, this solution will regularly check the storage status of the data and perform data migration as needed. Specifically, the basic attributes of the target data in the node are obtained according to a preset period.

[0081] It should be noted that the data type of the target data in the node in this embodiment may include JPEG images, MOV video files, JSON structured data or other binary data, which is not limited in this embodiment. The target data is pre-stored in the node, and the basic attributes of the target data in the node are obtained every preset period, specifically the data type and data size of the target data. In this embodiment, there is no restriction on the preset period, for example, it is set at the low-frequency use period of the node by users at night every day, depending on the specific implementation situation.

[0082] S11: Based on the data migration evaluation model, determine the evaluation result of the target data according to the basic attributes of the target data.

[0083] Among them, the data migration assessment model is a pre-created deep learning model for evaluating the importance, access speed requirements and backup requirements of data.

[0084] The data migration assessment model is a pre-created deep learning model that is used to evaluate feature vectors extracted from the basic attributes of the target data to obtain the importance, access speed requirements, and backup requirements of the target data.

[0085] It should be noted that importance refers to the importance of data. For example, there are some high-importance data that need to be saved; access speed requirement refers to the frequency of access to data. For example, there are some high-importance data whose access frequency is often high; backup requirement refers to whether the data needs to be backed up. The above information is an important criterion for whether to migrate data before data migration. Therefore, in this embodiment, based on the data migration evaluation model, the evaluation result of the target data is determined according to the basic attributes of the target data, that is, the importance of the target data, the access speed requirement and the backup requirement are determined, so as to judge whether the target data needs to be migrated in the future.

[0086] In this embodiment, there is no limitation on the specific process of determining the evaluation result of the target data through the data migration evaluation model, which depends on the specific implementation situation.

[0087] S12: Determine whether the target data needs to be migrated based on the evaluation result; if so, proceed to step S13; if not, return to step S10.

[0088] S13: Develop a migration strategy for the target data based on the evaluation results.

[0089] The migration strategy at least includes the target node, migration time and migration method.

[0090] S14: Migrate the target data to the target node according to the migration strategy.

[0091] After obtaining the evaluation result of the target data, it is determined whether the target data needs to be migrated according to the evaluation result. In this embodiment, the specific process of determining whether the target data needs to be migrated according to the evaluation result is not limited and depends on the specific implementation situation.

[0092] If the target data does not need to be migrated according to the evaluation result, the process returns to S10 and the basic attributes of the target data in the node are obtained again according to the preset period. If the target data needs to be migrated according to the evaluation result, a migration strategy for the target data is formulated according to the evaluation result.

[0093] It should be noted that the formulated migration strategy at least includes the target node for migration, the migration time, and the migration method. For example, the migration strategy may specifically include migrating from the solid state drive (SSD) of the current node to the mechanical hard disk (HDD) of the target node, migrating during the period when the load of the current node and the target node is low, and using an efficient transmission protocol for migration to ensure that the data is continuously available during the migration process. In this embodiment, there is no restriction on the specific content included in the migration strategy, which depends on the specific implementation situation.

[0094] Finally, the target data is migrated to the target node according to the established migration strategy, thus achieving the migration of the target data. In this embodiment, the specific process of migrating the target data according to the migration strategy is not limited and depends on the specific implementation situation.

[0095] In this embodiment, the solution does not need to rely on the client's data migration request. It actively monitors and obtains the basic attributes of the target data in the node according to the preset period, and determines the evaluation results of the target data based on the basic attributes of the target data based on the data migration evaluation model, so as to judge whether the target data needs to be migrated according to the evaluation results. When it is confirmed that the target data needs to be migrated, a migration strategy for the target data is formulated according to the evaluation results, and the migration of the target data to the target node is realized in combination with the migration strategy. Data migration has the advantage of being more flexible, does not generate additional network load and delay, and realizes efficient management of data in the distributed storage system.

[0096] On the basis of the above embodiments, in some embodiments, based on the data migration evaluation model, the evaluation result of the target data is determined according to the basic attributes of the target data, including:

[0097] S110: Extracting feature vectors according to basic attributes of target data.

[0098] The feature vector includes data type, data size, user usage frequency and modification record.

[0099] S111: Input the feature vector into the data migration assessment model to output the importance label, access speed requirement and backup requirement corresponding to the target data.

[0100] In the specific implementation, in order to determine the evaluation results of the target data, the feature vector is extracted based on the basic attributes of the target data. It should be noted that the feature vector specifically includes data type, data size, user usage frequency and modification record. The feature vector is further input into the data migration evaluation model to output the importance label, access speed requirement and backup requirement corresponding to the target data. The specific process of the data migration evaluation model outputting the importance label, access speed requirement and backup requirement is described in detail below:

[0101] The data migration evaluation model of the present invention is a Transform-based intelligent data migration multi-label evaluation method. The model generates multi-label classification results such as data importance, access speed requirements, and backup requirements through multi-dimensional analysis of data features. The feature vector is input into the data migration evaluation model to output the importance label, access speed requirement, and backup requirement corresponding to the target data, specifically including:

[0102] S1110: Mapping the feature vector into the vector space to obtain a mapped feature vector;

[0103] First, the basic attributes of the target data are extracted, and the obtained feature vectors are as follows:

[0104] ;

[0105] in, For data The characteristic vector of is the data type (such as JPEG format, MOV format, etc.), is the data size in MB, Frequency of use for users, To modify the record.

[0106] The feature vector is further mapped into the vector space to obtain the mapped feature vector:

[0107] ;

[0108] in, is the mapping matrix, is the feature vector after mapping.

[0109] S1111: Calculate the attention score of the data transfer evaluation model based on the query matrix, key matrix, value matrix, vector dimension and activation function;

[0110] Specifically, in the Transform model, the attention mechanism is used to calculate the importance and relevance of data. The attention score is calculated by dot product as follows:

[0111] ;

[0112] in, Score for attention, is the query matrix, is the key matrix, is the value matrix, is the dimension of the vector, is the activation function.

[0113] S1112: performing a dot product operation on the data feature vector and the node feature vector in the mapped feature vector to obtain a feature similarity score;

[0114] The data feature vector and the node feature vector in the mapped feature vector are multiplied to obtain the feature similarity score, as follows:

[0115] ;

[0116] in, data and nodes The feature similarity score of data The characteristic vector of node The feature vector of .

[0117] S1113: Determine the transition probability between the target data and the node according to the path between the target data and the node and the occurrence probability of the edge in the path;

[0118] Determine the transition probability between target data and nodes:

[0119] ;

[0120] in, For data With Node The transition probability between For target data Storage Node The path between is the probability of an edge appearing in the path.

[0121] S1114: Determine a local topological structure similarity score according to the node feature vector and the feature vectors of neighboring nodes;

[0122] Aggregate feature vectors to obtain local topological structure features and calculate local topological structure similarity scores as follows:

[0123] ;

[0124] in, For data and nodes The local topological structure similarity score of For data The feature vector of neighbor node k, For data The set of neighbor nodes.

[0125] S1115: determining a credibility score according to the feature similarity score, the transition probability and the local topological structure similarity score;

[0126] The feature similarity score, transition probability and local topology similarity score are further multiplied and sent to the activation function to obtain the credibility score:

[0127] ;

[0128] in, For data and nodes The credibility score of is the sigmoid activation function.

[0129] S1116: retaining neighbor nodes whose credibility scores are not less than a threshold to obtain a neighbor node set;

[0130] Neighbor nodes with credibility scores below the threshold are pruned to ensure the robustness of the model, as follows:

[0131] ;

[0132] in, is the set of neighbor nodes after pruning, is the pruning threshold.

[0133] S1117: Determine a weight value of a neighbor node under a data migration evaluation model according to the attention score;

[0134] Further calculate the weight value of the neighbor node under the Transform model:

[0135] ;

[0136] in, Neighbor node The weight value of .

[0137] S1118: Determine a final vector representation of the target data according to the weight value and the node feature vector of the neighbor node in the neighbor node set;

[0138] Specifically combining the representations of multiple paths, the final vector representation of the target data is obtained:

[0139] ;

[0140] in, For target data The final vector representation of .

[0141] S1119: Input the final vector representation into the fully connected layer to output the importance label, access speed requirement, and backup requirement corresponding to the target data.

[0142] Finally, the final vector representation is input into the fully connected layer, mapped to the probability distribution space, and the multi-label classification result is output, that is, the importance label, access speed requirement, and backup requirement corresponding to the target data:

[0143] ;

[0144] in, and are the weights and biases of the fully connected layer, It’s data Multi-label classification results.

[0145] In this way, the importance label, access speed requirement and backup requirement corresponding to the target data are obtained. It should be noted that the data migration evaluation model of the present invention is optimized using the following loss function, specifically, the cross entropy loss function and the contrast loss function, and the final optimization target is as follows:

[0146] ;

[0147] in, is the cross entropy loss, is the comparison loss.

[0148] Based on the above embodiments, in some embodiments, judging whether the target data needs to be migrated according to the evaluation result includes:

[0149] S121: Obtaining an initial importance label, initial access speed requirement, and initial backup requirement of target data in the node;

[0150] Among them, the initial importance label, initial access speed requirement, and initial backup requirement are generated by the data migration evaluation model when the target data is first stored in the node;

[0151] S122: Determine whether the importance label of the target data is the same as the initial importance label, and the access speed requirement is the same as the initial access speed requirement, and the backup requirement is the same as the initial backup requirement; if so, proceed to step S123; otherwise, proceed to step S124;

[0152] S123: confirming that the target data does not need to be migrated;

[0153] S124: Confirm that the target data needs to be migrated.

[0154] In order to determine whether the target data needs to be migrated, the initial importance label, initial access speed requirement and initial backup requirement of the target data in the node are specifically obtained in this embodiment. It should be noted that the initial importance label, initial access speed requirement and initial backup requirement are generated by the data migration evaluation model when the target data is first stored in the node. The specific evaluation process is the same as the evaluation process of the target data before migration in the above embodiment, which is output by the model after extracting the feature vector and inputting it into the data migration evaluation model, which will not be repeated in this embodiment.

[0155] After obtaining the initial importance label, initial access speed requirement and initial backup requirement of the target data, determine whether the importance label of the target data is the same as the initial importance label, and the access speed requirement is the same as the initial access speed requirement, and the backup requirement is the same as the initial backup requirement. If it is confirmed that the importance label of the target data is the same as the initial importance label, and the access speed requirement is the same as the initial access speed requirement, and the backup requirement is the same as the initial backup requirement, then it is considered that the evaluation result of the target data has not changed, and there is no need to migrate the target data. If the importance label of the target data is not the same as the initial importance label, and / or the access speed requirement is not the same as the initial access speed requirement, and / or the backup requirement is not the same as the initial backup requirement, then it is considered that the evaluation result of the target data has changed, and the target data is no longer suitable for storage in the current node, so it is confirmed that the target data needs to be migrated.

[0156] In this embodiment, by obtaining the initial importance label, initial access speed requirement and initial backup requirement of the target data in the node; judging whether the importance label of the target data is the same as the initial importance label, and the access speed requirement is the same as the initial access speed requirement, and the backup requirement is the same as the initial backup requirement; if so, it is confirmed that the target data does not need to be migrated; otherwise, it is confirmed that the target data needs to be migrated. Accurate judgment on whether the target data needs to be migrated is achieved.

[0157] Based on the above embodiments, in some embodiments, a migration strategy for target data is formulated according to the evaluation results, including:

[0158] S131: when the importance label of the target data is different from the initial importance label, and / or the access speed requirement is different from the initial access speed requirement, and / or the backup requirement is different from the initial backup requirement, determining the target node and the target storage medium for migration according to the importance label of the target data and / or the access speed requirement and / or the backup requirement;

[0159] S132: Setting the migration time and migration method for the target data.

[0160] In order to formulate a migration strategy for the target data, when the importance label of the target data is different from the initial importance label, and / or the access speed requirement is different from the initial access speed requirement, and / or the backup requirement is different from the initial backup requirement, the target node and target storage medium for migration are determined based on the importance label of the target data and / or the access speed requirement and / or the backup requirement.

[0161] For example, for target data that is of high importance and needs to be accessed quickly, a low-latency SSD node will be selected; for data that is of high importance but has a low access frequency, a high-durability HDD node will be selected. At the same time, for target data that needs to be backed up, the target node will be instructed to back up and store the target data.

[0162] At the same time, it is necessary to set the migration time and migration method for the target data. For example, the migration time should be a period of time with low node load, and the migration method should use an efficient transmission protocol, such as Hypertext Transfer Protocol (HHTP) or File Transfer Protocol (FTP).

[0163] In this embodiment, the target node and target storage medium for migration are determined based on the importance label and / or access speed requirement and / or backup requirement of the target data, and the migration time and migration method for migrating the target data are set, thereby implementing the setting of the migration strategy to facilitate the migration of the target data according to the migration strategy.

[0164] Based on the above embodiments, in some embodiments, migrating the target data to the target node according to the migration strategy includes:

[0165] S141: Determine the data type of the target data;

[0166] S142: Based on the migration time and the migration method, the target data is migrated to the target storage medium of the target node using a consistent hashing algorithm.

[0167] In the specific implementation, in order to migrate the target data to the target node, it is first necessary to determine the data type of the target data, such as JPEG images, MOV video files, JSON structured data or other binary data. Further, based on the migration time and migration method in the migration strategy, the consistent hashing algorithm is used to migrate the target data to the target storage medium of the target node.

[0168] It should be noted that the Distributed Hash Table (DHT) algorithm is a distributed system structure with high self-organization, scalability and fault tolerance. It is widely used in distributed storage systems to achieve fast and efficient data location and access. DHT mainly uses hashing technology to evenly distribute data on multiple nodes and provide efficient query and management. Its specific workflow is as follows:

[0169] The target data is migrated to the target storage medium of the target node using a consistent hashing algorithm, including:

[0170] S1421: Calculate the hash value of the target data according to the hash function;

[0171] S1422: Determine a target node for storing target data corresponding to the hash value according to a routing algorithm;

[0172] S1423: Migrate the target data to the target storage medium of the corresponding target node.

[0173] First, each node is assigned one or more hash value ranges, which define the set of keys that the node is responsible for storing. The initial hash function is:

[0174] ;

[0175] Among them, the key Mapped to a range from 0 to integer, is the size of the hash space.

[0176] When you need to store a key-value pair (that is, the target data), first use the hash function to calculate the key The hash value of ; Use a routing algorithm to determine which destination node is responsible for the hash value ; This usually involves finding the person responsible or its nearest range node; key-value pair is stored on this target node.

[0177] It should be noted that the hash function in this embodiment can avoid storage conflicts and ensure that keys are evenly distributed on all nodes. The hash function is:

[0178] ;

[0179] in, is the number of nodes.

[0180] At the same time, for the routing algorithm, each node maintains a table that records the information of itself and its neighboring nodes. , the node determines the storage The routing process may require multiple jumps, each of which gradually approaches the target node. In order to improve data availability and fault tolerance, data is usually not stored on only one target node. Through replication strategies, data can be stored on multiple target nodes. Using consistent hashing algorithms can keep data accessible in the event of node failure.

[0181] In addition, the data search process in the target node is similar to the storage process. First, the hash value is calculated. , use the routing algorithm to query the person responsible for storing the hash value The node from which to retrieve the value . This enables the search for data in the target node.

[0182] In this embodiment, the data type of the target data is determined; based on the migration time and migration method, a consistent hashing algorithm is used to migrate the target data to the target storage medium of the target node, thereby ensuring the balanced distribution of the target data among the target nodes.

[0183] Based on the above embodiments, in some embodiments, after the target data is migrated to the target node according to the migration strategy, the method further includes:

[0184] S15: Generate a checksum of the migrated target data according to a preset checksum algorithm;

[0185] S16: Determine whether the checksum of the target data before migration is the same as the checksum of the target data after migration; if so, proceed to step S17; otherwise, proceed to step S18;

[0186] S17: Confirm that the target data after migration is correct;

[0187] S18: Confirm that the target data after migration is incorrect.

[0188] After completing the migration of the target data, in order to ensure the integrity and consistency of the target data, a checksum of the migrated target data is generated according to a preset checksum algorithm. In this embodiment, there is no restriction on the preset algorithm, such as a cyclic redundancy check (CRC) algorithm or a message digest (MD5) algorithm.

[0189] It is further determined whether the checksum of the target data before migration is the same as the checksum of the target data after migration. If it is confirmed that the checksum of the target data before migration is the same as the checksum of the target data after migration, it is confirmed that the target data after migration is correct and the data has integrity and consistency. If it is confirmed that the checksum of the target data before migration is not the same as the checksum of the target data after migration, it is confirmed that the target data after migration is incorrect.

[0190] In this way, the integrity and consistency of the migrated target data are verified.

[0191] Based on the above embodiment, in some embodiments, if it is confirmed that the target data after migration is incorrect, the method further includes:

[0192] S19: Obtaining the original image file of the target data;

[0193] S20: Migrate the original image file of the target data back to the target node.

[0194] In order to ensure the integrity and consistency of the data, in the specific implementation, when it is confirmed that the target data after migration is incorrect, the original image file of the target data is obtained, and the original image file is the image file of the target data before migration. By re-migrating the original image file of the target data to the target node, the correctness of the data migration is guaranteed.

[0195] Based on the above embodiment, in some embodiments, after the target data is migrated to the target node according to the migration strategy, the method further includes:

[0196] S21: monitor the load status of each target node;

[0197] The load status includes at least capacity, data access frequency and processing performance;

[0198] S22: Control the target data to be evenly distributed among the target nodes according to the load status.

[0199] During the operation of the distributed storage system, load balancing is an important part of ensuring the efficient and stable operation of the system. Specifically, the load status of each target node is continuously monitored. It should be noted that the load status at least includes the capacity, data access frequency and processing performance of the target node.

[0200] Furthermore, the storage method of data between each target node is dynamically adjusted according to the load status of each target node, so as to control the balanced distribution of target data between each target node, and the stable operation of the distributed storage system can be ensured. It should be noted that the balanced distribution in this embodiment can specifically be storing the target data evenly in each target node, and can also be combined with the specific load status of each target node, and the target data can be stored in each target node with a data amount suitable for the corresponding load status, so as to achieve the best storage state, which is not limited in this embodiment.

[0201] Based on the above embodiments, in some embodiments, the method further includes:

[0202] S23: when a new node is detected to be added to the distributed storage system, the load status of the new node is obtained;

[0203] S24: Controlling the target data to be evenly distributed between the new node and the target node according to the load status of the new node and the load status of the target node;

[0204] S25: When it is detected that a target node exits the distributed storage system, the target data is controlled to be evenly distributed among the remaining target nodes.

[0205] In a specific implementation, the number of nodes in a distributed storage system may change, and may increase or decrease. In order to maintain a balanced distribution of data, in this embodiment, when a new node is detected to be added to the distributed storage system, the load status of the new node is obtained. It can be understood that the load status at least includes the capacity, data access frequency and processing performance of the target node. Further, according to the load status of the new node and the load status of the target node, the target data is controlled to be evenly distributed between the new node and the target node. In this way, the new node can share the responsibility of data storage and ensure the load balance of the system.

[0206] When it is detected that a target node has exited the distributed storage system, the target data that the target node is responsible for cannot be lost. The system will transfer the responsibility for this data to other existing target nodes and control the balanced distribution of the target data among the remaining target nodes to ensure data availability and system reliability.

[0207] Based on the above embodiment, in some embodiments, before obtaining the basic attributes of the target data in the node according to the preset period, the method further includes:

[0208] S26: Generate a unique identifier for the target data;

[0209] S27: Tracking the location information and processing status of the target data according to the unique identifier of the target data.

[0210] In order to better understand the location of the target data in the distributed storage system, before obtaining the basic attributes of the target data in the node according to the preset cycle, when the target data is stored in the node, the system will generate a unique identifier for the target data. It should be noted that the unique identifier will be transmitted along with the target data and exist as an independent identifier in the processing flow of the entire system, and the data can be traced at any time. Therefore, the location information and processing status of the target data can be tracked based on the unique identifier of the target data.

[0211] In order to help those skilled in the art better understand the present solution, the implementation process of the present solution is described in detail below with reference to specific examples:

[0212] (1) When data is first stored in a system node, the input data includes various types. Input data example:

[0213] JPEG images: 2MB in size, frequently accessed by users.

[0214] MOV video file: size 500MB, access frequency is low.

[0215] JSON data: 200KB in size, with moderate access frequency.

[0216] Identify the data format and extract basic properties, including data type and size. After preliminary processing, the data is stored in a temporary buffer.

[0217] (2) Use the pre-trained data migration evaluation model to evaluate the input data. The model considers characteristics such as data type, size, user usage frequency, and modification history, and outputs the importance label and storage requirements of the data.

[0218] Evaluation model formula:

[0219] ;

[0220] Simulation results:

[0221] JPEG images: high importance and high need for fast access.

[0222] MOV video files: medium importance, low access speed requirements.

[0223] JSON data: low importance, high backup requirements.

[0224] (3) Develop a storage strategy based on the evaluation results. Use a consistent hashing algorithm to ensure that data is evenly distributed among storage nodes.

[0225] Storage policy decisions:

[0226] JPEG images: stored on SSD.

[0227] MOV video files: stored in HDD.

[0228] JSON data: multi-node backup.

[0229] (4) Store data according to the storage strategy. Each node regularly performs data sorting and optimization tasks.

[0230] Storage Execution:

[0231] JPEG images are stored on SSDs with low latency and high input / output operations per second.

[0232] MOV video files are stored on HDD which has high durability.

[0233] JSON data is replicated on multiple nodes.

[0234] (5) Regularly check the storage status of the evaluation node and update the migration strategy based on the load and data importance. For example, as the access frequency of JPEG images decreases, the system decides to migrate them from SSD to HDD to optimize the use efficiency of SSD. The evaluation process is the same as the inspection and evaluation process in the above storage process.

[0235] (6) The system implements load balancing to ensure no single point of failure. Load balancing operation: Analyze the capacity and processing load of storage nodes and automatically adjust data distribution.

[0236] Simulation environment settings: Number of nodes: 10.

[0237] Data types: JPEG, MOV, JSON.

[0238] Total data volume: 10TB.

[0239] Simulated daily visits: 100,000 times.

[0240] Performance evaluation: By adjusting the data storage location and media, the access latency of JPEG images was reduced from 100ms to 50ms. The storage cost of MOV video files was reduced by 20%. The recovery time of JSON data was reduced from 5 minutes to 1 minute.

[0241] In the above embodiments, the data migration method is described in detail. The present invention also provides a corresponding embodiment of the data migration device.

[0242] Figure 2 A schematic diagram of a data migration device provided by an embodiment of the present invention. The device is applied to a distributed storage system; Figure 2 As shown, the device comprises:

[0243] The acquisition module 10 is used to acquire basic attributes of the target data in the node according to a preset period; wherein the basic attributes at least include data type and data size;

[0244] A determination module 11 is used to determine the evaluation result of the target data according to the basic attributes of the target data based on the data migration evaluation model; wherein the data migration evaluation model is a pre-created deep learning model for evaluating the importance, access speed requirements and backup requirements of the data;

[0245] The judgment module 12 is used to judge whether the target data needs to be migrated according to the evaluation result; if so, the strategy formulation module 13 is triggered;

[0246] A strategy formulation module 13 is used to formulate a migration strategy for the target data according to the evaluation results; wherein the migration strategy at least includes the target node, migration time and migration method of the migration;

[0247] The migration module 14 is used to migrate the target data to the target node according to the migration strategy.

[0248] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0249] In addition, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned data migration method when executed by a processor.

[0250] Figure 3 A schematic diagram of a data migration device provided by an embodiment of the present invention. Figure 3 As shown, the data migration device includes:

[0251] A memory 20, used for storing computer programs;

[0252] The processor 21 is used to implement the steps of the data migration method mentioned in the above embodiment when executing the computer program.

[0253] The data migration device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer, or a desktop computer.

[0254] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0255] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the data migration method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. Data 203 may include but is not limited to data involved in the data migration method.

[0256] In some embodiments, the data migration device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0257] Those skilled in the art will understand that Figure 3 The structure shown in the figure does not constitute a limitation on the data migration device, and may include more or fewer components than those shown in the figure.

[0258] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0259] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0260] The above is a detailed introduction to a data migration method, product, device and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

[0261] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A data migration method, characterized in that: Applied to a distributed storage system; the method comprises: Acquire basic attributes of target data in the node according to a preset period; wherein the basic attributes at least include data type and data size; Based on the data migration evaluation model, determining the evaluation result of the target data according to the basic attributes of the target data; wherein the data migration evaluation model is a pre-created deep learning model for evaluating the importance, access speed requirements and backup requirements of the data; Determining whether the target data needs to be migrated according to the evaluation result; If yes, a migration strategy for the target data is formulated according to the evaluation result; wherein the migration strategy at least includes a target node for migration, a migration time, and a migration method; Migrating the target data to the target node according to the migration strategy; The step of determining the evaluation result of the target data based on the data migration evaluation model according to the basic attributes of the target data includes: Extracting feature vectors according to basic attributes of the target data; wherein the feature vectors include data type, data size, user usage frequency and modification records; Inputting the feature vector into the data migration assessment model to output the importance label, access speed requirement and backup requirement corresponding to the target data; The step of inputting the feature vector into the data migration assessment model to output the importance label, access speed requirement and backup requirement corresponding to the target data includes: Mapping the feature vector into a vector space to obtain the mapped feature vector; Calculate an attention score of the data transfer evaluation model based on the query matrix, the key matrix, the value matrix, the vector dimension, and the activation function; Performing a dot multiplication operation on the data feature vector and the node feature vector in the mapped feature vector to obtain a feature similarity score; Determine the transition probability between the target data and the node according to the path between the target data and the node and the occurrence probability of the edge in the path; Determining a local topological structure similarity score based on the node feature vector and the feature vectors of neighboring nodes; Determining a credibility score according to the feature similarity score, the transition probability and the local topological structure similarity score; Retain the neighbor nodes whose credibility scores are not less than a threshold to obtain a neighbor node set; Determine a weight value of the neighbor node under the data migration evaluation model according to the attention score; Determining a final vector representation of the target data according to the weight value and the node feature vector of the neighbor node in the neighbor node set; The final vector representation is input into a fully connected layer to output the importance label, access speed requirement and backup requirement corresponding to the target data.

2. The data migration method according to claim 1, characterized in that: The determining whether the target data needs to be migrated according to the evaluation result includes: Obtaining an initial importance label, an initial access speed requirement, and an initial backup requirement of the target data in the node; wherein the initial importance label, the initial access speed requirement, and the initial backup requirement are generated by the data migration evaluation model when the target data is first stored in the node; Determine whether the importance label of the target data is the same as the initial importance label, the access speed requirement is the same as the initial access speed requirement, and the backup requirement is the same as the initial backup requirement; If so, confirming that the target data does not need to be migrated; If not, it is confirmed that the target data needs to be migrated.

3. The data migration method according to claim 2, characterized in that: The formulating a migration strategy for the target data according to the evaluation result includes: When the importance label of the target data is different from the initial importance label, and / or the access speed requirement is different from the initial access speed requirement, and / or the backup requirement is different from the initial backup requirement, determining the target node and target storage medium for migration according to the importance label of the target data and / or the access speed requirement and / or the backup requirement; Set the migration time and migration method for the target data.

4. The data migration method according to claim 3, characterized in that: Migrating the target data to the target node according to the migration strategy includes: Determine the data type of the target data; Based on the migration time and the migration method, the target data is migrated to the target storage medium of the target node using a consistent hashing algorithm.

5. The data migration method according to claim 4, characterized in that: The step of migrating the target data to the target storage medium of the target node by using a consistent hashing algorithm includes: Calculating a hash value of the target data according to a hash function; Determine the target node for storing the target data corresponding to the hash value according to the routing algorithm; The target data is migrated to the target storage medium of the corresponding target node.

6. The data migration method according to claim 1, characterized in that: After migrating the target data to the target node according to the migration strategy, the method further includes: Generate a checksum of the target data after migration according to a preset checksum algorithm; Determining whether a checksum of the target data before migration and a checksum of the target data after migration are the same; If yes, confirm that the target data after migration is correct; If not, it is confirmed that the target data after migration is incorrect.

7. The data migration method according to claim 6, characterized in that: If it is confirmed that the target data after migration is incorrect, it also includes: Obtaining an original image file of the target data; The original image file of the target data is re-migrated to the target node.

8. The data migration method according to claim 1, characterized in that: After migrating the target data to the target node according to the migration strategy, the method further includes: Monitoring the load status of each of the target nodes; wherein the load status at least includes capacity, data access frequency and processing performance; The target data is controlled to be evenly distributed among the target nodes according to the load status.

9. The data migration method according to any one of claims 1 to 8, characterized in that: Also includes: When a new node is added to the distributed storage system, the load status of the new node is obtained; According to the load status of the new node and the load status of the target node, controlling the target data to be evenly distributed between the new node and the target node; When it is detected that the target node exits in the distributed storage system, the target data is controlled to be evenly distributed among the remaining target nodes.

10. The data migration method according to claim 1, characterized in that: Before acquiring the basic attributes of the target data in the node according to the preset period, the method further includes: generating a unique identifier for the target data; The location information and processing status of the target data are tracked according to the unique identifier of the target data.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the data migration method described in any one of claims 1 to 10 are implemented.

12. A data migration device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data migration method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 10 are implemented.