Data migration methods, apparatus, equipment and storage media
By concatenating data attribute parameters and retention time characteristics, data is clustered into hot, warm, and cold data. Migration tasks are then constructed based on the data lifecycle, solving the problem of aging data occupying expensive storage resources and realizing intelligent hierarchical storage and efficient migration of data.
Patent Information
- Application Number
- CN202310883223.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-18
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-07-18
AI Technical Summary
In existing technologies, large amounts of aging data consume expensive storage resources, and data transfer requires the assistance of multiple parties, resulting in low processing efficiency.
By acquiring the attribute parameters and retention time features of the data and concatenating them, clustering is performed to divide the data into hot data, warm data, and cold data. Based on the lifecycle, migration tasks are constructed to automatically migrate the data to the corresponding storage media.
It enables hierarchical storage of massive amounts of data, identifies low-value data and stores it in low-value resources, improves the utilization efficiency of high-value storage resources, and realizes intelligent adaptation and storage migration of data assets.
Smart Images

Figure CN116974470B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data governance technology, and in particular to a data migration method, apparatus, device, and storage medium. Background Technology
[0002] With rapid business growth, enterprise data volume is increasing exponentially, leading to a continuous rise in the cost and power consumption of storage systems. The massive resources are placing higher demands on existing storage media. Reducing the cost and power consumption of storage resources while meeting business performance requirements, and implementing automated and intelligent storage migration, have become key priorities and urgent issues that need to be addressed.
[0003] Currently, there is no standardized definition of data lifecycle or data migration procedures, resulting in a large amount of aging data consuming expensive storage resources. In most cases, operations personnel only intervene subjectively to determine the lifecycle of aging data when storage resource alerts are triggered. This manual approach lacks standardized operating procedures, requires multi-party assistance, and leads to lengthy processing cycles. Often, after discovering a problem, operations personnel, lacking business expertise, require assistance from developers or even business experts to assess whether the data can be deleted. Extensive coordination and processes result in long processing times and low efficiency. Summary of the Invention
[0004] The main objective of this invention is to provide a data migration method, apparatus, device, and storage medium, aiming to solve the technical problems in the prior art where large amounts of aging data occupy expensive storage resources, data transfer requires the assistance of multiple parties, and data processing efficiency is low.
[0005] To achieve the above objectives, the present invention provides a data migration method, the method comprising the following steps:
[0006] Obtain the attribute parameters of each original data at the reference granularity, and perform feature concatenation on the attribute parameters and the retention time of each original data to obtain sample data;
[0007] The sample data is clustered, and the sample data is divided into hot data, warm data and cold data according to the clustering results.
[0008] According to the preset storage strategy, the hot data, warm data, and cold data are stored in the corresponding source data storage medium;
[0009] Obtain the lifecycle of the source data in the source data storage medium, construct a migration task based on the lifecycle, and migrate the source data according to the migration task.
[0010] Optionally, the step of obtaining the attribute parameters of each original data at the reference granularity, and performing feature concatenation on the attribute parameters and the retention time of each original data to obtain sample data, includes:
[0011] Obtain the raw data and divide it into daily, weekly, and monthly granularities to obtain the raw datasets of each granularity.
[0012] Determine the original dataset corresponding to a reference granularity among multiple original datasets with different granularities, and extract attribute parameters from each original data in the original dataset corresponding to the reference granularity. The attribute parameters include at least the frequency of use and the delay requirement.
[0013] Sample data is obtained by feature concatenation of the attribute parameters and the retention time of each original data.
[0014] Optionally, the step of clustering the sample data and dividing the sample data into hot data, warm data, and cold data based on the clustering results includes:
[0015] The sample features of the sample data are normalized to obtain reference features. The sample features include attribute features and retention time features. The attribute features include at least the mean access rate, median access rate, standard deviation of access rate, mean delay requirement, median delay requirement, and standard deviation of delay requirement.
[0016] An adjacency matrix is constructed based on the reference features; a degree matrix is constructed based on the adjacency matrix; and a Laplace matrix is constructed based on the adjacency matrix and the degree matrix.
[0017] Based on the eigenvalues and eigenvectors of the Laplacian matrix, the sample data is clustered to obtain hot data, warm data, and cold data.
[0018] Optionally, the step of clustering the sample data based on the feature values and feature vectors to obtain hot data, warm data, and cold data includes:
[0019] The feature values and feature vectors are sorted from smallest to largest, and the first K feature values and feature vectors are clustered to obtain a preset number of data clusters;
[0020] Calculate the feature mean of each data cluster, and sum the feature means to obtain the initial data heat of each data cluster;
[0021] The initial data heat of each data cluster is aligned in the same direction to obtain the data heat of each data cluster;
[0022] Compare the data popularity of each data cluster, and obtain hot data, warm data, and cold data based on the comparison results.
[0023] Optionally, obtaining the lifecycle of the source data in the source data storage medium, constructing a migration task based on the lifecycle, and migrating the source data according to the migration task includes:
[0024] When the current time meets the preset period, the life cycle of the source data is determined according to the retention time and attribute parameters of the source data in each source data storage medium;
[0025] A migration task is constructed based on the lifecycle, and the source data is migrated according to the migration task. The target storage medium is a source data storage medium that stores warm data or a source data storage medium that stores cold data.
[0026] Optionally, the step of constructing a migration task based on the lifecycle and migrating the source data according to the migration task includes:
[0027] Determine whether the lifecycle is within the preset migration period range, and use the source data whose lifecycle is within the preset migration period range as the data to be migrated;
[0028] The target storage medium is obtained based on the lifecycle timer of the data to be migrated;
[0029] A migration task is constructed based on the data to be migrated, the attribute parameters of the data to be migrated, and the target storage medium.
[0030] The source data is migrated according to the migration task.
[0031] Optionally, migrating the source data according to the migration task includes:
[0032] The migration task is parsed to obtain the current storage medium and the target storage medium of the data to be migrated;
[0033] When the current storage medium is a hot data storage medium, delete the data to be migrated in the current storage medium and migrate the data to be migrated to the storage medium corresponding to the warm data.
[0034] When the current storage location is a storage medium for warm data, delete the data to be migrated from the current storage medium and migrate the data to the storage medium corresponding to the cold data.
[0035] Furthermore, to achieve the above objectives, the present invention also proposes a data migration apparatus, the data migration apparatus comprising:
[0036] The data partitioning module is used to obtain the attribute parameters of each original data at the reference granularity, and to perform feature concatenation on the attribute parameters and the retention time of each original data to obtain sample data.
[0037] The data segmentation module is also used to cluster the sample data and divide the sample data into hot data, warm data and cold data according to the clustering results.
[0038] The data migration module is used to store the hot data, warm data and cold data to the corresponding source data storage medium according to a preset storage strategy;
[0039] The data migration module is further configured to obtain the lifecycle of the source data in the source data storage medium, construct a migration task based on the lifecycle, and migrate the source data according to the migration task.
[0040] In addition, to achieve the above objectives, the present invention also proposes a data migration device, the data migration device comprising: a memory, a processor, and a data migration program stored in the memory and executable on the processor, the data migration program being configured to implement the steps of the data migration method as described above.
[0041] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a data migration program, which, when executed by a processor, implements the steps of the data migration method described above.
[0042] This invention automatically classifies data lifecycles through standardized definitions and corresponding storage strategies, automatically migrating data from each stage to the corresponding storage medium. This enables tiered storage of massive amounts of data, identifies low-value data and stores it in low-value resources, and achieves efficient utilization of high-value storage resources. Through multi-dimensional analysis and judgment of application assets, including time, access rate, and response rate, it adapts storage to data intelligence, realizing the digitalization of data asset storage and migration. Attached Figure Description
[0043] Figure 1 This is a schematic diagram of the structure of the data migration device for the hardware operating environment involved in the embodiments of the present invention;
[0044] Figure 2 This is a flowchart illustrating the first embodiment of the data migration method of the present invention;
[0045] Figure 3 This is an architecture diagram of a data migration system in one embodiment of the data migration method of the present invention;
[0046] Figure 4 This is a schematic diagram of data migration under metadata management in one embodiment of the data migration method of the present invention;
[0047] Figure 5 This is a schematic diagram of a data migration strategy in one embodiment of the data migration method of the present invention;
[0048] Figure 6 This is a flowchart illustrating the second embodiment of the data migration method of the present invention;
[0049] Figure 7 This is a flowchart illustrating the third embodiment of the data migration method of the present invention;
[0050] Figure 8 This is a schematic diagram of the complete data migration process in one embodiment of the data migration method of the present invention;
[0051] Figure 9 This is a structural block diagram of the first embodiment of the data migration device of the present invention.
[0052] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0053] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.
[0054] Reference Figure 1 , Figure 1 This is a schematic diagram of the data migration device structure in the hardware operating environment involved in the embodiments of the present invention.
[0055] like Figure 1 As shown, the data migration device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.
[0056] Those skilled in the art will understand that Figure 1The structure shown does not constitute a limitation on the data migration device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0057] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a data migration program.
[0058] exist Figure 1 In the data migration device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the data migration device of the present invention can be set in the data migration device, and the data migration device calls the data migration program stored in the memory 1005 through the processor 1001 and executes the data migration method provided in the embodiment of the present invention.
[0059] This invention provides a data migration method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of a data migration method according to the present invention.
[0060] In this embodiment, the data migration method includes the following steps:
[0061] It should be noted that the data migration method in this embodiment is applied to a data migration system, and the architecture of the data migration system can be referred to Figure 3 Metadata management in the diagram: Unified metadata, managing the classification and hierarchical information of global data, basic information of global data assets, asset classification, business lines, offline tasks and application information and their deployment resources.
[0062] The algorithm model can utilize algorithms to calculate the lifecycle classification of data based on data retention period, usage rate, latency requirements, and intelligent scoring algorithms. Combining the characteristics of the classified data, and based on the global resource capacity, usage, and data scale of full resource management, a storage resource strategy definition is formed. Metadata management can record and manage global data asset information, including basic asset information, structural information, deployment information, and lifecycle information.
[0063] Task scheduling can be based on data lifecycle storage strategies to switch between hot, warm, and cold data storage, and generate execution parsing statements and scheduling tasks that automatically migrate and clean up relevant data after the lifecycle rules are met.
[0064] The task execution can be based on the parsing results to perform migration operations from the data source to the target. This can be achieved by extracting and migrating the target data, standardizing and transforming the data, and cleaning up the migrated data, thus realizing the final actual data migration action.
[0065] It should be noted that a data asset is abstracted into an object. Since data retains different data lifecycle hierarchies in different storage resources, the global hierarchical data is abstracted into a data asset, and each hierarchical data is an asset instance.
[0066] Each data asset has one and only one instance when the data is generated. This instance is the data generation instance, which is usually hot data when it is generated. Over time, it cools down to lukewarm data and awaits migration. Instance management of data assets includes information such as asset real name, data center, instance status, lifecycle, asset instance type, storage medium, distribution source, period range, timestamp, storage rules, storage suffix rules, partition fields, partition type, database, and deployment user.
[0067] In metadata management, in addition to managing asset information, it is necessary to manage the basic attribute data that provides lifecycle grading and storage policy definitions. Asset data that has undergone lifecycle and storage policy definitions will enter the automatic task creation phase. After the task is created and executed, the generated warm / cold grading period data asset instances will be automatically populated back into the metadata management system. For details, please refer to [link / reference]. Figure 4 The example in the image is generated based on automatic migration according to lifecycle hierarchy and is not editable.
[0068] Data asset application information management manages the usage and application information of asset data, including offline and real-time applications. The application information is mainly used to collect access information of data assets.
[0069] Offline applications can be configured through offline tasks to form a data asset lineage structure. The invocation rate of tasks that depend on this asset can be obtained. Real-time applications can be a list of real-time applications of the data asset; by filling in the application post-evaluation script, the access rate of the available applications can be obtained through the application post-evaluation.
[0070] Step S10: Obtain the attribute parameters of each original data at the reference granularity, and perform feature concatenation on the attribute parameters and the retention time of each original data to obtain sample data.
[0071] Understandably, the reference granularity includes daily, weekly, and monthly granularity, and the raw data can be all stored data or all data.
[0072] It should be noted that the raw data is divided into daily, weekly, and monthly granularity. The raw data at different granularities are further divided into hot data, warm data, and cold data. Granularity can be understood as period.
[0073] It should be noted that the attribute parameters can include usage frequency and latency requirements. Furthermore, the usage frequency can be obtained through the mean of business access rate, the median of business access rate, the standard deviation of business access rate, the mean of latency requirements, the median of latency requirements, and the standard deviation of latency requirements; the latency requirements can be obtained through the mean of latency requirements, the median of latency requirements, and the standard deviation of latency requirements.
[0074] It should be understood that each piece of raw data has a retention period. This retention period can be determined by prediction or historical experience to determine how long each application can be stored. Simply put, it can be the time from when an application is downloaded to when it is deleted.
[0075] It should be further explained that the sample data obtained by feature concatenation of the attribute parameters and the retention time of each original data can be obtained by concatenating the attribute features and retention time of each original data at the parameter granularity. It can be understood that the number of sample data can be the same as the number of original data.
[0076] In specific implementation, obtaining the attribute parameters of each original data at the reference granularity and performing feature concatenation on the attribute parameters and the retention time of each original data to obtain sample data can be achieved by: obtaining the original data, dividing the original data according to daily, weekly, and monthly granularity to obtain original datasets of each granularity; determining the original dataset corresponding to the reference granularity among multiple original datasets of different granularities, and obtaining the sample data by performing feature concatenation on the attribute parameters and the retention time of each original data in the original dataset corresponding to the reference granularity, wherein the attribute parameters include at least usage frequency and delay requirements; and obtaining sample data by performing feature concatenation on the attribute parameters and the retention time of each original data.
[0077] Step S20: Cluster the sample data and divide the sample data into hot data, warm data and cold data according to the clustering results.
[0078] It is understood that the sample data can be clustered using the k-means clustering algorithm (kmeans) based on the clustering results.
[0079] It should be understood that the clustering result obtained by k-means clustering can be three sample data clusters, and the data in the three sample data clusters can be defined as hot data, warm data and cold data respectively.
[0080] It should be noted that the popularity score can be calculated separately for each of the three sample data clusters, and the three sample data clusters can be defined based on the popularity score.
[0081] Understandably, the value of data, from its creation to its destruction, whether in business use or statistical analysis, gradually declines over time until it is destroyed or archived. Generally, the older the data, the lower the access rate, and a lower access rate usually allows for some compromise on the timeliness of response. Therefore, for historical data, some timeliness can be sacrificed to ensure application stability and resource investment. Thus, data access rate and response latency requirements are important indicators defining the data lifecycle. In statistics, the mean, median, and standard deviation are used to describe the central tendency of a numerical sequence X. By extending basic statistical indicators through the physical characteristics of data, we can analyze the convergence and divergence trends of data lifecycle indicators from a statistical perspective.
[0082] Step S30: Store the hot data, warm data, and cold data to the corresponding source data storage medium according to the preset storage strategy.
[0083] It is understandable that a preset storage strategy can be understood as a data storage strategy. A data storage strategy can include the selection of storage media and storage resources, and select appropriate storage media in combination with the characteristics of the data tiered lifecycle.
[0084] Understandably, the lifecycle can be understood as the retention time, that is, the time from the generation of a piece of data to its deletion. For example, the lifecycle of a piece of data may be 100 days. The lifecycle is divided into three levels: day 1-25 is level 1, day 26-60 is level 2, and day 61-100 is level 3. Based on the different levels, the data is classified into corresponding categories: level 1 is hot data, level 2 is warm data, and level 3 is cold data.
[0085] It should be noted that the preset storage strategy may be to store data in high-end storage (mostly all-in-one machines or high-end disk arrays) during the hot data stage to efficiently support data storage, computation, and retrieval; store data in lower-value storage (mostly databases based on distributed system infrastructure) during the warm data stage to ensure the reliability of file migration while also taking into account the potential utilization value of the data; and store data in inexpensive storage media during the cold data stage to prepare for the possibility of data backtracking or reuse of historical data later.
[0086] It should be further explained that the scalable storage configuration allows users to freely choose the media and storage period based on the temperature and coldness of the data; simultaneously, resource selection enables global resource management, unifying the management of all storage resources. Statistical management is performed on resource names, locations, capacities, usage, and utilization rates. Based on the principle of balanced resource utilization, resources are sorted according to remaining capacity and proximity, providing a sorted resource list for users to select their target resource.
[0087] In implementation, the target asset is selected, and the lifecycle definition process begins. This process offers users two options: calculation via the system's intelligent definition algorithm, and manual definition via a separate channel to ensure functional fault tolerance. The algorithm captures relevant indicators of the target asset (retention period, usage rate, and latency requirements), performs intelligent scoring calculations, and labels each lifecycle of the data asset as "hot," "warm," or "cold," thus classifying the asset's lifecycle. After calculation, user confirmation is required. If the user approves the calculation result, they can proceed to the next step; otherwise, manual intervention is possible to adjust the asset's lifecycle. After asset classification is confirmed, a classification instance (hot instance, warm instance, cold instance) for this data asset will be automatically generated. The user then selects an instance to proceed to the next step: storage strategy definition. Storage media and data lifecycle classification are strongly correlated. The storage media and storage lifecycle of the selected asset's lifecycle instance are displayed in a list. After confirming the storage media and storage lifecycle, the target storage resource must be selected. Given sufficient data size, the target resource is selected from resources sorted by utilization rate, remaining capacity, and distance. Specific storage strategy definitions can be found in [reference needed]. Figure 5 The figure also shows that after classifying the data, there are unreasonable classifications, which can be considered as requiring adjustments to the data classification.
[0088] Step S40: Obtain the lifecycle of the source data in the source data storage medium, construct a migration task based on the lifecycle, and migrate the source data according to the migration task.
[0089] Understandably, after dividing the raw data into three levels—hot, warm, and cold—it is stored in different storage media, and the data in each storage media can be the source data.
[0090] It should be noted that the lifecycle of the source data in each storage medium can be obtained according to a certain time period, which can be one day, two days or one week.
[0091] Understandably, a migration task can include the data to be migrated, the target storage medium, and the current storage medium. The lifecycle of the source data in each storage medium is periodically retrieved. Based on the lifecycle of the source data and its attribute parameters, migration tasks can be automatically created and then executed to migrate the data to the target storage medium.
[0092] It should be emphasized that during the data migration process, after the original data is divided into three levels—hot, warm, and cold—it is stored in different storage media. The database storing hot-level data can be a hot database, the database storing warm-level data can be a warm database, and the database storing cold-level data can be a cold database. Furthermore, the data migration can involve migrating data from the hot database to the warm database, migrating data from the warm database to the cold database, and deleting data from the cold database that has reached the end of its life cycle.
[0093] This embodiment automatically implements data lifecycle grading through standardized data lifecycle definitions and corresponding storage strategies. It automatically migrates data from each grading period to the corresponding storage medium, enabling tiered storage of massive amounts of data. It identifies low-value data and stores it in low-value resources, achieving efficient utilization of high-value storage resources. Through multi-dimensional analysis and judgment of application assets, such as time dimension, access rate, and response rate, it adapts storage for data intelligence, realizing the digitalization of data asset storage and migration.
[0094] refer to Figure 6 , Figure 6 This is a flowchart illustrating a second embodiment of a data migration method according to the present invention.
[0095] Based on the first embodiment described above, the data migration method of this embodiment includes the following in step S20:
[0096] Step S21: Normalize the sample features of the sample data to obtain reference features. The sample features include attribute features and retention time features. The attribute features include at least the mean access rate, median access rate, standard deviation of access rate, mean delay requirement, median delay requirement, and standard deviation of delay requirement.
[0097] It is understood that the sample characteristics of the sample data include attribute characteristics and retention time characteristics. The attribute characteristics include at least the mean access rate, median access rate, standard deviation of access rate, mean delay requirement, median delay requirement, and standard deviation of delay requirement.
[0098] It should be understood that the following formula can be used to normalize sample features:
[0099]
[0100] Where value is the original value of the feature, min is the minimum value of the feature, and max is the maximum value of the feature. Normalization can prevent the influence of the dimensions of each feature. Normalization of each feature maps it to the range [0,1].
[0101] Step S22: Construct an adjacency matrix based on the reference features, construct a degree matrix based on the adjacency matrix, and construct a Laplace matrix based on the adjacency matrix and the degree matrix.
[0102] Understandably, the adjacency matrix can be constructed using the following formula based on multiple reference features:
[0103]
[0104] Among them, W ij This represents the distance between any two samples calculated based on Euclidean distance. The similarity value between sample i and sample j is between [0,1]. The larger the similarity value, the more similar the samples are.
[0105] Furthermore, the degree matrix can be constructed based on the adjacency matrix using the following formula:
[0106]
[0107] in,
[0108] Furthermore, the Laplace matrix can be constructed by subtracting the adjacency matrix from the degree matrix based on the adjacency matrix and the degree matrix.
[0109] It should be noted that an adjacency matrix is a matrix representing the adjacency relationships between vertices. Representing a graph using an adjacency matrix makes it easy to determine whether any two vertices are connected by an edge. Therefore, graph adjacency matrices are used to describe the similarity between samples for statistical indicators such as visit rate and response latency. A degree matrix is constructed based on the adjacency matrix. The degree matrix is a diagonal matrix, with the diagonal elements representing the degree of each vertex. The degree of a vertex represents the number of nodes associated with that vertex. The degree matrix is calculated using a weighted adjacency matrix. Then, a Laplacian matrix is used for dimensionality reduction, thereby reducing computational complexity.
[0110] Step S23: Based on the eigenvalues and eigenvectors of the Laplacian matrix, cluster the sample data to obtain hot data, warm data, and cold data.
[0111] Understandably, clustering is performed based on the eigenvalues and eigenvectors of the Laplacian matrix for each sample data.
[0112] It should be noted that before clustering the sample data, the eigenvalues and eigenvectors of the Laplacian matrix of multiple sample data are sorted from smallest to largest. The first K eigenvalues and eigenvectors are clustered to obtain a preset number of data clusters. The value of K can be adjusted according to the number of sample data. This embodiment does not limit this. The preset number can be 3 or other numbers. This embodiment uses 3 data clusters as an example for explanation. However, it is also possible to divide the data into four levels according to the life cycle of the data, and then cluster the sample data accordingly to obtain four data clusters.
[0113] Calculate the mean of features for each data cluster. Here, the mean of features can be understood as the popularity characteristic of each sample data, i.e., the characteristics of access rate and latency requirements. Summing these mean features yields the initial data popularity for each data cluster. Simply put, this involves calculating the mean of features in each data cluster and then summing them to obtain the popularity score for each data cluster. Since the characteristics of each feature are different—for example, a higher value for a business access rate-related indicator indicates higher data popularity, while a lower latency requirement also indicates higher data popularity—the direction of each value must be consistent during the summation. The final popularity score SC for each cluster can be referenced from the following formula:
[0114]
[0115] Where k represents the k indicators with larger values, indicating higher data popularity, and m represents the m indicators with smaller values, indicating higher data popularity. This represents the mean of this feature across all samples within the cluster.
[0116] The initial data heat direction of each data cluster is unified, and the data heat of each data cluster is compared. Based on the comparison results, hot data, warm data and cold data are obtained. Among them, the data in the data cluster with the highest data heat score is regarded as hot data, the data in the data cluster with the lowest data heat score is regarded as cold data, and the data in the middle is regarded as warm data.
[0117] In practice, samples within a cluster are labeled as "hot," "warm," or "cold" based on their cluster popularity scores. The highest score is "hot," the second highest is "warm," and the lowest is "cold." After clustering using the k-means clustering algorithm, the cluster score is calculated based on popularity characteristics, such as access rate and latency requirements. Assuming the popularity scores SC1, SC2, and SC3 for the three clusters are SC1, SC2, and SC3, and SC1 > SC2 > SC3, then all samples in cluster SC1 are labeled as "hot," all samples in cluster SC2 are labeled as "warm," and all samples in cluster SC3 are labeled as "cold."
[0118] This embodiment clusters the eigenvalues and eigenvectors of the Laplacian matrix of the sample data. Based on the access rate and latency requirements of the data in each cluster, a heat score is obtained for each data cluster. Based on the heat score, the sample data is accurately divided into three levels. Different levels of data are stored in different storage media, which improves the classification and storage rationality of data assets. This replaces the original model of using high-end storage for all data assets, reduces the storage cost and power consumption of warm and cold data, relieves the storage pressure on core storage, and improves the classification and storage rationality of data assets.
[0119] refer to Figure 7 , Figure 7 This is a flowchart illustrating a third embodiment of a data migration method according to the present invention.
[0120] Based on the first embodiment described above, the data migration method of this embodiment includes the following in step S40:
[0121] Step S41: When the current time meets the preset period, determine the life cycle of the source data according to the retention time and attribute parameters of the source data in each source data storage medium.
[0122] Understandably, the current time can be based on the time of the last data migration, and the preset period can be one day or two days. For example, if the first data migration was at 20:25 on July 14, 2023, and the preset period is one day, then the preset period is satisfied if the current time is at 20:25 on July 15, 2023.
[0123] It should be understood that the data retention time can be a specific time such as 100 days. The data generation time and the time that the data has existed can be obtained through the data attribute parameters. The data lifecycle can be determined by whether the data is currently in the hot, warm, or cold lifecycle stage.
[0124] Understandably, the source data can be data from the initially partitioned hot database, data from the query database, or data from the cold database.
[0125] It is easy to understand that if the lifecycle of the data is still at the same level as the lifecycle of the database, then the data remains unchanged and is not migrated.
[0126] Step S42: Construct a migration task according to the lifecycle, and migrate the source data according to the migration task. The target storage medium is a source data storage medium that stores warm data or a source data storage medium that stores cold data.
[0127] Understandably, once the lifecycle grading and storage strategy definitions are completed, details of the data migration task will be provided, such as the data source and target information for the migration operation, the migration cycle, etc.
[0128] The system will automatically create, parse, and execute data migration actions according to the details of the migration task and on a periodic basis. Data migration can consist of a scheduling module and an execution module in the data migration system.
[0129] Furthermore, the scheduling module can be responsible for scheduling the task parsing service, controlling the task queue, and sending task execution requests and execution parameters; the execution module can receive execution parameters, perform operations such as data extraction, file push, data storage, and data cleaning, extract data by connecting and deploying resources according to source information, automatically create directories or tables according to target information, push data to the target, and clear the source data.
[0130] Furthermore, the data migration system may also include a task monitoring module. This module can view the migration status of tasks, including task status information, step information, and detailed execution logs and scripts. The complete data migration process can be found by referring to... Figure 8 .
[0131] It should be noted that the step of constructing a migration task based on the lifecycle and migrating the source data based on the migration task may involve determining whether the lifecycle is within a preset migration period and using the source data whose lifecycle is within the preset migration period as the data to be migrated.
[0132] The target storage medium is determined based on the lifecycle timer of the data to be migrated. If the current storage medium for the data to be migrated is a hot data storage medium, and the lifecycle of the data to be migrated is in the warm data stage, then the warm database (i.e., the storage medium storing warm data) is the target storage medium. If the current storage medium for the data to be migrated is a warm data storage medium, and the lifecycle of the data to be migrated is in the warm data stage, then the cold database (i.e., the storage medium storing warm data) is the target storage medium. If the current storage medium for the data to be migrated is a cold data storage medium, and the lifecycle of the data to be migrated has exceeded the retention time of the data, then the data is deleted.
[0133] Understandably, data in a hot database can be migrated to a warm database, and data in a warm database can be migrated to a cold database. Understandably, if data migration has not been performed for a long time, it can also be directly migrated to the corresponding database based on the data level in each database. If data migration has not been performed for a long time and the data level in the hot database is in the cold data stage, the data can be directly migrated to the cold database.
[0134] A migration task is constructed based on the data to be migrated, the attribute parameters of the data to be migrated, and the target storage medium.
[0135] The source data is migrated according to the migration task. First, the migration task is parsed to obtain the current storage medium and target storage medium of the data to be migrated. If the current storage medium is a hot data storage medium, the data to be migrated is deleted from the current storage medium and migrated to the storage medium corresponding to warm data. If the current storage location is a warm data storage medium, the data to be migrated is deleted from the current storage medium and migrated to the storage medium corresponding to cold data.
[0136] In practice, data migration can be divided into task creation and parsing, and task scheduling and execution. Task creation and parsing includes: once the data lifecycle and storage strategy are defined, the basic information required for migration is ready, and the data migration scheduling service will automatically parse and schedule migration tasks for execution according to the cycle.
[0137] Based on the data's hierarchical partitioning and storage strategy, after the user clicks to confirm the migration, the task creation phase will begin. The data migration task will be created automatically.
[0138] The migration task source is defined by data hierarchy, identifying the periodic ranges of hot, warm, and cold data. Hot data serves as the data source for warm data, and warm data serves as the data source for cold data. Metadata management allows for the retrieval of deployment information and deployment type of the data source.
[0139] The asset instance that the user clicks to confirm the migration is the target of the migration task. By defining the storage policy, the deployment information and deployment type of the target data can be obtained.
[0140] Data migration requires retrieving data from the thermal reference temperature (thermal data storage) and migrating it to the target temperature data storage, while deleting the original thermal data's target period range. Since the period ranges for hot, warm, and cold data have been defined after data grading, the period conditions of the data to be migrated can be directly extracted. After migrating this period's data to the target location, the data for this period in the source data will be deleted.
[0141] After a task is created, the single-cycle task instance corresponding to each cycle needs to be parsed and corresponding subtask information needs to be generated. A subtask is a major step in the entire data migration process, breaking down the operations performed on the target instance. For example, the entire data migration process might involve clearing tables / partitions / deleting data, and dragging data. The smaller steps within these major steps, such as data dragging being divided into data extraction and data import, are the subtask's step information and are reflected in the steps themselves.
[0142] Parsing the specific steps of the subtask: Parsing the step information needs to be combined with the scenario. The data migration scenario is mainly related to the storage type of the source and target (even scenarios using the same medium may differ; for example, in a Hive scenario, different versions may require different parameters). Different scenarios require different parameter information for each step. Based on the storage types of the source and target, determine the scenario of this data migration task and obtain the scenario steps and parameter information.
[0143] The task scheduling and execution include: Overall execution is offline, using offline tasks to synchronize data periodically. File interaction is employed. Source data is extracted into files, sent to the target storage resource, and the source data is simultaneously deleted. Real-time parsing scheduling involves calling the task initialization service during task creation. This function is used for the initial data migration, requiring all periodic data meeting migration conditions to be created and parsed together, migrating all historical periodic data to the target. Scheduled parsing scheduling involves calling the single-period task initialization service daily. Subsequent tasks will create incremental data migration tasks periodically, periodically migrating and deleting data that has undergone lifecycle changes. Data task queue control establishes execution resource management, controlling the task execution queue based on execution time and resource availability.
[0144] The data task execution scheduler moves parsed tasks into the execution phase, uploads data task parameters to the execution service, schedules tasks to be executed and their parameters to be uploaded to the execution engine, and executes the tasks; data task retry is performed for exceptions, and tasks with execution errors are automatically retried; data migration task execution: connects to the storage resources where the source data is located, queries and extracts data to generate files according to periodic conditions; pushes the files to the cluster where the target storage resources are located according to the scenario; inserts the files into the target location (path) or performs an inbound operation; deletes data in the source data that meets the migration period conditions.
[0145] Task monitoring includes overall monitoring of task execution in different cycles, monitoring of subtasks within a single task cycle, monitoring of steps within a single subtask cycle, and viewing the scripts corresponding to each step.
[0146] This embodiment periodically determines the lifecycle of source data in each storage medium and, based on the lifecycle of the source data, determines the data to be migrated and the target storage medium. It automatically creates, parses, and executes data migration, realizing the digitalization of data asset storage and migration. It establishes an intelligent classification and automatic migration mechanism for application asset data, providing more efficient data support and services for upper-layer applications. The application asset data classification and storage migration mechanism enhances the core data storage capabilities of the data center, serves metadata management, and provides efficient support for upper-layer applications based on the formulation of data classification strategies and intelligent storage strategies.
[0147] Furthermore, embodiments of the present invention also propose a storage medium storing a data migration program, which, when executed by a processor, implements the steps of the data migration method described above.
[0148] Reference Figure 9 , Figure 9 This is a structural block diagram of the first embodiment of the data migration device of the present invention.
[0149] like Figure 9 As shown, the data migration apparatus proposed in this embodiment of the invention includes:
[0150] Data partitioning module 10 is used to obtain the attribute parameters of each original data at the reference granularity, and to perform feature concatenation on the attribute parameters and the retention time of each original data to obtain sample data;
[0151] The data segmentation module 10 is also used to cluster the sample data and divide the sample data into hot data, warm data and cold data according to the clustering results.
[0152] Data migration module 20 is used to store the hot data, warm data and cold data to the corresponding source data storage medium according to a preset storage strategy;
[0153] The data migration module 20 is further configured to obtain the lifecycle of the source data in the source data storage medium, construct a migration task based on the lifecycle, and migrate the source data according to the migration task.
[0154] This embodiment automatically implements data lifecycle grading through standardized data lifecycle definitions and corresponding storage strategies. It automatically migrates data from each graded period to the corresponding storage medium, achieving tiered storage of massive amounts of data. It identifies low-value data and stores it in low-value resources, enabling efficient utilization of high-value storage resources. Through multi-dimensional analysis and judgment of application assets, such as time dimension, access rate, and response rate, it adapts storage for data intelligence, realizing the digitalization of data asset storage and migration.
[0155] In one embodiment, the data partitioning module 10 is further configured to acquire raw data and partition the raw data according to daily granularity, weekly granularity and monthly granularity to obtain raw datasets of each granularity.
[0156] Determine the original dataset corresponding to a reference granularity among multiple original datasets with different granularities, and extract attribute parameters from each original data in the original dataset corresponding to the reference granularity. The attribute parameters include at least the frequency of use and the delay requirement.
[0157] Sample data is obtained by feature concatenation of the attribute parameters and the retention time of each original data.
[0158] In one embodiment, the data partitioning module 10 is further configured to normalize the sample features of the sample data to obtain reference features. The sample features include attribute features and retention time features. The attribute features include at least the mean access rate, median access rate, standard deviation of access rate, mean delay requirement, median delay requirement, and standard deviation of delay requirement.
[0159] An adjacency matrix is constructed based on the reference features; a degree matrix is constructed based on the adjacency matrix; and a Laplace matrix is constructed based on the adjacency matrix and the degree matrix.
[0160] Based on the eigenvalues and eigenvectors of the Laplacian matrix, the sample data is clustered to obtain hot data, warm data, and cold data.
[0161] In one embodiment, the data partitioning module 10 is further configured to sort the feature values and feature vectors from smallest to largest, and cluster the first K feature values and feature vectors to obtain a preset number of data clusters;
[0162] Calculate the feature mean of each data cluster, and sum the feature means to obtain the initial data heat of each data cluster;
[0163] The initial data heat of each data cluster is aligned in the same direction to obtain the data heat of each data cluster;
[0164] Compare the data popularity of each data cluster, and obtain hot data, warm data, and cold data based on the comparison results.
[0165] In one embodiment, the data migration module 20 is further configured to determine the lifecycle of the source data based on the retention time and attribute parameters of the source data in each source data storage medium when the current time meets the preset period;
[0166] A migration task is constructed based on the lifecycle, and the source data is migrated according to the migration task. The target storage medium is a source data storage medium that stores warm data or a source data storage medium that stores cold data.
[0167] In one embodiment, the data migration module 20 is further configured to determine whether the lifecycle is within a preset migration period range, and to use source data whose lifecycle is within the preset migration period range as data to be migrated.
[0168] The target storage medium is obtained based on the lifecycle timer of the data to be migrated;
[0169] A migration task is constructed based on the data to be migrated, the attribute parameters of the data to be migrated, and the target storage medium.
[0170] The source data is migrated according to the migration task.
[0171] In one embodiment, the data migration module 20 is further configured to parse the migration task to obtain the current storage medium and the target storage medium of the data to be migrated;
[0172] When the current storage medium is a hot data storage medium, delete the data to be migrated in the current storage medium and migrate the data to be migrated to the storage medium corresponding to the warm data.
[0173] When the current storage location is a storage medium for warm data, delete the data to be migrated from the current storage medium and migrate the data to the storage medium corresponding to the cold data.
[0174] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.
[0175] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.
[0176] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.
[0177] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0178] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0179] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.
[0180] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A data migration method, characterized by, The data migration method includes: Obtain the raw data, and divide the raw data according to different granularities to obtain the raw datasets of each granularity; Determine the original dataset corresponding to a reference granularity among multiple original datasets with different granularities, and extract attribute parameters from each original data in the original dataset corresponding to the reference granularity. The attribute parameters include at least the frequency of use and the delay requirement. The sample data is obtained by concatenating the attribute parameters and the retention time of each original data. The sample features of the sample data are normalized to obtain reference features. The sample features include attribute features and retention time features. The attribute features include at least the mean access rate, median access rate, standard deviation of access rate, mean delay requirement, median delay requirement, and standard deviation of delay requirement. An adjacency matrix is constructed based on the reference features; a degree matrix is constructed based on the adjacency matrix; and a Laplace matrix is constructed based on the adjacency matrix and the degree matrix. Based on the eigenvalues and eigenvectors of the Laplacian matrix, the sample data is clustered to obtain hot data, warm data, and cold data. According to the preset storage strategy, the hot data, warm data, and cold data are stored in the corresponding source data storage medium; Obtain the lifecycle of the source data in the source data storage medium, construct a migration task based on the lifecycle, and migrate the source data according to the migration task.
2. The data migration method of claim 1, wherein, The step of clustering the sample data based on the feature values and feature vectors to obtain hot data, warm data, and cold data includes: The feature values and feature vectors are sorted from smallest to largest, and the first K feature values and feature vectors are clustered to obtain a preset number of data clusters; Calculate the feature mean of each data cluster, and sum the feature means to obtain the initial data heat of each data cluster; The initial data heat of each data cluster is aligned in the same direction to obtain the data heat of each data cluster; Compare the data popularity of each data cluster, and obtain hot data, warm data, and cold data based on the comparison results.
3. The data migration method of claim 1, wherein, The process of obtaining the lifecycle of the source data in the source data storage medium, constructing a migration task based on the lifecycle, and migrating the source data according to the migration task includes: When the preset period is met at the current time, the lifecycle of the source data is determined based on the retention time and attribute parameters of the source data in each source data storage medium. A migration task is constructed based on the lifecycle, and the source data is migrated according to the migration task. The target storage medium is either the source data storage medium storing warm data or the source data storage medium storing cold data.
4. The data migration method of claim 3, wherein, The step of constructing a migration task based on the lifecycle and migrating the source data according to the migration task includes: Determine whether the lifecycle is within the preset migration period range, and use the source data whose lifecycle is within the preset migration period range as the data to be migrated; The target storage medium is obtained based on the lifecycle timer of the data to be migrated; A migration task is constructed based on the data to be migrated, the attribute parameters of the data to be migrated, and the target storage medium. The source data is migrated according to the migration task.
5. The data migration method of claim 4, wherein, The migration of the source data according to the migration task includes: The migration task is parsed to obtain the current storage medium and the target storage medium of the data to be migrated; When the current storage medium is a hot data storage medium, delete the data to be migrated in the current storage medium and migrate the data to be migrated to the storage medium corresponding to the warm data. When the current storage medium is a storage medium for warm data, delete the data to be migrated in the current storage medium and migrate the data to be migrated to the storage medium corresponding to cold data.
6. A data migration apparatus, characterized by comprising: The data migration device includes: The data partitioning module is used to acquire raw data, partition the raw data according to daily, weekly, and monthly granularity to obtain raw datasets of each granularity; determine the raw dataset corresponding to a reference granularity among multiple raw datasets of different granularities, and extract attribute parameters from each raw data in the raw dataset corresponding to the reference granularity, the attribute parameters including at least usage frequency and latency requirements; and perform feature concatenation on the attribute parameters and the retention time of each raw data to obtain sample data. The data partitioning module is further used to normalize the sample features of the sample data to obtain reference features. The sample features include attribute features and retention time features. The attribute features include at least the mean access rate, median access rate, standard deviation of access rate, mean latency requirement, median latency requirement, and standard deviation of latency requirement. An adjacency matrix is constructed based on the reference features. A degree matrix is constructed based on the adjacency matrix. A Laplacian matrix is constructed based on the adjacency matrix and the degree matrix. The sample data is clustered based on the eigenvalues and eigenvectors of the Laplacian matrix to obtain hot data, warm data, and cold data. The data migration module is used to store the hot data, warm data and cold data to the corresponding source data storage medium according to a preset storage strategy; The data migration module is further configured to obtain the lifecycle of the source data in the source data storage medium, construct a migration task based on the lifecycle, and migrate the source data according to the migration task.
7. A data migration device, characterized in that, The device includes: a memory, a processor, and a data migration program stored in the memory and executable on the processor, the data migration program being configured to implement the data migration method as described in any one of claims 1 to 5.
8. A storage medium, characterized in that, The storage medium stores a data migration program, which, when executed by a processor, implements the data migration method as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Content distribution network caching method and system based on spectral clustering
CN113225380A
Cold and hot data separation method and device based on big data, equipment and medium
CN116150184A
Hierarchical energy-saving storage system and method based on data life cycle management
CN116301584A
