Hierarchical cloud storage adaptive migration method based on data popularity, medium and equipment

Abstract

The invention discloses a hierarchical cloud storage adaptive migration method based on data popularity, a medium and equipment. The method comprises the following steps that a migration management platform composed of a statistics and alarm module, a heat model calculation module, a strategy control center module, a metadata management module and a migration action module is added to a middle layer of service side storage and archiving side storage; when the data written by the client reaches an alarm threshold, the modules cooperate with each other to adaptively migrate the data from the service side storage to the archiving side storage according to the data popularity; and when the client needs to relocate the data, the modules cooperate with each other to store the relocated data from the filing side to the service side. The invention provides a self-adaptive migration method according to data popularity in a hierarchical cloud storage environment, and the overall performance and the resource utilization rate of a storage system are improved.

Description

Technical Field

[0001] The present invention belongs to the field of data storage, and in particular relates to a hierarchical cloud storage adaptive migration method, medium and device based on data popularity. Background Art

[0002] With the rapid development of digital media and internet technologies, the broadcasting and television industry is facing unprecedented data growth and diverse business demands. Massive amounts of video, audio, image, and other media content not only involve live broadcasts and on-demand content, but also program archiving, content distribution, and intelligent recommendations. This makes data storage and management extremely complex. This is especially true in multi-tenant environments, where different business departments, channels, or regions need to share storage resources on the same platform while ensuring data isolation and security to meet their respective business needs and service-level agreements (SLAs).

[0003] With the rapid growth of data volumes, storage systems are flooded with files. Their access frequency (popularity) is becoming increasingly dynamic and unevenly distributed across different tenant environments. Traditional storage management methods struggle to adapt to these rapidly changing access patterns, resulting in a mix of high- and low-frequency access files on the same storage tier, impacting overall system performance and resource utilization. This has led to an increasing demand for data migration between different cloud storage systems.

[0004] However, in cloud storage systems that adopt a hierarchical storage architecture, how to balance storage costs and access performance is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] The present invention aims to address the aforementioned issues in the prior art and provide a method, medium, and device for adaptive migration of hierarchical cloud storage based on data popularity. This invention addresses the dynamic scheduling and optimized management of data storage resources in a hierarchical cloud storage environment. By monitoring data access popularity, it enables intelligent data migration between different cloud storage platforms, thereby improving the overall performance and resource utilization of the storage system.

[0006] The specific technical solutions adopted in the present invention are as follows:

[0007] In a first aspect, the present invention provides a hierarchical cloud storage adaptive migration method based on data popularity. The method collaboratively implements the write data archiving process and the read data migration process by setting up a migration management platform at the intermediate layer between the business-side storage and the archive-side storage. The migration management platform includes a statistics and alarm module, a popularity model calculation module, a policy control center module, a metadata management module, and a migration action module.

[0008] In the data archiving process, the statistics and alarm module obtains the data writing status of the client to the business side storage in real time. When the data written by the client reaches the alarm threshold, the policy control center module verifies the first migration condition and then calls the heat model calculation module to calculate the file heat. The migration action module is called to migrate the files with the lowest heat from the business side storage to the archiving side storage. The metadata management module records the migration metadata information.

[0009] In the data read and migration process, the migration management platform receives the data migration request initiated by the client, and finds the target file requested to be migrated by the client based on the migration metadata information recorded in the metadata management module. The policy control center module then verifies the second migration condition and selects the target file to be migrated according to the preset migration file screening policy. The migration action module is called to migrate the target file from the archive side storage to the business side storage, and the metadata management module records the migration metadata information.

[0010] As a preferred embodiment of the first aspect, the data archiving process is as follows:

[0011] The client mounts the business-side storage shared directory and writes data. When the amount of data written by the client reaches the alarm threshold assigned by the business-side storage file system control center to the client shared directory, an alarm is generated and sent to the statistics and alarm module. The alarm information includes the alarm time, the business-side storage directory shared path, the total directory quota, the directory quota used, and the alarm reason.

[0012] The statistics and alarm module sends the generated alarm information to the policy control center module in real time. At the same time, it obtains the operating status of the business-side storage file system control center and the archiving-side storage file system control center in real time and sends the operating status to the policy control center module. The operating status includes whether the total capacity has reached the read-only threshold, whether the link bandwidth has reached the bottleneck, and whether the network connectivity is abnormal.

[0013] The policy control center module analyzes the received operating status to determine whether the first migration condition is met. If so, the business side storage directory shared path in the alarm information is sent to the heat model calculation module; the heat model calculation module calls the log system of the business side storage file system control center according to the received business side storage directory shared path, and uses the heat model to calculate the heat of all files under the business side storage directory shared path based on the log of the log system, generates a heat value for each file, and sends the calculated file path, file name, and file heat value of each file to the policy control center module.

[0014] The policy control center module selects target files to be migrated based on the file heat value and the preset migration file screening policy, and sends the file path and file name of the target files that meet the screening conditions to the migration action module. The migration action module migrates the corresponding file data from the business side storage shared directory to the archive side storage shared directory, and writes the file migration metadata information to the metadata management module to record the archiving status; the migration metadata information includes migration status, migration time, business side storage directory shared path, file name, archive side storage directory shared path and file size;

[0015] The migration management platform synchronizes key migration metadata information, including migration time, file name, and migration status, to the migration record table in the client.

[0016] As a preferred embodiment of the first aspect, the read data migration process is specifically as follows:

[0017] If the client cannot find the target file in the shared directory on the business side, it will search its own migration record table. If it finds the target file there, it will initiate a data migration request to the migration management platform.

[0018] Based on the received data migration request, the migration management platform retrieves the migration metadata information from the metadata management module, finds the target file metadata that needs to be migrated, and sends it to the policy control center module;

[0019] After receiving the target file metadata, the policy control center module needs to obtain the business-side storage shared directory quota attributes and the operating status of the business-side storage file system control center and the archive-side storage file system control center from the statistics and alarm module, and determine whether the second migration condition is met. If so, the target file metadata is sent to the migration action module;

[0020] The migration action module receives the metadata of the target file to be relocated and executes the migration action of the target file, migrating the target file from the archive-side storage shared directory to the business-side storage shared directory. At the same time, the metadata information of the relocated target file is updated in the metadata management module.

[0021] The migration management platform synchronizes key migration metadata information, including migration time, file name, and migration status, to the migration record table in the client.

[0022] As a preferred embodiment of the first aspect, in the heat model calculation module, the heat model for calculating the heat of a file is as follows:

[0023]

[0024] Where: w iis the preset weight corresponding to the event category of the i-th event; t now is the current time, t i is the occurrence time of the i-th event, λ and μ are preset decay constants; T(x) is the weight corresponding to different tenant levels, which is determined by the tenant level to which the sub-path x of the currently calculated file belongs. The higher the tenant level, the greater the weight; B(t now ) is a scheduling factor function determined by the time difference between the current time and the time of the last write operation. When the time difference does not exceed the time threshold, B(t now ) is 1, otherwise B(t now ) decays exponentially based on the difference between the time difference and the time threshold.

[0025] As a preference of the above first aspect, the first migration condition includes two conditions that need to be satisfied simultaneously, where:

[0026] The first condition is that both the business-side storage and the archive-side storage satisfy that the total storage capacity has not reached the read-only threshold, the link bandwidth has not reached the bottleneck, and the network connectivity is normal;

[0027] The second condition is that the shared path of the business-side storage directory in the alarm information obtained by the statistics and alarm module belongs to the business-side storage shared path library PL monitored in the policy control center module, and the alarm reason is reaching the hard threshold, reaching the soft threshold, or reaching the recommended threshold.

[0028] As a preference of the above first aspect, the second migration condition includes three conditions that need to be satisfied simultaneously, where:

[0029] The first condition is that both the business-side storage and the archive-side storage satisfy that the total storage capacity has not reached the read-only threshold, the link bandwidth has not reached the bottleneck, and the network connectivity is normal;

[0030] The second condition is that the shared path of the business-side storage directory in the target file metadata belongs to the business-side storage shared path library PL monitored in the policy control center module, and the alarm reason is reaching the hard threshold, reaching the soft threshold, or reaching the recommended threshold;

[0031] The third condition is that the currently used quota CUQ of the business-side storage shared directory is less than the hard threshold assigned by the business-side storage file system control center to the client shared directory.

[0032] As a preference of the above first aspect, the alarm thresholds assigned by the business-side storage file system control center to the client shared directory are divided into three types: hard threshold, soft threshold, or recommended threshold, and the alarm reasons in the generated alarm information also correspond to three types: reaching the hard threshold, reaching the soft threshold, or reaching the recommended threshold.

[0033] As a preferred embodiment of the above-mentioned first aspect, the migration file screening strategy is: according to the file heat value of each file in the shared directory stored on the business side and the preset file size threshold, the alarm cause is identified from the current alarm information, and the heat threshold preset for the alarm cause is determined, and all files in the shared directory whose file heat value is lower than the heat threshold and whose file size exceeds the file size threshold are screened out, and use them as files to be migrated.

[0034] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for hierarchical cloud storage adaptive migration based on data heat as described in any one of the first aspects above is implemented.

[0035] In a third aspect, the present invention provides a computer electronic device, characterized in that it includes a memory and a processor;

[0036] The memory is used to store computer programs;

[0037] The processor is configured to implement the data heat-based hierarchical cloud storage adaptive migration method as described in any one of the first aspects above when executing the computer program.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention provides a hierarchical cloud storage adaptive migration method based on data heat. This method adds a migration management platform consisting of a statistics and alarm module, a heat model calculation module, a policy control center module, a metadata management module, and a migration action module to the intermediate layer between business-side storage and archive-side storage. When the data written by the client reaches the alarm threshold, the modules cooperate with each other to adaptively migrate the data from the business-side storage to the archive-side storage based on the data heat. When the client needs to migrate the data back, the modules cooperate with each other to migrate the data back from the archive-side storage to the business-side storage. By implementing adaptive file screening and migration based on data heat in a hierarchical cloud storage environment, the present invention improves the overall performance and resource utilization of the storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a schematic diagram of the module composition in the migration management platform;

[0041] Figure 2 The diagram shows the coordination relationship between modules corresponding to the data archiving process. The dotted line represents the storage process, and the solid line represents the data interaction between modules.

[0042] Figure 3This is a diagram of the inter-module coordination relationship corresponding to the read data migration process. The dotted line represents the storage process, and the solid line represents the data interaction between modules.

[0043] Figure 4 A flowchart of an exemplary implementation of the data archiving process;

[0044] Figure 5 This is a flowchart of an exemplary implementation of the read data migration process. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0046] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0047] The present invention provides a hierarchical cloud storage adaptive migration method based on data popularity, which is applied to a storage system adopting a hierarchical storage architecture. Such a storage system consists of business-side storage and archive-side storage. Taking the radio and television industry as an example, the business-side storage is mainly enterprise-level distributed NAS storage, which is suitable for storing popular data with high frequency access, ensuring low latency and high response speed. The archive-side storage system adopts a hybrid media architecture to build a hierarchical storage system in which the storage area network (SAN) and the tape library work together. The front end of the system integrates a multimodal access interface through a protocol conversion gateway: it supports traditional file system protocols (including NFS and CIFS), and the back-end storage layer realizes intelligent hierarchical management of warm data (SAN) and cold data (tape) through an automated data migration engine, which significantly reduces storage costs while ensuring data accessibility. It is suitable for storing historical programs, archival materials and long-term backup data.

[0048] In a preferred embodiment of the present invention, for a hierarchical cloud storage consisting of business-side storage and archive-side storage, a migration management platform is set up on the middle layer between the business-side storage and the archive-side storage to collaboratively implement the write data archiving process and the read data migration process. In an embodiment of the present invention, both the business-side storage and the archive-side storage can be implemented using NAS (Network Attached Storage) storage. Therefore, the business-side storage and the archive-side storage can respectively use business-side NAS storage and archive-side NAS storage. Both the business-side NAS storage and the archive-side NAS storage can provide shared directories to the client and the migration management platform. The present invention refers to the shared directories on the business-side NAS storage and the archive-side NAS storage as business-side NAS shared directories and archive-side NAS shared directories, respectively. At the same time, NAS storage generally has its own file system control center, which is a file system responsible for the allocation of storage resources, optimization of file services, and user permission management. For the business-side NAS shared directory allocated to each client, the NAS file system control center needs to pre-specify the total quota that can be used (i.e., the storage capacity limit of the directory), and can also set multiple thresholds to implement quota usage alarms. Generally speaking, each shared directory can have a hard threshold, soft threshold, or recommended threshold. Each threshold corresponds to a percentage. For example, the hard threshold, soft threshold, and recommended threshold can be 100%, 90%, and 80% of the total quota of the business-side NAS shared directory, respectively. The specific thresholds can also be adjusted based on actual needs.

[0049] It should be noted that the client in the present invention may be a terminal or a service program.

[0050] like Figure 1 As shown, the migration management platform set up on the middle layer between the business-side storage and the archive-side storage includes a statistics and alarm module, a heat model calculation module, a policy control center module, a metadata management module and a migration action module. It should be noted that the statistics and alarm module, the heat model calculation module, the policy control center module, the metadata management module and the migration action module are core functional modules integrated into the migration management platform, and the migration management platform, as a management platform, provides corresponding login, management, visualization, API interface call and other services to cloud storage tenants at all levels. The client directly interacts with the various modules on the migration management platform through the migration management platform. The data interaction relationship between the business-side storage, archive-side storage and various modules in the migration management platform is shown in the figure. Figure 2As shown in the figure. When the data written by the client reaches the alarm threshold, the modules in the migration management platform work together to adaptively migrate data from business-side storage to archive-side storage based on data popularity. When the client needs to re-migrate data, the modules in the migration management platform work together to re-migrate data from archive-side storage to business-side storage. The following describes in detail how each module in the migration management platform works together.

[0051] In the data archiving process of the present invention, the specific collaboration and cooperation of each module is as follows: the statistics and alarm module obtains the data writing status of the client to the business side storage in real time. When the data written by the client reaches the alarm threshold, the policy control center module verifies the first migration condition and then calls the heat model calculation module to calculate the file heat, and calls the migration action module to migrate the part of the file with the lowest file heat from the business side storage to the archiving side storage, and the metadata management module records the migration metadata information.

[0052] In the data read and migration process of the present invention, the specific collaboration and cooperation of each module is as follows: the migration management platform receives the data migration request initiated by the client, and finds the target file requested to be migrated by the client based on the migration metadata information recorded in the metadata management module. The policy control center module then performs a second migration condition verification and filters the target files that need to be migrated according to the preset migration file screening policy. The migration action module is called to migrate the target files from the archive side storage to the business side storage, and the metadata management module records the migration metadata information.

[0053] The core of the present invention is to adaptively migrate files based on their popularity, thereby improving the overall performance and resource utilization of the storage system. Each module on the migration management platform is a software functional module, and the coordination between them can be reasonably adjusted and optimized according to actual needs.

[0054] In an embodiment of the present invention, a specific implementation of the above-mentioned data archiving process is provided. For the specific process, see Figure 2 As shown, the details are as follows:

[0055] S11. The client (terminal / business program) mounts the business-side NAS shared directory and writes data. When the amount of data written by the client reaches the hard threshold, soft threshold, or recommended threshold assigned to the client shared directory by the business-side NAS file system control center, an alarm is generated and sent to the statistics and alarm module. The alarm information includes the alarm time, the business-side NAS directory sharing path, the total directory quota, the directory quota used, and the alarm reason. The alarm reason can be reaching the hard threshold, reaching the soft threshold, or reaching the recommended threshold.

[0056] S12. The statistics and alarm module sends the generated alarm information to the policy control center module in real time, and at the same time obtains the operating status of the business side NAS file system control center and the archiving side NAS file system control center in real time, and sends the operating status to the policy control center module; the operating status includes whether the total capacity reaches the read-only threshold, whether the link bandwidth reaches the bottleneck, and whether the network connectivity is abnormal.

[0057] S13. The policy control center module analyzes the operating status of the business-side NAS file system control center and the archiving-side NAS file system control center to determine whether the first migration condition is met. If so, the business-side NAS directory sharing path in the alarm information is sent to the heat model calculation module.

[0058] S14. The heat model calculation module calls the log system of the business side NAS file system control center according to the received business side NAS directory sharing path, uses the heat model to calculate the heat of all files under the business side NAS directory sharing path based on the log of the log system, and generates the heat value of each file.

[0059] S15. The heat model calculation module sends the calculated file path, file name, and file heat value of each file to the policy control center module.

[0060] S16. The policy control center module selects target files to be migrated according to the file heat value and the preset migration file screening policy, and sends the file paths and file names of the target files that meet the screening conditions to the migration action module.

[0061] S17. The migration action module migrates the corresponding file data from the business-side NAS shared directory to the archiving-side NAS shared directory based on the received file path and file name, and writes the file's migration metadata information to the metadata management module to record the archiving status; the migration metadata information includes the migration status, migration time, business-side NAS directory shared path, file name, archiving-side NAS directory shared path, and file size.

[0062] S18. The migration metadata information recorded in the metadata management module is visualized on the migration management platform in a directory tree structure. Each file includes information such as migration status, migration time, business-side NAS shared directory path, file name, archive-side NAS shared directory path, and file size.

[0063] S19. The migration management platform synchronizes key migration metadata information including migration time, file name, and migration status to the migration record table in the client.

[0064] It should be noted that the first migration condition verification is to verify whether all migration conditions required for migrating files from business-side storage to archive-side storage are met. The first migration condition contains multiple migration conditions, and the file migration task can only be executed if all migration conditions are met. In an embodiment of the present invention, the first migration condition contains two conditions that need to be met simultaneously, where:

[0065] The first condition is that both the business-side storage and the archive-side storage meet the following conditions: the total storage capacity does not reach the read-only threshold, the link bandwidth does not reach the bottleneck, and the network connectivity is normal;

[0066] The second condition is that the business side NAS directory shared path in the alarm information obtained by the statistics and alarm module belongs to the business side NAS shared path library PL monitored in the policy control center module, and the alarm reason is reaching the threshold alarm (in this embodiment, including hard threshold, reaching soft threshold or reaching recommended threshold).

[0067] In an embodiment of the present invention, a specific implementation of the above-mentioned read data migration process is provided. For the specific process, see Figure 3 As shown, the details are as follows:

[0068] S21. When the client (terminal / business application) cannot find the target file to be written in the business-side NAS shared directory, it will search its own stored migration record table. If the target file is found in the migration record table, it will initiate a data migration request to the migration management platform. It should be noted that the client can initiate a data migration request to the migration management platform manually by logging into the migration management platform, or automatically by calling the migration management platform API to initiate a data migration request to the migration management platform to locate the file data to be migrated.

[0069] S22. The migration management platform retrieves the migration metadata information in the metadata management module according to the received data migration request, and finds the target file metadata that needs to be migrated back.

[0070] S23. The metadata management module sends the found target file metadata to the policy control center module.

[0071] S24. After receiving the target file metadata from the metadata management module, the policy control center module obtains the business-side NAS storage shared directory quota attributes and the operating status of the business-side NAS file system control center and the archiving-side NAS file system control center from the statistics and alarm module, and determines whether the second migration condition is met. If so, the target file metadata is sent to the migration action module. The operating status of the business-side NAS file system control center and the archiving-side NAS file system control center includes whether the total capacity has reached the read-only threshold, whether the link bandwidth has reached a bottleneck, and whether the network connectivity is abnormal.

[0072] S25. The migration action module receives metadata of the target file to be migrated.

[0073] S26. The migration action module executes the migration action of the target file based on the received target file metadata, migrates the target file from the archive side NAS shared directory to the business side NAS shared directory, and updates the metadata information of the migrated target file in the metadata management module.

[0074] S27. The metadata management module visualizes the updated metadata information in a directory tree structure on the migration management platform. Each file includes information such as migration status, migration time, business-side NAS shared directory path, file name, archive-side NAS shared directory path, and file size.

[0075] S28. The migration management platform synchronizes key migration metadata information including migration time, file name, and migration status to the migration record table in the client.

[0076] It should be noted that the migration record table in the client does not record complete migration metadata information, but only records key information such as migration time, file name, and migration status.

[0077] It should be noted that the second migration condition verification is to verify whether all migration conditions required for migrating files from archive-side storage to business-side storage are met. The second migration condition contains multiple migration conditions, and the file migration task can only be executed if all migration conditions are met. In an embodiment of the present invention, the second migration condition needs to include three conditions that need to be met simultaneously, including:

[0078] The first condition is that both the business-side storage and the archive-side storage meet the following conditions: the total storage capacity does not reach the read-only threshold, the link bandwidth does not reach the bottleneck, and the network connectivity is normal;

[0079] The second condition is that the business-side NAS directory shared path in the target file metadata belongs to the business-side NAS shared path library PL monitored in the policy control center module (a policy library is pre-constructed in the policy control center module, and the business-side NAS shared paths that need to be monitored are specified to form the path library PL), and the alarm reason is reaching a threshold alarm (in this embodiment, including hard threshold, reaching soft threshold or reaching recommended threshold).

[0080] The third condition is that the currently used quota CUQ of the business-side NAS storage shared directory is less than the hard threshold allocated to the client shared directory by the business-side NAS file system control center.

[0081] In the data migration process of the present invention, the key is to achieve adaptive migration according to the heat of file data, that is, to migrate some files with lower heat from the business-side NAS directory to the archival-side NAS directory, while the files with higher heat remain in the business-side NAS directory, thereby improving the overall performance and resource utilization rate of the storage system. In the present invention, the heat of a file needs to be calculated depending on the heat model in the heat model calculation module. In an embodiment of the present invention, the heat model for calculating the file heat can calculate the heat of each file in the following manner:

[0082] First, an exponential decay function is used as the basic decay function to reflect that the impact of each event of a file on the file heat gradually weakens over time. The basic weight at any i-th time is e -λ·Δt , where Δt = t now -t i is the difference between the current time t now and the event occurrence time t i , and λ is the decay constant.

[0083] [[ID=1--6]]Secondly, the scheduling factor function B(t) at any time t is calculated based on the basic decay function as follows:

[0084]

[0085] where: T last_write is the time of the last write operation of the file; Δt1 represents the "high demand" time window used by the file, and the value of Δt1 can be adjusted according to actual needs. μ controls the faster decline when it is farther from the time of the last write operation.

[0086] The above scheduling factor function B(t) can be used to reflect the file demand relative to any time. Generally speaking, when calculating the file heat, it is calculated based on the current time t now , so the corresponding scheduling factor B(t now ) can be calculated by setting t = t now . According to the formula of B(t now ) above, it can be seen that the scheduling factor function B(t now ) is determined by the time difference t now -T last_write between the current time and the time of the last write operation. When the time difference t now -T last_write does not exceed the time threshold Δt1, B(t now ) is 1, otherwise B(t now ) decays exponentially based on the difference between the time difference t now -T last_write and the time threshold Δt1. Thus, when t now is close to T last_writeWhen B(t) takes the value of 1, it means that the file demand increases; when t now Stay away from T last_write , B(t) gradually decreases (even approaches 0), reflecting the current low usage demand of the file.

[0087] Then calculate the multi-tenant enhancement factor function T(x), where T(x) is the weight corresponding to different tenant levels, and x is the sub-path of the business-side NAS directory shared path P (corresponding to the file whose popularity is calculated). In this embodiment, it is assumed that tenants are divided into three levels, P1 is the first-level tenant, P2 is the second-level tenant, and P3 is the third-level tenant. The corresponding T(x) assignment principles are as follows:

[0088]

[0089] Among them: T1, T2, and T3 are the weights of first-level tenants, second-level tenants, and third-level tenants respectively, satisfying T1>T2>T3.

[0090] In summary, a hybrid function model consisting of a basic decay function, a scheduling factor function, and a multi-tenant enhancement factor function can be established. The file popularity calculated by this model needs to be calculated based on each event of the file (including four types of events: creation, reading, writing, and renaming. All events can be obtained from the log system of the NAS file system control center on the business side). The calculation formula is as follows:

[0091]

[0092] Where: w i The preset weight of the event category corresponding to the i-th event. Each type of event in create, read, write, and rename can be preset with a weight, which is an adjustable value. i It can be used to distinguish the contribution of different types of access or operations to file popularity. now is the current time, t i is the time when the i-th event occurs (which can be determined from the log), λ and μ are preset decay constants, which are adjustable values; T(x) is the weight corresponding to the aforementioned different tenant levels, which is determined by the tenant level to which the subpath x of the currently calculated file belongs according to the aforementioned formula.

[0093] The above heat model can not only reflect the attenuation effect of historical access data, but also appropriately adjust the "heat" increase generated when a file is modified, and distinguish the degree of importance attached to different tenant levels.

[0094] In order to better understand the write data archiving process and the read data migration process in the above-mentioned data popularity-based hierarchical cloud storage adaptive migration method of the present invention, exemplary implementation methods of the two processes are specifically demonstrated below.

[0095] For the convenience of description, some concepts are defined below:

[0096] Definition 1: The total storage capacity, link bandwidth, and network connectivity are AC (All_Capacity), BW (Bandwidth), and LO (LinkOk), respectively. Their values are:

[0097]

[0098] Definition 2: The service-side NAS shared directory path in the alarm information in the statistics and alarm module is P (Path), X is the subpath of P, the hard threshold alarm, soft threshold alarm, and recommended threshold alarm are AHT (Alarm_Hard_Threshold), AST (Alarm_Soft_Threshold), and ART (Alarm_Recommended_Threshold), respectively.

[0099] Definition 3: The service-side NAS shared path library monitored in the policy control center module is PL (Path_Library).

[0100] Definition 4: The hard threshold, soft threshold, and recommended threshold are 100%, 90%, and 80% of the business-side shared directory quota, respectively. The corresponding storage capacities are denoted as Q1, Q2, and Q3, respectively, and Q1>Q2>Q3.

[0101] Definition 5: The first-level heat threshold, the second-level heat threshold, and the third-level heat threshold are H1, H2, and H3 respectively, and H1>H2>H3.

[0102] Definition 6: The used capacity of the current directory is CUQ (Currently_Used_Quota), and the file size is Size.

[0103] Definition 7: The aforementioned heat model H(x) in the heat model calculation module is denoted as Model.

[0104] Based on the above definition, exemplary preferred implementations of the write data archiving process and the read data migration process are as follows: Figure 4 and Figure 5 shown.

[0105] See also Figure 4 As shown, with the cooperation of various functional modules, the process of writing data archiving strategy is as follows:

[0106] Step 1-1: Obtain the AC, BW, and LO values for the service-side NAS storage and the archive-side NAS storage from the Statistics and Alarms module. Calculate whether AC*BW*LO equals 1. If AC*BW*LO equals 1 on both sides, continue; otherwise, terminate the migration task.

[0107] In steps 1-2, query whether the service-side NAS storage shared path P obtained from the Statistics and Alarms module belongs to the service-side NAS shared path library PL monitored by the Policy Control Center module. Also, determine whether the alarm is caused by reaching a threshold (i.e., the alarm is caused by reaching a hard threshold, a soft threshold, or a recommended threshold). If the result is yes in all three cases, continue; otherwise, terminate the migration task.

[0108] Steps 1-3: Call the heat model Model to calculate the heat value H corresponding to all files in the current business-side NAS directory shared path P (each file corresponds to a specific sub-path x). Because the multi-tenant enhancement factor function T(x) is introduced in the model, different x in P correspond to different T(x) values.

[0109] Step 1-4: Query the currently used quota CUQ of the business-side NAS storage shared path P obtained from the statistics and alarm module. According to the position of the current CUQ value in Q1, Q2, and Q3, finally filter out the files that meet the heat thresholds H1, H2, and H3 corresponding to Q1, Q2, and Q3 and whose size is greater than 50MB. These files j are the files to be migrated. In an embodiment of the present invention, the screening process needs to be divided into three judgment steps. If CUQ=Q1, then the files with a heat value H less than the heat threshold H1 and a file size greater than 50MB are filtered out from the business-side NAS storage shared path P as the files to be migrated; if Q2≤CUQ <Q1,则从业务侧NAS存储共享路径P中筛选出热度值H小于热度阈值H2且文件大小Size大于50MB的文件作为待迁移文件;若Q3≤CUQ<Q2,则从业务侧NAS存储共享路径P中筛选出热度值H小于热度阈值H3且文件大小Size大于50MB的文件作为待迁移文件。每个待迁移文件的文件名以及文件路径发送到迁移动作模块,执行迁移任务,将文件从业务侧NAS存储迁移到归档侧NAS存储。

[0110] It should also be noted that the above steps 1-1 to 1-4 only describe some details. For unclear parts of the data transmission process within each module and between each other, please refer to the above S11 to S19.

[0111] See also Figure 5 As shown, with the cooperation of various functional modules, the read data migration strategy process is as follows:

[0112] Step 2-1: Obtain the AC, BW, and LO values for the business-side NAS storage and the archive-side NAS storage from the Statistics and Alarms module. Calculate whether AC*BW*LO equals 1. If AC*BW*LO equals 1 on both sides, continue; otherwise, terminate the migration task.

[0113] Step 2-2: Check whether the service-side NAS storage shared path P obtained from the Statistics and Alarms module belongs to the service-side NAS shared path library PL monitored by the Policy Control Center module. Also, determine whether the alarm is caused by reaching a threshold (i.e., the alarm is caused by reaching a hard threshold, a soft threshold, or a recommended threshold). If the result is yes in all three cases, continue; otherwise, terminate the migration task.

[0114] Step 2-3: Query the current used quota CUQ of the business-side NAS storage shared path P obtained from the statistics and alarm module. If CUQ is less than Q1, the data to be migrated will be sent to the migration action module; otherwise, the migration task will be terminated.

[0115] It should also be noted that the above steps 2-1 to 2-3 only describe some details. For unclear parts of the data transmission process within each module and between each other, please refer to the above S21 to S28.

[0116] The embodiments described above are merely some preferred implementations of the present invention and are not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A hierarchical cloud storage adaptive migration method based on data heat, characterized in that, This method implements the data archiving process and the data migration process in a coordinated manner by setting up a migration management platform on the middle layer between the business-side storage and the archiving-side storage. The migration management platform includes a statistics and alarm module, a heat model calculation module, a policy control center module, a metadata management module, and a migration action module. In the data archiving process, the statistics and alarm module obtains the data writing status of the client to the business side storage in real time. When the data written by the client reaches the alarm threshold, the policy control center module verifies the first migration condition and then calls the heat model calculation module to calculate the file heat. The migration action module is called to migrate the files with the lowest heat from the business side storage to the archiving side storage. The metadata management module records the migration metadata information. In the data read and migration process, the migration management platform receives the data migration request initiated by the client, and finds the target file requested to be migrated by the client based on the migration metadata information recorded in the metadata management module. The policy control center module then verifies the second migration condition and selects the target file to be migrated according to the preset migration file screening policy. The migration action module is called to migrate the target file from the archive side storage to the business side storage, and the metadata management module records the migration metadata information.

2. The hierarchical cloud storage adaptive migration method based on data heat as claimed in claim 1, wherein The data archiving process is as follows: The client mounts the business-side storage shared directory and writes data. When the amount of data written by the client reaches the alarm threshold assigned by the business-side storage file system control center to the client shared directory, an alarm is generated and sent to the statistics and alarm module. The alarm information includes the alarm time, the business-side storage directory shared path, the total directory quota, the directory quota used, and the alarm reason. The statistics and alarm module sends the generated alarm information to the policy control center module in real time. At the same time, it obtains the operating status of the business-side storage file system control center and the archiving-side storage file system control center in real time and sends the operating status to the policy control center module. The operating status includes whether the total capacity has reached the read-only threshold, whether the link bandwidth has reached the bottleneck, and whether the network connectivity is abnormal. The policy control center module analyzes the received operating status to determine whether the first migration condition is met. If so, it sends the service-side storage directory shared path in the alarm information to the heat model calculation module; The heat model calculation module calls the log system of the business side storage file system control center based on the received business side storage directory shared path, uses the heat model to calculate the heat of all files under the business side storage directory shared path based on the log of the log system, generates the heat value of each file, and sends the calculated file path, file name, and file heat value of each file to the policy control center module; The policy control center module selects the target files to be migrated based on the file popularity value and the preset migration file screening policy, and sends the file path and file name of the target files that meet the screening conditions to the migration action module. The migration action module migrates the corresponding file data from the business-side storage shared directory to the archive-side storage shared directory, and writes the file migration metadata information to the metadata management module to record the archiving status; The migration metadata information includes migration status, migration time, business side storage directory shared path, file name, archive side storage directory shared path and file size; The migration management platform synchronizes key migration metadata information, including migration time, file name, and migration status, to the migration record table in the client.

3. The hierarchical cloud storage adaptive migration method based on data popularity according to claim 1, characterized in that: The read data migration process is as follows: If the client cannot find the target file in the shared directory on the business side, it will search its own migration record table. If it finds the target file there, it will initiate a data migration request to the migration management platform. Based on the received data migration request, the migration management platform retrieves the migration metadata information from the metadata management module, finds the target file metadata that needs to be migrated, and sends it to the policy control center module; After receiving the target file metadata, the policy control center module needs to obtain the business-side storage shared directory quota attributes and the operating status of the business-side storage file system control center and the archive-side storage file system control center from the statistics and alarm module, and determine whether the second migration condition is met. If so, the target file metadata is sent to the migration action module; The migration action module receives the metadata of the target file to be relocated and executes the migration action of the target file, migrating the target file from the archive-side storage shared directory to the business-side storage shared directory. At the same time, the metadata information of the relocated target file is updated in the metadata management module. The migration management platform synchronizes key migration metadata information, including migration time, file name, and migration status, to the migration record table in the client.

4. The hierarchical cloud storage adaptive migration method based on data popularity according to claim 1, characterized in that: In the heat model calculation module, the heat model for calculating file heat is as follows: Where: w i is the preset weight of the event category corresponding to the i-th event; t now is the current time, t i is the occurrence time of the i-th event, λ and μ are preset decay constants; T(x) is the weight corresponding to different tenant levels, determined by the tenant level to which the sub-path x of the currently calculated file belongs, and the higher the tenant level, the greater the weight; B(t now ) is a scheduling factor function determined by the time difference between the current time and the time of the last write operation. When the time difference does not exceed the time threshold, B(t now ) is 1, otherwise B(t now ) decays exponentially based on the difference between the time difference and the time threshold.

5. The hierarchical cloud storage adaptive migration method based on data popularity as claimed in claim 1, wherein, The first migration condition includes two conditions that need to be met simultaneously, wherein: The first condition is that both the business-side storage and the archive-side storage meet the following conditions: the total storage capacity does not reach the read-only threshold, the link bandwidth does not reach the bottleneck, and the network connectivity is normal; The second condition is that the business side storage directory shared path in the alarm information obtained by the statistics and alarm module belongs to the business side storage shared path library monitored in the policy control center module, and the alarm reason is reaching the threshold alarm.

6. The hierarchical cloud storage adaptive migration method based on data heat as claimed in claim 1, wherein The second migration condition includes three conditions that need to be met simultaneously, among which: The first condition is that both the business-side storage and the archive-side storage meet the following conditions: the total storage capacity does not reach the read-only threshold, the link bandwidth does not reach the bottleneck, and the network connectivity is normal; The second condition is that the business-side storage directory shared path in the target file metadata belongs to the business-side storage shared path library monitored by the policy control center module, and the alarm reason is reaching the threshold alarm; The third condition is that the currently used quota of the storage shared directory on the service side is less than the hard threshold allocated by the file system control center of the service side storage to the client shared directory.

7. The hierarchical cloud storage adaptive migration method based on data popularity according to claim 1, characterized in that: The warning thresholds allocated by the file system control center of the service side storage to the client shared directory are divided into three types: hard threshold, soft threshold, or recommended threshold, and the warning reasons in the generated warning messages also correspond to reaching the hard threshold, reaching the soft threshold, or reaching the recommended threshold.

8. The hierarchical cloud storage adaptive migration method based on data heat as claimed in claim 1, wherein The migration file screening strategy is as follows: Based on the file heat value of each file in the storage shared directory on the service side and a preset file size threshold, identify the warning reason from the current warning message, determine the preset heat threshold for this warning reason, screen out all files in the shared directory whose file heat value is lower than the heat threshold and whose file size exceeds the file size threshold, and use them as files to be migrated.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the hierarchical cloud storage adaptive migration method based on data heat as described in any one of claims 1 to 8.

10. A computer electronic device, characterized in that, It includes a memory and a processor; The memory is used to store a computer program; The processor is used to implement the hierarchical cloud storage adaptive migration method based on data heat as described in any one of claims 1 to 8 when executing the computer program.

Images