A file data hierarchical storage method, device, medium and electronic equipment
Through the file data hierarchical storage method, the file is stored on an appropriate storage device according to the access characteristics of the file, which solves the problem that traditional storage systems cannot meet different file requirements, realizes efficient storage management and resource utilization, and improves user experience and System performance.
Patent Information
- Application Number
- CN202411170677.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-26
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2044-08-26
AI Technical Summary
Traditional single storage systems cannot meet the needs of different files. High frequency and important files require high performance storage, while low frequency and non-important files can accept low performance storage, resulting in a sharp increase in storage costs and poor system performance, especially in business loads. At high a time, resource consumption is too large, which affects the normal access of the storage device.
Using the file data hierarchical storage method, by configuring the hierarchical strategy, the file migration strategy is determined based on the access characteristics of the file (such as access popularity, time, number of read and write operations and data volume), and the high frequency and important files are stored in high-performance storage devices. On the other hand, low frequency and non-important files are stored on low-cost and high-capacity devices and dynamically adjust the migration rate to avoid high loads.
It improves file access speed, reduces access delay, avoids the waste of expensive storage resources, improves the overall performance and scalability of the system, and ensures efficient storage management.
Smart Images
Figure CN118672520B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data storage, and in particular relates to a file data hierarchical storage method, device, medium and electronic equipment. Background Art
[0002] With the advent of the information age, enterprises and organizations generate a large number of files in their daily operations, and these files vary in importance and access frequency. Traditional single storage systems cannot meet the needs of different files. For example, high-frequency access and very important files require faster access speeds and higher security, while low-frequency access and non-important files can accept slower access speeds and lower security.
[0003] And with the development of hardware technology in recent years, new storage devices continue to emerge, such as SATASSD, PCIeSSD, etc., which can provide high-performance data storage and data access, but their prices are relatively high. Relatively speaking, traditional hard disks have poor performance, but are cheap. In addition, many studies on large-scale storage systems have shown that among all files, only a small number of files are used every day, while most files are hardly used, that is, file access is local. Therefore, if all files are stored on high-performance storage devices, although high-performance storage and access of files can be guaranteed, the storage cost will rise sharply due to the huge amount of data; and if all files are stored on large-capacity storage devices, the overall performance of the system cannot be guaranteed. Due to fewer considerations, the hierarchical storage effect cannot be fully utilized, especially when the business load is high, it is not conducive to the resource consumption caused by the hierarchical migration of file data, and affects the normal business access of storage devices. Therefore, there is an urgent need for a hierarchical storage method and device for files in multiple storage systems to solve the above problems. Summary of the invention
[0004] The purpose of the present invention is to address the above-mentioned problems and propose a file data hierarchical storage method, device, medium and electronic device, which can increase access speed, reduce access delay, improve user experience, avoid wasting expensive storage resources, and improve overall system utilization efficiency and scalability.
[0005] To achieve the above object, the technical solution adopted by the present invention is:
[0006] The present invention provides a method, device, medium and electronic device for hierarchical storage of file data, comprising the following steps:
[0007] S1. Configure a tiering strategy, which includes at least two levels of storage devices, and the higher the level of the storage device, the better the performance;
[0008] S2. Respond to data access requests and record data access logs;
[0009] S3, collect and analyze data access logs, and count file access logs;
[0010] S4, analyzing access features in the file access log, and deciding a preset migration strategy for the file data based on the access features and the grading strategy, wherein the access features include at least one of access popularity, access time, number of accesses, number of read and write operations per second, and amount of read and write data per unit time, and the preset migration strategy includes a list of files to be migrated and corresponding target storage devices;
[0011] S5. According to the preset migration strategy and the preset migration time window, the migration task is started to migrate the file data according to the preset migration strategy, and the load status of the storage device is monitored during the migration process. The migration rate of the file data is dynamically adjusted according to the load status to complete the hierarchical storage of the file data. The migration rate of the file data is adjusted by increasing or decreasing the number of threads for concurrent file data migration or increasing or decreasing the number of file data migration read and write requests per unit time.
[0012] Preferably, the tiered strategy includes two levels of storage devices, which are respectively referred to as first-level storage devices and second-level storage devices, as follows:
[0013] When the files are divided according to storage performance, a first access threshold is preset for the files. The first-level storage device uses NVMe SSD as the storage medium to store files greater than or equal to the first access threshold. The second-level storage device uses HDD as the storage medium to store files less than the first access threshold.
[0014] When the files are divided according to the access heat, a first heat threshold is preset for the file. If the access heat of the file is greater than or equal to the first heat threshold, the file is stored in the first-level storage device. If the access heat of the file is less than the first heat threshold, the file is stored in the second-level storage device. If the storage location of the file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration.
[0015] When the files are divided according to the number of read and write operations per second and the bandwidth constraint, a first OPS threshold is preset for the small files. If the number of read and write operations per second of the small files is greater than or equal to the first OPS threshold, the files are stored in the first-level storage device. If the number of read and write operations per second of the small files is less than the first OPS threshold, the files are stored in the second-level storage device. If the storage location of the small files is incorrect, it indicates that the files need to be migrated and are added to the list of files to be migrated and wait for migration. A first bandwidth threshold is preset for the large files. If the amount of read and write data of the large files per unit time is greater than or equal to the first bandwidth threshold, the files are stored in the first-level storage device. If the amount of read and write data of the large files per unit time is less than the first bandwidth threshold, the large files are stored in the second-level storage device. If the storage location of the large files is incorrect, it indicates that the files need to be migrated and are added to the list of files to be migrated and wait for migration. Small files are files less than the first classification threshold, and large files are files greater than or equal to the first classification threshold.
[0016] When the files are divided according to the interval from the last access time, a first expiration archiving time threshold is preset. If the file is accessed within the first expiration archiving time threshold, it is stored in the first-level storage device. If the file is not accessed within the first expiration archiving time threshold, it is stored in the second-level storage device. If the storage location of the file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration.
[0017] Or the tiered strategy includes three levels of storage devices, which are respectively recorded as first-level storage devices, second-level storage devices, and third-level storage devices, as follows:
[0018] When the files are divided according to storage performance, a second access threshold and a third access threshold are preset for the files, and the third access threshold is greater than the second access threshold. The first-level storage device uses NVMe SSD as a storage medium for storing files greater than or equal to the third access threshold. The second-level storage device uses SAS SSD as a storage medium for storing files less than the third access threshold but greater than or equal to the second access threshold. The third-level storage device uses HDD as a storage medium for storing files less than the second access threshold.
[0019] When dividing according to access heat, a second heat threshold and a third heat threshold are preset for the file, and the third heat threshold is greater than the second heat threshold. If the access heat of the file is greater than or equal to the third heat threshold, it is stored in the first-level storage device. If the access heat of the file is less than the third heat threshold but greater than or equal to the second heat threshold, it is stored in the second-level storage device. If it is less than the second heat threshold, the file is stored in the third-level storage device. If the storage location of the file is incorrect, it indicates that the file needs to be migrated, and it is added to the list of files to be migrated and wait for migration;
[0020] When divided according to the number of read and write operations per second and bandwidth constraints, a second OPS threshold and a third OPS threshold are preset for small files, and the third OPS threshold is greater than the second OPS threshold. If the number of read and write operations per second of the small file is greater than or equal to the third OPS threshold, it is stored in the first-level storage device. If the number of read and write operations per second of the small file is less than the third OPS threshold but greater than or equal to the second OPS threshold, it is stored in the second-level storage device. If the number of read and write operations per second of the small file is less than the second OPS threshold, it is stored in the third-level storage device. If the storage location of the small file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration. A second OPS threshold is preset for large files. bandwidth threshold and the third bandwidth threshold, and the third bandwidth threshold is greater than the second bandwidth threshold. If the amount of read and written data of the large file in a unit time is greater than or equal to the third bandwidth threshold, it is stored in the first-level storage device. If the amount of read and written data of the large file in a unit time is less than the third bandwidth threshold but greater than or equal to the second bandwidth threshold, it is stored in the second-level storage device. If the amount of read and written data of the large file in a unit time is less than the second bandwidth threshold, the large file is stored in the third-level storage device. If the storage location of the large file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and wait for migration. Small files are files less than the first classification threshold, and large files are files greater than or equal to the first classification threshold.
[0021] When divided according to the interval from the last access time, a second expiration archiving time threshold and a third expiration archiving time threshold are preset, and the third expiration archiving time threshold is greater than the second expiration archiving time threshold. If the file is accessed within the second expiration archiving time threshold, it is stored in the first-level storage device. If the file is not accessed within the second expiration archiving time threshold but is accessed within the third expiration archiving time threshold, the file is stored in the second-level storage device. If the file is not accessed within the third expiration archiving time threshold, the file is stored in the third-level storage device. If the storage location of the file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration.
[0022] Preferably, the file data hierarchical storage method further includes:
[0023] When file data in a high-level storage device is migrated to a low-level storage device, the low-level storage device uses the same file naming method and storage path. The high-level storage device retains the file's metadata information and modifies the file attribute information describing the content data to point to the storage address of the migrated file data in the target storage device. When file data in a low-level storage device is migrated to a high-level storage device, the corresponding file data in the low-level storage device is deleted. The high-level storage device serves as the default storage device and is used to respond to data access requests. The file data is transparent to user services.
[0024] Preferably, the data access log is recorded as follows:
[0025] S201, recording data access logs into a log file, wherein the format of the data access logs recorded in the log file includes access time, accessor, file identifier, access type and operation data volume, wherein the access type includes read operation, write operation, file attribute access operation and non-data read / write access operation, and the operation data volume of the non-data read / write access operation is calculated according to a fixed value;
[0026] S202, asynchronously collecting and aggregating data access logs, parsing them according to the format of the data access logs, and recording the access information into the log database using the file identifier as the unique index. If the log database already has the index of the file, the number of accesses to the file is incremented, the amount of operation data is accumulated, and the last operation time is updated. The access information includes the access time, the visitor, the file identifier, the access type, and the amount of operation data.
[0027] S203, monitor the log file in real time, read it in real time when a new data access log is added to the log file, and return to execute step S202.
[0028] Preferably, data access logs are collected and parsed, and file access logs are counted, as follows:
[0029] Step S301: Collect data access logs in a preset time window, where the preset time window is in days or weeks;
[0030] Step S302: Clean the collected data access logs to remove invalid data, where invalid data refers to log statements that do not contain access types.
[0031] Step S303: According to the cleaned data access log, the access characteristics of the files in the current time unit are counted. The access heat in the access characteristics is calculated as follows:
[0032] Calculated using the weighted summation formula:
[0033] Q i = M i / N i ;
[0034] Or use the time decay formula:
[0035] Q i = M i / N i + Q i-1 * e -λt ;
[0036] Among them, Q i-1represents the access popularity of the i-1th time unit, Q i Indicates the access popularity of the i-th time unit, with an initial value of 0, M i represents the number of visits in the i-th time unit, N i It represents the total amount of operation data in the i-th time unit, i is a positive integer, λ is the attenuation coefficient, t Indicates the time since the file was first accessed or recorded. e represents the base of natural logarithms;
[0037] Or a neural network model is used for calculation, where the neural network model is a Markov model or a multi-layer perceptron;
[0038] Step S304, recording the access characteristics of the file in the current time unit into the log database, and setting the number of accesses and the total amount of operation data to zero;
[0039] Step S305: sort the access heat in the log database.
[0040] Preferably, the file data hierarchical storage method further includes the following steps:
[0041] After dynamically adjusting the file data migration rate according to the load status so that the storage device does not fall into a high-load state, file data migration is performed in a business priority mode or a migration priority mode. The business priority mode refers to migrating file data without affecting business access, and the migration priority mode refers to migrating file data with maximum utilization of storage resources, which include at least one of a processor and a memory.
[0042] Preferably, the load status includes the status of CPU, memory, disk, number of access operations, bandwidth, and latency, and when at least one of them exceeds the corresponding preset threshold, the storage device is considered to be in a high-load state, and the migration rate of file data is dynamically adjusted according to the load status so that the storage device does not fall into a high-load state; the token bucket algorithm is used to increase or decrease the number of file data migration read and write requests per unit time.
[0043] A file data hierarchical storage device includes a configuration module, an agent module, a statistics module, a migration decision module and a migration control module, wherein:
[0044] Configuration module, used to configure the grading strategy;
[0045] The proxy module is used to respond to data access requests and record data access logs;
[0046] Statistics module, used to collect and analyze data access logs and count file access logs;
[0047] A migration decision module is used to analyze the access features in the file access log and decide on a preset migration strategy for the file data based on the access features and the grading strategy. The access features include access popularity, access time, number of accesses, number of read and write operations per second, and the amount of read and write data per unit time. The preset migration strategy includes a list of files to be migrated and the corresponding target storage device.
[0048] The migration control module is used to start the migration task according to the preset migration strategy and the preset migration time window, migrate the file data according to the preset migration strategy, monitor the load status of the storage device during the migration process, and dynamically adjust the migration rate of the file data according to the load status so that the storage device does not fall into a high load state, thereby completing the hierarchical storage of the file data. The migration rate of the file data is adjusted by increasing or decreasing the number of threads for concurrent file data migration or increasing or decreasing the number of file data migration read and write requests per unit time.
[0049] A file data hierarchical storage medium is used to store a computer program. When the computer program is executed by a processor, any of the above-mentioned file data hierarchical storage methods is implemented.
[0050] An electronic device for hierarchical storage of file data comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, any of the above-mentioned methods for hierarchical storage of file data is implemented.
[0051] Compared with the prior art, the present invention has the following beneficial effects:
[0052] When responding to data access requests, the present application records data access logs, collects and parses data access logs, analyzes access characteristics, decides whether to migrate file data and the migration direction based on a preset classification strategy, starts a migration task based on the preset migration strategy and migration time window, completes data migration according to the decision, monitors the load of the storage device during the migration process, and adjusts the performance limit of data migration according to the load, thereby maintaining efficient storage management; according to different classification strategies, frequently accessed files are stored on high-speed storage devices, which improves access speed and reduces access latency, thereby improving user experience, and infrequently accessed files are stored on low-cost, high-capacity storage devices, thereby avoiding wasting expensive storage resources and improving overall system utilization efficiency; and can manage and access a variety of different data storage systems, and is independent of the internal functions of the data storage, thereby improving the scalability of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 It is a flow chart of the file data hierarchical storage method of the present invention;
[0054] Figure 2 This is a flow chart of recording data access logs in step S2 of the present invention;
[0055] Figure 3 is a flow chart of step S3 of the present invention;
[0056] Figure 4 It is a schematic diagram of the structure of the file data hierarchical storage device of the present invention. DETAILED DESCRIPTION
[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0058] It should be noted that when a component is referred to as being "connected" to another component, it may be directly connected to the other component or there may be an intermediate component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art in the technical field of this application. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. Example
[0059] like Figure 1-Figure 3 As shown, a method for hierarchical storage of file data includes the following steps:
[0060] S1. Configure a tiering strategy, where the tiering strategy includes at least two levels of storage devices, and the higher the level of the storage device, the better the performance.
[0061] In one embodiment, the tiering strategy includes two levels of storage devices, which are respectively referred to as first-level storage devices and second-level storage devices, as follows:
[0062] When the files are divided according to storage performance, a first access threshold is preset for the files. The first-level storage device uses NVMe SSD as the storage medium to store files greater than or equal to the first access threshold. The second-level storage device uses HDD as the storage medium to store files less than the first access threshold.
[0063] When the files are divided according to the access heat, a first heat threshold is preset for the file. If the access heat of the file is greater than or equal to the first heat threshold, the file is stored in the first-level storage device. If the access heat of the file is less than the first heat threshold, the file is stored in the second-level storage device. If the storage location of the file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration.
[0064] When the files are divided according to the number of read and write operations per second and the bandwidth constraint, a first OPS threshold is preset for the small files. If the number of read and write operations per second of the small files is greater than or equal to the first OPS threshold, the files are stored in the first-level storage device. If the number of read and write operations per second of the small files is less than the first OPS threshold, the files are stored in the second-level storage device. If the storage location of the small files is incorrect, it indicates that the files need to be migrated and are added to the list of files to be migrated and wait for migration. A first bandwidth threshold is preset for the large files. If the amount of read and write data of the large files per unit time is greater than or equal to the first bandwidth threshold, the files are stored in the first-level storage device. If the amount of read and write data of the large files per unit time is less than the first bandwidth threshold, the large files are stored in the second-level storage device. If the storage location of the large files is incorrect, it indicates that the files need to be migrated and are added to the list of files to be migrated and wait for migration. Small files are files less than the first classification threshold, and large files are files greater than or equal to the first classification threshold.
[0065] When the files are divided according to the interval from the last access time, a first expiration archiving time threshold is preset. If the file is accessed within the first expiration archiving time threshold, it is stored in the first-level storage device. If the file is not accessed within the first expiration archiving time threshold, it is stored in the second-level storage device. If the storage location of the file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration.
[0066] Or the tiered strategy includes three levels of storage devices, which are respectively recorded as first-level storage devices, second-level storage devices, and third-level storage devices, as follows:
[0067] When the files are divided according to storage performance, a second access threshold and a third access threshold are preset for the files, and the third access threshold is greater than the second access threshold. The first-level storage device uses NVMe SSD as a storage medium for storing files greater than or equal to the third access threshold. The second-level storage device uses SAS SSD as a storage medium for storing files less than the third access threshold but greater than or equal to the second access threshold. The third-level storage device uses HDD as a storage medium for storing files less than the second access threshold.
[0068] When dividing according to access heat, a second heat threshold and a third heat threshold are preset for the file, and the third heat threshold is greater than the second heat threshold. If the access heat of the file is greater than or equal to the third heat threshold, it is stored in the first-level storage device. If the access heat of the file is less than the third heat threshold but greater than or equal to the second heat threshold, it is stored in the second-level storage device. If it is less than the second heat threshold, the file is stored in the third-level storage device. If the storage location of the file is incorrect, it indicates that the file needs to be migrated, and it is added to the list of files to be migrated and wait for migration;
[0069] When divided according to the number of read and write operations per second and bandwidth constraints, a second OPS threshold and a third OPS threshold are preset for small files, and the third OPS threshold is greater than the second OPS threshold. If the number of read and write operations per second of the small file is greater than or equal to the third OPS threshold, it is stored in the first-level storage device. If the number of read and write operations per second of the small file is less than the third OPS threshold but greater than or equal to the second OPS threshold, it is stored in the second-level storage device. If the number of read and write operations per second of the small file is less than the second OPS threshold, it is stored in the third-level storage device. If the storage location of the small file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration. A second OPS threshold is preset for large files. bandwidth threshold and the third bandwidth threshold, and the third bandwidth threshold is greater than the second bandwidth threshold. If the amount of read and written data of the large file in a unit time is greater than or equal to the third bandwidth threshold, it is stored in the first-level storage device. If the amount of read and written data of the large file in a unit time is less than the third bandwidth threshold but greater than or equal to the second bandwidth threshold, it is stored in the second-level storage device. If the amount of read and written data of the large file in a unit time is less than the second bandwidth threshold, the large file is stored in the third-level storage device. If the storage location of the large file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and wait for migration. Small files are files less than the first classification threshold, and large files are files greater than or equal to the first classification threshold.
[0070] When divided according to the interval from the last access time, a second expiration archiving time threshold and a third expiration archiving time threshold are preset, and the third expiration archiving time threshold is greater than the second expiration archiving time threshold. If the file is accessed within the second expiration archiving time threshold, it is stored in the first-level storage device. If the file is not accessed within the second expiration archiving time threshold but is accessed within the third expiration archiving time threshold, the file is stored in the second-level storage device. If the file is not accessed within the third expiration archiving time threshold, the file is stored in the third-level storage device. If the storage location of the file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration.
[0071] It is easy to understand that the level of storage devices can also be adjusted according to actual needs. It is preferred to use the first-level storage device as the highest level, that is, as the storage device with the best performance. For example, the first-level storage device has high performance characteristics and uses NVMeSSD as the storage medium to store file data with high access frequency. The second-level storage device has large capacity and low performance characteristics and uses HDD as the storage medium to store file data with low access frequency. The performance of the storage devices decreases in sequence, and when divided according to storage performance, other storage media in the prior art, such as SATA SSD, RAID, etc., can also be selected. The size of each threshold can be adjusted according to actual needs. OPS represents the number of read and write operations per second. By setting the number of read and write operations per second (OPS) and bandwidth constraint levels, it is possible to avoid the problem of excessive consumption of storage resources caused by the hierarchical migration of file data, which affects the service access performance.
[0072] S2. Respond to data access requests and record data access logs.
[0073] In one embodiment, data access logs are recorded as follows:
[0074] S201, recording data access logs into a log file, wherein the format of the data access logs recorded in the log file includes access time, accessor, file identifier, access type and operation data volume, wherein the access type includes read operation, write operation, file attribute access operation and non-data read / write access operation, and the operation data volume of the non-data read / write access operation is calculated according to a fixed value;
[0075] S202, asynchronously collecting and aggregating data access logs, parsing them according to the format of the data access logs, and recording the access information into the log database using the file identifier as the unique index. If the log database already has the index of the file, the number of accesses to the file is incremented, the amount of operation data is accumulated, and the last operation time is updated. The access information includes the access time, the visitor, the file identifier, the access type, and the amount of operation data.
[0076] S203, monitor the log file in real time, read it in real time when a new data access log is added to the log file, and return to execute step S202.
[0077] A third-party log collection tool can be used to collect and aggregate data access logs. The amount of operation data for non-data read and write access operations is calculated according to a fixed value, such as a fixed value of 4KB.
[0078] S3: Collect and parse data access logs and count file access logs.
[0079] In one embodiment, data access logs are collected and parsed, and file access logs are counted, as follows:
[0080] Step S301: Collect data access logs in a preset time window, where the preset time window is in days or weeks;
[0081] Step S302: Clean the collected data access logs to remove invalid data, where invalid data refers to log statements that do not contain access types.
[0082] Step S303: According to the cleaned data access log, the access characteristics of the files in the current time unit are counted, and the access heat in the access characteristics is calculated as follows:
[0083] Calculated using the weighted summation formula:
[0084] Q i = M i / N i ;
[0085] Or use the time decay formula:
[0086] Q i = M i / N i + Q i-1 * e -λt ;
[0087] Among them, Q i-1 represents the access popularity of the i-1th time unit, Q i Indicates the access popularity of the i-th time unit, with an initial value of 0, M i represents the number of visits in the i-th time unit, N i It represents the total amount of operation data in the i-th time unit, i is a positive integer, λ is the attenuation coefficient, t Indicates the time since the file was first accessed or recorded. e represents the base of natural logarithms;
[0088] Or a neural network model is used for calculation, where the neural network model is a Markov model or a multi-layer perceptron;
[0089] Step S304, recording the access characteristics of the file in the current time unit into the log database, and setting the number of accesses and the total amount of operation data to zero;
[0090] Step S305: sort the access heat in the log database.
[0091] Among them, the contribution of each factor in the access heat of the file in the current time unit to the access heat value can be adjusted according to actual needs. When using the neural network model for calculation, various access characteristics of the file (such as the number of accesses, access time, etc.) can be used as input to output the predicted value of the access heat of the file. The access heat can also be obtained by other methods in the prior art, such as by calculating the support vector machine algorithm. The access heat in the log database is sorted to facilitate the division of access heat to achieve fast hierarchical storage of files.
[0092] S4, analyzing the access features in the file access log, and deciding the preset migration strategy of the file data based on the access features and the hierarchical strategy, wherein the access features include at least one of access popularity, access time, number of accesses, number of read and write operations per second, and amount of read and write data per unit time, and the preset migration strategy includes a list of files to be migrated and the corresponding target storage device. According to the statistical access features, the preset migration strategy of the file data is decided by the hierarchical strategy configured in advance, that is, whether the file is to be migrated and the migration direction are determined and wait for migration.
[0093] S5. According to the preset migration strategy and the preset migration time window, the migration task is started to migrate the file data according to the preset migration strategy, and the load status of the storage device is monitored during the migration process. The migration rate of the file data is dynamically adjusted according to the load status to complete the hierarchical storage of the file data. The migration rate of the file data is adjusted by increasing or decreasing the number of threads for concurrent file data migration or increasing or decreasing the number of file data migration read and write requests per unit time.
[0094] In one embodiment, the load status includes the status of CPU, memory, disk, access operation number, bandwidth, and latency, and when at least one of them exceeds the corresponding preset threshold, the storage device is considered to be in a high-load state, and the migration rate of file data is dynamically adjusted according to the load status so that the storage device does not fall into a high-load state; the token bucket algorithm is used to increase or decrease the number of file data migration read and write requests per unit time.
[0095] Specifically, if the CPU usage exceeds a certain threshold (generally 75%), the storage device is considered to be in a high-load state; or if the memory usage exceeds a certain threshold (generally recommended to be 90%), and the disk utilization exceeds a certain threshold (generally recommended to be 90%), the storage device is considered to be in a high-load state. Similarly, the load influencing factors and thresholds can be adjusted according to actual needs. For example, a token bucket with a fixed capacity is maintained through the token bucket algorithm. A certain number of tokens are put into the token bucket every second. Assuming it is 2000, the number of requests passing through per unit time is limited to 2000. When a request arrives, if there is an idle token in the token bucket, the request is allowed to pass and consumes a token; if there is no token in the token bucket or the token is insufficient, the request waits.
[0096] Among them, according to the preset classification strategy, it is decided whether the file data is migrated and the migration direction. After the file data to be migrated is put into the list of files to be migrated, the migration task is started according to the preset migration time window to migrate the file data in the list of files to be migrated to the corresponding target storage device, and the load status of the storage device is monitored during the migration process. The performance limit of data migration is adjusted according to the load level, which is conducive to improving the overall responsiveness of the system. The preset migration time window can be adjusted according to actual needs. The migration rate of file data is adjusted by increasing or decreasing the number of threads for concurrent file data migration or the number of file data migration read and write requests per unit time. It is recommended that the default number of threads is 20. Specifically, by increasing or decreasing the number of threads for concurrent file data migration, the width of the file data migration channel is constrained, thereby adjusting the level of storage device resource consumption; or by increasing or decreasing the number of migration read and write requests per unit time, the frequency of sending file data migration read and write requests is constrained, thereby adjusting the level of storage device resource consumption.
[0097] In one embodiment, the file data hierarchical storage method further includes:
[0098] When file data in a high-level storage device is migrated to a low-level storage device, the low-level storage device uses the same file naming method and storage path. The high-level storage device retains the file's metadata information and modifies the file attribute information describing the content data to point to the storage address of the migrated file data in the target storage device. When file data in a low-level storage device is migrated to a high-level storage device, the corresponding file data in the low-level storage device is deleted. The high-level storage device serves as the default storage device and is used to respond to data access requests. The file data is transparent to user services.
[0099] Among them, high-level storage devices and low-level storage devices are relative, and are divided according to different levels. For example, when it is a three-level storage device, the first-level storage device is a high-level storage device of the second-level storage device, and the second-level storage device is a high-level storage device of the third-level storage device.
[0100] In one embodiment, the file data hierarchical storage method further includes the following steps:
[0101] After dynamically adjusting the file data migration rate according to the load status so that the storage device does not fall into a high-load state, file data migration is performed in a business priority mode or a migration priority mode. The business priority mode refers to migrating file data without affecting business access, and the migration priority mode refers to migrating file data with maximum utilization of storage resources, which include at least one of a processor and a memory.
[0102] It should be understood that although Figure 1-Figure 3The steps in the flowchart are shown in sequence according to the arrows, but these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps. Example
[0103] like Figure 4 As shown, a file data hierarchical storage device includes a configuration module, an agent module, a statistics module, a migration decision module and a migration control module, wherein:
[0104] Configuration module, used to configure the grading strategy;
[0105] The proxy module is used to respond to data access requests and record data access logs;
[0106] Statistics module, used to collect and analyze data access logs and count file access logs;
[0107] A migration decision module is used to analyze the access features in the file access log and decide on a preset migration strategy for the file data based on the access features and the grading strategy. The access features include access popularity, access time, number of accesses, number of read and write operations per second, and the amount of read and write data per unit time. The preset migration strategy includes a list of files to be migrated and the corresponding target storage device.
[0108] The migration control module is used to start the migration task according to the preset migration strategy and the preset migration time window, migrate the file data according to the preset migration strategy, monitor the load status of the storage device during the migration process, and dynamically adjust the migration rate of the file data according to the load status so that the storage device does not fall into a high load state, thereby completing the hierarchical storage of the file data. The migration rate of the file data is adjusted by increasing or decreasing the number of threads for concurrent file data migration or increasing or decreasing the number of file data migration read and write requests per unit time.
[0109] The file data hierarchical storage device includes a configuration module 401, an agent module 402, a statistics module 403, a migration decision module 404 and a migration control module 405. For the specific definition of the file data hierarchical storage device, please refer to the definition of the file data hierarchical storage method in Example 1, which will not be repeated here. Each module in the above-mentioned file data hierarchical storage device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. Example
[0110] A file data hierarchical storage medium is used to store a computer program. When the computer program is executed by a processor, the file data hierarchical storage method described in any one of the above-mentioned embodiments 1 is implemented.
[0111] The processor may be an integrated circuit chip with data processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor implements the file data hierarchical storage method in Embodiment 1 of the present invention by running a computer program. Example
[0112] An electronic device for hierarchical storage of file data comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for hierarchical storage of file data described in any one of the above-mentioned embodiments 1 is implemented.
[0113] The memory and the processor are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory stores a computer program that can be run on the processor, and the processor implements the file data hierarchical storage method in Embodiment 1 of the present invention by running the computer program stored in the memory.
[0114] The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory is used to store a computer program, and the processor executes the computer program after receiving an execution instruction.
[0115] The processor may be an integrated circuit chip with data processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0116] In summary, the present application has the following advantages: 1) Performance optimization: High-frequency access files are stored on high-speed storage media to improve access speed and accelerate system response time; low-frequency access files use low-cost, high-capacity storage media to avoid wasting expensive storage resources and improve overall system performance. 2) Improved resource utilization: According to the file classification strategy, storage resources are intelligently allocated so that high-frequency files and low-frequency files are reasonably allocated storage space, improving the utilization efficiency of storage resources. And the use of storage media of different costs can reduce the overall storage cost while meeting performance requirements. 3) Intelligent data management: Real-time monitoring of file access patterns (such as high-frequency access, low-frequency access), dynamic adjustment of file storage levels, so that the storage system can adapt to changes in file access characteristics and maintain efficient storage management. 4) Improved user experience: The storage of high-frequency files on high-speed media improves the reading and writing speed of files and improves user experience, especially in application scenarios with high real-time requirements; through intelligent scheduling and storage optimization, the delay in file access can be reduced and the overall responsiveness of the system can be improved. 5) System scalability: It is suitable for file storage systems of various sizes, can flexibly respond to the growth of storage needs, and improve the scalability of the system.
[0117] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0118] The above-described embodiments only express the more specific and detailed embodiments described in this application, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of this application, which all belong to the protection scope of this application. Therefore, the protection scope of the patent application shall be based on the attached claims.
Claims
1. A method for hierarchical storage of file data, characterized in that: The file data hierarchical storage method comprises the following steps: S1. Configure a tiering strategy, wherein the tiering strategy includes at least two levels of storage devices, and the higher the level of the storage device, the better the performance; S2. Respond to data access requests and record data access logs; S3, collect and analyze data access logs, and count file access logs; S4, analyzing access features in the file access log, and deciding a preset migration strategy for the file data based on the access features and the grading strategy, wherein the access features include at least one of access popularity, access time, number of accesses, number of read and write operations per second, and amount of read and write data per unit time, and the preset migration strategy includes a list of files to be migrated and corresponding target storage devices; S5. According to the preset migration strategy and the preset migration time window, the migration task is started to migrate the file data according to the preset migration strategy, and the load status of the storage device is monitored during the migration process. The migration rate of the file data is dynamically adjusted according to the load status to complete the hierarchical storage of the file data. The migration rate of the file data is adjusted by increasing or decreasing the number of threads for concurrent migration of the file data or increasing or decreasing the number of migration read and write requests of the file data per unit time; When the file data in the high-level storage device is migrated to the low-level storage device, the low-level storage device adopts the same file naming method and storage path, and the high-level storage device retains the metadata information of the file and modifies the content data described in the file attribute information to point to the storage address of the migrated file data in the target storage device; When the file data in the low-level storage device is migrated to the high-level storage device, the corresponding file data in the low-level storage device is deleted, and the high-level storage device is used as the default storage device to respond to data access requests. The file data is transparent to user services. The data access log is recorded as follows: S201, recording a data access log into a log file, wherein the format of the data access log recorded in the log file includes access time, accessor, file identifier, access type and operation data volume, wherein the access type includes a read operation, a write operation, a file attribute access operation, and a non-data read-write access operation, and the operation data volume of the non-data read-write access operation is calculated according to a fixed value; S202, asynchronously collecting and aggregating data access logs, parsing them according to the format of the data access logs, and recording the access information into the log database using the file identifier as a unique index. If the log database already has an index of the file, the number of accesses to the file is incremented, the amount of operation data is accumulated, and the last operation time is updated. The access information includes the access time, the visitor, the file identifier, the access type, and the amount of operation data. S203, monitor the log file in real time, read it in real time when a new data access log is added to the log file, and return to execute step S202; The data access logs are collected and analyzed, and the file access logs are counted, as follows: Step S301: Collect data access logs in a preset time window, where the preset time window is in days or weeks; Step S302: Clean the collected data access logs to remove invalid data, where the invalid data is log statements that do not contain access types; Step S303: According to the cleaned data access log, the access characteristics of the files in the current time unit are counted. The access heat in the access characteristics is calculated as follows: Calculated using the time decay formula: Q i = M i / N i + Q i-1 * e -λt ; Among them, Q i-1 represents the access popularity of the i-1th time unit, Q i Indicates the access popularity of the i-th time unit, with an initial value of 0, M i Indicates the number of visits in the i-th time unit, N i It represents the total amount of operation data in the i-th time unit, i is a positive integer, λ is the attenuation coefficient, t Indicates the time since the file was first accessed or recorded. e represents the base of natural logarithms; Step S304, recording the access characteristics of the file in the current time unit into the log database, and setting the number of accesses and the total amount of operation data to zero; Step S305: sort the access heat in the log database.
2. The file data hierarchical storage method according to claim 1, characterized in that: The tiered strategy includes two levels of storage devices, which are respectively referred to as first-level storage devices and second-level storage devices, as follows: When divided according to storage performance, a first access threshold is preset for the file, the first-level storage device uses NVMeSSD as the storage medium for storing files greater than or equal to the first access threshold, and the second-level storage device uses HDD as the storage medium for storing files less than the first access threshold; When the files are divided according to the access heat, a first heat threshold is preset for the file. If the access heat of the file is greater than or equal to the first heat threshold, the file is stored in the first-level storage device. If the access heat of the file is less than the first heat threshold, the file is stored in the second-level storage device. If the storage location of the file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration. When the files are divided according to the number of read and write operations per second and the bandwidth constraint, a first OPS threshold is preset for the small file. If the number of read and write operations per second of the small file is greater than or equal to the first OPS threshold, the small file is stored in the first-level storage device. If the number of read and write operations per second of the small file is less than the first OPS threshold, the small file is stored in the second-level storage device. If the storage location of the small file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration. A first bandwidth threshold is preset for the large file. If the amount of read and write data of the large file per unit time is greater than or equal to the first bandwidth threshold, the large file is stored in the first-level storage device. If the amount of read and write data of the large file per unit time is less than the first bandwidth threshold, the large file is stored in the second-level storage device. If the storage location of the large file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration. The small file is a file less than the first classification threshold, and the large file is a file greater than or equal to the first classification threshold. When the files are divided according to the interval from the last access time, a first expiration archiving time threshold is preset. If the file is accessed within the first expiration archiving time threshold, it is stored in the first-level storage device. If the file is not accessed within the first expiration archiving time threshold, it is stored in the second-level storage device. If the storage location of the file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration. Or the grading strategy includes three levels of storage devices, which are respectively recorded as first-level storage devices, second-level storage devices and third-level storage devices, as follows: When the files are divided according to storage performance, a second access threshold and a third access threshold are preset for the files, and the third access threshold is greater than the second access threshold. The first-level storage device uses NVMe SSD as a storage medium for storing files greater than or equal to the third access threshold. The second-level storage device uses SAS SSD as a storage medium for storing files less than the third access threshold but greater than or equal to the second access threshold. The third-level storage device uses HDD as a storage medium for storing files less than the second access threshold. When dividing according to access heat, a second heat threshold and a third heat threshold are preset for the file, and the third heat threshold is greater than the second heat threshold. If the access heat of the file is greater than or equal to the third heat threshold, it is stored in the first-level storage device. If the access heat of the file is less than the third heat threshold but greater than or equal to the second heat threshold, it is stored in the second-level storage device. If it is less than the second heat threshold, the file is stored in the third-level storage device. If the storage location of the file is incorrect, it indicates that the file needs to be migrated, and it is added to the list of files to be migrated and wait for migration; When divided according to the number of read and write operations per second and bandwidth constraints, a second OPS threshold and a third OPS threshold are preset for small files, and the third OPS threshold is greater than the second OPS threshold. If the number of read and write operations per second of the small file is greater than or equal to the third OPS threshold, it is stored in the first-level storage device. If the number of read and write operations per second of the small file is less than the third OPS threshold but greater than or equal to the second OPS threshold, it is stored in the second-level storage device. If the number of read and write operations per second of the small file is less than the second OPS threshold, it is stored in the third-level storage device. If the storage location of the small file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration. Preset the second bandwidth for large files The first-level storage device stores the large file in the first-level storage device. The first-level storage device stores the large file in the first-level storage device. The first-level storage device stores the large file in the second-level storage device. The first-level storage device stores the large file in the first-level storage device. The first-level storage device stores the large file in the second-level storage device. The first-level storage device stores the large file in the first-level storage device. The first-level storage device stores the large file in the second-level storage device. The first-level storage device stores the large file in the third ... When divided according to the interval from the last access time, a second expiration archiving time threshold and a third expiration archiving time threshold are preset, and the third expiration archiving time threshold is greater than the second expiration archiving time threshold. If the file is accessed within the second expiration archiving time threshold, it is stored in the first-level storage device. If the file is not accessed within the second expiration archiving time threshold but is accessed within the third expiration archiving time threshold, the file is stored in the second-level storage device. If the file is not accessed within the third expiration archiving time threshold, the file is stored in the third-level storage device. If the storage location of the file is incorrect, it indicates that the file needs to be migrated and is added to the list of files to be migrated and waits for migration.
3. The file data hierarchical storage method according to claim 1, characterized in that: The file data hierarchical storage method also includes the following steps: After dynamically adjusting the migration rate of file data according to the load status so that the storage device does not fall into a high-load state, file data migration is performed in a business priority mode or a migration priority mode. The business priority mode refers to migrating file data without affecting business access, and the migration priority mode refers to migrating file data with maximum utilization of storage resources, and the storage resources include at least one of a processor and a memory.
4. The file data hierarchical storage method according to claim 3, characterized in that: The load status includes the status of CPU, memory, disk, access operation number, bandwidth, and latency. When at least one of them exceeds the corresponding preset threshold, the storage device is considered to be in a high-load state. The migration rate of file data is dynamically adjusted according to the load status so that the storage device does not fall into a high-load state. The token bucket algorithm is used to increase or decrease the number of file data migration read and write requests per unit time.
5. A file data hierarchical storage device, characterized in that: The file data hierarchical storage device comprises a configuration module, an agent module, a statistical module, a migration decision module and a migration control module to implement the file data hierarchical storage method according to any one of claims 1 to 4 above, wherein: The configuration module is used to configure the classification strategy; The proxy module is used to respond to data access requests and record data access logs; The statistical module is used to collect and analyze data access logs and count file access logs; The migration decision module is used to analyze the access features in the file access log, and decide the preset migration strategy of the file data based on the grading strategy according to the access features, wherein the access features include access popularity, access time, number of accesses, number of read and write operations per second, and amount of read and write data per unit time, and the preset migration strategy includes a list of files to be migrated and corresponding target storage devices; The migration control module is used to start the migration task according to the preset migration strategy and the preset migration time window, migrate the file data according to the preset migration strategy, monitor the load status of the storage device during the migration process, and dynamically adjust the migration rate of the file data according to the load status so that the storage device does not fall into a high load state and completes the hierarchical storage of the file data. The migration rate of the file data is adjusted by increasing or decreasing the number of threads for concurrent migration of the file data or increasing or decreasing the number of migration read and write requests for the file data per unit time.
6. A file data hierarchical storage medium, characterized in that: The file data hierarchical storage medium is used to store a computer program, and when the computer program is executed by a processor, the file data hierarchical storage method described in any one of claims 1 to 4 is implemented.
7. An electronic device for hierarchical storage of file data, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor runs the computer program, the file data hierarchical storage method described in any one of claims 1 to 4 is implemented.