A hybrid high-performance storage method, system, and medium

Abstract

The application relates to a hybrid high-performance storage method, system and medium, which comprises the following steps: receiving a data reading or writing request of a client; determining a storage position of the requested data through a metadata server; acquiring or storing the data from a block storage cache layer or an object storage layer according to the storage position; dynamically migrating data between the block storage cache layer and the object storage layer based on a predetermined strategy, wherein the block storage cache layer is used for storing hot data, and the object storage layer is used for storing cold data; and automatically adjusting the distribution of data between the block storage cache layer and the object storage layer according to preset data level factors through an intelligent hierarchical management mechanism. The application combines low-cost object storage and high-performance block storage, which is beneficial to solving the contradiction between storage performance and cost in the prior art, improves the intelligence and flexibility of data management, and adapts to the needs of different business scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of storage, in particular to a hybrid high-performance storage method, system and medium. BACKGROUND

[0002] With the rapid development of cloud computing technology, storage systems have become a core component in data management and architecture design. In the fields of big data processing, cloud storage and enterprise-level data management, how to balance storage cost and performance has become a key issue. Currently, block storage and object storage are two main schemes for cloud storage. Block storage is widely used in scenarios that require fast response due to its low latency and high performance, while object storage is the preferred choice for long-term storage due to its strong scalability and low cost.

[0003] However, the existing storage scheme has certain limitations when dealing with peak and valley business demands. When business load fluctuates, the storage system has difficulty in flexibly switching between block storage and object storage, which may lead to an increase in storage cost. In addition, there is a high time and cost overhead in the data migration process. When both storage schemes are needed, there is a lack of intelligent data management and automatic migration mechanism, and users have to manually adjust the storage strategy, increasing the management complexity and not fully optimizing resources. SUMMARY

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a hybrid high-performance storage method, system and medium, which combines low-cost object storage with high-performance block storage, not only to solve the contradiction between storage performance and cost in the prior art, but also to improve the intelligence and flexibility of data management, and to adapt to the needs of different business scenarios.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions.

[0006] In a first aspect, the present application provides a hybrid high-performance storage method, which adopts the following technical solutions:

[0007] receiving a data read or write request of a client;

[0008] determining the storage location of the requested data through a metadata server;

[0009] According to the storage location, the data is obtained or stored from the block storage cache layer or the object storage layer;

[0010] Based on a predetermined strategy, the data is dynamically migrated between the block storage cache layer and the object storage layer, the block storage cache layer is used to store hot data, and the object storage layer is used to store cold data;

[0011] And, through an intelligent hierarchical management mechanism, the distribution of data between the block storage cache layer and the object storage layer is automatically adjusted according to preset data level factors.

[0012] Further, in the above hybrid high-performance storage method, further comprising:

[0013] The storage capacity of the block storage cache layer is expanded or reduced on demand to adapt to fluctuations in business load.

[0014] Only the changed data is written back to the block storage cache layer to reduce redundant transmission during data migration.

[0015] Further, in the above hybrid high-performance storage method, the predetermined strategy includes:

[0016] A dynamically adjustable threshold is set according to the access frequency of the data, and when the data access frequency exceeds the threshold, the data is migrated from the object storage layer to the block storage cache layer.

[0017] When the data access frequency is lower than the threshold, the data is migrated from the block storage cache layer to the object storage layer.

[0018] Further, in the above hybrid high-performance storage method, further comprising:

[0019] Setting a time to live for data stored in the block storage cache layer.

[0020] When the time to live of the data expires, the data is migrated from the block storage cache layer to the object storage layer.

[0021] Further, in the above hybrid high-performance storage method, the intelligent hierarchical management mechanism includes:

[0022] Classify data according to its importance, life cycle, and access pattern.

[0023] Based on the classification results, different storage levels and migration strategies are assigned to different categories of data.

[0024] Further, in the above hybrid high-performance storage method, further comprising:

[0025] Implement an intelligent prefetching mechanism to predict data that will be accessed according to the historical access pattern and data usage trends of the client.

[0026] The predicted data is loaded from the object storage layer to the block storage cache layer in advance.

[0027] Further, in the above hybrid high-performance storage method, the prediction of data that will be accessed according to the historical access pattern and data usage trends of the client includes:

[0028] Collect historical data access records from the client;

[0029] Machine learning algorithms are used to analyze the historical data access records to identify time patterns and correlations in data access.

[0030] Based on the identified time patterns and correlations, predict the data that may be accessed in the future.

[0031] Furthermore, the aforementioned hybrid high-performance storage method also includes:

[0032] Monitor system load and storage resource usage;

[0033] Based on monitoring results, the data distribution ratio between the block storage cache layer and the object storage layer is automatically adjusted.

[0034] Furthermore, the aforementioned hybrid high-performance storage method also includes:

[0035] Batch processing of write requests combines multiple small write operations into a larger write operation;

[0036] Before writing data to the object storage layer, the data is compressed and deduplicated to reduce storage space usage and data transfer volume.

[0037] Secondly, the present invention provides a hybrid high-performance storage system, which adopts the following technical solution:

[0038] The client interface is used to receive data read or write requests.

[0039] Metadata server, used to determine the storage location of requested data;

[0040] A block storage cache layer is used to store hot data;

[0041] The object storage layer is used to store cold data;

[0042] The data migration module is used to dynamically migrate data between the block storage cache layer and the object storage layer based on a predetermined strategy;

[0043] It also includes an intelligent hierarchical management module, which automatically adjusts the distribution of data between the block storage cache layer and the object storage layer based on preset data level factors.

[0044] Furthermore, the aforementioned hybrid high-performance storage system also includes:

[0045] The storage capacity adjustment module is used to expand or reduce the storage capacity of the block storage cache layer as needed to adapt to fluctuations in business load;

[0046] A data write-back optimization module is configured to write back only changed data to the block storage cache layer, thereby reducing redundant transmissions during data migration.

[0047] A readable storage medium, characterized in that the readable storage medium stores computer instructions, the computer instructions are executed by a processor to implement the hybrid high-performance storage method of any one of claims 1-9.

[0048] In a third aspect, the present application provides a readable storage medium, which adopts the following technical solution:

[0049] A readable storage medium, characterized in that the readable storage medium stores computer instructions, the computer instructions are executed by a processor to implement the hybrid high-performance storage method of any one of claims 1-9.

[0050] In summary, compared with the prior art, the present application has at least one of the following beneficial technical effects:

[0051] The present application combines the advantages of block storage and object storage, achieving efficient data management and access. The block storage cache layer provides fast data read-write capability, suitable for processing hot data; the object storage layer provides a large-capacity, low-cost storage solution, suitable for storing cold data. Through intelligent hierarchical management and dynamic data migration, a balance between performance and cost can be achieved to meet the needs of different business scenarios. At the same time, the method also improves the flexibility and scalability of the storage system, better coping with the challenges of data volume growth and access mode changes. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1 is a flow chart of a specific embodiment of a hybrid high-performance storage method of the present application.

[0054] Figure 2 is a flow chart of a specific embodiment of a hybrid high-performance storage method of the present application.

[0055] Figure 3 is a flow chart of another specific embodiment of a hybrid high-performance storage method of the present application.

[0056] Figure 4 is a flow chart of another specific embodiment of a hybrid high-performance storage method of the present application.

[0057] Figure 5 is a structural schematic diagram of one specific embodiment of a hybrid high-performance storage system. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application. In addition, it should be understood that the specific embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0059] It should be noted that the description order of the following embodiments is not used to limit the preferred order of the embodiments of the present application. In addition, the description of each embodiment in the following embodiments has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0060] The execution order of the method steps in the embodiments of the present application can be executed according to the order described in the specific embodiments, or the execution order of each step can be adjusted on the premise of solving the technical problems according to actual needs, which is not listed one by one here.

[0061] With reference to Figure 1 , the embodiments of the present application provide a hybrid high-performance storage method, comprising:

[0062] S1, receiving a data read or write request of a client;

[0063] S2, determining a storage location of the requested data through a metadata server;

[0064] S3, acquiring or storing the data from a block storage cache layer or an object storage layer according to the storage location;

[0065] S4, dynamically migrating data between the block storage cache layer and the object storage layer based on a predetermined strategy, the block storage cache layer being used for storing hot data, and the object storage layer being used for storing cold data;

[0066] S5, automatically adjusting the distribution of data between the block storage cache layer and the object storage layer according to a preset data level factor through an intelligent hierarchical management mechanism.

[0067] In some embodiments, the hybrid high-performance storage method adopts a three-layer architecture, including a client, a service end (cache layer), and an object storage. The method includes receiving a data read or write request of the client, determining the storage location of the requested data through a metadata server, and obtaining or storing data from the block storage cache layer or the object storage layer according to the storage location.

[0068] After receiving the client request, the metadata server queries the metadata information maintained by itself to determine the storage location and related attributes of the target data, such as the data server where the data is located, the cache state, and the like. The metadata server then returns the query result to the client, and the client sends a specific data request to the corresponding data server according to the positioning information.

[0069] Data is divided into data blocks of a fixed size, and all operations are performed in units of data blocks. In the data storage and acquisition process, the block storage cache layer is used to store hot data, and the object storage layer is used to store cold data. Based on a predetermined strategy, data is dynamically migrated between the block storage cache layer and the object storage layer.

[0070] In some embodiments, the predetermined strategy includes setting a dynamically adjustable threshold according to the access frequency of the data. When the data access frequency exceeds the threshold, the data is migrated from the object storage layer to the block storage cache layer; when the data access frequency is lower than the threshold, the data is migrated from the block storage cache layer to the object storage layer. This dynamic migration mechanism ensures that frequently accessed data can be quickly responded to, while reducing storage costs.

[0071] The intelligent hierarchical management mechanism automatically adjusts the distribution of data between the block storage cache layer and the object storage layer according to the preset data level factors. In some embodiments, the intelligent hierarchical management mechanism includes classifying data according to the importance, life cycle, and access mode of the data, and assigning different storage levels and migration strategies to different categories of data based on the classification results.

[0072] The hybrid high-performance storage method balances high performance and low cost by combining the advantages of block storage and object storage. The intelligent hierarchical management and dynamic data migration mechanism ensure efficient use of storage resources while providing flexible data access performance. In addition, the method can adapt to different business load fluctuations, ensuring data access speed while optimizing storage costs, and providing an efficient storage solution for various cloud computing and big data application scenarios.

[0073] Reference Figure 2In some embodiments, the hybrid high-performance storage method includes receiving a data read request from a client. Upon receiving the data read request, the metadata server determines the storage location of the requested data. The metadata server queries the metadata information it maintains to determine the storage location and related attributes of the target data, such as the data server where it is located, the cache status, etc. The metadata server then returns the query result to the client.

[0074] The client sends a specific data request to the corresponding data server according to the positioning information. Upon receiving the file content request from the client, the data server checks whether the file content exists in the local cache. If the file content is cached, the data server directly returns the file content to the client. If the file content is not in the cache, the data server initiates a request to the object storage, reads the target data from the object storage, and updates or creates a copy of the file content in the local cache.

[0075] When the data server obtains the target file, it packages the file content and returns it to the client. After receiving and verifying the returned data, the client performs corresponding operations according to business needs, such as reading the file content for processing, presenting it to the user, or further storing it in other systems.

[0076] Reference Figure 3 In some embodiments, the hybrid high-performance storage method includes receiving a data write request from a client. Upon receiving the data write request, the metadata server checks the permission of data writing and verifies the storage status of the data.

[0077] The metadata server checks whether the existing chunk needs to be replaced according to the request and allocates storage space for the data. In some embodiments, if the current data needs to be updated in chunks, appropriate storage locations are allocated for the data according to space availability. If the storage space of the data chunk is full, the space is re-allocated or the storage scheme is adjusted.

[0078] In some embodiments, the data is divided into multiple data chunks and stored in local nodes. These data chunks are distributed to another adjacent node for backup to prevent data loss due to failure of the local node. This backup process ensures high availability and fault tolerance of the data.

[0079] After the data chunking and backup are completed, the data server updates the distribution information of the file and synchronizes it to the metadata server. The metadata server then maintains the latest storage status to ensure that the data can be quickly located.

[0080] In some embodiments, the metadata server periodically checks in the background data that has been cached but not yet written back to the object store. For these data, processing is done according to the configured policy: data chunks are written back to the object store, ensuring long-term preservation of the data; according to the policy, data chunks that have been successfully written back are deleted or kept as needed to save storage space.

[0081] In some embodiments, data is written back to the object store asynchronously in the background, reducing the impact on performance during the write process. After writing back, whether to keep a copy or delete the data is determined by the storage policy to ensure storage efficiency and cost optimization.

[0082] After the data is successfully written, the data server returns a write completion response to the client. After the client confirms that the data has been successfully written, it continues with the subsequent operation.

[0083] In some embodiments, the hybrid high-performance storage method includes expanding or reducing the storage capacity of the block storage cache layer on demand to adapt to fluctuations in business load. By dynamically adjusting the storage capacity, this method can provide sufficient cache space during high load while releasing excess resources during low load, thereby optimizing resource utilization and reducing costs.

[0084] Specifically, the adjustment of storage capacity can be achieved in various ways. In some embodiments, elastic scaling technology is used to automatically increase or decrease storage nodes according to preset load thresholds. When the load is detected to exceed the upper threshold, new storage nodes are automatically added to expand the capacity; when the load is below the lower threshold, the excess nodes are automatically removed to reduce the capacity.

[0085] In other embodiments, the capacity is adjusted by dynamically allocating and reclaiming storage space. In high load, additional space is allocated from a reserved storage pool to the block storage cache layer; in low load, unused space is returned to the storage pool. This method can achieve rapid capacity adjustment without changing the number of physical nodes.

[0086] To further optimize storage efficiency, the method also includes data write-back optimization. In some embodiments, only the changed data is written back to the block storage cache layer to reduce redundant transmission during data migration. This incremental write-back mechanism significantly reduces data transmission and improves write-back efficiency.

[0087] In specific implementation, a difference comparison algorithm can be used to identify changed data chunks. When data is updated, the new data is compared with the original data, and only the changed part is transmitted. For unchanged data chunks, the original storage state is retained to avoid unnecessary data transmission.

[0088] In some embodiments, data compression and deduplication techniques can also be used in combination to further reduce the amount of data that needs to be written back. By compressing the changed data and identifying and removing duplicate data, the data transmission and storage overhead can be significantly reduced.

[0089] In addition, in some embodiments, a batch write-back strategy can be employed to combine multiple small write operations into larger write operations, reducing the number of I / Os and improving write efficiency. By setting appropriate write-back intervals and trigger conditions, the write-back efficiency can be maximized while ensuring data consistency.

[0090] By combining dynamic adjustment of storage capacity and data write-back optimization, this hybrid high-performance storage method can effectively improve storage resource utilization, reduce operating costs, and adapt to changes in load under different business scenarios while ensuring performance.

[0091] In some embodiments, the predetermined strategy includes setting a dynamically adjustable threshold based on the access frequency of the data. This threshold is used to determine the migration of data between the block storage cache layer and the object storage layer.

[0092] Specifically, when the data access frequency exceeds the set threshold, the data is migrated from the object storage layer to the block storage cache layer; when the data access frequency is below the threshold, the data is migrated from the block storage cache layer to the object storage layer. This dynamic migration mechanism ensures that frequently accessed data can be quickly responded to while reducing storage costs.

[0093] The threshold is set using a dynamic adjustment mechanism to adapt to different data access patterns and storage requirements. In some embodiments, the initial value of the threshold is set based on historical data access statistics and storage capacity. Subsequently, by continuously monitoring the data access pattern and storage layer usage, the threshold is adjusted regularly.

[0094] The threshold adjustment can take into account multiple factors, including but not limited to:

[0095] 1. Determined by data access frequency distribution, specifically including analyzing the overall data access pattern to determine the appropriate threshold range;

[0096] 2. Determined by storage layer capacity utilization, specifically including adjusting the threshold based on the current usage of the block storage cache layer and the object storage layer to balance the load of the two storage layers;

[0097] 3. Determined by performance indicators, specifically including monitoring data access latency and throughput, and adjusting the threshold based on performance goals;

[0098] 4. Determined based on cost factors, including considering the cost of different storage layers to optimize the threshold to achieve a balance between performance and cost.

[0099] In some embodiments, a multi-level threshold strategy is adopted. For example, three threshold levels, high, medium, and low, are set, corresponding to different data migration priorities. Data above the highest threshold is immediately migrated to the block storage cache layer; data below the lowest threshold is migrated to the object storage layer; data in between is determined whether to migrate according to the current storage capacity and load situation.

[0100] During threshold adjustment, to avoid performance overhead caused by frequent migration, a cooling period mechanism can be introduced. Within a certain time after data migration, even if the data access frequency changes, a new migration operation is not triggered temporarily to ensure the stability of the storage layer.

[0101] In addition, in some embodiments, threshold adjustment also considers the life cycle and business importance of data. For data with shorter life cycle or higher business importance, the migration threshold can be appropriately lowered to make it easier to remain in the block storage cache layer, providing faster access speed.

[0102] Through this dynamic threshold adjustment and data migration strategy, efficient use of storage resources is achieved, ensuring fast response of high-frequency access data while optimizing overall storage cost. This method can adapt to changes in data access patterns in different business scenarios, providing flexible and efficient storage solutions.

[0103] In some embodiments, the mixed high-performance storage method further comprises setting a time to live for data stored in the block storage cache layer. Time to live refers to the maximum time data can remain in the block storage cache layer. By setting the time to live, the data in the cache layer can be effectively managed, ensuring efficient use of storage resources.

[0104] The setting of the time to live can be based on multiple factors, including but not limited to data access frequency, importance, data size, and current storage capacity. In some embodiments, a dynamic time to live strategy is adopted, automatically adjusting the length of the time to live based on real-time monitoring of data access patterns and storage usage.

[0105] In specific implementation, a timestamp attribute can be added to each data block to record the time the data enters the block storage cache layer. The timestamp of the data block is checked regularly to calculate the length of time it has been stored in the cache layer and compared with the preset time to live.

[0106] When the time to live of the data expires, the data is migrated from the block storage cache layer to the object storage layer. The migration process includes the following steps:

[0107] 1. Regularly scan the data in the block storage cache layer to identify data blocks that have exceeded the time to live;

[0108] 2. The identified expired data is packaged and prepared for migration. In some embodiments, the data can be compressed or encrypted to optimize transmission efficiency and security;

[0109] 3. The prepared data package is transmitted to the object storage layer. The transmission process is asynchronous to minimize the impact on normal business operations;

[0110] 4. After receiving the data at the object storage layer, integrity verification is performed to ensure lossless transmission of the data;

[0111] 5. After migration is complete, metadata information is updated to record the new location and status of the data;

[0112] 6. The successfully migrated data is deleted from the block storage cache layer to release storage space.

[0113] In some embodiments, to avoid the impact of data migration on performance, a migration window period can be set. Data migration operations are performed during periods of low business activity to minimize disruption to normal business operations.

[0114] In addition, during the data migration process, an incremental migration strategy can also be implemented. For data with a survival time that is about to expire but is still frequently accessed, only a portion of the data can be migrated, and the most recently accessed portion can be retained in the block storage cache layer to maintain access performance.

[0115] By implementing a survival time-based data management and migration strategy, the hybrid high-performance storage method can ensure data access efficiency while optimizing storage resource utilization and achieving automatic hierarchical storage of cold and hot data. This method not only improves storage efficiency but also reduces management complexity, providing the most suitable storage environment for different types of data.

[0116] In some embodiments, the intelligent hierarchical management mechanism classifies data based on its importance, life cycle, and access pattern, and assigns different storage levels and migration strategies to different categories of data based on the classification results.

[0117] The data classification process involves multiple dimensions of evaluation. Data importance can be determined by predefined business rules or machine learning algorithms, taking into account factors such as the degree of impact on business operations, regulatory compliance requirements, etc. Life cycle assessment is based on data creation time, last access time, and expected retention period. Access pattern analysis is completed by monitoring access frequency, access time distribution, and other indicators.

[0118] Based on these classification results, appropriate storage tiers are assigned to data of different categories. For example, data with high importance, short life cycle, and frequent access can be assigned to a high-performance block storage cache tier to ensure fast access and processing. Conversely, data with low importance, long life cycle, and low access frequency can be assigned to a lower-cost object storage tier.

[0119] The formulation of migration policies is also based on data classification results. For data with high importance but gradually decreasing access frequency, a gradual migration policy can be adopted, with some data first migrated to a lower-level storage and then completely migrated as access frequency further decreases. For data with a short life cycle, a more aggressive migration policy can be adopted to remove it from the high-performance storage tier in a timely manner before it expires.

[0120] In some embodiments, migration policies are customized according to time windows and data life cycles. For example, data migration can be performed during business off-peak hours, or triggered at a specific time point after data creation. This flexible policy configuration can adapt to the needs of different business scenarios, optimizing storage resource utilization while ensuring data access performance.

[0121] The intelligent hierarchical management mechanism dynamically adjusts classification results and migration policies by continuously monitoring and analyzing data characteristics. This adaptive mechanism ensures efficient use of storage resources while meeting the storage needs of different types of data. Through fine-grained data management, a balance between storage performance and cost is achieved, providing flexible and efficient storage solutions for different business scenarios.

[0122] In some embodiments, the hybrid high-performance storage method includes implementing an intelligent prefetching mechanism that predicts data to be accessed based on the historical access patterns and data usage trends of clients. This mechanism identifies potential access patterns by analyzing client data access behavior, thereby loading data that may be accessed from the object storage tier to the block storage cache tier in advance.

[0123] The implementation of the intelligent prefetching mechanism involves several steps. First, historical data access records of clients are collected and stored, including data types accessed, access time, access frequency, and other information. These records form the basis of the prediction model dataset.

[0124] Based on the collected data, machine learning algorithms are used to analyze historical data access records to identify temporal patterns and correlations of data access. For example, the algorithm can find that certain data is frequently accessed within a certain time period, or that there is a correlation between the access order of certain data. These patterns and correlations provide important basis for predicting future data access.

[0125] Based on the identified temporal patterns and correlations, the data that is likely to be accessed in the future is predicted. The prediction process takes into account multiple factors, including but not limited to: historical access frequency, recent access time, correlation between data, current time, etc. The prediction result includes a list of data that is likely to be accessed and their priority.

[0126] According to the prediction result, data is preloaded from the object storage layer to the block storage cache layer. The loading process takes into account multiple factors, such as the predicted access probability, data size, available space in the current cache layer, etc. In some embodiments, a hierarchical loading strategy is adopted, which prioritizes the loading of data with higher access probability, while reserving certain cache space for real-time response to unexpected data access requests.

[0127] To optimize the efficiency of pre-fetching, in some embodiments, a dynamic adjustment mechanism is implemented. This mechanism continuously monitors the actual usage of pre-fetched data, and evaluates the accuracy of the prediction. Based on the evaluation result, the parameters of the prediction model are dynamically adjusted to improve the prediction accuracy. For example, if it is found that some pre-fetched data has not been accessed for a long time, the weight of the related prediction rule may be reduced; on the contrary, if the pre-fetched data is frequently accessed, the weight of the corresponding rule is increased.

[0128] In addition, in some embodiments, the intelligent pre-fetching mechanism can also consider the overall load situation of the storage system. During periods of low load, the number of pre-fetched data may be increased; while during periods of high load, the pre-fetching operation may be reduced to avoid affecting normal data access.

[0129] By implementing the intelligent pre-fetching mechanism, the hybrid high-performance storage method can pre-load the data that is likely to be accessed from the object storage layer to the block storage cache layer, thereby reducing data access latency and improving overall storage performance. This prediction method based on historical access patterns and data usage trends enables the storage system to more intelligently manage data and provide clients with faster and more efficient data access experience.

[0130] In some embodiments, the hybrid high-performance storage method includes predicting the data that will be accessed based on the historical access patterns and data usage trends of the client. This method analyzes the data access behavior of the client to identify potential access patterns, thereby pre-loading the data that is likely to be accessed from the object storage layer to the block storage cache layer.

[0131] Specifically, based on the collection of data (including information such as the type of data accessed, access time, access frequency, etc.), a machine learning algorithm is used to analyze historical data access records to identify temporal patterns and correlations of data access. For example, the algorithm identifies that certain data is frequently accessed within a certain time period, or there is a correlation between the access order of certain data. These patterns and correlations provide important basis for predicting future data access.

[0132] Based on the identified temporal patterns and correlations, the method predicts the data that is likely to be accessed in the future. The prediction process takes into account multiple factors, including but not limited to: historical access frequency, recent access time, correlation between data, current time, etc. The prediction result includes a list of data that is likely to be accessed and their priority.

[0133] In some embodiments, the prediction method employs multiple machine learning algorithms, such as time series analysis, association rule mining, clustering analysis, etc. Time series analysis is used to identify periodic patterns of data access, such as daily, weekly, or monthly access regularities. Association rule mining helps discover access correlations between different data, for example, accessing data A usually leads to accessing data B. Clustering analysis is used to group data with similar access patterns for batch prediction and processing.

[0134] The prediction method can also consider the attributes and metadata information of the data. For example, for structured data, consider the attributes such as data type, size, creation time, etc.; for unstructured data, it may consider file type, content keywords, etc. These additional information helps improve the accuracy of prediction. In some embodiments, the prediction method employs an incremental learning strategy. As new access records are continuously generated, the prediction model is updated regularly to adapt to changes in data access patterns. This dynamic adjustment mechanism ensures that the prediction model can reflect the latest data access trends. In some embodiments, the prediction method also considers contextual information, such as current time, user identity, device type, etc. These contextual information helps improve the accuracy of prediction. For example, for different user groups or different types of devices, the prediction method uses different model parameters. In some embodiments, the prediction method employs a multi-model fusion strategy. Multiple prediction models run in parallel, each focusing on different data access patterns or time scales. The final prediction result is obtained by integrating the outputs of multiple models, improving the robustness and accuracy of prediction.

[0135] Further, the prediction method also includes an adaptive threshold mechanism. According to the feedback of prediction accuracy, the confidence threshold of prediction is dynamically adjusted. When the prediction accuracy is high, the threshold is lowered to prefetch more data; when the accuracy decreases, the threshold is raised to reduce unnecessary prefetching.

[0136] Through this prediction method based on historical access patterns and data usage trends, the hybrid high-performance storage method can more intelligently manage data and provide clients with faster and more efficient data access experience.

[0137] For example, with reference to Figure 4Based on the prediction method described in the above embodiments, the metadata server predicts the data that the client is likely to need to access through intelligent analysis or preset rules and decides whether to read the corresponding data from the storage system. When the data server 1 receives a request, it first tries to obtain the data from its own cache. If the data is not hit, it requests the corresponding data block from the object storage and stores it in the local cache, so that subsequent access can be directly obtained from the cache, improving access efficiency. Subsequently, the metadata server updates the file layout information to ensure that subsequent data access can quickly find the location of the data, reducing query time and unnecessary storage calls.

[0138] After completing data acquisition and cache updating, the metadata server will again perform intelligent / rule prediction based on access patterns and historical data to analyze which data is likely to become hot data. These data predicted to have high access frequency are marked as hot data blocks and transferred from the data server 1 to the data server 2, realizing the distribution of hot data. This strategy helps to balance the load and avoid a single data server becoming a performance bottleneck, while also improving data availability. When the data is successfully stored in the data server 2, the metadata server will update the file layout again to record the new storage location of the data to optimize the access path for subsequent data requests.

[0139] In some embodiments, the hybrid high-performance storage method includes monitoring load and storage resource usage and automatically adjusting the data distribution ratio between the block storage cache layer and the object storage layer based on the monitoring results.

[0140] Specifically, the monitoring process involves collecting multiple indicators, including but not limited to: CPU usage, memory usage, storage I / O throughput, network bandwidth utilization, etc. For storage resources, the monitoring content includes the capacity usage of the block storage cache layer and the object storage layer, read / write operation frequency, data access pattern, etc. These indicators are obtained through a combination of regular sampling and real-time monitoring, forming a comprehensive performance and resource usage profile.

[0141] Based on the monitoring data, the automatic adjustment mechanism assesses whether the current data distribution is reasonable. In some embodiments, a preset threshold is used to trigger adjustment. For example, when the usage rate of the block storage cache layer exceeds 80%, or the read operation frequency of the object storage layer significantly increases, the data redistribution process is started.

[0142] The data redistribution process involves multiple steps. In some embodiments, first, identify the data with low access frequency in the block storage cache layer and migrate these data to the object storage layer to release high-performance storage space. At the same time, identify the data with rising access frequency from the object storage layer and migrate it to the block storage cache layer to improve access speed.

[0143] During the adjustment process, multiple factors are considered to determine the optimal data distribution ratio. These factors include but are not limited to: current load level, expected load change trend, cost-benefit ratio of different storage layers, changes in data access patterns, etc. By comprehensively analyzing these factors, the optimal data distribution ratio is dynamically calculated.

[0144] In some embodiments, the automatic adjustment mechanism adopts a gradual strategy to avoid large-scale data migration affecting performance. The adjustment process is carried out in multiple small batches, and the system performance is evaluated after each batch is completed to decide whether to continue the adjustment of the next batch.

[0145] Through this dynamic monitoring and automatic adjustment mechanism, the hybrid high-performance storage method can flexibly adjust the distribution of data between different storage layers according to the actual load and resource usage. This method not only optimizes the utilization of storage resources, but also adapts to load changes in different business scenarios, providing continuous high-performance storage services.

[0146] In some embodiments, the hybrid high-performance storage method includes batch processing of write requests, combining multiple small write operations into larger write operations. Through the batch processing mechanism, the overhead required for processing each small write operation individually is reduced, improving overall write efficiency. The batch processing process involves temporarily storing received write requests in a memory buffer, and when the accumulated data volume reaches a preset threshold or a predetermined time interval is reached, triggering batch write operations. In the batch write process, multiple small write requests are combined into one large write operation, reducing the number of interactions with the storage layer, thereby reducing latency and improving throughput.

[0147] In some embodiments, the hybrid high-performance storage method compresses and deduplicates data before writing it to the object storage layer to reduce storage space occupancy and data transmission volume. The compression process uses suitable compression algorithms such as LZ4 or Snappy to ensure compression efficiency while minimizing CPU overhead. Deduplication processing identifies and eliminates duplicate data blocks by calculating the hash value of the data block, storing only unique data blocks. Through compression and deduplication, not only is the storage cost reduced, but also the data transmission time is reduced, improving overall storage efficiency.

[0148] In some implementations, the hybrid high-performance storage method employs an asynchronous background write mechanism to write data to the object storage layer. Asynchronous writing allows the main thread to return immediately after initiating a write request, while the actual write operation is handled by a background thread. This approach reduces the impact of write operations on the main thread's performance and improves overall response speed. During background writing, a queue mechanism is used to manage the data to be written, ensuring that data is written in order, and metadata information is updated after the write is complete. The asynchronous write mechanism also includes failure retries and data consistency checks to ensure data reliability and integrity.

[0149] The hybrid high-performance storage method described in this invention achieves a high-performance, low-cost, and highly scalable storage solution through a hybrid storage architecture, intelligent cache management, and data tiering optimization. It employs a combination of block storage and object storage, ensuring that frequently accessed data is stored in a high-performance cache layer while infrequently accessed data is stored in low-cost object storage, effectively balancing storage performance and cost. The introduction of a clustered caching mechanism enhances data access concurrency, reduces single-node storage bottlenecks, and improves the overall throughput of the storage system. Furthermore, the solution utilizes intelligent data migration, dynamically adjusting storage locations based on access frequency to optimize storage resource utilization. Simultaneously, it incorporates intelligent prefetching strategies to reduce data access latency and improve response speed. Moreover, this solution supports incremental data migration, reducing data writing and migration costs and improving the efficiency and stability of the storage system. In summary, this invention can effectively improve data access performance while reducing storage and management costs in application scenarios such as cloud storage, big data processing, and high-performance computing, achieving efficient and intelligent data storage management.

[0150] This invention also discloses a hybrid high-performance storage system.

[0151] Reference Figure 5 The hybrid high-performance storage system includes a client interface 1, a metadata server 2, a block storage cache layer 3, an object storage layer 4, a data migration module 5, an intelligent tiered management module 6, a storage capacity adjustment module 7, and a data write-back optimization module 8.

[0152] Client interface 1 is used to receive data read or write requests. Client interface 1 can be an API or a graphical user interface, supporting multiple protocols such as HTTP and FTP to meet the needs of different clients.

[0153] Metadata server 2 is used to determine the storage location of the requested data. Metadata server 2 maintains metadata such as data location information and access permissions, and provides a fast query service. Metadata server 2 employs a distributed architecture to ensure high availability and scalability.

[0154] Block storage cache tier 3 is used for storing hot data. Block storage cache tier 3 employs high-performance storage devices such as SSDs, providing low-latency, high-throughput data access. Block storage cache tier 3 supports full-SSD mode, in which all data is stored in SSDs for optimal performance.

[0155] Object storage tier 4 is used for storing cold data. Object storage tier 4 employs large-capacity, low-cost storage devices suitable for long-term storage of data with low access frequency. Object storage tier 4 supports data compression and deduplication to improve storage efficiency.

[0156] Data migration module 5 is used to dynamically migrate data between block storage cache tier 3 and object storage tier 4 based on predetermined policies. Data migration module 5 monitors data access patterns and determines data migration timing based on pre-set thresholds. The data migration process is in an asynchronous manner, minimizing the impact on system performance.

[0157] Intelligent tiering management module 6 is used to automatically adjust the distribution of data between block storage cache tier 3 and object storage tier 4 according to pre-set data level factors. Intelligent tiering management module 6 considers factors such as data access frequency, importance, and life cycle to dynamically optimize data distribution. Intelligent tiering management module 6 supports custom storage policies, allowing users to adjust data tiering rules according to business needs.

[0158] In some embodiments, the hybrid high-performance storage system adopts a clustered caching mechanism. Compared with single-machine caching mechanisms, the clustered caching mechanism provides higher performance and reliability through the collaborative work of multiple nodes. The clustered caching mechanism supports data sharding and replication, ensuring load balancing and fault recovery capabilities.

[0159] In some embodiments, the hybrid high-performance storage system also includes a storage capacity adjustment module 7. Storage capacity adjustment module 7 is used to expand or reduce the storage capacity of block storage cache tier 3 on demand to adapt to fluctuations in business load. Storage capacity adjustment module 7 monitors system load and resource usage to automatically trigger capacity adjustment operations.

[0160] In some embodiments, the hybrid high-performance storage system also includes a data write-back optimization module 8. Data write-back optimization module 8 is used to write back only changed data to block storage cache tier 3 to reduce redundant transmission during data migration. Data write-back optimization module 8 uses an incremental update strategy to identify and transmit changed data blocks, improving write-back efficiency.

[0161] Through the collaborative work of these components, the hybrid high-performance storage system realizes efficient data management and access, balancing performance and cost, and meeting the needs of different business scenarios. The flexibility and scalability of the system make it applicable to various storage needs, from small businesses to large-scale data centers.

[0162] The embodiment of the present application further discloses a readable storage medium.

[0163] A readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the hybrid high-performance storage method according to any one of the above embodiments. The computer readable storage medium can include any entity or device capable of carrying the computer program, a recording medium, a USB flash disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a Read-Only Memory (ROM), a Random Access Memory (RAM), and a software distribution medium, etc. The computer program includes computer program code. The computer program code can be in a form of source code, an object code, an executable file, or some intermediate form, etc. The computer readable storage medium can include any entity or device capable of carrying the computer program code, a recording medium, a USB flash disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a Read-Only Memory (ROM), a Random Access Memory (RAM), and a software distribution medium, etc.

[0164] Any process or method descriptions or any other information in connection with the flow diagrams described herein can be understood as representing any manner of activity or actions by, or within, the present application, including structural computer readable storage media encoded with computer readable program code to perform the process or method, and / or in structure, hardware, firmware, microcode, state machines, or combinations thereof, providing commands for or otherwise tangibly transforming a physical device or apparatus.

[0165] The logic and / or steps represented in the flow diagrams and / or otherwise described herein, for example, can be considered as a list of executable instructions to cause a machine to perform the described logic, and such can be stored in any computer readable medium, for execution by or to control the operation of an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processing module, or other systems that can fetch instructions from the instruction execution system, apparatus, or device and execute instructions, or combinations thereof.

[0166] The above embodiments are only used to illustrate the technical solutions of the present application, not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A hybrid high performance storage method, characterized by, Comprising: Receiving data read or write requests from clients; Determining the storage location of the requested data through the metadata server; According to the storage location, obtaining or storing the data from the block storage cache layer or the object storage layer; Based on a predetermined strategy, dynamically migrating data between the block storage cache layer and the object storage layer, the block storage cache layer is used to store hot data, and the object storage layer is used to store cold data; And through an intelligent hierarchical management mechanism, automatically adjusting the distribution of data between the block storage cache layer and the object storage layer according to preset data level factors; Wherein, the predetermined strategy includes: setting a dynamically adjustable threshold according to the access frequency of the data, when the data access frequency exceeds the threshold, migrating the data from the object storage layer to the block storage cache layer; when the data access frequency is lower than the threshold, migrating the data from the block storage cache layer to the object storage layer; The dynamically adjustable threshold uses a multi-level threshold strategy, data above the highest threshold is immediately migrated to the block storage cache layer; data below the lowest threshold is migrated to the object storage layer; data in between is determined whether to migrate according to the current storage capacity and load situation; Setting the lifetime of the data stored in the block storage cache layer; when the lifetime of the data expires, migrating the data from the block storage cache layer to the object storage layer; Wherein, when migrating data to the object storage layer, a migration window period is set, and the data migration operation is performed during the business low peak period; When migrating data to the object storage layer, an incremental migration strategy is implemented, for data whose lifetime is about to expire but is still frequently accessed, the recently accessed part is retained in the block storage cache layer; Implementing an intelligent prefetching mechanism to predict data that will be accessed according to the historical access pattern and data usage trend of the client, including: Collecting the historical data access records of the client; using a machine learning algorithm to analyze the historical data access records to identify the time pattern and correlation of data access; The use of machine learning algorithms to analyze the historical data access records to identify the time pattern and correlation of data access includes: Adaptive threshold mechanism, dynamically adjusting the confidence threshold of the prediction according to the feedback of the prediction accuracy; the predicted data is loaded in advance from the object storage layer to the block storage cache layer; Based on the prediction structure, the metadata server updates the file layout information; Data predicted to have high access frequency will be marked as hot data blocks and transferred from data server (1) to data server (2) to achieve the distribution of hot data; the metadata server will update the file layout again to record the new storage location of the data.

2. The hybrid high performance storage method of claim 1, wherein, Also including: Expanding or reducing the storage capacity of the block storage cache layer as needed to adapt to the fluctuations in business load; Only write the changed data back to the block storage cache layer to reduce redundant transmission during data migration.

3. The hybrid high performance storage method of claim 1, wherein, The intelligent hierarchical management mechanism includes: Classifying data according to its importance, life cycle and access pattern; Based on the classification results, different storage levels and migration strategies are assigned to different categories of data.

4. The hybrid high performance storage method of claim 1, wherein, Also including: Monitoring system load and storage resource usage; According to the monitoring result, the data distribution ratio between the block storage cache layer and the object storage layer is automatically adjusted.

5. The hybrid high performance storage method of claim 1, wherein, Further comprising: Batch processing of write requests, combining multiple small write operations into larger write operations; Before writing data into the object storage layer, the data is compressed and deduplicated to reduce storage space occupation and data transmission volume.

6. A hybrid high performance storage system applied to the hybrid high performance storage method according to any one of claims 1 to 5, characterized in that, The system comprises: A client interface for receiving data read or write requests; A metadata server for determining the storage location of the requested data; A block storage cache layer for storing hot data; An object storage layer for storing cold data; A data migration module for dynamically migrating data between the block storage cache layer and the object storage layer based on a predetermined strategy; And an intelligent tier management module for automatically adjusting the distribution of data between the block storage cache layer and the object storage layer according to predetermined data level factors.

7. The hybrid high-performance storage system of claim 6, wherein, The system further comprises: A storage capacity adjustment module for expanding or reducing the storage capacity of the block storage cache layer as needed to adapt to fluctuations in business load; A data write-back optimization module for writing only changed data back to the block storage cache layer to reduce redundant transmission during data migration.

8. A readable storage medium, characterized by, The readable storage medium stores computer instructions, which are executed by the processor to implement the hybrid high-performance storage method of any one of claims 1-5.

Images