Cloud computing-oriented big data asset storage and retrieval method and system

Abstract

The invention relates to the technical field of cloud computing data storage and retrieval, and discloses a cloud computing-oriented big data asset storage and retrieval method and system.The access event information of a data block is obtained, the instantaneous access strength of the data block is evaluated in a preset short time window, and when the instantaneous access strength reaches a preset strength threshold value, the data block is retrieved. An access peak indication is generated. According to the method, an instantaneous burst use mode formed by high-intensity access of a micro service unit and an automatic task to specific data content in an extremely short time can be effectively captured, and the problem of misjudgment of instantaneous peak access caused by adoption of a relatively long information collection period and a relatively long statistical range in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of cloud computing data storage and retrieval technology, and in particular to a method and system for storing and retrieving big data assets for cloud computing. Background Technology

[0002] In modern cloud computing environments, a tiered data storage strategy is typically employed to effectively manage massive amounts of data and balance storage costs with access speed. This strategy relies on assessing data activity levels, storing frequently used data (common data) on high-performance storage devices and migrating less frequently used data (infrequently used data) to lower-cost archive storage. However, with the continuous development of cloud service platform technologies, particularly the widespread application of microservice units and automated tasks, traditional data activity assessment mechanisms face significant challenges. These new applications often experience high-intensity access to specific data content within extremely short periods, creating a burst of usage patterns. Existing mechanisms, due to their inherent statistical methods, struggle to accurately capture such dynamic changes, leading to misjudgments of true data activity and consequently causing a series of performance and management problems.

[0003] One of the core functions of a large cloud service platform is to provide enterprise customers with storage and retrieval services for unstructured big data content. The platform was initially designed to serve a few large, relatively stable enterprise applications. The data usage patterns of these applications are continuous and stable. For example, a traditional Enterprise Resource Planning (ERP) system continuously reads and writes historical transaction records around the clock, with minimal fluctuations in data usage frequency over a relatively long period. To optimize storage costs and ensure data performance, the platform has established a tiered data storage mechanism, with determining whether data is frequently or infrequently used being a key component.

[0004] The platform's data hot / cold indexing system relies heavily on a usage logger. Initially, this logger was configured to collect information over longer time periods and cover a wider statistical range. For example, it might only analyze data block usage events every few hours and calculate the average usage frequency on a daily basis. This design was reasonable at the time because it effectively minimized the impact of small fluctuations in consistent, stable data usage patterns, thus providing a stable and accurate assessment of the "activity" of data that has been used or unused for extended periods.

[0005] However, with the continuous development of cloud service platform technology and the increasing complexity of customer business needs, the application ecosystem within the platform has undergone fundamental changes. The original large applications have been gradually broken down into hundreds or even thousands of independent micro-service units. Simultaneously, the platform has introduced numerous automated data analysis processes and real-time business report generation tasks. These micro-service units and automated tasks operate in a completely different way than traditional applications. They often occur in concentrated bursts at specific times; for example, at the top of each hour, at midnight, or at the beginning of each week, a large number of micro-service units are triggered simultaneously, performing numerous concurrent read and write operations on specific historical unstructured data sets to generate real-time analysis reports or update business metrics. These usage behaviors are characterized by extremely short durations, perhaps only a few minutes, but their intensity and frequency of simultaneous processing are very high, forming a "peak usage" pattern.

[0006] Because the platform used the original usage logger with a longer information collection cycle and statistical range, it exhibited significant limitations when faced with this new type of short-duration, high-intensity usage peak. When a dataset experienced tens of thousands of uses in an extremely short period (e.g., 5 minutes), but the logger's statistical range remained 24 hours, these high-intensity usage events were averaged over the entire 24-hour period. This resulted in the calculated average usage count being far lower than the actual instantaneous usage intensity. Consequently, data that was actually used periodically and frequently was incorrectly classified as "infrequently used" data by the system. This misjudgment of the true data usage pattern led to a significant discrepancy between the system's understanding of data "activity" and the actual usage situation.

[0007] Based on the platform's established data lifecycle management rules, data incorrectly judged as "infrequently used" is automatically moved from the fast but slow-responding "frequently used" storage layer (e.g., NVMe SSD-based storage devices) to lower-cost but significantly slower archive storage devices (e.g., large-capacity HDD devices or tape libraries). This migration is intended to optimize storage costs, but when the next round of small service unit requests or automated tasks re-access this "archived" data, the system is forced to read the data from the slow-responding archive storage. This leads to a significant increase in the response time of many services. For example, a real-time analytics dashboard that previously had a second-level response time may now take several minutes to load, resulting in a general performance degradation across the entire platform and severely impacting business efficiency and user experience.

[0008] To address the aforementioned problems, existing technologies urgently need improvement. Summary of the Invention

[0009] This invention provides a method and system for storing and retrieving big data assets for cloud computing. It aims to solve the problem that in the existing cloud computing environment, traditional data hierarchical storage strategies are unable to accurately capture the true activity of data when faced with short-term, high-intensity access patterns generated by small service units and automated tasks, leading to data misjudgment, and consequently causing performance degradation and low management efficiency.

[0010] The technical solution of this application is as follows: Firstly, this application discloses a method for storing and retrieving big data assets for cloud computing, including: Retrieve access event information for data blocks, including the data block identifier, access type, and access time; Based on access event information, the instantaneous access intensity of the data block is evaluated within a preset short time window. When the instantaneous access intensity reaches a preset intensity threshold, an access peak indication is generated. In response to a peak access signal, initiate an evaluation process that samples access events over a short period and assesses activity levels within a preset statistical window; When the results of the evaluation process consistently meet the preset activity conditions, the data block is marked as a peak activity state, and a retention period is set for the data block in the peak activity state. During the retention period, prevent data blocks in peak active states from being moved from the high-performance storage layer to the archive storage layer; If the data block does not trigger another access peak indication after the retention period ends, the storage tier of the data block will be evaluated based on the long-term average access activity.

[0011] Through this technical solution, this application can effectively identify and respond to instantaneous high-intensity access to data blocks, avoid misjudgment of activity caused by traditional long-term average statistical methods, thereby preventing peak active data from being mistakenly migrated to low-performance storage layers. This solves the problem that data tiering strategies in existing technologies cannot adapt to new application models, and significantly improves data access performance and user experience.

[0012] Furthermore, in response to a peak access indication, the long-term average access frequency statistics for data blocks are paused, and an evaluation process is initiated to sample access events over a short period and assess activity levels within a preset statistical window, including: Inside each storage node of the high-performance storage layer, a data block access counter is configured to record the number of times a data block is accessed. When the number of accesses recorded by the data block access counter reaches the preset instantaneous peak threshold, the local metadata of the data block is updated, and the data block is marked as a peak active state. The status update callback mechanism is triggered to notify the local hot / cold data judgment module. The local hot / cold data judgment module assigns the highest priority to the peak activity status of the data block and adjusts the storage level management strategy of the data block. Set a retention period for data blocks in peak activity states, and ensure that data blocks in peak activity states remain in the high-performance storage layer during the retention period; After the retention period ends, the activity of the data blocks will be reassessed.

[0013] Through this technical solution, this application can achieve rapid response and status update for instantaneous peak access by configuring data block access counters inside the storage node, and give the highest priority to peak active status, ensuring that data stays in the high-performance storage layer, thereby effectively avoiding data being misjudged as cold data and migrated during peak periods, and further ensuring the real-time performance and stability of data access.

[0014] Building upon the above, this application further proposes, during the retention period, preventing the relocation of data blocks in peak active states from the high-performance storage layer to the archive storage layer, including: The high-performance storage tier is divided into multiple virtual storage partitions, and each virtual storage partition is bound to one or a set of data. Predict the capacity requirements of the data set within each virtual storage partition and reserve physical capacity for each virtual storage partition; Place data blocks in peak active states into the physical resource range corresponding to the bound virtual storage partition; Resource isolation is implemented for each virtual storage partition. When a virtual storage partition is in the predicted peak period, a cross-partition resource scheduling and priority arbitration mechanism is initiated to prevent data blocks in peak active state from being moved from the high-performance storage layer to the archive storage layer.

[0015] Through this technical solution, this application effectively guarantees the storage resources of peak-active data blocks by using virtual storage partitioning and resource isolation mechanisms, combined with cross-partition resource scheduling and priority arbitration, preventing them from being forcibly migrated during the predicted peak period, thereby improving resource utilization efficiency and data access stability.

[0016] In some preferred embodiments, during the retention period, preventing the relocation of data blocks in peak activity states from the high-performance storage tier to the archive storage tier includes: Identify peak active data blocks located within the damaged storage area; Based on the business priority, data sensitivity, and fault type of peak active data blocks, determine the emergency protection strategy; Perform emergency data processing tasks to copy or migrate affected peak-active data blocks to other high-performance storage areas, or perform encrypted backups; Update the metadata and data access index of the peak active data block, and reactivate the peak active retention period of the peak active data block.

[0017] Through this technical solution, this application can promptly activate emergency protection strategies for peak active data blocks in the damaged storage area based on their business priority and data sensitivity, perform data replication, migration or encrypted backup, and reactivate the retention period, thereby effectively ensuring the availability and security of peak active data in the event of storage media failure.

[0018] As a technical improvement, during the retention period, measures are taken to prevent data blocks in peak activity states from being moved from the high-performance storage layer to the archive storage layer, including: Continuously monitor access event information for data blocks, including the data block identifier, access type, and access time; Calculate the activity trend of data blocks based on access event information; Identify the match between activity trends and the current retention period setting; Based on the trend of activity changes and the preset adjustment rules, the retention period is dynamically adjusted to obtain the adjusted retention period; Update the metadata of the data block, which includes the adjusted retention period; During the adjusted retention period, prevent data blocks in peak active states from being moved from the high-performance storage layer to the archive storage layer.

[0019] Through this technical solution, this application can dynamically adjust the retention period according to the activity trend of data blocks, making the storage level management of data blocks more flexible and accurate, avoiding resource waste or data misjudgment that may be caused by fixed periods, and further optimizing the utilization efficiency of storage resources.

[0020] To enhance functionality, during the retention period, prevent the relocation of data blocks in peak activity states from the high-performance storage tier to the archive storage tier, including: Deploy a separate storage tier management command channel, configure dedicated network bandwidth resources through the separate storage tier management command channel, and adopt a high-priority data transmission protocol; When the system generates an instruction to prevent data blocks from being moved from the high-performance storage layer to the archive storage layer, the instruction is transmitted through an independent storage layer management instruction channel, and the instruction is verified in real time to obtain the verification result. Based on the verification results, an instruction execution confirmation mechanism is configured at the receiving end of the independent storage tier management instruction channel. After successfully receiving and processing the instruction, an confirmation signal is returned to the sending end of the independent storage tier management instruction channel. If no confirmation signal is received within the preset time, the command retransmission mechanism will be triggered to retransmit the command.

[0021] This technical solution, by deploying an independent storage-level management command channel, configuring dedicated network bandwidth and high-priority transmission protocols, and introducing a real-time verification and command execution confirmation mechanism, ensures the reliable transmission and execution of data migration commands, effectively avoiding data mismigration problems caused by command transmission failure or delay.

[0022] As a further improvement, during the retention period, the relocation of data blocks in peak active states from the high-performance storage layer to the archive storage layer is prevented, including: Allocate independent storage resource quotas to each tenant; To prevent instructions from moving data blocks from the high-performance storage layer to the archive storage layer, a global highest execution priority is set; When the system receives an instruction to prevent a data block from being moved from the high-performance storage layer to the archive storage layer, the instruction is routed to a dedicated instruction processing queue. This dedicated instruction processing queue is independent of other tenants' regular data operation queues and has priority scheduling rights. Deploy a policy arbitration module on each storage node so that the policy arbitration module can monitor the resource usage and command execution status among tenants in real time; When the policy arbitration module detects that an instruction to prevent data blocks from being moved from the high-performance storage layer to the archive storage layer conflicts with a high-priority tenant’s data migration or resource allocation operation, it will forcibly interrupt or delay the operation of other tenants. A dedicated communication channel is provided for the instructions. This dedicated communication channel is configured with independent network bandwidth and uses protocols that ensure the reliability of instruction transmission.

[0023] This technical solution sets the highest global execution priority for the data migration prevention instruction and provides it with a dedicated instruction processing queue and communication channel. Combined with the real-time monitoring of resource usage among tenants by the policy arbitration module, it ensures the priority execution of the instruction in a multi-tenant environment, effectively avoiding conflicts with other high-priority operations, thereby guaranteeing the stability of the storage hierarchy for peak active data.

[0024] Building upon the above, this application further proposes, during the retention period, preventing the relocation of data blocks in peak active states from the high-performance storage layer to the archive storage layer, including: Continuously acquire the operating status information of each storage medium within the high-performance storage layer. The operating status information includes the read / write error rate, number of bad blocks, remaining lifetime, and temperature of the storage medium. Based on the operating status information, assess the health status of the storage medium. When the health status is lower than the preset health threshold, generate a media anomaly warning. In response to media anomaly warnings, identify peak active data blocks, determine the data protection level based on the business priority and data sensitivity of the peak active data blocks, and determine emergency handling operations based on the type of media anomaly warning; Perform emergency response operations to copy or migrate affected peak-active data blocks to other healthy storage media, or perform data backup; Update the metadata and data access index of peak active data blocks and reactivate the retention period.

[0025] Through this technical solution, this application continuously monitors the operating status and assesses the health status of the storage medium, and can generate early warnings in a timely manner when the medium is abnormal. Based on the business priority and data sensitivity of the peak active data blocks, emergency processing operations are performed to migrate or back up the data to the healthy medium, thereby effectively ensuring the security and availability of peak active data in the event of storage medium failure.

[0026] Furthermore, during the retention period, prevent data blocks in peak active states from being moved from the high-performance storage tier to the archive storage tier, including: Deploy a data consistency coordinator, configure a cross-cloud data transmission channel, and the cross-cloud data transmission channel adopts a data sharding transmission method. The data sharding transmission method divides the data block into shards and performs independent verification and encryption on each data shard. Deploy a data proxy module in the storage environment of each cloud service provider. The data proxy module identifies and adapts to the storage interfaces and data processing logic of different cloud service providers. The data broker module retrieves metadata and data content of peak-active data blocks from storage environments of different cloud service providers. The metadata includes the checksum, version information, and timestamp of the data block. Based on the metadata and data content of peak active data blocks, perform cross-cloud data consistency verification. Cross-cloud data consistency verification includes comparing the checksums, version information and timestamps of data blocks between different cloud service providers. When cross-cloud data consistency verification finds data inconsistencies, the consistent data version is determined according to the preset conflict resolution strategy. The data broker module synchronizes consistent data versions to the storage environment of cloud service providers where inconsistencies exist and updates the corresponding metadata. During data synchronization, the data consistency coordinator continuously monitors the transmission status and data integrity of the cross-cloud data transmission channel, and triggers data retransmission operations when transmission is abnormal or data is corrupted.

[0027] This technical solution, through the deployment of a data consistency coordinator and a data broker module, enables consistency verification and synchronization of peak active data blocks in a cross-cloud environment. It effectively solves the problem of data consistency in a multi-cloud environment and ensures the reliability and availability of peak active data in different cloud service provider storage environments.

[0028] Secondly, this application also discloses a big data asset storage and retrieval system for cloud computing, comprising: The input end is used to obtain access event information of data blocks. The access event information includes the identifier of the data block, the access type, and the access time. Based on the access event information, the instantaneous access intensity of the data block is evaluated within a preset short time window. When the instantaneous access intensity reaches a preset intensity threshold, an access peak indication is generated. The response end is used to respond to access peak indications and initiate an evaluation process that samples access events over a short period of time and assesses the level of activity within a preset statistical window. When the results of the evaluation process continuously meet the preset activity conditions, the data block is marked as a peak activity state, and a retention period is set for the data block in the peak activity state. The adjustment mechanism is used to prevent data blocks in a peak activity state from being moved from the high-performance storage tier to the archive storage tier during the retention period. After the retention period ends, if the data block does not trigger the access peak indication again, the storage tier of the data block is evaluated based on the long-term average access activity.

[0029] Through this technical solution, the system of this application can achieve accurate identification, status marking and storage-level protection of instantaneous access peaks of data blocks by working together at the input end, response end and adjustment end. It effectively solves the shortcomings of traditional systems in handling short-term high-intensity access patterns, thereby improving the efficiency and reliability of big data asset storage and retrieval. Beneficial effects

[0030] This application provides a method for storing and retrieving big data assets for cloud computing. It acquires access event information for data blocks and evaluates the instantaneous access intensity of the data blocks within a preset short time window. When the instantaneous access intensity reaches a preset intensity threshold, an access peak indication is generated. This method effectively captures the instantaneous burst usage patterns formed by high-intensity access to specific data content by small service units and automated tasks within a very short period, overcoming the problem of misjudging instantaneous peak access caused by the use of longer information collection cycles and statistical ranges in existing technologies.

[0031] In response to a peak access indication, this application initiates an evaluation process that samples access events over a short period and assesses activity levels within a preset statistical window. When the evaluation results consistently meet preset activity conditions, the data block is marked as being in a peak activity state, and a retention period is set for this peak activity state. This mechanism ensures that the system can promptly and accurately identify the true activity level of data blocks, avoiding the erroneous classification of periodically frequently used data as "infrequently used" data.

[0032] During the retention period, this application prevents data blocks in peak activity states from being moved from the high-performance storage layer to the archive storage layer. This crucial step effectively solves the problem in existing technologies where misjudgment of data leads to its migration from the high-performance storage layer to the slow-responding archive storage layer, resulting in significantly extended service response times and a general decrease in platform performance. By forcing data to remain in the high-performance storage layer during the peak activity retention period, this application ensures that data can be accessed quickly when needed, significantly improving business efficiency and user experience.

[0033] If the data block does not trigger another access peak indication after the retention period ends, its storage tier is evaluated based on the long-term average access activity. This strategy, which combines instantaneous peak judgment with long-term average activity evaluation, makes data tier management more refined and intelligent. It can cope with sudden high-intensity access while also taking into account long-term storage cost optimization, achieving an effective balance between performance and cost.

[0034] In summary, this application effectively solves the technical challenges of existing big data asset storage and retrieval methods in cloud computing environments when dealing with short-term, high-intensity access patterns by introducing instantaneous access intensity assessment, peak activity status marking, and retention period mechanisms. It avoids misjudgment of data activity and the resulting performance degradation, significantly improves data access efficiency and user experience, and has significant progressiveness and practicality. Attached Figure Description

[0035] Figure 1 This is a flowchart of a big data asset storage and retrieval method for cloud computing provided by an embodiment of the present invention; Figure 2 This is a flowchart of a method for evaluating activity levels within a preset statistical window, provided by an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of a big data asset storage and retrieval system for cloud computing provided in an embodiment of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Reference Figure 1 , Figure 1 This is a flowchart of a big data asset storage and retrieval method for cloud computing provided by an embodiment of the present invention, including the following steps: S11, Obtain access event information for the data block, wherein the access event information includes the identifier of the data block, the access type, and the access time; S12, Based on the access event information, evaluate the instantaneous access intensity of the data block within a preset short time window, and generate an access peak indication when the instantaneous access intensity reaches a preset intensity threshold. S13, in response to the access peak indication, initiate an evaluation process that samples access events over a short period of time and assesses the level of activity within a preset statistical window; S14, when the results of the evaluation process continuously meet the preset activity conditions, the data block is marked as a peak activity state, and a retention period is set for the data block in the peak activity state; S15, during the retention period, prevent the data block in the peak active state from being moved from the high-performance storage layer to the archive storage layer; S16, after the retention period ends, if the data block does not trigger the access peak indication again, the storage level of the data block is evaluated based on the long-term average access activity.

[0038] This application effectively identifies and responds to the instantaneous high-intensity access patterns of data blocks by introducing instantaneous access intensity assessment, peak activity status marking, and retention period mechanisms. This prevents data blocks from being mistakenly migrated to low-performance storage layers during active periods, and significantly improves the storage and retrieval efficiency of big data assets in a cloud computing environment.

[0039] To better understand the technical solution proposed in this application, some key terms involved will be explained first.

[0040] In a distributed storage system, a "data block" is a basic unit of storage into which data is divided. Each data block typically has a unique "identifier" used for identification and location within the system.

[0041] "Access event information" refers to the records generated when operations are performed on data blocks (such as reading, writing, modifying, etc.). It includes an "identifier" to indicate the data block being operated on, an "access type" to describe the specific operation behavior, and an "access time" to record the time when the operation occurred.

[0042] "High-performance storage tier" typically refers to a storage area that uses high-speed storage media such as solid-state drives (SSDs) or non-volatile memory (NVMe), which has extremely low access latency and high throughput, and is suitable for storing frequently accessed hot data.

[0043] "Archived storage tier" typically refers to a storage area that uses low-cost, high-capacity storage media such as hard disk drives (HDDs), tape libraries, or object storage. It has higher access latency and lower throughput, and is suitable for storing cold data or archived data that is not frequently accessed.

[0044] "Instantaneous access intensity" refers to the frequency or number of times a data block is accessed within a very short time window, reflecting the activity level of the data block at a specific moment.

[0045] "Peak activity state" refers to a data block that has experienced high-intensity access in a short period of time and is identified by the system as being in a highly active state.

[0046] "Retention period" refers to a time period set for data blocks that are in a state of peak activity, during which the system will take measures to prevent them from being migrated to a low-performance storage layer.

[0047] The method proposed in this application aims to optimize the storage and retrieval efficiency of big data assets in a cloud computing environment, especially when facing instantaneous high-intensity access patterns. The implementation methods of its main technical features will be described in detail below.

[0048] First, it's necessary to obtain access event information for the data blocks. This information is fundamental for determining the activity level of the data blocks. One implementation is to deploy a log collection agent on each storage node. This agent monitors all read and write operations on local storage data blocks in real time, recording the data block identifier, access type (e.g., read, write, delete), and precise access time for each operation, forming an access event log. These logs can be periodically uploaded to a central log processing service for further analysis. Another implementation is to modify the storage system's I / O stack so that when a data block is accessed, the operating system or storage driver directly generates access event information and sends it to a message queue for subsequent modules to consume. For example, when a user requests to read a data block, the storage system records the unique ID of the data block, that this is a "read" operation, and the specific timestamp of the operation.

[0049] Secondly, based on the acquired access event information, the instantaneous access intensity of the data block needs to be evaluated within a preset short time window. One implementation is to set a sliding time window, such as 5 minutes. The system continuously counts the total number of times each data block is accessed within the current sliding window. When the number of accesses to a data block within the current 5-minute window exceeds a preset intensity threshold (e.g., 1000 accesses every 5 minutes), the instantaneous access intensity of that data block is considered to have reached a peak, and an access peak indication is generated. Another implementation is to combine the number of accesses with the amount of data accessed to evaluate the instantaneous access intensity. For example, if the number of accesses and / or the amount of data accessed both exceed a preset threshold within the preset short time window, an access peak indication is generated.

[0050] Next, in response to the access peak indication, an evaluation process needs to be initiated to sample access events over a short period and assess activity levels within a preset statistical window. When the system detects a peak in the instantaneous access intensity of a data block, a more granular activity evaluation mechanism will be immediately triggered. One implementation is to suspend the regular long-term activity statistics for that data block and instead initiate a dedicated short-term sampling process for that data block. For example, the sampling frequency can be increased from once per hour to once per second, and the access frequency or volume of the data block can be continuously calculated within a shorter statistical window (e.g., 1 minute). Another implementation is that, upon receiving the access peak indication, the system allocates a separate evaluation thread for that data block. This thread collects access events for the data block at a higher granularity (e.g., every 10 seconds) over the next few minutes and calculates its average access frequency within each short statistical window.

[0051] Subsequently, when the evaluation process consistently meets the preset activity conditions, the data block is marked as peak active, and a retention period is set for the data block in peak active state. For example, if the above short-term sampling evaluation process shows that the average access frequency of the data block exceeds a certain activity condition (e.g., 500 accesses per minute) within three consecutive 1-minute statistical windows, the system will officially mark the data block as peak active. Simultaneously, the system will set a retention period for the data block, such as 2 hours. During this period, the data block will be considered hot data, and even if its access volume decreases at a brief moment, it will not be immediately downgraded.

[0052] The key is to prevent data blocks in a peak-activity state from being moved from the high-performance storage tier to the archive storage tier during the retention period. Once a data block is marked as peak-activity and a retention period is set, the system takes enforcement measures to ensure it remains in the high-performance storage tier. One implementation is to set a high-priority locking flag for data blocks in a peak-activity state in the storage tier management module. When an automatic or manual data migration task attempts to migrate the data block from the high-performance storage tier to the archive storage tier, this locking flag will prevent the operation until the retention period ends. Another implementation is to add a "migration prohibited" field to the data block's metadata and set it to true. When the storage tier management system makes data migration decisions, it will first check this field; if it is true, the migration operation for that data block will be skipped.

[0053] Finally, after the retention period ends, if the data block does not trigger the access peak indication again, its storage tier is evaluated based on long-term average access activity. When the retention period expires, the system removes the migration block restriction. At this point, if the data block does not trigger the access peak indication again within a certain period after the retention period ends, the system resumes its long-term average access activity evaluation. For example, the system re-enables a 24-hour sliding window to calculate the average access frequency of the data block. Based on this long-term average activity, the system re-determines whether the data block should continue to be retained in the high-performance storage tier or can be safely migrated to the archive storage tier to save costs.

[0054] The big data asset storage and retrieval method proposed in this application for cloud computing works by using a multi-stage, multi-granularity activity assessment mechanism to accurately identify the actual usage patterns of data blocks, especially when faced with instantaneous high-intensity access.

[0055] Through the interconnected steps described above, the overall technical solution of this application can effectively identify and respond to the instantaneous high-intensity access patterns of big data assets in a cloud computing environment, avoiding data migration errors and performance degradation caused by misjudgment of activity in traditional data tiering strategies, thereby significantly improving the storage and retrieval efficiency of big data assets and the user experience.

[0056] The big data asset storage and retrieval method proposed in this application for cloud computing aims to solve the problems of data misjudgment and performance degradation caused by inaccurate activity judgment when traditional data hierarchical storage strategies face instantaneous high-intensity access patterns.

[0057] In some embodiments described above in this application, an evaluation process is proposed to initiate an evaluation process in response to an access peak indication, which samples access events over a short period of time and assesses activity levels within a preset statistical window. However, in its implementation, if an efficient and real-time mechanism is lacking to accurately identify and immediately respond to instantaneous access peaks of data blocks, the storage hierarchy management strategy for data blocks may not be adjusted in a timely manner during the evaluation period, thereby affecting data access performance.

[0058] In this regard, refer to Figure 2 , Figure 2 This is a flowchart of a method for evaluating activity levels within a preset statistical window, provided by an embodiment of the present invention, including: S131, In each storage node of the high-performance storage layer, a data block access counter is configured to record the number of times a data block is accessed; S132, when the number of accesses recorded by the data block access counter reaches the preset instantaneous peak threshold, the local metadata of the data block is updated and the data block is marked as a peak active state. S133, trigger the status update callback mechanism, notify the local hot / cold judgment module, and assign the peak activity status of the data block to the highest priority through the local hot / cold judgment module, and adjust the storage level management strategy of the data block. S134 sets a retention period for data blocks in peak active state, ensuring that data blocks in peak active state remain in the high-performance storage layer during the retention period; S135, after the retention period ends, reassess the activity of the data blocks.

[0059] Specifically, a data block access counter can be understood as a lightweight software or hardware counter deployed within each high-performance storage node. Its purpose is to track the access frequency of a specific data block in real time and with high accuracy within a very short period. For example, this counter could be an in-memory hash table, with the data block identifier as the key and the access count as the value, atomically incrementing with each access. The preset instantaneous peak threshold is a pre-defined value; when a data block is accessed more than this threshold within a short period, it is considered to have experienced an instantaneous access peak. This threshold can be dynamically configured or statically set based on business needs, data importance, and system load capacity. In practical applications, updating the local metadata of the data block and marking it as peak active means that upon detecting an instantaneous peak, immediately modifying the locally stored description information about the data block, for example, adding a boolean flag "is_peak_active" to the metadata and recording the peak start time.

[0060] Furthermore, the state update callback mechanism can be understood as an event-driven notification mechanism. When the state of a data block changes (e.g., from off-peak activity to peak activity), a preset callback function or message is automatically triggered to notify relevant system components. The local hot / cold data determination module is the core component responsible for determining the storage tier of a data block based on its access patterns and activity levels. By assigning the highest priority to peak activity states, the storage tier decision for peak activity data blocks can be ensured to be processed before any other hot / cold data determination logic. Adjusting the storage tier management strategy for a data block, such as temporarily disabling the cold data migration strategy for that data block or raising its migration priority to the highest level, ensures its residency in the high-performance storage tier. The retention period refers to a time window set for data blocks in the peak activity state. During this period, the data block will be forcibly retained in the high-performance storage tier, unaffected by the regular hot / cold data migration strategy. The setting of this period should comprehensively consider the continuity of business access patterns, the importance of the data, and the cost-effectiveness of storage resources. After the retention period ends, the activity of the data block will be reassessed. This means that the system will resume regular activity monitoring and evaluation of the data block to determine its subsequent storage tier. This may include restarting long-term average access frequency statistics or making a comprehensive judgment based on new access patterns.

[0061] This application's solution achieves real-time, fine-grained monitoring of the instantaneous access intensity of data blocks by configuring a data block access counter within each storage node of the high-performance storage layer. When the number of accesses reaches a preset instantaneous peak threshold, the local metadata of the data block is immediately updated, marking it as peak active, thus avoiding the latency that may be caused by traditional long-term statistics. Subsequently, by triggering a status update callback mechanism, the local hot / cold data judgment module is quickly notified and given the highest priority, ensuring that its storage tier management strategy can respond in time and effectively preventing data blocks from being mistakenly judged as cold data and migrated to a low-performance storage layer during peak access periods. In addition, by setting a retention period, the system provides a stable high-performance storage environment for peak active data blocks, ensuring their access performance during critical business periods. After the retention period ends, the activity of the data block is reassessed, allowing storage tier management to flexibly adapt to changes in data access patterns and avoiding long-term unreasonable occupation of resources.

[0062] Through the above technical solution, this application can achieve rapid and accurate identification and response to peak access times for big data assets. Compared with methods that rely solely on long-term average access frequency statistics, this solution significantly shortens the time interval between the occurrence of a peak access time and the system taking protective measures by introducing an instantaneous access counter and a real-time status update mechanism. This effectively avoids the risk of data blocks being incorrectly downgraded to the storage tier due to evaluation lag during peak periods. Therefore, it ensures that critical business data always resides in the high-performance storage tier during peak access times, greatly improving the stability and response speed of data access, thereby optimizing user experience and ensuring business continuity.

[0063] In some preferred embodiments, a specific example is given below. Suppose that in a cloud computing environment, a data block (e.g., an image or video of a popular e-commerce product) is frequently accessed by a large number of users within a short period. First, within the high-performance storage layer node hosting the data block, a data block access counter is configured and continuously records the number of accesses to the data block. For example, within a short time window of 10 seconds, the number of accesses to the data block rapidly increases from a few times per second to hundreds of times per second. When the number of accesses recorded by the data block access counter reaches a preset instantaneous peak threshold (e.g., more than 500 accesses within 10 seconds), the system immediately updates the local metadata of the data block, marking it as "peak active state." Subsequently, a status update callback mechanism is triggered, sending a notification to the local hot / cold data determination module.

[0064] Upon receiving the notification, the local hot / cold data assessment module immediately assigns the highest priority to the peak activity status of the data block and adjusts its storage tier management strategy. For example, it temporarily disables any strategies that might migrate the data block to the archive storage tier and ensures its residency in the high-performance storage tier. Simultaneously, the system sets a retention period, such as 30 minutes, for this peak activity data block. During these 30 minutes, regardless of how the regular hot / cold data assessment logic evaluates it, the data block will be forcibly retained in the high-performance storage tier. After the 30-minute retention period ends, the system will reassess the activity of the data block. If the access volume has returned to normal, the data block will be reinstated under the regular hot / cold data management strategy; if the access volume remains high and triggers the instantaneous peak threshold again, the retention period will be reactivated. In this way, this application can flexibly and efficiently address the dynamic access patterns of big data assets.

[0065] In some embodiments described above, this application proposes preventing data blocks from being moved from the high-performance storage layer to the archive storage layer after they are marked as peak-active and a retention period is set. However, in real-world cloud computing environments, the high-performance storage layer is typically a shared resource. Simply "preventing" data migration may not effectively address complex resource contention and dynamic load changes, especially when multiple datasets experience simultaneous access peaks, potentially leading to resource allocation conflicts and impacting the performance assurance of peak-active data blocks. If these issues are not addressed, peak-active data blocks may be unexpectedly degraded during the retention period due to insufficient resources or improper scheduling, affecting user experience and business continuity. Therefore, this application further proposes a more refined and reliable method for preventing data block migration. By implementing virtualization management and resource reservation for the high-performance storage layer, the storage stability of peak-active data blocks during the retention period is ensured.

[0066] During the aforementioned retention period, preventing the transfer of data blocks in peak activity states from the high-performance storage layer to the archive storage layer specifically includes the following steps: First, the high-performance storage layer is divided into multiple virtual storage partitions, and each virtual storage partition is bound to one or a set of data sets. A virtual storage partition can be understood as an independent storage space logically allocated from physical storage resources, with each partition specifically used to store a data set of a particular type or for a specific business purpose. This division aims to achieve fine-grained management and isolation of storage resources, avoiding resource contention between different data sets.

[0067] Secondly, the capacity requirements of the data set within each virtual storage partition are predicted, and physical capacity is reserved for each virtual storage partition. Specifically, capacity requirement prediction can be based on factors such as historical access patterns, data growth trends, and business planning, for example, using time series analysis or machine learning models. Reserved physical capacity refers to pre-allocating and locking a certain amount of physical storage space for each virtual storage partition in the high-performance storage layer to ensure sufficient resources are available during peak data periods and prevent data migration triggered by insufficient capacity.

[0068] Next, the data blocks in the peak active state are placed within the physical resource range corresponding to the bound virtual storage partition. This means that once a data block is identified as being in a peak active state, and its data set has been bound to a virtual storage partition, the data block will be ensured to be stored within the physical capacity reserved by that virtual storage partition, thereby obtaining dedicated storage resource protection.

[0069] Finally, resource isolation is implemented for each virtual storage partition. When a virtual storage partition is in a predicted peak period, a cross-partition resource scheduling and priority arbitration mechanism is initiated to prevent the transfer of the peak-active data blocks from the high-performance storage layer to the archive storage layer. Resource isolation refers to ensuring that resources between different virtual storage partitions do not interfere with each other through software-defined storage (SDS) or virtualization technologies. When a peak access period is predicted for a virtual storage partition, the system initiates an advanced resource scheduling and arbitration mechanism. This mechanism coordinates resource requests between different virtual storage partitions and assigns a higher priority to the virtual storage partition containing the peak-active data blocks, thereby forcibly preventing any operations that could cause the peak-active data blocks to migrate from the high-performance storage layer to the archive storage layer.

[0070] This application's solution effectively addresses the limitations of basic solutions in complex cloud computing environments, where peak-active data blocks may be unexpectedly degraded due to resource contention or mismanagement. Specifically, by introducing virtual storage partitioning, capacity demand prediction, physical capacity reservation, and resource isolation, as well as cross-partition resource scheduling and priority arbitration mechanisms, it effectively addresses these limitations. Specifically, dividing the high-performance storage layer into virtual storage partitions and binding them to data sets allows for fine-grained management of storage resources, preventing resource sharing and contention between different data sets. Predicting capacity demand and reserving physical capacity ensures sufficient storage space for peak-active data blocks during peak data periods, fundamentally eliminating the risk of migration due to insufficient capacity. Placing peak-active data blocks within the physical resource range of their bound virtual storage partitions provides them with dedicated storage protection. Furthermore, resource isolation of virtual storage partitions and initiation of cross-partition resource scheduling and priority arbitration mechanisms during predicted peak periods ensure that when multiple partitions face resource pressure simultaneously, the system prioritizes the storage stability of peak-active data blocks, forcibly preventing their migration to the archive storage layer. This ensures that peak-active data blocks remain in the high-performance storage layer for the duration of the retention period.

[0071] Through the above technical solutions, this application provides a more robust and reliable mechanism to prevent peak-active data blocks from being moved from the high-performance storage layer to the archive storage layer during the retention period. Specifically, by using virtual storage partitioning and capacity reservation, it effectively avoids resource contention and capacity shortages that may occur in a shared storage environment, ensuring that peak-active data blocks can obtain the necessary storage resources under any circumstances. Furthermore, the introduction of cross-partition resource scheduling and priority arbitration mechanisms enables the system to intelligently allocate resources and resolve conflicts when facing complex and variable workloads, prioritizing the storage layer of critical data blocks, significantly improving the stability and performance of big data asset storage, and effectively addressing potential deficiencies in resource management and migration prevention in the basic solution.

[0072] In some preferred embodiments, it is assumed that the high-performance storage layer of a cloud computing platform is divided into three virtual storage partitions: partition A for storing financial transaction data, partition B for storing e-commerce user behavior logs, and partition C for storing medical image data. The system continuously predicts the capacity requirements of the data sets within each partition. For example, based on historical data analysis, it is predicted that the 15th of each month is the peak settlement period for financial transaction data, and the capacity requirements of partition A will increase significantly. Therefore, the system reserves additional physical capacity for partition A before the 10th of each month to cope with the upcoming peak. When financial transaction data block 100 is identified as being in a peak activity state, it will be placed within the physical resource range reserved for partition A. During peak settlement periods, even though partitions B and C may experience some access, the resources of partition A have been isolated and reserved, and the system has activated a cross-partition resource scheduling and priority arbitration mechanism. Any instruction attempting to migrate financial transaction data block 100 from the high-performance storage layer to the archive storage layer will be blocked. At the same time, this mechanism ensures that partition A has sufficient I / O bandwidth and storage space to meet the high-performance access requirements of financial transaction data block 100, thereby guaranteeing the smooth operation of financial business.

[0073] In some embodiments described above, this application proposes a scheme to prevent the relocation of data blocks in a peak active state from the high-performance storage layer to the archive storage layer during a retention period. However, in its implementation, if the high-performance storage layer itself is damaged or fails, simply preventing the data block's tier relocation may not be sufficient to guarantee the integrity and availability of the peak active data blocks. If the above problem is not addressed, even if a data block is identified as peak active and its degradation is prevented, its security within the damaged storage area may still be threatened, thus affecting business continuity. Therefore, this application further proposes a scheme for proactively protecting peak active data blocks in the event of storage media damage, to ensure data security and business continuity.

[0074] In this regard, this application further proposes the following steps to prevent the relocation of data blocks in a peak active state from the high-performance storage layer to the archive storage layer during the retention period: Identify the peak active data blocks located within the damaged storage area; Based on the business priority, data sensitivity, and fault type of the peak active data block, an emergency protection strategy is determined. Perform emergency data processing tasks to copy or migrate affected peak-active data blocks to other high-performance storage areas, or perform encrypted backups; Update the metadata and data access index of the peak active data block, and reactivate the peak active retention period of the peak active data block.

[0075] Specifically, identifying the aforementioned high-activity data blocks located within the damaged storage area involves continuously scanning various storage media in the high-performance storage layer, such as solid-state drives (SSDs) or hard disk drives (HDDs), using the storage system's health monitoring module to detect physical damage, logical errors, bad blocks, abnormally high read / write error rates, and other forms of damage. Once a damaged area is detected, the system cross-references the identifiers of the data blocks stored within these areas with the identifiers of the data blocks currently marked as high-activity, thereby accurately identifying the affected high-activity data blocks.

[0076] The emergency protection strategy, determined based on the business priority, data sensitivity, and fault types of the aforementioned peak-active data blocks, can be understood as the system comprehensively considering multiple dimensions to formulate the most appropriate protection measures. Business priority refers to the importance of the data block to core business processes; for example, critical transaction data typically has the highest priority. Data sensitivity refers to the potential negative impact of data leakage or damage; for example, user personal privacy data or financial data is typically highly sensitive. Fault type refers to the specific manifestation of storage media damage; for example, whether it is a recoverable soft error, an unrecoverable hard error, or a warning that the media's lifespan is about to expire. Based on this information, the system can dynamically choose whether to perform data replication (creating a copy without affecting the source data), data migration (moving data from the damaged area to a healthy area), or encrypted backup (encrypting the data and storing it in a secure location when data migration or replication is not feasible).

[0077] In practical applications, emergency data processing tasks are performed by copying or migrating affected peak-activity data blocks to other high-performance storage areas, or by performing encrypted backups. For example, corresponding operations can be automatically triggered based on a defined emergency protection strategy. If the strategy is data migration, the system will seamlessly transfer peak-activity data blocks from the affected area to other healthy storage nodes or storage media in the high-performance storage tier, ensuring uninterrupted service during the migration process. If the strategy is encrypted backup, the data blocks will be encrypted and backed up to an independent, secure backup storage system to prevent data loss or unauthorized access.

[0078] Furthermore, updating the metadata and data access index of the aforementioned peak-active data blocks and reactivating their peak-active retention period aims to ensure that the data block's status information promptly reflects the latest storage location and protection policy after protection or migration. Metadata updates include the data block's physical storage location, checksum, and version information. Updating the data access index ensures that applications can access the data block through the new path. Reactivating the peak-active retention period means the data block will receive a new protection period during which it will continue to be prevented from being moved from the high-performance storage tier to the archive storage tier, thus ensuring it continues to receive high-priority protection in the new storage environment.

[0079] This application's solution effectively compensates for the shortcomings of basic solutions in extreme storage failure scenarios by introducing a proactive identification and protection mechanism for peak-active data blocks within damaged storage areas. Through this technical solution, this application can significantly improve the robustness and reliability of big data asset storage and retrieval methods for cloud computing. This solution not only prevents the tier degradation of peak-active data blocks, but also proactively identifies and implements emergency protection measures when the high-performance storage layer is damaged, thereby effectively avoiding data loss or service interruption due to storage media failure. This is of great significance for ensuring the continuity and availability of critical business data, especially in cloud computing environments where the complexity of storage infrastructure and potential failure points increase; this solution can provide users with a higher level of data security.

[0080] In some preferred embodiments, a specific example is given below. Suppose that the high-performance storage layer of a cloud computing platform consists of multiple solid-state drive (SSD) arrays. One storage node in one of the SSD arrays reports a continuous increase in read / write errors and bad blocks, indicating that the area is damaged. The system monitoring module will immediately generate a media anomaly warning. At this time, the system will identify data blocks stored on the damaged SSD array that are currently marked as being in a high-activity state, such as real-time transaction log data blocks of a financial institution's customer.

[0081] Specifically, the transaction log data block was assessed as having extremely high business priority and data sensitivity, and the failure type was irrecoverable hardware damage. Based on this, the system automatically determined the emergency protection strategy to be "immediate migration." Subsequently, the system will initiate an emergency data processing task to copy and migrate the affected transaction log data block from the damaged SSD array to another healthy SSD array with sufficient redundancy in the high-performance storage tier. During the migration process, data consistency is strictly verified. After the migration is completed, the metadata (including its new physical storage location) and data access index of the transaction log data block will be updated immediately to ensure that all applications can access it normally through the new path. At the same time, the peak activity retention period of the transaction log data block will be reactivated to ensure that it continues to enjoy the highest priority data protection in the new storage location, preventing it from being adjusted to the archive storage tier during the new retention period, thereby effectively ensuring the continuity and security of financial transaction data.

[0082] In some embodiments described above, a retention period is proposed for data blocks in a peak activity state to prevent them from being moved from the high-performance storage layer to the archive storage layer. However, in practical applications, the access activity of data blocks may change significantly within the set retention period. If the retention period is fixed, it may lead to inflexible resource allocation. For example, if the activity of a data block drops rapidly within the retention period, keeping it in the high-performance storage layer will waste storage resources; conversely, if the activity unexpectedly increases and the retention period is about to end, the data block may be migrated to the archive storage layer prematurely, affecting user experience and business continuity.

[0083] In response, this application further proposes a method for preventing the transfer of data blocks in a peak active state from the high-performance storage layer to the archive storage layer during the retention period, comprising: Continuously monitor the access event information of the data block, the access event information including the identifier of the data block, the access type, and the access time; Calculate the activity change trend of the data block based on the access event information; Identify the match between the activity change trend and the currently set retention period; Based on the activity change trend and the preset adjustment rules, the retention period is dynamically adjusted to obtain the adjusted retention period; Update the metadata of the data block, the metadata including the adjusted retention period; During the adjusted retention period, the data blocks in the peak active state are prevented from being moved from the high-performance storage layer to the archive storage layer.

[0084] Specifically, continuously monitoring the access event information of the data block refers to the system uninterruptedly collecting all access operation records related to the data block. The access event information can be understood as including the data block's unique identifier, the specific access operation type (e.g., read, write, delete), and the precise timestamp of the access operation. This information is the basic data for evaluating the data block's activity. Calculating the activity trend of the data block based on the access event information means that the system uses the collected access event information and statistical analysis methods to periodically calculate the changes in the data block's access frequency, access volume, or access pattern over a period of time.

[0085] For example, it can calculate the number of accesses per unit time, the growth or decline rate of access traffic, or identify periodic changes in access patterns. The purpose is to quantify the dynamic evolution of data block activity. In practical applications, identifying the match between the activity change trend and the currently set retention period means that the system compares and analyzes the calculated activity change trend with the current retention period set for that data block. For example, if the activity shows a significant downward trend while the retention period is still relatively long, there may be a mismatch; if the activity continues to rise while the retention period is about to expire, there may also be a mismatch. The purpose is to determine the reasonableness of the current retention period.

[0086] Furthermore, based on the activity change trend and preset adjustment rules, the retention period is dynamically adjusted. The adjusted retention period refers to the system's operation of extending, shortening, or maintaining the current retention period based on a series of predefined strategies or algorithms combined with the activity change trend. For example, if the activity continues to decline, the retention period can be shortened according to the rules; if the activity continues to rise, the retention period can be extended. The adjustment rules can be based on machine learning models, expert systems, or simple threshold judgment logic. The purpose is to make the retention period flexibly adapt to the actual activity of the data block. Therefore, the metadata of the data block is updated. The metadata containing the adjusted retention period means that after the retention period is dynamically adjusted, the system will immediately update the metadata record corresponding to the data block. The metadata is key information describing the attributes of the data block, including the adjusted retention period, ensuring that all storage management modules can obtain the latest retention period setting. The purpose is to ensure the real-time performance and consistency of the storage management strategy.

[0087] Ultimately, preventing the transfer of peak-active data blocks from the high-performance storage layer to the archive storage layer during the adjusted retention period means that the system will strictly adhere to the updated retention period. During this period, even if the long-term average activity assessment result of the data block may be lower than the standard of the high-performance storage layer, it will not be migrated to the archive storage layer. The purpose is to ensure that peak-active data blocks always enjoy high-performance storage resources during their active period.

[0088] This application's solution effectively addresses the inflexible resource allocation issue that can arise from fixed retention periods in basic solutions by introducing a mechanism for continuously monitoring and dynamically adjusting the retention period based on data block activity trends. Through this technical solution, this application enables dynamic optimization of storage tier management strategies for peak-activity data blocks. Compared to the fixed retention period approach in basic solutions, this application significantly improves storage resource utilization efficiency and system responsiveness by continuously monitoring data block activity trends and dynamically adjusting the retention period accordingly. Specifically, when the actual activity of a data block changes within the retention period, such as a decrease in activity, the system can promptly shorten the retention period, thereby avoiding unnecessary resource consumption and releasing high-performance storage space for other more active data blocks. Conversely, when the activity of a data block remains high, the system can extend the retention period, ensuring stable access to high-performance storage services throughout the peak activity period, effectively preventing premature migration and potential service interruptions due to excessively short retention periods. This dynamic adjustment mechanism enables storage tier management strategies to more accurately match the lifecycle and access patterns of data blocks, thereby maximizing the optimization of storage resource allocation in the cloud computing environment while ensuring service quality.

[0089] In some embodiments described above, a retention period is proposed to prevent data blocks in a peak active state from being moved from the high-performance storage layer to the archive storage layer. However, in real-world cloud computing environments, the transmission and execution of storage tier management instructions may face problems such as network congestion, instruction loss, processing delays, or execution errors, resulting in instructions preventing data block migration not being executed in a timely and reliable manner. If these problems are not addressed, even if the system has identified a data block as being in a peak active state and set a retention period, the data block may still be erroneously migrated to the low-performance archive storage layer due to unreliable instruction transmission or execution, thereby affecting its access performance and business continuity. To address this, this application proposes an optimized instruction transmission and execution mechanism to ensure that storage tier management instructions for peak active data blocks can be reliably transmitted and executed.

[0090] In this regard, this application further proposes the following steps to prevent the transfer of data blocks in a peak active state from the high-performance storage layer to the archive storage layer during the aforementioned retention period: Deploy an independent storage tier management command channel, configure dedicated network bandwidth resources through the independent storage tier management command channel, and adopt a high-priority data transmission protocol; When the system generates an instruction to prevent the data block from being moved from the high-performance storage layer to the archive storage layer, the instruction is transmitted through the independent storage layer management instruction channel, and the instruction is verified in real time to obtain the verification result. Based on the verification result, an instruction execution confirmation mechanism is configured at the receiving end of the independent storage tier management instruction channel. After successfully receiving and processing the instruction, an confirmation signal is returned to the sending end of the independent storage tier management instruction channel. If the confirmation signal is not received within the preset time, the instruction retransmission mechanism is triggered to retransmit the instruction.

[0091] Specifically, deploying an independent storage tier management command channel refers to establishing a dedicated communication path within a cloud computing storage architecture for storage tier management commands (especially commands that prevent data block migration), separate from regular data transmission and control commands. This command channel is configured with dedicated network bandwidth resources, meaning that fixed network capacity is reserved for this channel, unaffected by other data traffic, to ensure low latency and high throughput for command transmission. Simultaneously, this channel employs high-priority data transmission protocols; for example, based on QoS (Quality of Service) mechanisms, packets containing such commands can be marked as having the highest priority, thereby gaining priority access to transmission resources during network congestion and further ensuring timely delivery of commands.

[0092] Specifically, when the system generates an instruction to prevent data blocks from being moved from the high-performance storage layer to the archive storage layer, this instruction will be transmitted through the aforementioned independent storage tier management instruction channel. During transmission, the instruction will be verified in real time. Real-time verification can be understood as instantly verifying the integrity and correctness of the instruction content, as well as the legitimacy of the sender's identity. This can be achieved through methods such as calculating the instruction's checksum, digital signature, or encryption verification, to prevent the instruction from being tampered with or corrupted during transmission and to ensure the reliability of the instruction's source.

[0093] In practical applications, based on the verification results, an instruction execution confirmation mechanism is configured at the receiving end of an independent storage tier management instruction channel. This mechanism aims to ensure that after an instruction is successfully received and processed at the receiving end, it returns an confirmation signal to the sending end. The return of the confirmation signal indicates that the instruction has reached the target storage node and that the corresponding operation to prevent data block migration has begun or been completed. For example, after successfully updating the metadata of a data block or adjusting the storage policy, the receiving end generates and sends a confirmation message with the instruction ID and execution status.

[0094] Furthermore, if the aforementioned confirmation signal is not received within a preset time, a command retransmission mechanism will be triggered. The purpose of this mechanism is to automatically resend the original command when an anomaly occurs during command transmission or execution (such as network packet loss, receiver failure, or loss of confirmation signal). The preset time can be dynamically configured based on factors such as network latency, system load, and command importance to balance the timeliness of retransmission with the consumption of system resources. Retransmission can significantly improve the delivery rate and execution success rate of critical commands.

[0095] This application's solution fundamentally improves the reliability and timeliness of transmitting instructions that prevent data block migration by deploying an independent storage-level management command channel and configuring it with dedicated network bandwidth resources and high-priority data transmission protocols. Due to the isolation of the command transmission path and resource guarantees, critical commands can bypass the competition of regular data traffic and reach the target storage node first. Simultaneously, real-time verification of commands ensures their integrity and legitimacy during transmission, preventing erroneous operations caused by command corruption or forgery. Furthermore, the introduction of a command execution confirmation mechanism allows the sender to clearly know whether the command has been successfully processed by the receiver, forming a closed-loop feedback control. When the confirmation signal is not received within a preset time, the command retransmission mechanism is triggered, further mitigating the potential failure risk of a single transmission. This ensures that even in complex cloud environments, commands preventing peak-activity data blocks from being moved from the high-performance storage layer to the archive storage layer can be executed reliably and accurately, effectively preventing performance degradation or service interruption caused by erroneous migration of critical data.

[0096] Through the above technical solutions, this application significantly improves the reliability and execution efficiency of storage layer management commands for peak-active data blocks in big data asset storage and retrieval methods for cloud computing. Specifically, independent command channels, dedicated network bandwidth, and high-priority protocols ensure low latency and high success rate in command transmission, effectively preventing command loss or delay in complex network environments. Real-time verification mechanisms guarantee the integrity and security of commands, preventing the execution of erroneous commands. Command execution confirmation mechanisms and command retransmission mechanisms together construct a robust command delivery guarantee system, ensuring that critical migration prevention commands are ultimately successfully executed even in the event of partial component failures or network anomalies. Therefore, this application can more effectively maintain peak-active data blocks in the high-performance storage layer, ensuring their continuous high access performance and business continuity, and significantly reducing business risks and potential losses caused by misadjustments to the storage layer.

[0097] Traditional methods for storing and retrieving big data assets in cloud computing may face challenges in maintaining data blocks in high-performance storage layers during peak periods, especially in multi-tenant environments with shared resources. Specifically, instructions to prevent data blocks from being moved from high-performance storage to archive storage layers may conflict with other tenants' data migration and resource allocation operations, or cause delays or failures due to resource contention. If these issues are not addressed, peak-performing data blocks may not be protected in a timely manner or may even be incorrectly downgraded, impacting the performance and availability of critical business operations.

[0098] To address this issue, this application proposes an optimization scheme designed to ensure that instructions preventing data blocks from being moved from the high-performance storage layer to the archive storage layer during peak activity periods can be executed reliably and efficiently in a multi-tenant cloud computing environment. This scheme effectively resolves instruction execution conflicts and reliability issues by introducing fine-grained resource management, instruction priority scheduling, and dedicated communication mechanisms.

[0099] In some embodiments described above in this application, a method is proposed to prevent the transfer of data blocks in a peak active state from the high-performance storage layer to the archive storage layer during the retention period, specifically including: Allocate independent storage resource quotas to each tenant; To prevent instructions from moving the data block from the high-performance storage layer to the archive storage layer, a global highest execution priority is set; When the system receives an instruction to prevent the data block from being moved from the high-performance storage layer to the archive storage layer, the instruction is routed to a dedicated instruction processing queue. This dedicated instruction processing queue is independent of the regular data operation queues of other tenants and has priority scheduling rights. Deploy a policy arbitration module on each storage node so that the policy arbitration module can monitor the resource usage and instruction execution status among tenants in real time; When the policy arbitration module detects that the instruction to prevent the data block from being moved from the high-performance storage layer to the archive storage layer conflicts with the data migration or resource allocation operation of a high-priority tenant, it forcibly interrupts or delays the operation of other tenants. A dedicated communication channel is provided for the instructions, the dedicated communication channel is configured with independent network bandwidth and adopts a protocol to ensure the reliability of instruction transmission.

[0100] Specifically, allocating independent storage resource quotas to each tenant means reserving and isolating available storage capacity, IOPS (input / output operations per second), and network bandwidth for different tenants in a cloud computing environment. This aims to ensure that each tenant's basic storage needs are met even in a multi-tenant shared infrastructure environment, and to provide a basis for subsequent priority management.

[0101] Furthermore, a globally highest execution priority is set for the instruction that prevents the data block from being moved from the high-performance storage layer to the archive storage layer. This means that the instruction is given the highest processing authority in the entire storage management system. The purpose is to ensure that the instruction can be scheduled and executed with priority over all other storage operations (such as data migration, backup, deletion, etc.), thereby avoiding the erroneous degradation of peak active data blocks due to resource contention or scheduling delays.

[0102] Specifically, when the system receives an instruction to prevent the data block from being moved from the high-performance storage layer to the archive storage layer, it routes the instruction to a dedicated instruction processing queue. This dedicated instruction processing queue is independent of other tenants' regular data operation queues and has priority scheduling rights. This dedicated instruction processing queue is a buffer specifically designed for high-priority instructions, isolated from queues handling other tenants' or low-priority operations. This isolation and priority scheduling mechanism ensures that the instruction preventing migration is not blocked or delayed by other regular operations, thereby improving its response speed and execution determinism.

[0103] In practical applications, a policy arbitration module is deployed on each storage node. This module monitors resource usage and instruction execution status among tenants in real time. The policy arbitration module is an intelligent decision-making unit deployed on each storage node, responsible for collecting and analyzing current storage system operational data, including each tenant's resource consumption, currently executing tasks, and instruction priorities. Its purpose is to provide real-time decision support to address complex resource contention scenarios.

[0104] Furthermore, when the policy arbitration module detects a conflict between the instruction to prevent the data block from being moved from the high-performance storage layer to the archive storage layer and a data migration or resource allocation operation of a high-priority tenant, it forcibly interrupts or delays the operations of other tenants. This means that in extreme cases, to ensure the storage tier of peak-active data blocks, the system can proactively intervene and adjust the non-critical operations of other tenants. This forced interruption or delay mechanism is the ultimate guarantee to ensure that the highest-priority instructions are executed. Although it may have a temporary impact on other tenants, its core purpose is to maintain the critical business continuity of the overall system.

[0105] In a preferred embodiment, a dedicated communication channel is provided for the instructions. This dedicated communication channel is configured with independent network bandwidth and employs a protocol that ensures the reliability of instruction transmission. A dedicated communication channel refers to a network path that is physically or logically isolated from other data traffic; it is specifically used to transmit instructions that prevent data block migration. By configuring independent network bandwidth, the impact of network congestion on instruction transmission can be avoided. Employing a protocol that ensures the reliability of instruction transmission (e.g., a protocol with retransmission, acknowledgment, and error checking mechanisms) ensures that instructions are not lost or corrupted during transmission, thereby further improving the reliability of instruction execution.

[0106] This application's solution effectively addresses the execution conflicts and reliability issues that may arise when instructions preventing peak-active data blocks from being moved from the high-performance storage layer to the archive storage layer in a shared cloud computing environment by introducing a fine-grained resource management and instruction priority scheduling mechanism. Through this technical solution, the application significantly improves the execution efficiency and reliability of peak-active data block storage layer management strategies in a multi-tenant cloud computing environment. Compared to basic solutions, this application effectively solves the problem of critical instructions being blocked, delayed, or conflicted by other operations in a shared resource environment by introducing tenant resource quotas, globally highest instruction priority, dedicated instruction processing queues, policy arbitration modules, and dedicated communication channels. This ensures that peak-active data blocks remain stably in the high-performance storage layer within the retention period, avoiding unexpected degradation due to resource contention or system overload, thereby guaranteeing high availability and access performance of critical business data, improving user experience, and enhancing system stability.

[0107] In some embodiments described above, this application proposes preventing data blocks from being moved from the high-performance storage layer to the archive storage layer when they are in a peak active state, in order to ensure the availability of frequently accessed data. However, in its implementation, the basic scheme mainly focuses on managing data levels based on access activity, without fully considering the health status of the storage media itself within the high-performance storage layer. If the storage media in the high-performance storage layer fails or its performance degrades, even if the data block is marked as peak active and its migration is prevented, its data integrity, availability, and access performance may still be threatened. If the above problems are not addressed, important data may be at risk of being lost, or the expected quality of service may not be provided during peak periods. To address this, this application further proposes a scheme to proactively protect peak active data blocks by continuously monitoring the health status of the storage media and responding to anomaly alerts during the retention period.

[0108] The aforementioned prevention of moving data blocks in peak activity states from the high-performance storage layer to the archive storage layer specifically includes: The system continuously acquires the operating status information of each storage medium within the high-performance storage layer. The operating status information includes the read / write error rate, number of bad blocks, remaining lifetime, and temperature of the storage medium. Based on the operating status information, the health status of the storage medium is evaluated. When the health status is lower than a preset health threshold, a media anomaly warning is generated. In response to the media anomaly warning, the peak active data block is identified, the data protection level is determined based on the service priority and data sensitivity of the peak active data block, and emergency handling operations are determined based on the type of media anomaly warning. Perform the emergency handling operations to copy or migrate the affected peak active data blocks to other healthy storage media, or perform data backup; Update the metadata and data access index of the peak active data block, and reactivate the retention period.

[0109] Specifically, the operational status information refers to a series of indicators used to describe the current working condition of the storage medium. For example, the read / write error rate reflects the success rate and stability of data read / write operations on the storage medium; the number of bad blocks indicates the number of unusable storage units on the storage medium; remaining lifetime predicts the expected service life of the storage medium; and temperature reflects the operating environment of the storage medium, with excessively high temperatures potentially indicating hardware failure risks. This information is continuously collected to monitor the health status of the storage medium in real time. The health status can be understood as the overall reliability and performance level of the storage medium, comprehensively evaluated based on the aforementioned operational status information. The preset health threshold is a baseline value set by the system administrator or a predefined policy. When the health status of the storage medium falls below this threshold, it indicates a potential failure or performance degradation, requiring attention. In practical applications, the media anomaly warning refers to a notification or event automatically triggered by the system when the health status assessment result of the storage medium falls below the preset health threshold. This warning aims to promptly alert the system management module or automated processing flow to the existence of an abnormal situation with the storage medium.

[0110] Furthermore, the data protection level is determined comprehensively based on the business priority and data sensitivity of the peak-active data block. For example, for critical data of core business operations, the data protection level may be set to the highest, requiring immediate data replication or migration; while for sensitive user data, encrypted backup may be necessary. The emergency handling operations are specific measures taken in response to different types of media anomaly warnings (e.g., increased read / write error rate, increased number of bad blocks, excessively high temperature, or low remaining lifetime) and data protection levels, aiming to maximize data security and availability. Therefore, after executing the emergency handling operations, the metadata and data access index of the peak-active data block will be updated to reflect the new storage location or backup status of the data block. Simultaneously, the retention period will be reactivated to ensure that the peak-active data block still enjoys high-priority protection on the new storage medium, avoiding erroneous degradation.

[0111] This application's solution addresses the insufficient data protection in cases of storage media anomalies in basic solutions by introducing continuous monitoring of the operational status of storage media within the high-performance storage layer. Through this technical solution, this application significantly improves the storage reliability and availability of big data assets in cloud computing environments. Compared to basic solutions that rely solely on hierarchical management based on access activity, this application proactively monitors the health of storage media, enabling early warning and response to potential hardware failures, thus allowing for preventative measures before or in the early stages of a failure. This ensures that peak-active data blocks receive timely and effective protection even when facing media anomalies within the high-performance storage layer, avoiding the risk of data loss or service interruption. Furthermore, determining the data protection level based on the business priority and sensitivity of data blocks and executing corresponding emergency handling operations ensures that critical data receives the highest level of protection, further enhancing the resilience of data assets. The mechanism for reactivating the retention period guarantees that the protection strategy for peak-active states continues after data blocks are migrated or backed up, thereby continuously optimizing the storage and retrieval efficiency and security of big data assets.

[0112] As a specific implementation, assuming a cloud computing environment, the high-performance storage tier consists of multiple solid-state drives (SSDs). The system continuously acquires the operational status information of each SSD through the SMART (Self-Monitoring, Analysis and Reporting Technology) tool, including its read / write error rate, the number of allocated bad blocks, the percentage of remaining lifespan, and internal temperature. The system sets a health threshold; for example, when the read / write error rate of any SSD exceeds 0.01%, or the number of bad blocks exceeds 0.1% of the total capacity, or the remaining lifespan is less than 20%, or the temperature remains above 60 degrees Celsius, a media anomaly warning is generated. For example, when the system detects a sudden increase in the read / write error rate of a certain SSD, it triggers a media anomaly warning. At this time, the storage tier management module will immediately identify all peak-active data blocks stored on the abnormal SSD. For a data block marked as "core business critical data" and "highly sensitive," the system will determine its data protection level to the highest level based on its business priority and data sensitivity. The corresponding emergency handling operation is determined to be "immediate replication and migration." The system then initiates a data replication task, completely copying the data block to another healthy SSD in the high-performance storage tier. After replication, the original data block is deleted from the faulty SSD. Simultaneously, the data block's metadata and data access index are updated to ensure subsequent accesses point to the new storage location. Furthermore, the data block's retention period is reactivated, ensuring it remains considered high-activity data on the new SSD and continues to enjoy high-performance storage benefits until the new retention period ends and its activity decreases. In this way, even if the underlying storage media fails, the availability and integrity of high-activity data blocks are effectively guaranteed.

[0113] Traditional cloud-based big data asset storage and retrieval methods typically focus on policy control within a single storage environment when preventing peak-activity data blocks from being moved from the high-performance storage tier to the archive storage tier. However, in real-world cloud environments, big data assets may be distributed across multiple cloud service providers' storage environments to achieve high availability, disaster recovery, or compliance requirements. In this multi-cloud deployment scenario, simply preventing the hierarchical adjustment of data blocks cannot fully guarantee the consistency and integrity of the peak-activity data block across different cloud service providers within the retention period. Inconsistencies in data replicas between different cloud service providers may lead to data access errors, business logic corruption, or even data loss, thereby impacting the reliability and availability of big data assets.

[0114] In response, this application further proposes a method for storing and retrieving big data assets for cloud computing, wherein, during the aforementioned retention period, preventing the transfer of data blocks in peak active states from the high-performance storage layer to the archive storage layer includes: Deploy a data consistency coordinator, which is configured with a cross-cloud data transmission channel. The cross-cloud data transmission channel adopts a data sharding transmission method, which shards the data block and performs independent verification and encryption on each data shard. A data proxy module is deployed in the storage environment of each cloud service provider. The data proxy module identifies and adapts to the storage interfaces and data processing logic of different cloud service providers. The data proxy module obtains the metadata and data content of the peak active data blocks from the storage environments of different cloud service providers. The metadata includes the checksum, version information and timestamp of the data blocks. Based on the metadata and data content of the peak active data blocks, perform cross-cloud data consistency verification, which includes comparing the checksums, version information and timestamps of data blocks between different cloud service providers; When the cross-cloud data consistency check finds data inconsistency, the consistent data version is determined according to the preset conflict resolution strategy. The data broker module synchronizes the consistent data version to the storage environment of the cloud service provider where inconsistencies exist, and updates the corresponding metadata. During the data synchronization process, the data consistency coordinator continuously monitors the transmission status and data integrity of the cross-cloud data transmission channel, and triggers data retransmission operations when transmission is abnormal or data is corrupted.

[0115] Specifically, the data consistency coordinator is a core component whose main function is to manage and coordinate data consistency across multiple cloud service providers. This coordinator is configured with a cross-cloud data transmission channel dedicated to the efficient and secure transfer of data between different cloud environments. To optimize transmission efficiency and security, this channel employs a data fragmentation transmission method, breaking down the original data block into multiple smaller data fragments. Each data fragment can be transmitted, verified, and encrypted independently. Independent checksums ensure that each fragment is not tampered with or corrupted during transmission, while independent encryption enhances data security during cross-cloud transmission.

[0116] The data broker module is deployed in the storage environment of each cloud service provider. These data broker modules have the ability to identify and adapt to the storage interfaces and data processing logic of different cloud service providers. This means that regardless of the specific API and data model of the underlying cloud storage service (such as Amazon S3, Azure Blob Storage, or Google Cloud Storage), the data broker module can interact effectively, thereby achieving unified management and operation of peak-active data blocks.

[0117] In practical applications, the data broker module can retrieve metadata and data content of peak-active data blocks from storage environments of different cloud service providers. This metadata contains key information describing the data block, typically including the block's checksum, version information, and timestamp. The checksum verifies data integrity, the version information tracks the block's modification history, and the timestamp records the last modification time.

[0118] Furthermore, based on the metadata and data content of the acquired peak active data blocks, cross-cloud data consistency verification is performed. The core of this verification process is comparing the checksums, version information, and timestamps of data blocks from different cloud service providers. By comparing these key pieces of information, discrepancies between data replicas in different cloud environments can be effectively detected.

[0119] When cross-cloud data consistency checks detect inconsistencies, the system determines a consistent data version based on a pre-defined conflict resolution strategy. Conflict resolution strategies can include various methods, such as selecting the latest version, selecting the primary replica version, or using a majority voting mechanism. Once a consistent data version is determined, the data broker module is responsible for synchronizing this version to the storage environments of the cloud service providers where inconsistencies exist and updating the corresponding metadata to ensure that the data content and metadata of all replicas remain synchronized.

[0120] Furthermore, the data consistency coordinator continuously monitors the transmission status and data integrity of the cross-cloud data transmission channel throughout the entire data synchronization process. This includes monitoring the transmission rate, packet loss rate, and data fragmentation verification results. Once a transmission anomaly or data corruption is detected, the data consistency coordinator will immediately trigger a data retransmission operation to ensure that the data is transmitted completely and accurately and ultimately achieves consistency.

[0121] This application's solution effectively addresses data consistency issues that may arise in multi-cloud environments when peak-active data blocks are prevented from being moved from the high-performance storage layer to the archive storage layer. By introducing a data consistency coordinator and a data proxy module, this application significantly improves the reliability and robustness of big data asset storage and retrieval methods for cloud computing. Especially in multi-cloud deployment scenarios, this solution effectively solves the problems of cross-cloud data inconsistency and data integrity loss that traditional methods may encounter when preventing data hierarchical adjustments. Specifically, data fragmentation transmission, independent verification, and encryption mechanisms greatly enhance the security and efficiency of cross-cloud data transmission; cross-cloud data consistency verification and conflict resolution strategies ensure that replicas of peak-active data blocks remain consistent across different cloud service providers, avoiding business interruptions or data errors caused by data inconsistency. Furthermore, continuous monitoring of transmission status and data integrity, along with an anomaly retransmission mechanism, further guarantees the reliability of data synchronization. Therefore, the solution proposed in this application can not only effectively prevent the hierarchical adjustment of peak active data blocks, but also ensure the high availability, integrity and consistency of data in complex distributed multi-cloud environments, thereby providing users with more stable and reliable big data asset storage and retrieval services.

[0122] As a specific implementation, suppose an enterprise deploys its core big data assets in storage environments of two different cloud service providers, A and B, to achieve off-site disaster recovery and load balancing. When a critical data block 100 is identified as being in a peak active state and a retention period is set, the method of this application will be initiated. First, a data consistency coordinator 200 will be deployed and activated, and its configured cross-cloud data transmission channel 300 will be ready. Data proxy module A and data proxy module B are deployed in the storage environments of cloud service provider A and cloud service provider B, respectively. During the retention period, the data consistency coordinator 200 will periodically instruct data proxy modules A and B to retrieve the metadata (including checksum, version information, and timestamp) and data content of data block 100 from their respective storage environments. For example, data proxy module A retrieves the metadata and content of copy A of data block 100 from cloud service provider A, and data proxy module B retrieves the metadata and content of copy B of data block 100 from cloud service provider B. Subsequently, the data consistency coordinator 200 performs cross-cloud data consistency verification, comparing the checksums, version information, and timestamps of replicas A and B. If it is found that replica A's version is V2.0, while replica B's version is still V1.0, it indicates a data inconsistency. At this point, according to the preset "select latest version" conflict resolution strategy, the data consistency coordinator 200 determines V2.0 as the consistent data version. Next, the data consistency coordinator 200 instructs the data broker module B to synchronize replica A (V2.0) to the cloud service provider B's storage environment through the cross-cloud data transmission channel 300, and updates replica B's metadata to V2.0. During the synchronization process, the data consistency coordinator 200 continuously monitors the transmission status and data integrity of the transmission channel 300. For example, if a data shard transmission failure is detected, a retransmission will be triggered. Through this series of operations, even if there are copies of data block 100 between different cloud service providers, its data content and metadata can be kept highly consistent throughout the retention period, thereby effectively preventing data inconsistency problems caused by the complexity of multi-cloud environments and ensuring the stability and reliability of data block 100 during peak periods.

[0123] refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of a big data asset storage and retrieval system for cloud computing provided in an embodiment of the present invention, including: The input terminal is used to obtain access event information of data blocks, the access event information including the identifier of the data block, access type and access time; based on the access event information, the instantaneous access intensity of the data block is evaluated within a preset short time window, and when the instantaneous access intensity reaches a preset intensity threshold, an access peak indication is generated. The response end is used to respond to the access peak indication by initiating an evaluation process that samples access events over a short period of time and assesses the activity level within a preset statistical window; when the results of the evaluation process continuously meet the preset activity level conditions, the data block is marked as a peak activity state, and a retention period is set for the data block in the peak activity state. The adjustment terminal is used to prevent the data block in the peak activity state from being moved from the high-performance storage layer to the archive storage layer during the retention period; after the retention period ends, if the data block does not trigger the access peak indication again, the storage level of the data block is evaluated based on the long-term average access activity.

[0124] This system, through the collaborative work of structured components, can effectively identify and respond to instantaneous high-intensity access patterns of data blocks, thereby preventing data blocks from being incorrectly migrated to low-performance storage layers during active periods. This significantly improves the storage and retrieval efficiency of big data assets in a cloud computing environment. The input end is responsible for the initial data collection and instantaneous activity assessment, the response end further refines the evaluation and marks peak activity states, and the adjustment end ensures that the storage level of data blocks is not incorrectly downgraded during critical periods, ultimately achieving intelligent optimization and allocation of storage resources.

[0125] The system proposed in this application aims to optimize the storage and retrieval efficiency of big data assets in a cloud computing environment, especially when facing instantaneous high-intensity access patterns. The implementation methods of its main technical features will be described in detail below.

[0126] Specifically, the input terminal is used to acquire access event information of data blocks and, based on the access event information, evaluate the instantaneous access intensity of the data blocks within a preset short time window. When the instantaneous access intensity reaches a preset intensity threshold, an access peak indication is generated. The specific methods for acquiring access event information of data blocks, evaluating instantaneous access intensity, and generating access peak indications have been described in the above embodiments and will not be repeated here. It is important to emphasize that the input terminal can be implemented as an independent data acquisition and preprocessing module, which can be configured with dedicated sensors or interfaces for real-time capture of I / O operation logs in the storage system. For example, the input terminal can be a lightweight agent program deployed on each storage node, responsible for listening to file system events or network storage protocol traffic and sending the captured access event information to a central processing unit. Alternatively, the input terminal can be integrated into the underlying storage management software, acquiring access event information by intercepting system calls and using built-in computing logic to locally evaluate the instantaneous access intensity.

[0127] Furthermore, the response end is used to respond to the access peak indication by initiating an evaluation process that samples access events over a short period and assesses activity levels within a preset statistical window. When the results of the evaluation process continuously meet preset activity conditions, the data block is marked as being in a peak active state, and a retention period is set for the data block in the peak active state. The specific methods for responding to the access peak indication, initiating the evaluation process, marking the peak active state, and setting the retention period have been described in the above embodiments, and will not be repeated here. It should be emphasized that the response end can be implemented as an independent activity evaluation service, which receives the access peak indication from the input end and initiates more refined evaluation logic. For example, the response end can be an asynchronous processing unit based on a message queue. When it receives the access peak indication, it reads the data block identifier from the message queue and initiates a high-frequency data sampling task. As another implementation, the response end can be a centralized decision engine that maintains a data block status table and dynamically updates the status and retention period of the data block based on the instantaneous access data provided by the input end and the preset activity conditions.

[0128] Furthermore, the adjustment terminal is used to prevent the data block in the peak activity state from being moved from the high-performance storage layer to the archive storage layer during the retention period; after the retention period ends, if the data block does not trigger the access peak indication again, the storage level of the data block is evaluated based on the long-term average access activity. The specific methods for preventing data block migration during the retention period and re-evaluating the storage level after the retention period have been described in the above embodiments, and will not be repeated here. It should be emphasized that the adjustment terminal can be implemented as a storage level management policy execution module, which interacts with the underlying data migration service of the storage system. For example, the adjustment terminal can be a high-priority data migration arbitrator. When it receives a peak activity state flag from the response terminal, it sets a "migration lock" flag for the corresponding data block in its internal policy table, thereby preventing any automatic or manual data degradation operation. As another implementation, the adjustment terminal can be a distributed policy agent deployed on each storage controller, which monitors the metadata of the data block in real time and refuses to execute any instruction to remove the data block from the high-performance storage layer when it finds that the data block is in a peak activity state and within the retention period.

[0129] The big data asset storage and retrieval system proposed in this application for cloud computing aims to solve the problems of data misjudgment and performance degradation caused by inaccurate activity judgment when traditional data hierarchical storage strategies face instantaneous high-intensity access patterns.

[0130] Traditional data tiered storage systems typically rely on average access frequency over long periods to determine data "hot" or "cold" status, such as daily or weekly statistics. While this approach performs well with data exhibiting consistent and stable access patterns, its limitations become apparent when faced with scenarios like microservices or automated tasks that experience high-intensity access to specific data within extremely short timeframes. Due to the excessively long statistical periods, instantaneous peak access is averaged out, causing the system to incorrectly classify actually active data as "cold" and migrate it to a low-performance archive storage layer. When the next peak access arrives, the data has to be read from the slow-responding archive storage, leading to severe performance bottlenecks, prolonged service response times, and a negative impact on user experience.

[0131] The core innovation of this application lies in the introduction of a structured input, response, and adjustment mechanism to achieve an evaluation mechanism for the "instantaneous access intensity" of data blocks and a strategy for maintaining the "peak activity state." Unlike existing systems that rely solely on long-term average activity, the input of this application can evaluate the instantaneous access intensity of the data block within a preset short time window, enabling timely detection of explosive access behavior of the data block within a very short period. When the instantaneous access intensity reaches a preset intensity threshold, the system immediately generates an access peak indication, which is equivalent to issuing an early warning for potentially active data.

[0132] Furthermore, upon responding to an access peak indication, the response end of this application initiates an evaluation process that samples access events over a short period and assesses activity levels within a preset statistical window. This refined evaluation process confirms the instantaneous peak, avoiding misjudgments caused by accidental access. Only when the evaluation results consistently meet the preset activity conditions will the data block be officially marked as being in a peak activity state, and a retention period will be set for it.

[0133] Most importantly, the adjustment mechanism of this application can prevent data blocks in a peak active state from being moved from the high-performance storage layer to the archive storage layer during the retention period. This mandatory blocking mechanism is key to solving the pain points of existing technologies. It ensures that even after the instantaneous peak period ends, data blocks can continue to remain in the high-performance storage layer for a preset period of time, thereby effectively avoiding the problem of erroneous degradation due to a temporary drop in activity, leading to a decrease in subsequent access performance. Compared to the strategy in existing systems that immediately migrates data once it is determined to be "cold," the retention period mechanism of this application provides a buffer period, greatly improving the stability and reliability of data access.

[0134] After the retention period ends, if the data block does not trigger the access peak indication again, the adjustment terminal will evaluate the storage tier of the data block based on the long-term average access activity. This reflects the flexibility and resource optimization capabilities of this application, ensuring performance while also controlling storage costs.

[0135] In summary, by introducing functional modules such as input, response, and adjustment, this application can effectively identify and respond to the instantaneous high-intensity access patterns of data blocks, thereby preventing data blocks from being mistakenly migrated to low-performance storage layers during active periods. This significantly improves the storage and retrieval efficiency of big data assets in a cloud computing environment and solves the performance bottleneck problem caused by inaccurate data activity judgment in existing technologies, demonstrating significant progress and practical value.

[0136] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for storing and retrieving big data assets for cloud computing, characterized in that, include: Obtain access event information for a data block, the access event information including the identifier of the data block, access type, and access time; Based on the access event information, the instantaneous access intensity of the data block is evaluated within a preset short time window. When the instantaneous access intensity reaches a preset intensity threshold, an access peak indication is generated. In response to the access peak indication, an evaluation process is initiated to sample access events over a short period of time and assess the level of activity within a preset statistical window; When the results of the evaluation process continuously meet the preset activity conditions, the data block is marked as a peak activity state, and a retention period is set for the data block in the peak activity state; During the retention period, prevent the data blocks in the peak active state from being moved from the high-performance storage layer to the archive storage layer; If the data block does not trigger the access peak indication again after the retention period expires, the storage level of the data block is evaluated based on the long-term average access activity.

2. The method for storing and retrieving big data assets for cloud computing according to claim 1, characterized in that, In response to the access peak indication, the long-term average access frequency statistics for the data block are paused, and an evaluation process that samples access events over a short period and assesses activity levels within a preset statistical window is initiated, including: Within each storage node of the high-performance storage layer, a data block access counter is configured to record the number of times the data block is accessed. When the number of accesses recorded by the data block access counter reaches a preset instantaneous peak threshold, the local metadata of the data block is updated, and the data block is marked as a peak active state. The status update callback mechanism is triggered to notify the local hot / cold data judgment module. The local hot / cold data judgment module assigns the peak activity status of the data block to the highest priority and adjusts the storage level management strategy of the data block. A retention period is set for the data blocks in the peak active state, and during the retention period, the data blocks in the peak active state are ensured to remain in the high-performance storage layer; After the retention period ends, the activity of the data block is reassessed.

3. The method for storing and retrieving big data assets for cloud computing according to claim 1, characterized in that, During the retention period, preventing the data blocks in the peak active state from being moved from the high-performance storage layer to the archive storage layer includes: The high-performance storage layer is divided into multiple virtual storage partitions, and each virtual storage partition is bound to one or a set of data. Predict the capacity requirements of the data set within each of the virtual storage partitions, and reserve physical capacity for each of the virtual storage partitions; Place the data blocks in the peak active state into the physical resource range corresponding to the bound virtual storage partition; Resource isolation is performed on each virtual storage partition. When the virtual storage partition is in the predicted peak period, a cross-partition resource scheduling and priority arbitration mechanism is initiated to prevent the data blocks in the peak active state from being adjusted from the high-performance storage layer to the archive storage layer.

4. The method for storing and retrieving big data assets for cloud computing according to claim 1, characterized in that, During the retention period, preventing the data blocks in the peak active state from being moved from the high-performance storage layer to the archive storage layer includes: Identify the peak active data blocks located within the damaged storage area; Based on the business priority, data sensitivity, and fault type of the peak active data block, an emergency protection strategy is determined. Perform emergency data processing tasks to copy or migrate affected peak-active data blocks to other high-performance storage areas, or perform encrypted backups; Update the metadata and data access index of the peak active data block, and reactivate the peak active retention period of the peak active data block.

5. A method for storing and retrieving big data assets for cloud computing according to claim 1, characterized in that, During the retention period, preventing the data blocks in the peak active state from being moved from the high-performance storage layer to the archive storage layer includes: Continuously monitor the access event information of the data block, the access event information including the identifier of the data block, the access type, and the access time; Calculate the activity change trend of the data block based on the access event information; Identify the match between the activity change trend and the currently set retention period; Based on the activity change trend and the preset adjustment rules, the retention period is dynamically adjusted to obtain the adjusted retention period; Update the metadata of the data block, the metadata including the adjusted retention period; During the adjusted retention period, the data blocks in the peak active state are prevented from being moved from the high-performance storage layer to the archive storage layer.

6. The method for storing and retrieving big data assets for cloud computing according to claim 1, characterized in that, During the retention period, preventing the data blocks in the peak active state from being moved from the high-performance storage layer to the archive storage layer includes: Deploy an independent storage tier management command channel, configure dedicated network bandwidth resources through the independent storage tier management command channel, and adopt a high-priority data transmission protocol; When the system generates an instruction to prevent the data block from being moved from the high-performance storage layer to the archive storage layer, the instruction is transmitted through the independent storage layer management instruction channel, and the instruction is verified in real time to obtain the verification result. Based on the verification result, an instruction execution confirmation mechanism is configured at the receiving end of the independent storage tier management instruction channel. After successfully receiving and processing the instruction, an confirmation signal is returned to the sending end of the independent storage tier management instruction channel. If the confirmation signal is not received within the preset time, the instruction retransmission mechanism is triggered to retransmit the instruction.

7. A method for storing and retrieving big data assets for cloud computing according to claim 1, characterized in that, During the retention period, preventing the data blocks in the peak active state from being moved from the high-performance storage layer to the archive storage layer includes: Allocate independent storage resource quotas to each tenant; To prevent instructions from moving the data block from the high-performance storage layer to the archive storage layer, a global highest execution priority is set; When the system receives an instruction to prevent the data block from being moved from the high-performance storage layer to the archive storage layer, the instruction is routed to a dedicated instruction processing queue. This dedicated instruction processing queue is independent of other tenants' regular data operation queues and has priority scheduling rights. Deploy a policy arbitration module on each storage node so that the policy arbitration module can monitor the resource usage and instruction execution status among tenants in real time; When the policy arbitration module detects that the instruction to prevent the data block from being moved from the high-performance storage layer to the archive storage layer conflicts with the data migration or resource allocation operation of a high-priority tenant, it forcibly interrupts or delays the operation of other tenants. A dedicated communication channel is provided for the instructions, the dedicated communication channel is configured with independent network bandwidth and adopts a protocol to ensure the reliability of instruction transmission.

8. A method for storing and retrieving big data assets for cloud computing according to claim 1, characterized in that, During the retention period, preventing the data blocks in the peak active state from being moved from the high-performance storage layer to the archive storage layer includes: The system continuously acquires the operating status information of each storage medium within the high-performance storage layer. The operating status information includes the read / write error rate, number of bad blocks, remaining lifetime, and temperature of the storage medium. Based on the operating status information, the health status of the storage medium is evaluated. When the health status is lower than a preset health threshold, a media anomaly warning is generated. In response to the media anomaly warning, the peak active data block is identified, the data protection level is determined based on the service priority and data sensitivity of the peak active data block, and emergency handling operations are determined based on the type of media anomaly warning. Perform the emergency handling operations to copy or migrate the affected peak active data blocks to other healthy storage media, or perform data backup; Update the metadata and data access index of the peak active data block, and reactivate the retention period.

9. A method for storing and retrieving big data assets for cloud computing according to claim 1, characterized in that, During the retention period, preventing the data blocks in the peak active state from being moved from the high-performance storage layer to the archive storage layer includes: Deploy a data consistency coordinator, which is configured with a cross-cloud data transmission channel. The cross-cloud data transmission channel adopts a data sharding transmission method, which shards the data block and performs independent verification and encryption on each data shard. A data proxy module is deployed in the storage environment of each cloud service provider. The data proxy module identifies and adapts to the storage interfaces and data processing logic of different cloud service providers. The data proxy module obtains the metadata and data content of the peak active data blocks from the storage environments of different cloud service providers. The metadata includes the checksum, version information and timestamp of the data blocks. Based on the metadata and data content of the peak active data blocks, perform cross-cloud data consistency verification, which includes comparing the checksums, version information and timestamps of data blocks between different cloud service providers; When the cross-cloud data consistency check finds data inconsistency, the consistent data version is determined according to the preset conflict resolution strategy. The data broker module synchronizes the consistent data version to the storage environment of the cloud service provider where inconsistencies exist, and updates the corresponding metadata. During the data synchronization process, the data consistency coordinator continuously monitors the transmission status and data integrity of the cross-cloud data transmission channel, and triggers data retransmission operations when transmission is abnormal or data is corrupted.

10. A big data asset storage and retrieval system for cloud computing, characterized in that, include: The input terminal is used to obtain access event information of the data block, the access event information including the identifier of the data block, the access type, and the access time; Based on the access event information, the instantaneous access intensity of the data block is evaluated within a preset short time window. When the instantaneous access intensity reaches a preset intensity threshold, an access peak indication is generated. The response end is used to respond to the access peak indication and initiate an evaluation process that samples access events over a short period of time and assesses the level of activity within a preset statistical window; When the results of the evaluation process continuously meet the preset activity conditions, the data block is marked as a peak activity state, and a retention period is set for the data block in the peak activity state; The adjustment terminal is used to prevent the data block in the peak activity state from being moved from the high-performance storage layer to the archive storage layer during the retention period; after the retention period ends, if the data block does not trigger the access peak indication again, the storage level of the data block is evaluated based on the long-term average access activity.

Images