Data object management method and device and storage medium

Abstract

The embodiment of the invention discloses a data object management method and device and a storage medium, and belongs to the technical field of computers. The method comprises the steps that on the basis of historical operation corresponding to a first data object, historical operation information corresponding to the first data object is generated, and the historical operation comprises historical read operation and historical write operation; based on the historical operation information corresponding to the first data object, a corresponding operation mode of the first data object in a future preset time period is predicted, and the operation mode indicates execution modes of read operation and write operation; and adjusting the storage position of the first data object in the storage system based on the corresponding operation mode of the first data object in the future preset time period. According to the embodiment of the invention, the technical problem that the storage cost and the performance are difficult to balance due to complex and changeable I / O operation modes is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to data object management methods, apparatus and storage media. Background Technology

[0002] In the field of large-scale data storage management, a common technical challenge lies in balancing storage costs and performance due to the complexity and variability of I / O operation modes. This problem becomes increasingly severe in high-performance computing scenarios such as model training, especially for large models. These models access the storage system frequently during training, and their data read and write requirements are often highly dynamic and complex, making the trade-off between storage costs and performance due to the complex and variable I / O operation modes even more prominent. Summary of the Invention

[0003] This application provides a data object management method, apparatus, and storage medium to solve at least one of the aforementioned technical problems.

[0004] According to one aspect of the embodiments of this application, a data object management method is provided, the method comprising: Based on the historical operations corresponding to the first data object, historical operation information corresponding to the first data object is generated, wherein the historical operations include historical read operations and historical write operations. Based on the historical operation information corresponding to the first data object, the operation mode corresponding to the first data object in the future preset time period is predicted, and the operation mode indicates the execution mode of read operation and write operation. Based on the operation mode of the first data object within a future preset time period, the storage location of the first data object in the storage system is adjusted.

[0005] In one exemplary implementation, predicting the operation mode of the first data object within a future preset time period based on the historical operation information corresponding to the first data object includes: The historical operation information is input into the operation prediction model to obtain the heat value output by the operation prediction model. The heat value is used to indicate the operation mode of the first data object in a future preset time period. The operation prediction model is a model that predicts future operation behavior patterns based on the historical behavior time sequence patterns in the input data.

[0006] In one exemplary embodiment, adjusting the storage location of the first data object in the storage system based on the operation mode corresponding to the first data object within a preset future time period includes: If the first data object does not belong to the target storage layer, a corresponding migration task is generated based on the popularity value of the first data object. The migration task is used to migrate the first data object to the target storage layer, which belongs to the storage system and is the storage layer with the highest read and write efficiency in the storage system. Execute the migration task.

[0007] In one exemplary implementation, generating a corresponding migration task based on the popularity value corresponding to the first data object includes: If the popularity value corresponding to the first data object meets the preset conditions, the migration task is generated. Under the preset conditions, the first data object is determined to be accessed more frequently than a preset threshold within the preset future time period.

[0008] In one exemplary embodiment, the method further includes: Determine the business weight of the business to which the first data object belongs; Determine the migration cost of the first data object and the data volume of the first data object; Based on the popularity value of the first data object, the migration cost of the first data object, the data volume of the first data object, and the business weight of the business to which the first data object belongs, the migration value of the first data object is obtained. Executing the migration task includes: executing the migration task based on the migration value of the first data object.

[0009] In one exemplary implementation, performing the migration task based on the migration value of the first data object includes: Obtain the migration task corresponding to each of the multiple second data objects, and the migration value of each second data object; Based on each of the migration values, the migration tasks corresponding to the first data object and the migration tasks corresponding to each of the second data objects are sorted to obtain a migration task sequence; Execute each migration task in the migration task sequence.

[0010] In one exemplary implementation, generating a corresponding migration task based on the popularity value corresponding to the first data object includes: Obtain the popularity value of each of multiple third data objects, wherein the third data objects do not belong to the target storage layer; Determine the migration cost, access importance, and data volume of each target data object, where both the first data object and the third data object belong to the target data object; Obtain the remaining capacity and current migration bandwidth of the target storage layer; With the constraint of maximizing migration benefits and minimizing migration costs, multiple data objects to be migrated are selected from each of the target data objects based on the popularity value, migration cost, access importance, data volume of each target data object, as well as the remaining capacity and current migration bandwidth of the target storage layer. The multiple data objects to be migrated include the first data object. The migration benefits are determined based on the popularity value and access importance. Generate a corresponding migration task for each of the data objects to be migrated.

[0011] According to another aspect of the embodiments of this application, a data object management apparatus is provided, the apparatus comprising: The historical operation information acquisition module is used to generate historical operation information corresponding to the first data object based on the historical operations corresponding to the first data object. The historical operations include historical read operations and historical write operations. An operation prediction module is used to predict the operation mode of the first data object within a future preset time period based on the historical operation information corresponding to the first data object. The operation mode indicates the execution mode of read and write operations. The data object adjustment module is used to adjust the storage location of the first data object in the storage system based on the operation mode corresponding to the first data object in a future preset time period.

[0012] In one exemplary embodiment, the operation prediction module is configured to include: The historical operation information is input into the operation prediction model to obtain the heat value output by the operation prediction model. The heat value is used to indicate the operation mode of the first data object in a future preset time period. The operation prediction model is a model that predicts future operation behavior patterns based on the historical behavior time sequence patterns in the input data.

[0013] In one exemplary embodiment, the data object adjustment module is configured to: If the first data object does not belong to the target storage layer, a corresponding migration task is generated based on the popularity value of the first data object. The migration task is used to migrate the first data object to the target storage layer, which belongs to the storage system and is the storage layer with the highest read and write efficiency in the storage system. Execute the migration task.

[0014] In one exemplary embodiment, the data object adjustment module is configured to: If the popularity value corresponding to the first data object meets the preset conditions, the migration task is generated. Under the preset conditions, the first data object is determined to be accessed more frequently than a preset threshold within the preset future time period.

[0015] In one exemplary embodiment, the data object adjustment module is configured to: Determine the business weight of the business to which the first data object belongs; Determine the migration cost of the first data object and the data volume of the first data object; Based on the popularity value of the first data object, the migration cost of the first data object, the data volume of the first data object, and the business weight of the business to which the first data object belongs, the migration value of the first data object is obtained. The migration task is executed based on the migration value of the first data object.

[0016] In one exemplary embodiment, the data object adjustment module is configured to: Obtain the migration task corresponding to each of the multiple second data objects, and the migration value of each second data object; Based on each of the migration values, the migration tasks corresponding to the first data object and the migration tasks corresponding to each of the second data objects are sorted to obtain a migration task sequence; Execute each migration task in the migration task sequence.

[0017] In one exemplary embodiment, the data object adjustment module is configured to: Obtain the popularity value of each of multiple third data objects, wherein the third data objects do not belong to the target storage layer; Determine the migration cost, access importance, and data volume of each target data object, where both the first data object and the third data object belong to the target data object; Obtain the remaining capacity and current migration bandwidth of the target storage layer; With the constraint of maximizing migration benefits and minimizing migration costs, multiple data objects to be migrated are selected from each of the target data objects based on the popularity value, migration cost, access importance, data volume of each target data object, as well as the remaining capacity and current migration bandwidth of the target storage layer. The multiple data objects to be migrated include the first data object. The migration benefits are determined based on the popularity value and access importance. Generate a corresponding migration task for each of the data objects to be migrated.

[0018] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the data object management method described above.

[0019] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, the storage medium storing at least one instruction, the at least one instruction being loaded and executed by a processor to implement the above-described data object management method.

[0020] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform to implement the above-described data object management method.

[0021] The technical solution provided in this application can bring the following beneficial effects: This application proposes a method to predict future operation patterns of data objects based on their historical operations. The storage location of the data object in the storage system is then adjusted in advance according to these operation patterns, ensuring it is in a storage location adapted to the upcoming operation pattern. This solution offers the following significant advantages: Being proactive rather than reactive, it allows for adjustments to data storage locations based on upcoming changes in operational patterns. For example, if a data object is predicted to be accessed frequently, it can be migrated from a low-speed storage medium to a high-speed one in advance, significantly improving access efficiency. Conversely, if the frequency of operation on the same data object is predicted to decrease significantly, it can be moved to a lower-cost, higher-capacity storage area to optimize storage resource utilization and reduce costs.

[0022] As can be seen, this application achieves dynamic, forward-looking, and adaptive cost reduction and efficiency improvement, reasonably controls the balance between cost and performance, and solves the technical problem of difficulty in balancing storage cost and performance caused by the complexity and variability of I / O operation modes. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1This is a flowchart illustrating a data object management method provided in one embodiment of this application; Figure 2 This is a schematic diagram provided in one embodiment of the present application for a model training scenario; Figure 3 This is a schematic diagram illustrating the process of predicting operating modes using a Long Short-Term Memory (LSTM) network, provided in one embodiment of this application. Figure 4 This is a timing diagram of the implementation process in a model training scenario (AI training application) provided by one embodiment of this application; Figure 5 This is a schematic diagram of a method for determining a migration task provided in one embodiment of this application; Figure 6 This is a schematic diagram of a migration task generation process provided in one embodiment of this application; Figure 7 This is a block diagram of a data object management device provided in one embodiment of this application; Figure 8 This is a structural block diagram of a computer device provided in one embodiment of this application; Figure 9 This is a structural block diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. Thus, features defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments, unless otherwise stated, "a plurality of" means two or more. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0027] To make the objectives, technical solutions, and advantages disclosed in the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the embodiments of this application.

[0028] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be noted that all data used in the embodiments of this application have been fully authorized by the relevant parties before use. To facilitate understanding of this application, the relevant technical background will first be explained.

[0029] Artificial Intelligence (AI) refers to the development of computer systems capable of simulating human intelligence through the interdisciplinary application of computer science and other disciplines. These systems possess the abilities to learn, reason, plan, understand language, and perceive and interact, demonstrating performance similar to or even surpassing that of humans in specific tasks. In various application scenarios, AI can process massive amounts of data and extract valuable information to assist in decision-making or automate tasks. AI model training, in the context of AI technology, is the process of equipping an AI model with these capabilities using large-scale datasets and powerful computing resources, employing scientific training methods. Therefore, the AI ​​model training process can be understood as the learning process of artificial intelligence.

[0030] In the field of large-scale data storage management, specifically in high-performance computing scenarios such as AI model training, there is a common technical problem of balancing storage costs and performance due to the complexity and variability of I / O operation modes. This problem becomes increasingly serious as the size of the model increases, especially for large models. Large models refer to models with a large number of parameters and complex structures, whose training process requires processing massive amounts of data and frequently accessing the storage system. Common large models include large generative language models, or LLMs. These models access the storage system frequently during training, and their data read and write requirements are often highly dynamic, making the technical problem of balancing storage costs and performance due to the complexity and variability of I / O operation modes even more prominent.

[0031] With the rapid evolution of technologies such as Large Language Models (LLM), the scale of datasets required for model training has jumped from TB to PB levels. Logically, this training data is not monolithic; rather, it consists of various types of data with distinct I / O characteristics. I / O characteristics refer to the behavioral patterns of data in terms of read and write operations. For example, in a model training scenario, different data exhibit vastly different I / O operation patterns: (1) Training sample data: usually consists of a large number of small files, which are read sequentially or randomly in multiple epochs during the training period. The reading and writing operation pattern of the training sample data is periodic. Epoch refers to the complete traversal of a dataset during the model training process. Reading the training sample data often requires multiple epochs to ensure that the model can fully learn the features in the data.

[0032] (2) Model checkpoint: It is huge (ranging from GB to TB), written once with high bandwidth at a low frequency (e.g. every half hour), but read at an even lower frequency, and is usually only used for recovery after training is interrupted.

[0033] (3) Intermediate data such as model gradients: These are read and written frequently and with low latency in each training iteration, which places the most stringent requirements on I / O operation performance.

[0034] This application proposes that, in model training scenarios, at least I / O operations are required on the aforementioned training sample data, model checkpoint data, and model gradients. Ultra-high-performance storage media capable of meeting the gradient data read / write requirements (such as persistent memory SCM and NVMe SSD) have extremely high unit capacity costs, making it uneconomical to carry all data including the aforementioned three types. Meanwhile, lower-cost, high-capacity media (such as HDD and object storage OBS) are far from meeting the high-performance I / O demands of training. This "cost-performance" contradiction leads to GPU computing power often being idle while waiting for data loading in actual deployments, making the storage system a bottleneck restricting overall training efficiency and increasing total cost of ownership. Therefore, how to construct a storage system that can intelligently perceive the actual I / O demands of model training and automatically optimize data layout has become a key technical problem urgently needing to be solved in the field of data storage management, and also reflects the technical objective that this application aims to achieve.

[0035] This application proposes that, although several data storage optimization methods have been proposed in related technologies to solve the "cost-performance" contradiction, they are all difficult to adapt to the actual scenario of model training and have significant limitations: Option 1: Use an all-flash array: Store all data in high-performance storage. While this option offers extremely high performance, its procurement and maintenance costs are extremely high for petabyte-scale or even exabyte-scale data, making it unsuitable for large-scale deployment.

[0036] Option 2: Manual or Static-Based Data Tiling: Storage administrators migrate data between different storage tiers manually or by writing scripts with fixed rules, based on experience. This approach lacks intelligence, flexibility, and adaptability. It is labor-intensive, difficult to implement, and unsuitable for model training scenarios. During model training, the I / O operation patterns of different data change at different stages (e.g., data preprocessing, early training phase, convergence phase). Manual adjustments are severely delayed and burdensome to maintenance. Static strategies cannot capture these dynamic changes, often leading to data mismatches. For example, frequently I / O-intensive data might be stored in cold storage, while infrequently I / O-intensive data occupies hot storage space, resulting in low storage efficiency.

[0037] Option 3: Automatic tiering based on traditional caching / tiering algorithms (such as LRU / LFU): Most storage systems have built-in automatic tiering functionality based on algorithms such as "Least Recently Used" (LRU) or "Least Frequently Used" (LFU). The core drawback of this type of solution is its "passive reactivity." They only migrate data to a high-speed tier after it has been accessed frequently (i.e., "becomes" hot data). This type of solution is difficult to apply to model training scenarios because it leads to at least the following two problems: The first problem is cold start latency: at the end of the previous training cycle, the training sample data has few I / O operations and may be regarded as cold data and stored in a low-speed layer. This will cause significant I / O latency when the training sample data is first accessed intensively at the beginning of a new training cycle, because the training sample data is still in a low-speed layer, which slows down the initial training speed.

[0038] The second problem is the inability to understand semantics: This type of solution's inability to understand the "business semantics" of data can lead to incorrect data stratification. For example, a huge checkpoint file that has just been written may not be read for a long time in the future, which is a typical example of "hot write, cold read" data. However, this type of solution may misjudge it as hot data and keep it in expensive high-speed storage, resulting in wasted resources.

[0039] Option 4: This type of option is typically based on static rules (such as "archive data older than 90 days from the high-performance layer to the low-cost layer") and coarse-grained (entire file) management. This strategy is unsuitable for model training scenarios because the I / O behavior of the same file varies drastically, is highly dynamic, and is strongly correlated with the training period at different training stages.

[0040] In summary, the technical solutions proposed in this application are either too costly, too complex to manage, or unable to proactively adapt to the dynamic and periodic I / O behavior characteristics of data in model training scenarios due to their "passive response" mechanism. Therefore, they cannot fundamentally solve the deep contradiction between the performance and cost of I / O operations for data storage in model training scenarios.

[0041] To address the technical problem of balancing storage cost and performance caused by the complexity and variability of I / O operation modes, and especially to fundamentally resolve the deep-seated contradiction between I / O operation performance and cost in data storage during model training scenarios, this application provides a data object management method. The core advantage of this method lies in predicting the I / O operation mode of a first data object in future periods using its historical I / O operations. Based on the future I / O operation mode, the storage location of the first data object is adjusted to adapt to the future I / O operation mode. Therefore, this application solves the technical problem of balancing storage cost and performance caused by the complexity and variability of I / O operation modes through I / O operation mode prediction and timely storage location adjustment.

[0042] This technical solution is particularly suitable for scenarios with clear temporal characteristics. For example, if different data have different I / O operations at different stages, resulting in a complex overall I / O operation pattern, but the changing trends of the I / O operation patterns of each type of data at different stages are traceable, this application's embodiments are especially suitable for managing the first data object. For instance, model training scenarios are well-suited for the data object management method of this application's embodiments. In this model training scenario, the I / O operation patterns of different data are accompanied by Epochs, exhibiting periodic and dynamic changes with high predictability. Therefore, this application's embodiments can be used for dynamic, fine-grained, prediction-based data management.

[0043] Please refer to Figure 1 The diagram illustrates a flowchart of a data object management method according to an embodiment of this application. The method includes the following steps: S110. Based on the historical operations corresponding to the first data object, generate historical operation information corresponding to the first data object, wherein the historical operations include historical read operations and historical write operations.

[0044] This application does not limit the data object; a data object can refer to any entity with data attributes, such as a file, database record, data stream, or data block. In specific application scenarios, a data object can be a carrier of the aforementioned three types of data in model training, such as a record fragment recorded in a log file, or a data block in a storage system. In some implementations, the object targeted by read, write, migration, or other operations in any storage system can be used as a data object. For example, the data object can be an operation object in Ceph / RADOS distributed storage, an NVMe-oF-based storage network, or a parallel file system such as Lustre. This application does not limit the granularity of the data object; for example, it can be a block, a data entry, or a file to adapt to the underlying abstraction of different storage systems.

[0045] In this application embodiment, the first data object refers to any data object. The following focuses on the scenario of migrating data from a storage layer with lower read / write efficiency to a storage layer with higher read / write efficiency. In this scenario, the first data object refers to any data object that is not in the storage layer with higher read / write efficiency.

[0046] The historical operations in this application embodiment include read and write operations. By recording these historical operations, this application learns the changing patterns of the operational behavior of the first data object over time, thereby predicting the operational patterns within a preset future time period, and using these operational patterns as the basis for adjusting the data storage location. Therefore, the more detailed and abundant the historical operation information of the first data object generated based on historical operations, the more accurate the prediction, and the more prominent the advantages of this application embodiment become.

[0047] This application does not limit the scope of historical operation information. It can refer to various information indicating the read / write behavior patterns of a first data object in a historical period. For example, this historical operation information may include read / write type, the identifier of the first data object, operation time information, associated file path, etc. Furthermore, it may include the access frequency of the first data object, the duration of the operation, and the data volume of each operation. In addition, historical operation information may also involve details such as the storage node identifier related to the first data object, network transmission rate, or the success and failure status of the operation. This information collectively constitutes a comprehensive description of the operational behavior of the first data object in a historical period, providing a foundation for further analysis of its behavior patterns. By integrating this multi-dimensional information, the operational characteristics and changing trends of the first data object can be captured more accurately, thereby providing a reliable basis for subsequent prediction and optimization.

[0048] Please refer to Figure 2This diagram illustrates an embodiment of the present application applied to a model training scenario. The diagram depicts a scenario where a model training application or client (AI Training Application) is training a model, and the data object management method of this embodiment is executed. This management method is used to manage an intelligent hierarchical storage system. Specifically, this embodiment collects I / O requests issued by the AI ​​Training Application through a profiling agent. These I / O requests refer to historical operations (read and write operations).

[0049] For example, the ProfilingAgent can be deployed as a lightweight process or kernel module (such as eBPF) on each storage node of an intelligent tiered storage system. It is responsible for non-intrusively collecting data I / O operation data flowing through the node (such as read / write type, data object identifier, timestamp, and the amount of data involved in the I / O operation), and combining this data with file attribute data such as the file path and file type of the data being operated on to aggregate profile data corresponding to the data object. Taking the first data object as an example, the ProfilingAgent can collect its corresponding historical operation information (profile data) and send it to the machine learning prediction engine. This agent is designed with extremely low performance overhead to ensure minimal impact on the performance of upper-layer applications.

[0050] The profile data in this application embodiment can reflect time-series information. For example, by recording the timestamp of each operation, the access patterns and trends of data objects at different points in time can be clearly restored. For instance, in a model training task, this profile data can reflect the capture results of I / O trajectories. The user starts the model training task. When the training framework reads and writes data, the monitoring agent deployed on the storage node captures these I / O requests and records key information. The key information from discrete I / O events is aggregated into a time series for the data object (data block). For example, for a data block Block_X, its profile data may include [(t1,read),(t5,read),(t10,write)...].

[0051] S120. Based on the historical operation information corresponding to the first data object, predict the operation mode corresponding to the first data object in a future preset time period, wherein the operation mode indicates the execution mode of read operation and write operation.

[0052] This application's embodiments are applicable to scenarios where the operation patterns are predictable and regular. For example, if the operation pattern has strong periodicity and predictability, then this application's embodiments are applicable. Taking a model training scenario as an example, the aforementioned three types of data will exhibit strong regularity and predictability during multiple epochs of iteration. Based on this, a model for predicting operation patterns can be constructed, and based on this model, the operation pattern corresponding to the first data object in a future preset time period can be predicted.

[0053] In model training scenarios, access to model-related data during multiple epochs is often not completely random, but follows a specific, repeatable sequence. This application capitalizes on this characteristic, transforming the data management problem from a traditional "cache replacement problem" into a "time series prediction problem." Please refer to... Figure 2 Profiling can be achieved through a lightweight monitoring agent. The agent continuously detects profile data corresponding to each data object and sends this profile data to the machine learning prediction engine to predict the future operation patterns of these data objects. For example, this operation pattern may indicate which data blocks will be frequently read, which data blocks are less likely to be read, and which data blocks will be frequently written in the next few time steps. Through this prediction, the system can adjust caching strategies or resource allocation in advance, thereby significantly improving overall performance.

[0054] In one exemplary embodiment, predicting the operation mode of the first data object within a future preset time period based on the historical operation information corresponding to the first data object includes: inputting the historical operation information into an operation prediction model to obtain a popularity value output by the operation prediction model, wherein the popularity value is used to indicate the operation mode of the first data object within a future preset time period; wherein the operation prediction model is a model that predicts the behavior mode of future operations based on the historical behavior time sequence pattern in the input data.

[0055] This application does not limit the prediction model for this operation in the embodiments, so as to Figure 2 For example, this operational prediction model can be a machine learning prediction engine. To improve the prediction accuracy of this operational prediction model, it can be configured as a machine learning model that excels at capturing sequence dependencies, such as a Long Short-Term Memory (LSTM) network. Please refer to [reference needed]. Figure 3This diagram illustrates the process of predicting operational patterns using a Long Short-Term Memory (LSTM) network, as described in an embodiment of this application. First, the access time sequence information of historical operations for each data object (data block X) can be extracted. Then, corresponding profile data is constructed based on feature engineering. This profile data is input into the LSTM network model to capture long and short-term temporal dependencies, thereby obtaining the model's output (Qutput). The output result is the popularity value, which indicates the popularity value within a preset future time period T. In some implementations, this machine learning prediction engine can also employ other models with temporal analysis capabilities, such as Temporal Convolutional Networks (TCN), Transformer models, or a combination of traditional statistical models such as ARIMA and neural network models. Furthermore, complex models can be compressed into lighter inference models using techniques such as knowledge distillation to adapt to deployment environments with different resource constraints.

[0056] This application's embodiments use a popularity value to characterize the operation mode of a first data object within a preset future time period. For example, a high popularity value indicates that the first data object is likely to be frequently accessed or operated on within the preset future time period, suggesting a high utilization rate. Conversely, a low popularity value may mean that the data object is operated on less frequently and is a low-priority management object. In this way, the system can dynamically adjust resource allocation strategies based on popularity values, thereby optimizing overall operating efficiency. For example, in storage management, high-popularity data objects can be preferentially loaded into the cache, while low-popularity data objects may be migrated to slower storage devices to achieve rational utilization of storage resources.

[0057] Taking model training as an example, the machine learning prediction engine can output the popularity value of each data object to the policy and decision engine, which will then decide which data objects to migrate to a storage layer with high read efficiency to improve data read and write efficiency, and which data objects to migrate to a storage layer with low read efficiency to reduce costs.

[0058] S130. Based on the operation mode corresponding to the first data object in a future preset time period, adjust the storage location of the first data object in the storage system.

[0059] In practical applications, this adjustment mechanism can significantly optimize the utilization efficiency of storage resources. By analyzing historical operation records and predicting future access patterns, the engine can accurately identify which data objects are more likely to be accessed frequently, thus prioritizing their placement in high-performance storage areas. Simultaneously, data objects with low access frequency or no short-term access needs are migrated to lower-cost storage media. This approach not only improves the overall system response speed but also effectively controls storage overhead, providing users with a more flexible and economical solution. Furthermore, this mechanism supports dynamic adjustments, allowing for real-time updates to the data object storage strategy based on changes in business needs, ensuring the system is always in optimal operating condition.

[0060] This application proposes a method to predict future operation patterns of data objects based on their historical operations. The storage location of the data object in the storage system is then adjusted in advance according to these operation patterns, ensuring it is in a storage location adapted to the upcoming operation pattern. This solution offers the following significant advantages: Being proactive rather than reactive, it allows for adjustments to data storage locations based on upcoming changes in operational patterns. For example, if a data object is predicted to be accessed frequently, it can be migrated from a low-speed storage medium to a high-speed one in advance, significantly improving access efficiency. Conversely, if the frequency of operation on the same data object is predicted to decrease significantly, it can be moved to a lower-cost, higher-capacity storage area to optimize storage resource utilization and reduce costs.

[0061] As can be seen, this application achieves dynamic, forward-looking, and adaptive cost reduction and efficiency improvement, reasonably controls the balance between cost and performance, and solves the technical problem of difficulty in balancing storage cost and performance caused by the complexity and variability of I / O operation modes.

[0062] In one exemplary embodiment, adjusting the storage location of the first data object in the storage system based on the operation mode corresponding to the first data object within a preset future time period includes: when the first data object does not belong to the target storage layer, generating a corresponding migration task based on the popularity value corresponding to the first data object, the migration task being used to migrate the first data object to the target storage layer, the target storage layer belonging to the storage system, and the target storage layer being the storage layer with the highest read / write efficiency in the storage system; and executing the migration task.

[0063] In practical applications, this process is highly automated through intelligent algorithms. The system automatically triggers the generation of migration tasks based on the popularity value of the first data object and assigns the tasks to the corresponding execution modules. By executing migration tasks, high accuracy and stability in storage location adjustments can be ensured.

[0064] In some exemplary embodiments, generating a corresponding migration task based on the popularity value of the first data object includes: generating the migration task when the popularity value of the first data object meets a preset condition, wherein the first data object is determined to be accessed more frequently than a preset threshold within the preset future time period.

[0065] If the popularity value corresponding to the first data object meets the preset condition, it can be determined that frequent I / O operations will be performed on the first data object within a preset future time period, resulting in frequent reads and / or writes. This application embodiment does not limit the preset threshold, the preset future time period, or the preset condition. For example, the preset threshold can be the average access frequency calculated based on historical data, or a fixed value set based on business needs. The length of the preset future time period can be flexibly adjusted according to specific application scenarios, such as being set to ten seconds, one minute, one hour, etc. The preset condition can be set based on business characteristics.

[0066] If the heat value corresponding to the first data object meets the preset condition, it can be determined that the first data object will be frequently subjected to I / O operations in the future preset period of time, and then it needs to be migrated to the target storage layer with the highest read and write performance.

[0067] In some implementations, the migration task execution process can employ a parallel processing mechanism, handling multiple migration tasks in parallel to minimize the impact on overall system performance. Furthermore, to ensure the reliability of data migration, the system incorporates verification and rollback mechanisms. If an anomaly occurs during migration, the system can quickly restore the data to its pre-migration state, thereby guaranteeing data integrity and consistency. This refined management approach enables the storage system to maintain efficient and stable operation in complex and ever-changing business environments.

[0068] Taking model training as an example, the policy and decision engine can generate migration tasks and send them to the migration executor. The migration executor then executes the specific migration tasks, including migrating data objects that will be frequently accessed in the future but are not currently in the target storage layer to the target storage layer. For example, the first data object in this embodiment could be a data object that will be frequently accessed in the future but is not currently in the target storage layer. Furthermore, this embodiment can also migrate data objects that will not be frequently accessed in the future and are currently in the target storage layer out of the target storage layer to release storage resources and optimize overall performance. This is the reverse process relative to migrating the first data object to the target storage layer, and therefore will not be elaborated upon. This dynamic adjustment mechanism can rationally allocate storage resources based on prediction results, thereby improving the system's operating efficiency. Figure 2 The storage system comprises two storage layers: a high-performance layer (NVMe550), which can be used as the target storage layer, and a high-capacity layer (HDD / OB5), which is not the target layer. The storage system (heterogeneous storage pool) supports direct read and write operations on both the high-performance layer (NVMe550) and the high-capacity layer (HDD / OB5) by upper-layer applications.

[0069] Below, the applicant will discuss the matter. Figure 2 Further explanation of the key components: The Machine Learning Prediction Engine (PredictionEngine) is a centralized service that serves as the core computing unit of the system. It receives data from all monitoring agents and maintains and trains one or more Machine Learning Prediction Engines. This Prediction Engine periodically (e.g., every minute) outputs the predicted popularity value for all (or partially active) data blocks in the storage system over a future period.

[0070] The Policy & Decision Engine subscribes to the "heat index" reports published by the prediction engine. Based on the administrator's preset global policies (such as cost-first mode, performance-first mode, and balanced mode), and combined with the real-time status of each storage tier (such as capacity and load), it transforms the heat index into specific data migration tasks.

[0071] MigrationExecutor: Distributed across various storage nodes, it is responsible for executing migration tasks. It interacts with the underlying data management plane of the storage system to move data blocks between different physical media in a way that is transparent to upper-layer applications and has the lowest possible performance impact.

[0072] Please refer to Figure 4 This diagram illustrates the implementation process of an embodiment of this application in a model training scenario (AI training application). The diagram shows the collaborative process between the six core components of the system (AI training application, monitoring agent, prediction engine, decision engine, transfer executor, and heterogeneous storage), which consists of nine key steps and can be divided into three stages: "data capture and prediction," "decision and transfer," and "feedback and optimization." Phase 1: Data Capture and Prediction (Steps 1-4) 1. Capture I / O Trajectory: When the AI ​​training application reads and writes data to heterogeneous storage, the monitoring agent captures its I / O (input / output) operation trajectory and then returns the trajectory to the AI ​​training application.

[0073] 2. Send profile data: The monitoring agent sends the captured profile data to the prediction engine.

[0074] This application does not limit the information content of the portrait data, which may include the following information: (1) Time characteristics: access interval time sequence, time decay factor.

[0075] (2) I / O characteristics: read / write ratio, random / sequential I / O ratio, and statistical distribution of average I / O size.

[0076] (3) Data semantic features: Identify the data type (Checkpoint, training sample, gradient data, etc.) and the user / project group to which the read or write operation is performed through dynamic inference.

[0077] (4) Related system status characteristics: Current storage system load and network bandwidth utilization.

[0078] This application embodiment constructs an open "multi-dimensional profile" data. This profile data not only includes data semantics derived through dynamic inference, but also systematically covers time features (access interval, decay), I / O features (read-write ratio, random / sequential ratio, size distribution), and system state features (storage system load, network bandwidth utilization), providing rich input for the LSTM model, thereby improving the accuracy of popularity value prediction.

[0079] 3. Training / Inference Model: The prediction engine is based on time-series data and continuously learns the patterns of data access by training or inferring machine learning models (such as LSTM) through loops.

[0080] In some implementations, the prediction engine continuously collects profile data. The system employs a strategy combining periodic offline retraining with online learning. During the initial phase or system idle windows, the prediction engine performs offline training on the full dataset to generate a high-precision baseline model. During operation, to adapt to load drift, the system can perform online learning and mitigate catastrophic forgetting issues that may occur during incremental learning through mechanisms such as a replay buffer or multi-model integration, ensuring model stability.

[0081] 4. The output of the model (usually a value between 0 and 1) is the heat value of that data block.

[0082] In one specific implementation, profile data can be represented by structured log entries. Each log entry can contain the following field values: timestamp, block_id, io_type (read / write), io_size, process_id, and file_path. These fields represent the timestamp, block identifier (data object identifier), input / output type (read or write), input / output size, process identifier, and file path, respectively. Through the combination of these fields, the system can accurately record detailed information for each data access, providing a reliable foundation for further analysis.

[0083] For each data block, extract its N most recent log entries and process these log entries to generate a tensor that meets the input requirements of the LSTM model. Input the tensor into the trained LSTM model for forward inference to obtain the heat value corresponding to the data block.

[0084] 4. Publish popularity report: The prediction engine generates a data popularity report (i.e., the popularity value of different data blocks) and sends it to the decision engine.

[0085] Phase 2: Decision Making and Migration (Steps 5-7) 1. Generate migration plan: The decision engine combines preset storage strategies (such as "high-frequency access data is stored in high-speed storage, and low-frequency access data is stored in low-speed storage") and generates a data migration plan based on the heat report.

[0086] 2. Issue migration tasks: The decision engine issues specific data migration tasks to the migration executor.

[0087] 3. Perform cross-layer data migration: The migration executor performs tasks to migrate data between different layers of heterogeneous storage, realizing "intelligent hierarchical storage".

[0088] Phase 3: Feedback and Optimization (Steps 8-9) 1. Monitor and provide feedback on the true hit rate: The monitoring agent continuously monitors the "true hit rate" of data access (i.e. whether the predicted access behavior is consistent with the actual behavior) and provides feedback to the prediction engine.

[0089] For example, the monitoring agent not only monitors I / O requests but also the "hit rate" of data access (i.e., whether hot data is already in the high-performance layer when accessed). The prediction engine compares the actual access behavior with the model's prediction results to calculate the model's prediction accuracy. This information serves as feedback for the next round of iteration and optimization of the operational prediction model (prediction engine), forming an adaptive closed-loop system.

[0090] 2. Model Optimization: The prediction engine compares the "predicted results" with the "actual results" and optimizes them through a loop to improve the accuracy of subsequent predictions, forming a closed-loop iteration.

[0091] This diagram illustrates how a closed-loop process of "prediction-decision-transfer-feedback" enables AI training data to be automatically stored in tiers across heterogeneous storage based on access patterns, thereby balancing storage costs and access efficiency. This embodies the core of this application's embodiment—"intelligent storage scheduling"—in a typical application of AI training scenarios.

[0092] The data object management method proposed in this application essentially embodies a paradigm shift from "passive reaction" to "proactive prediction": the core of the embodiments of this application is "prediction" rather than "response." That is, by analyzing historical operations to predict future operation patterns, and based on the predicted operation patterns, data migration is performed in advance, so that the read and write efficiency of the storage layer where the data resides can match the future operation patterns. The object management method of this application covers multiple stages such as data collection related to historical operation behavior, profile data generation, popularity value prediction, migration decision, and migration execution. Furthermore, it can adjust the operation prediction model according to the actual data read and write hit situation, thereby realizing an automated and adaptive closed-loop optimization system, constructing a closed loop of "monitoring-prediction-decision-execution-feedback," and performing online adaptive optimization and rollback control.

[0093] In one exemplary implementation, please refer to Figure 5 This diagram illustrates a method for determining migration tasks in an embodiment of this application. The step of generating a corresponding migration task based on the popularity value corresponding to the first data object includes: S510. Obtain the heat value of each of the multiple third data objects, wherein the third data objects do not belong to the target storage layer.

[0094] A third data object refers to a data object that is different from the first data object. Taking an example where neither the first data object nor multiple third data objects belong to the target data layer, the solution of this application embodiment is described in detail.

[0095] S520. Determine the migration cost, access importance, and data volume of each target data object, wherein the first data object and the third data object both belong to the target data object.

[0096] Migration cost refers to the resources required to migrate a data object from one storage tier to another, including time cost, computing resources, and potential performance degradation. This parameter can be obtained by analyzing the characteristics of the storage medium, the size of the data object (data volume), network bandwidth, and the impact of the migration operation on the overall system performance. Access importance, on the other hand, is related to the importance of the data object to the relevant business and the importance of the business itself.

[0097] S530. Obtain the remaining capacity and current migration bandwidth of the target storage layer.

[0098] S540. With the constraint of maximizing migration benefits and minimizing migration costs, based on the popularity value, migration cost, access importance, data volume of each target data object, as well as the remaining capacity and current migration bandwidth of the target storage layer, select multiple data objects to be migrated from each target data object. The multiple data objects to be migrated include the first data object. The migration benefits are determined based on the popularity value and the access importance.

[0099] The total amount of data in the migrated data objects should be less than the remaining capacity, and the ratio of the total amount of data in the migrated data objects to the maximum allowable migration time when parallel migration occurs should be less than the current migration bandwidth. These are also constraints required to perform this step. In an exemplary implementation, the product of the popularity value and the access importance can be determined as the migration benefit, thereby planning multiple data objects to be migrated. Specifically, The above objectives can be achieved by constructing an optimization model. This model takes maximizing migration benefits and minimizing migration costs as its main optimization objectives, while introducing the aforementioned constraints. By solving this optimization model, an optimal set of data objects to be migrated can be selected. Furthermore, in practical applications, heuristic algorithms or dynamic programming methods can be combined to further improve computational efficiency and reduce complexity, thereby adapting to the needs of large-scale data management scenarios.

[0100] S550. Generate a corresponding migration task for each of the data objects to be migrated.

[0101] In one exemplary implementation, the data object to be migrated may include a first data object and several other second data objects. In one implementation, please refer to... Figure 6This diagram illustrates the migration task generation process according to an embodiment of this application. To generate a migration task for the first data object, the following operations need to be performed: S610. Determine the business weight of the business to which the first data object belongs; This application does not limit the method for determining the business weight of the business to which the first data object belongs. For example, it can be comprehensively evaluated by analyzing the historical access frequency, access duration, and importance of related businesses of the business. In addition, it can also be set directly by the user.

[0102] S620. Determine the migration cost of the first data object and the data volume of the first data object; The migration costs and data volume have been discussed earlier and will not be repeated here.

[0103] S630. Based on the popularity value corresponding to the first data object, the migration cost of the first data object, the data volume of the first data object, and the business weight of the business to which the first data object belongs, the migration value of the first data object is obtained; This application does not limit the method for determining the migration value based on popularity, migration cost, data volume, and business weight. For example, the migration value can be defined as: Score = (p_b * w_b - Cost_b) / size_b. Where p_b is the popularity value, w_b is the business weight, Cost_b is the migration cost, and size_b is the data volume.

[0104] Based on the same inventive concept as steps S610-S630, the migration value of each second data object can be obtained. Therefore, performing the migration task based on the migration value of the first data object includes: S640. Obtain the migration task corresponding to each of the multiple second data objects, and the migration value of each second data object; S650. Based on each of the migration values, sort the migration tasks corresponding to the first data object and the migration tasks corresponding to each of the second data objects to obtain a migration task sequence; The corresponding migration tasks can be arranged in descending order of migration values ​​to obtain a migration task sequence.

[0105] S660. Execute each migration task in the migration task sequence.

[0106] During the sequential execution of each migration task, concurrent execution is supported to improve overall migration efficiency. Simultaneously, the system can dynamically monitor the execution status of each migration task and adjust the concurrency strategy in real time based on resource usage, ensuring the stability and efficiency of the migration process. If a migration task is interrupted or fails due to an exception during execution, the system can record the task's status information and provide a retry mechanism to re-execute the task when resources allow.

[0107] To prevent data jitter, the system also introduces a minimum residency time. The minimum residency time mechanism ensures that when a data object's storage state changes, the system guarantees that the object remains in a particular storage state for a minimum time that meets a preset threshold before allowing it to move to the next state. For example, if the first data object's residency time in the high-speed layer is too short to reach the preset threshold, it cannot be migrated to another storage layer. This mechanism effectively avoids data instability caused by frequent state switching, ensuring that the time a data object remains in a particular storage state is not less than the set threshold, thereby preventing system resource waste and instability caused by frequent state switching.

[0108] In some implementations, if the hotness value of a data object located in the target storage layer is very low, it can be moved out of the target storage layer. The inventive concept of this process is consistent with the aforementioned inventive concept of moving data objects into the target storage layer, and will not be elaborated here.

[0109] This application's embodiments predict the future read / write operation patterns of data blocks and can migrate them to appropriate storage layers in advance. For example, taking model training as an example, by migrating data blocks predicted to become hot to the high-performance layer in advance, when the model training task actually needs to access these data, they are already located in the corresponding high-performance storage medium, thereby significantly reducing I / O latency and improving the effective utilization of computing units (such as GPUs). Moreover, by selecting the migrated data objects, the system can dynamically and globally optimize the data layout, ensuring that the high-performance storage medium is always preferentially occupied by the data blocks predicted to be accessed soon.

[0110] By intelligently and maximally utilizing low-cost, high-capacity storage media to store cold data, the total cost of ownership for storing massive amounts of data is significantly reduced. This is then combined with high-performance storage media to support access to hot data, satisfying the demands of high-performance computing while achieving efficient allocation of storage resources. This intelligent data management approach not only reduces the need for manual intervention but also enhances the system's adaptability, enabling it to better cope with diverse application scenarios and complex workload changes. Simulation results show that compared to all-flash solutions, it saves significant costs; compared to traditional tiered solutions, it significantly improves space utilization efficiency, reduces I / O latency, and increases I / O hit rate.

[0111] The entire process in this embodiment is completed automatically by the system without manual intervention, which greatly reduces the complexity of storage management and operation and maintenance costs.

[0112] Moreover, the embodiments of this application have strong adaptability and can automatically adapt to load changes in different models and different training stages, providing continuous and stable high performance without the need for manual tuning for each new task.

[0113] The storage system in this application embodiment must include at least two or more storage layers with different performance and cost characteristics: a high-performance storage layer and a low-performance storage layer. The purpose of this design is to achieve optimal resource allocation by storing data in layers based on its access frequency and importance. The high-performance storage layer is primarily used to handle frequently accessed "hot" data, ensuring the system can still provide fast response under high load; while the low-performance storage layer is responsible for handling less frequently accessed "cold" data, meeting long-term storage needs at a lower cost. Data objects are migrated based on their "hotness" value and current storage location.

[0114] Specifically, if a data object has a high accessibility value and is currently located in a low-performance storage layer, it should be migrated to a high-performance storage layer to ensure faster access and higher processing efficiency. Conversely, if a data object has a low accessibility value and is currently located in a high-performance storage layer, it should be migrated to a low-performance storage layer to free up space in the high-performance storage layer to better serve frequently accessed data. This dynamic migration mechanism relies on the system's real-time assessment of data accessibility, predicts data access patterns through intelligent algorithms, and makes optimal migration decisions accordingly. This multi-layered architecture not only enhances the overall flexibility of the storage system but also significantly reduces hardware investment and operating costs while maintaining performance. Furthermore, through dynamic monitoring and analysis of data access patterns, the system can intelligently adjust the distribution of data across different layers, further optimizing storage efficiency and performance.

[0115] Furthermore, the relevant components in the system implementing the data object management method in this application embodiment should possess the following capabilities: I / O monitoring capability: The storage system needs to provide an interface or mechanism that allows the monitoring agent to capture sufficiently detailed I / O operation information.

[0116] Data migration capability: The underlying storage system needs to support the ability to migrate data blocks at the lower level between different storage layers in a way that is transparent to upper-layer applications.

[0117] Computing resources: CPU / memory resources are required to run the prediction engine for popularity value prediction.

[0118] In one specific implementation, a feasible implementation environment is disclosed: the implementation environment is a typical distributed computing cluster containing multiple compute nodes (e.g., nodes configured with 8×A100 80GB GPUs) and a multi-tiered heterogeneous storage pool. The storage pool includes at least a high-performance tier (such as NVMe SSDs) and a high-capacity, low-cost tier (such as HDDs or object storage OBS).

[0119] The software in this system can be deployed in a containerized environment (such as Kubernetes v1.20). The monitoring agent can be implemented using eBPF technology to ensure low overhead; the prediction engine can be built using mainstream machine learning frameworks (such as PyTorch 2.x); communication between components can be accomplished through a message bus (such as Kafka or Redpanda); and data management can employ an embedded database (such as RocksDB) or a distributed coordination service (such as Etcd).

[0120] The effectiveness of the embodiments in this application can be verified through a variety of representative model training tasks, such as pre-training of large-scale language models (using WebDataset or streaming TFRecord format data), training of computer vision models (using ImageNet / COCO datasets), and training of recommendation system models (using Criteo datasets). During the verification process, the training tasks cover typical access patterns such as sequential data reading, random data reading, multi-epoch model iteration, and checkpoint recovery reading caused by training interruption.

[0121] This application proposes that, to ensure the consistency of data blocks during concurrent access, a copy-on-write or shadow paging mechanism can be employed, employing strict state management to ensure data consistency. During migration, data blocks are marked as "migrable," and only after migration is complete are they updated to the "available" state, thus avoiding data inconsistency issues caused by concurrent access. Furthermore, to further enhance system reliability, distributed locks or version control mechanisms can be introduced to ensure that only one thread or process can modify data blocks at a time. After migration, the system checks data integrity through a verification and validation mechanism; if an anomaly is detected, a rollback operation is triggered, restoring the data to its pre-migration state.

[0122] Please refer to Figure 7 The diagram illustrates a block diagram of a data object management apparatus. The apparatus includes logic modules for implementing various logical operations within the data object management system. The historical operation information acquisition module is used to generate historical operation information corresponding to the first data object based on the historical operations corresponding to the first data object. The historical operations include historical read operations and historical write operations. An operation prediction module is used to predict the operation mode of the first data object within a future preset time period based on the historical operation information corresponding to the first data object. The operation mode indicates the execution mode of read and write operations. The data object adjustment module is used to adjust the storage location of the first data object in the storage system based on the operation mode corresponding to the first data object in a future preset time period.

[0123] In one exemplary embodiment, the operation prediction module is configured to include: The historical operation information is input into the operation prediction model to obtain the heat value output by the operation prediction model. The heat value is used to indicate the operation mode of the first data object in a future preset time period. The operation prediction model is a model that predicts future operation behavior patterns based on the historical behavior time sequence patterns in the input data.

[0124] In one exemplary embodiment, the data object adjustment module is configured to: If the first data object does not belong to the target storage layer, a corresponding migration task is generated based on the popularity value of the first data object. The migration task is used to migrate the first data object to the target storage layer, which belongs to the storage system and is the storage layer with the highest read and write efficiency in the storage system. Execute the migration task.

[0125] In one exemplary embodiment, the data object adjustment module is configured to: If the popularity value corresponding to the first data object meets the preset conditions, the migration task is generated. Under the preset conditions, the first data object is determined to be accessed more frequently than a preset threshold within the preset future time period.

[0126] In one exemplary embodiment, the data object adjustment module is configured to: Determine the business weight of the business to which the first data object belongs; Determine the migration cost of the first data object and the data volume of the first data object; Based on the popularity value of the first data object, the migration cost of the first data object, the data volume of the first data object, and the business weight of the business to which the first data object belongs, the migration value of the first data object is obtained. The migration task is executed based on the migration value of the first data object.

[0127] In one exemplary embodiment, the data object adjustment module is configured to: Obtain the migration task corresponding to each of the multiple second data objects, and the migration value of each second data object; Based on each of the migration values, the migration tasks corresponding to the first data object and the migration tasks corresponding to each of the second data objects are sorted to obtain a migration task sequence; Execute each migration task in the migration task sequence.

[0128] In one exemplary embodiment, the data object adjustment module is configured to: Obtain the popularity value of each of multiple third data objects, wherein the third data objects do not belong to the target storage layer; Determine the migration cost, access importance, and data volume of each target data object, where both the first data object and the third data object belong to the target data object; Obtain the remaining capacity and current migration bandwidth of the target storage layer; With the constraint of maximizing migration benefits and minimizing migration costs, multiple data objects to be migrated are selected from each of the target data objects based on the popularity value, migration cost, access importance, data volume of each target data object, as well as the remaining capacity and current migration bandwidth of the target storage layer. The multiple data objects to be migrated include the first data object. The migration benefits are determined based on the popularity value and access importance. Generate a corresponding migration task for each of the data objects to be migrated.

[0129] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0130] Please refer to Figure 8 This diagram illustrates a structural block diagram of a computer device provided in one embodiment of this application, used to execute the aforementioned data object management method. The computer device may be a server. Specifically: Computer device 800 includes a central processing unit (CPU) 801, a system memory 804 including random access memory (RAM) 802 and read-only memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the CPU 801. Computer device 800 also includes a basic input / output system (I / O system) 806 that facilitates information transfer between various devices within the computer, and a mass storage device 807 for storing the operating system 813, application programs 814, and other program modules 815.

[0131] The basic input / output system 806 includes a display 808 for displaying information and an input device 809 for user input, such as a mouse or keyboard. Both the display 808 and the input device 809 are connected to the central processing unit 801 via an input / output controller 810 connected to the system bus 805. The basic input / output system 806 may also include the input / output controller 810 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 810 also provides output to a display screen, printer, or other types of output devices.

[0132] Mass storage device 807 is connected to central processing unit 801 via a mass storage controller (not shown) connected to system bus 805. Mass storage device 807 and its associated computer-readable media provide non-volatile storage for computer device 800. That is, mass storage device 807 may include computer-readable media (not shown) such as hard disk or CD-ROM (CompactDisc Read-Only Memory) drive.

[0133] Without loss of generality, computer-readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that computer storage media are not limited to the above-mentioned types. The system memory 804 and mass storage device 807 described above can be collectively referred to as memory.

[0134] According to various embodiments of this application, the computer device 800 can also be connected to a remote computer on a network, such as the Internet, for operation. That is, the computer device 800 can be connected to a network 812 via a network interface unit 811 connected to the system bus 805, or the network interface unit 811 can be used to connect to other types of networks or remote computer systems (not shown).

[0135] Figure 9 This is a block diagram of an electronic device according to an exemplary embodiment. The electronic device may be a terminal for executing the data object management method described above, and its internal structure diagram may be as follows. Figure 9 As shown, the device may include an RF (Radio Frequency) circuit 910, a memory 920 including one or more computer-readable storage media, an input unit 930, a display unit 940, a sensor 950, an audio circuit 960, a WiFi (Wireless Fidelity) module 970, a processor 980 including one or more processing cores, and a power supply 990, among other components. Those skilled in the art will understand that... Figure 9 The terminal structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The RF circuit 910 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and hands it over to one or more processors 980 for processing; additionally, it transmits uplink data to the base station. Typically, the RF circuit 910 includes, but is not limited to, an antenna, at least one amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, an LNA (Low Noise Amplifier), a duplexer, etc. Furthermore, the RF circuit 910 can also communicate wirelessly with networks and other terminals. Wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System for Mobile communication), GPRS (General Packet Radio Service), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), LTE (Long Term Evolution), email, SMS (Short Messaging Service), etc.

[0136] The memory 920 can be used to store software programs and modules. The processor 980 executes various functional applications and data processing by running the software programs and modules stored in the memory 920. The memory 920 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for the functions, etc.; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 920 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 920 may also include a memory controller to provide access to the memory 920 for the processor 980 and the input unit 930.

[0137] The input unit 930 can be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, the input unit 930 may include a touch-sensitive surface 931 and other input devices 932. The touch-sensitive surface 931, also known as a touch display screen or touchpad, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch-sensitive surface 931), and drive the corresponding connection device according to a pre-set program. Optionally, the touch-sensitive surface 931 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to the processor 980, and can receive and execute commands from the processor 980. In addition, the touch-sensitive surface 931 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 931, the input unit 930 may also include other input devices 932. Specifically, other input devices 932 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0138] The display unit 940 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the terminal. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit 940 may include a display panel 941, which may optionally be configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or similar display. Further, a touch-sensitive surface 931 may cover the display panel 941. When the touch-sensitive surface 931 detects a touch operation on or near it, it transmits the information to the processor 980 to determine the type of touch event. Subsequently, the processor 980 provides corresponding visual output on the display panel 941 according to the type of touch event. The touch-sensitive surface 931 and the display panel 941 can be two independent components to implement input and output functions; however, in some embodiments, the touch-sensitive surface 931 and the display panel 941 can be integrated to achieve both input and output functions.

[0139] The terminal may also include at least one sensor 950, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 941 according to the ambient light level, and the proximity sensor can turn off the display panel 941 and / or backlight when the terminal is moved to the ear. As a type of motion sensor, a gravity acceleration sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that identify the terminal's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured on the terminal, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0140] Audio circuitry 960, speaker 961, and microphone 962 provide an audio interface between the user and the terminal. Audio circuitry 960 converts received audio data into electrical signals, which are then transmitted to speaker 961, where they are converted into sound signals for output. Conversely, microphone 962 converts collected sound signals into electrical signals, which are received by audio circuitry 960, converted back into audio data, and then processed by processor 980 before being transmitted via RF circuitry 910 to, for example, another terminal, or output to memory 920 for further processing. Audio circuitry 960 may also include an earphone jack to facilitate communication between a peripheral headset and the terminal.

[0141] WiFi is a short-range wireless transmission technology. This terminal, through the WiFi module 970, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 9 The WiFi module 970 is shown, but it is understood that it is not a necessary component of the terminal and can be omitted as needed without changing the nature of the invention.

[0142] The processor 980 is the control center of the terminal, connecting various parts of the terminal via various interfaces and lines. It executes software programs and / or modules stored in the memory 920, and calls data stored in the memory 920, to perform various functions and process data, thereby enabling overall monitoring of the terminal. Optionally, the processor 980 may include one or more processing cores; preferably, the processor 980 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interaction area, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 980.

[0143] The terminal also includes a power supply 990 (such as a battery) to power various components. Preferably, the power supply can be logically connected to the processor 980 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 990 may also include one or more DC or AC power supplies, a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0144] Although not shown, the terminal may also include a camera, Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the terminal is a touch screen display, and the terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors of the instructions in the method embodiment of the present invention.

[0145] The aforementioned memory also includes a computer program stored in the memory and configured to be executed by one or more processors to implement the aforementioned data object management method.

[0146] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction, the at least one program segment, the code set, or the instruction set is executed by a processor to implement the data object management method.

[0147] Optionally, the computer-readable storage medium may include: ROM (Read Only Memory), RAM (Random Access Memory), SSD (Solid State Drives), or optical disc, etc. The random access memory may include ReRAM (Resistance Random Access Memory) and DRAM (Dynamic Random Access Memory).

[0148] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data object management method described above.

[0149] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0150] In addition, in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0151] The above are merely exemplary embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. A data object management method characterized by, The method comprises: generating historical operation information corresponding to the first data object based on historical operations corresponding to the first data object, the historical operations comprising historical read operations and historical write operations; predicting an operation mode corresponding to the first data object in a future preset time period based on the historical operation information corresponding to the first data object, the operation mode indicating an execution mode of read operations and write operations; adjusting a storage location of the first data object in a storage system based on the operation mode corresponding to the first data object in the future preset time period.

2. The method of claim 1, wherein, The prediction of the operation mode corresponding to the first data object in the future preset time period based on the historical operation information corresponding to the first data object comprises: inputting the historical operation information into an operation prediction model to obtain a heat value output by the operation prediction model, the heat value being used to indicate the operation mode corresponding to the first data object in the future preset time period; wherein the operation prediction model is a model for predicting a behavior mode of future operations based on historical behavior time sequence rules in input data.

3. The method of claim 2, wherein, The adjustment of the storage location of the first data object in the storage system based on the operation mode corresponding to the first data object in the future preset time period comprises: in a case where the first data object does not belong to a target storage layer, generating a corresponding migration task based on the heat value corresponding to the first data object, the migration task being used to migrate the first data object to the target storage layer, the target storage layer belonging to the storage system, and the target storage layer being a storage layer with the highest read-write efficiency in the storage system; executing the migration task.

4. The method of claim 3, wherein, The generation of the migration task based on the heat value corresponding to the first data object comprises: in a case where the heat value corresponding to the first data object meets a preset condition, generating the migration task, the first data object being determined to have a higher access frequency than a preset threshold in the future preset time period under the preset condition.

5. The method of claim 3, wherein, The method further comprises: determining a business weight of a business where the first data object is located; determining a migration cost of the first data object and a data amount of the first data object; obtaining a migration value of the first data object based on the heat value corresponding to the first data object, the migration cost of the first data object, the data amount of the first data object, and the business weight of the business where the first data object is located; The execution of the migration task comprises executing the migration task based on the migration value of the first data object.

6. The method of claim 5, wherein, The execution of the migration task based on the migration value of the first data object comprises: obtaining migration tasks corresponding to a plurality of second data objects respectively, and migration values of each of the second data objects; sorting the migration task corresponding to the first data object and the migration tasks corresponding to the second data objects respectively based on the migration values to obtain a migration task sequence; executing each of the migration tasks in the migration task sequence.

7. The method of claim 3, wherein, The generation of the migration task based on the heat value corresponding to the first data object comprises: obtain a plurality of third data objects each having a corresponding hotness value, the third data objects not belonging to the target storage tier; determine a migration cost, an access importance, and a data volume for each target data object, the first data object and the third data objects belonging to the target data objects; obtain a remaining capacity and a current migration bandwidth of the target storage tier; select a plurality of to-be-migrated data objects from the target data objects based on the hotness value, the migration cost, the access importance, and the data volume of each target data object, and the remaining capacity and the current migration bandwidth of the target storage tier, the to-be-migrated data objects including the first data object, the migration cost being determined based on the hotness value and the access importance, and the migration cost being maximized and the migration cost being minimized as constraints; generate a corresponding migration task for each to-be-migrated data object.

8. A data object management apparatus characterized by comprising: The apparatus comprises: a historical operation information obtaining module configured to generate historical operation information of the first data object based on historical operations of the first data object, the historical operations including historical read operations and historical write operations; an operation predicting module configured to predict an operation mode of the first data object in a future preset time period based on the historical operation information of the first data object, the operation mode indicating an execution mode of read operations and write operations; a data object adjusting module configured to adjust a storage location of the first data object in a storage system based on the operation mode of the first data object in the future preset time period.

9. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores at least one instruction, which is loaded and executed by the processor to implement the data object management method according to any one of claims 1 to 7.

10. A computer program product, characterised in that, The computer program product comprises computer instructions, and a processor of a computer device reads the computer instructions, and the processor of the computer device executes the computer instructions to implement the data object management method according to any one of claims 1 to 7.

Images