A data processing method and a data storage system

Abstract

The application provides a data processing method and a data storage system. The method comprises training, by the data storage system, a first streaming model according to a plurality of first access data features corresponding to the first file at an i-1th access request and a first access attribute parameter of the first file, then inputting a plurality of second access data features corresponding to the first file at an ith access request into the first streaming model to obtain a second access attribute parameter possessed by the first file at an i+1th access request, and then pre-fetching or migrating the first file according to the second access attribute parameter. In this way, the method trains the first streaming model using the plurality of first access data features, thereby consuming relatively few storage resources and computing resources, and enabling the first streaming model to adapt to changing file access patterns, thereby effectively improving the business performance of the data storage system.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method and a data storage system. Background Technology

[0002] Caching and tiering strategies are fundamental to building modern data storage systems. These strategies reduce latency when accessing data from cold storage media, extend the lifespan of flash memory devices, and reserve sufficient space in the cache to store new data. However, caching and tiering are highly sensitive to the workload of data storage systems (such as access requests), which are typically generated by many applications accessing thousands of files in parallel. Building such strategies requires understanding and leveraging file access patterns present in these workloads. Furthermore, the workloads handled by modern data storage systems are constantly changing, transforming through various file access patterns throughout their lifecycle. Therefore, effectively utilizing multiple file access patterns to predict future access variables for modern data storage systems is a complex task.

[0003] To address the aforementioned issues, modern data storage systems employ various models. One model (such as heuristic algorithms, neural networks, or Markov chains) processes multiple historical file access patterns of the file to be accessed, predicting the next access portion of the file (e.g., the file block or page requested in the next access request). Another model (such as reinforcement learning, neural networks, or gradient boosting trees) processes multiple historical access frequencies of the file to be accessed, predicting the access frequency of the file. However, the models involved in these solutions are trained offline, which consumes significant computing and storage resources. Furthermore, even offline-trained models tend to have lower accuracy in predicting the next access portion or access frequency of a file when file access patterns continuously change, leading to a decrease in the operational performance of the data storage system. Summary of the Invention

[0004] This application provides a data processing method and a data storage system to effectively improve the business performance of the data storage system.

[0005] In a first aspect, embodiments of this application provide a data processing method, which can be executed by a data storage system or a component (such as a chip system or circuit) capable of supporting the data storage system in implementing the functions required by the method. Optionally, taking the execution of the data processing method by a data storage system as an example, in this method, after obtaining multiple first access data features corresponding to the first file at the (i-1)th access request, the data storage system trains a first streaming model based on the multiple first access data features and the first access attribute parameters of the first file. Then, after obtaining multiple second access data features corresponding to the first file at the i-th access request, the data storage system inputs the multiple second access data features into the first streaming model to obtain the second access attribute parameters of the first file. Then, the data storage system can prefetch or migrate the first file based on the second access attribute parameters, wherein the first access attribute parameters are the actual access attribute parameters of the first file at the i-th access request, and the second access attribute parameters are the predicted access attribute parameters of the first file at the (i+1)-th access request.

[0006] In the above design, the data storage system trains the first streaming model online using multiple first access data features corresponding to the first file at the (i-1)th access request, instead of offline training the model required to predict the second access attribute parameters of the first file using massive historical sample data. Therefore, it consumes fewer storage and computing resources, enabling the data storage system to predict the second access attribute parameters of the first file at the (i+1)th access request using the first streaming model with limited storage and computing resources. Furthermore, since the first streaming model begins online training with each access request, it is constantly updated and can adapt to continuously changing file access patterns. This allows for more accurate prediction of the second access attribute parameters of the first file, effectively improving the business performance of the data storage system.

[0007] In one possible design, the data storage system acquires multiple first access data features corresponding to the first file at the (i-1)th access request, including:

[0008] The data storage system can obtain multiple first access data features corresponding to the first file at the (i-1)th access request by at least one of the following: request information, file attribute information, directory attribute information of the directory to which the first file belongs, or file format attribute information of the file format to which the first file belongs at the (i-1)th access request;

[0009] The data storage system obtains multiple second access data features corresponding to the first file at the time of the i-th access request, including:

[0010] The data storage system can obtain multiple second access data features corresponding to the first file at the time of the i-th access request through at least one of the following: request information, file attribute information, directory attribute information of the directory to which the first file belongs, or file format attribute information of the file format to which the first file belongs at the time of the i-th access request.

[0011] In the above design, the data storage system extracts multiple first access data features corresponding to the first file in the previous access request based on at least one piece of information. This ensures that the sample data required for online training of the first streaming model is up-to-date, enabling the first streaming model to capture changes in file access patterns in a timely manner. Consequently, the first streaming model can naturally adjust and adapt to changes in workload over time. Thus, by inputting the multiple first access data features extracted from at least one piece of information corresponding to the first file in the current access request into the trained first streaming model, the data storage system can more accurately determine the predicted access attribute parameters corresponding to the first file in the next access request.

[0012] In one possible design, the request information includes at least one of the following: request operation, request offset, or request length;

[0013] File attribute information includes at least one of the following: file identifier, file size, file creation time, last access time, last update time, most recent open times, or access frequency;

[0014] Directory attribute information includes at least one of the following: directory identifier, total number of multiple different files included in the directory, total access frequency of multiple files included in the directory, or percentage of access patterns of multiple files included in the directory;

[0015] File format attribute information includes at least one of the following: file format identifier, total number of multiple different files belonging to the file format, total access frequency of multiple files belonging to the file format, or percentage of access modes of multiple files belonging to the file format.

[0016] In the above design, the data storage system extracts access data features by using information of the first file at different information granularities (such as access request, file, directory to which the file belongs, and file format to which the file belongs). This allows for a more comprehensive acquisition of the data features corresponding to the first file at the time of the access request. Subsequently, the data storage system trains the first streaming model using more comprehensive data features, which improves the training accuracy of the first streaming model and thus improves its prediction accuracy.

[0017] In one possible design, the access mode includes at least one of the following: time-intensive mode, time-sparse mode, spatially ordered mode, spatially random mode, uniform length mode, variable length mode, file access frequency mode, or file full read frequency mode.

[0018] In one possible design, the data storage system acquires multiple second access data features corresponding to the first file at the time of the i-th access request, including:

[0019] The data storage system first determines multiple third access data features corresponding to the first file at the i-th access request. Then, the data storage system can select multiple second access data features from the multiple third access data features.

[0020] In the above design, the data storage system can effectively reduce the number of access data features by selecting a portion of the third access data features as second access data features and inputting them into the first streaming model for prediction. This helps to reduce the storage and computing resources consumed by the first streaming model during prediction, thereby improving the prediction efficiency of the first streaming model.

[0021] In one possible design, the data storage system selects multiple second-access data features from a plurality of third-access data features, including:

[0022] The data storage system can determine the P-values ​​of multiple third-access data features based on the P-value test, the chi-square values ​​of multiple third-access data features based on the chi-square test, and the Gini values ​​of multiple third-access data features based on the Gini metric. Then, the data storage system can perform weighted processing on the P-values, chi-square values, and Gini values ​​of each third-access data feature to determine the weighted value of each third-access data feature. Finally, the data storage system can select multiple second-access data features whose weighted values ​​are greater than or equal to a first set threshold from the multiple third-access data features.

[0023] In the above design, the data storage system can retain only the access data features that have a good impact on the prediction effect of the first streaming model by removing or deleting some unrepresentative access data features. This reduces the number of access data features required for the first streaming model to make predictions, helps to reduce the storage and computing resources consumed by the first streaming model to make predictions, and can reduce possible noise in the first streaming model prediction process, thereby effectively improving the prediction efficiency and accuracy of the first streaming model.

[0024] In one possible design, the data storage system selects multiple second-access data features from a plurality of third-access data features, including:

[0025] The data storage system first determines the correlation between any two third access data features among multiple third access data features. When any correlation is greater than a second set threshold, one of the two third access data features corresponding to that correlation is removed. After that, the data storage system can use the remaining third access data features other than the removed third access data features as multiple second access data features.

[0026] In the above design, the data storage system can reduce the possible noise in the first-stream model prediction process by removing or deleting some highly correlated access data features, and can also reduce the number of access data features required for the first-stream model prediction. This helps to reduce the storage and computing resources consumed by the first-stream model prediction, thereby effectively improving the prediction efficiency and accuracy of the first-stream model.

[0027] In one possible design, the first access attribute parameter is one of the following: actual request offset, actual request offset category, actual access popularity, or actual access popularity category; the second access attribute parameter is one of the following: predicted request offset, predicted request offset category, predicted access popularity, or predicted access popularity category.

[0028] Secondly, embodiments of this application provide a possible data storage system, the beneficial effects of which are described in the first aspect and will not be repeated here. The data storage system has the function of implementing the behavior in the method example of the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. In one possible design, the data storage system includes an acquisition module and a processing module. The acquisition module is used to acquire multiple first access data features corresponding to the first file at the (i-1)th access request. The processing module is used to train a first streaming model based on the multiple first access data features and the first access attribute parameters of the first file. The first access attribute parameters are the actual access attribute parameters of the first file at the i-th access request. The acquisition module is also used to acquire multiple second access data features corresponding to the first file at the i-th access request. The processing module is also used to input the multiple second access data features into the first streaming model to obtain the second access attribute parameters of the first file. The second access attribute parameters are the predicted access attribute parameters of the first file at the (i+1)th access request. The processing module is also used to prefetch or migrate the first file based on the second access attribute parameters. These modules can perform the corresponding functions in any of the possible designs in the first aspect described above, as detailed in the method examples, which will not be repeated here.

[0029] Thirdly, embodiments of this application provide a possible data storage system, which includes a communication interface and a processor. Optionally, the data storage system further includes a memory. The memory stores computer programs or instructions, and the processor is coupled to the memory and the communication interface. When the processor executes the computer program or instructions, the data storage system performs any of the possible design methods described in the first aspect above.

[0030] Fourthly, embodiments of this application provide a computer program product, which includes a computer program or instructions that, when executed on a computer, cause the computer to perform any of the possible designs in the first aspect described above.

[0031] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed by a computer, cause the computer to perform any of the possible designs in the first aspect described above.

[0032] In a sixth aspect, embodiments of this application also provide a chip coupled to a memory, the chip being used to read a computer program stored in the memory and execute the method in any of the possible designs of the first aspect described above.

[0033] In a seventh aspect, embodiments of this application also provide a chip system including a processor for supporting a computer device in implementing any of the possible designs in the first aspect described above. In one possible design, the chip system further includes a memory for storing programs and data necessary for the computer device. The chip system may be composed of chips or may include chips and other discrete devices.

[0034] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0035] Figure 1 An exemplary schematic diagram illustrates a possible application scenario provided by an embodiment of this application;

[0036] Figure 2 This illustration shows a functional module structure diagram of a data storage system provided in an embodiment of this application;

[0037] Figure 3 An exemplary flowchart of a data processing method provided in an embodiment of this application is shown.

[0038] Figure 4 An exemplary schematic diagram of a possible data storage system provided in an embodiment of this application is shown;

[0039] Figure 5 An exemplary schematic diagram of a possible data storage system provided in an embodiment of this application is shown. Detailed Implementation

[0040] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0041] The following describes possible application scenarios for this application. It should be noted that these descriptions are for the purpose of facilitating understanding by those skilled in the art and do not constitute a limitation on the scope of protection claimed in this application.

[0042] Figure 1 An illustrative diagram shows a possible application scenario to which this application applies. For example... Figure 1 As shown, this application scenario may include terminal 100 and data storage system 200 (such as a hybrid data storage system, or a multi-level storage system).

[0043] Terminal 100 can be a user-side entity with signal transmission and reception capabilities, providing users with services such as audio, video, voice, and data connectivity. Optionally, terminal 100 can also have data processing capabilities. For example, terminal 100 can send user-submitted data access requests to data storage system 200, allowing users to access relevant data stored in data storage system 200.

[0044] For example, terminal 100 may be a smartphone, tablet computer, desktop computer, computer with wireless transceiver capabilities (such as a laptop computer), PPD (Personal Digital Assistant), mobile internet device (MID), in-vehicle terminal (such as a cockpit infotainment system), wearable device with wireless communication capabilities (such as a smartwatch, smart bracelet, smart glasses, or smart helmet), virtual reality (VR) device, augmented reality (AR) device, or smart home device (such as a smart speaker or smart TV). It should be understood that this application does not limit the specific device form of the terminal.

[0045] Data storage system 200 can be an entity with data processing capabilities and the ability to store large amounts of data, providing users with data access services or data storage services. For example, after receiving a user's data access request, data storage system 200 can provide the user with the data they need.

[0046] Optionally, the terminal 100 and the data storage system 200 can communicate with each other. For example, the terminal 100 and the data storage system 200 can communicate with each other via a wired network or a wireless network, and this application embodiment does not limit this.

[0047] Optionally, when the terminal 100 and the data storage system 200 communicate via a wireless network, the wireless network can be a wireless local area network (WLAN), such as a wireless fidelity (Wi-Fi) network, a ZigBee network, a Bluetooth (BT) network, or a near field communication (NFC) network, or other forms of communication network. This application embodiment does not limit this.

[0048] It should be noted that the above Figure 1 The application scenario shown is merely an example, intended to more clearly illustrate the technical solutions of the embodiments of this application, and does not constitute a limitation on the application scenarios of the data processing method provided in this application. Furthermore, Figure 1 The forms and quantities of the various structures in the illustrated application scenarios are for illustrative purposes only and do not constitute a limitation of this application. Furthermore, Figure 1 The names of the structures shown in the application scenario are just examples. In the actual implementation, the names of the structures may be different, and this application does not make any specific restrictions on them.

[0049] As described in the background section, in existing data storage systems where file access patterns are constantly changing, the accuracy of using offline-trained models to predict the next access portion or access frequency of a file is relatively low, leading to a decrease in the business performance of the data storage system. Therefore, this application provides a data processing method to effectively improve the business performance of the data storage system.

[0050] based on Figure 1 As illustrated in the application scenario, this application also provides a functional module structure for a data storage system. See [link / reference] Figure 2Based on logical functions, a data storage system can be divided into the following functional modules: a data prediction module, a data caching module, a data grading module (or data migration module), or a multi-level storage module. Optionally, the data prediction module may include, but is not limited to, at least one of the following: an access pattern recognition submodule, a feature extraction submodule, a feature selection submodule, or a model training / prediction submodule. Optionally, the multi-level storage module may include, but is not limited to, at least one of the following: cache media, main memory media, solid-state drive (SSD) media, hard disk drive (HDD) media, magnetic tape media, or optical disc media.

[0051] It should be noted that, Figure 2 The connections between the functional modules shown are merely an example and do not constitute a limitation of this application. The functions of each module are described below.

[0052] The data prediction module is used to predict the access attribute parameters (such as request offset or access popularity) of a file in the (i+1)th access request based on the access data characteristics of the file in the i-th access request.

[0053] The access pattern recognition submodule is used to identify the access pattern of a file in the i-th access request after receiving the i-th access request submitted by the user for a certain file.

[0054] The feature extraction submodule is used to extract the request data features carried in the i-th access request (i.e., the data features used to characterize the i-th access request), or to extract the file data features contained in the metadata of the file (i.e., the data features used to characterize the file), or to extract the directory data features of the directory to which the file belongs (i.e., the data features used to characterize the directory to which the file belongs), or to extract the file format data features of the file format to which the file belongs (i.e., the data features used to characterize the file format to which the file belongs).

[0055] The feature selection submodule is used to select a target data feature (i.e., at least one data feature used for model prediction or training) from multiple data features extracted by the feature extraction submodule.

[0056] The model training / prediction submodule is used to train or predict the model based on the target data features selected by the feature selection submodule.

[0057] The data caching module is used to prefetch the aforementioned file according to the predicted request offset or request offset category corresponding to the (i+1)th access request. For example, taking the predicted request offset (e.g., request offset a) corresponding to a certain file (e.g., file 1) in the (i+1)th access request as an example, the data caching module can determine the data (e.g., file block 2) required for the (i+1)th access request of file 1 based on the predicted request offset a, and can determine the storage location of file block 2, that is, determine which storage medium file block 2 is stored in, such as storing file block 2 in an HDD medium. Since HDD media is a low-performance storage medium, the data caching module can prefetch file block 2 into a high-performance storage medium (such as SSD media), that is, copy file block 2 to the SSD media. In this way, when the (i+1)th access request for file 1 arrives, file block 2 has a relatively high probability of being hit on the SSD media, thus greatly shortening the access latency of the data to be accessed (such as file block 2), thereby effectively improving the business performance of the data storage system.

[0058] The data classification module is used to perform corresponding migration processing on the above files based on the predicted access popularity or access popularity category corresponding to the (i+1)th access request. For example, taking the predicted access popularity (e.g., access popularity 0.8) of a certain file (e.g., file 1) in the (i+1)th access request as an example, since an access popularity of 0.8 indicates that file 1 is accessed frequently, that is, file 1 is a hot file, therefore, if some of the multiple file blocks included in file 1 are stored in low-performance storage media and some are stored in high-performance storage media, the data classification module needs to migrate the file blocks stored in low-performance storage media to high-performance storage media; if all the file blocks included in file 1 are stored in low-performance storage media, the data classification module needs to migrate all the file blocks stored in low-performance storage media to high-performance storage media.

[0059] Multi-level storage modules are used to store corresponding data. For example, a portion of the data is stored in cache media, a portion in main memory media, a portion in SSD media, a portion in HDD media, a portion in magnetic tape media, and a portion in optical disc media.

[0060] The following is based on Figure 1 The application scenarios shown in this application provide a detailed description of the specific implementation of the data processing method in the embodiments of this application.

[0061] Figure 3This illustration shows a schematic flowchart of a data processing method provided in an embodiment of this application. The method is applicable to... Figure 1 The illustrated application scenario. This method can be executed by a data storage system or a component (such as a chip system or circuit) that supports the data storage system in implementing the required functions of this method. Optionally, the data storage system can be, for example,... Figure 1 The illustrated data storage system 200. To facilitate the introduction of the technical solutions provided in the embodiments of this application, the following description uses the data storage system performing a data processing method as an example. Figure 3 As shown, the method includes:

[0062] Step 301: The data storage system obtains multiple first access data features corresponding to the first file during the (i-1)th access request.

[0063] Optionally, the first access data feature may include the request data feature, file data feature, directory data feature, or file format data feature corresponding to the first file in the (i-1)th access request. Optionally, the first access data feature may also include the access frequency or the complete read frequency of the first file corresponding to the first file in the (i-1)th access request.

[0064] Specifically, the request data feature corresponding to the first file in the (i-1)th access request is obtained by the data storage system through feature extraction of the request information carried in the (i-1)th access request; the file data feature corresponding to the first file in the (i-1)th access request is obtained by the data storage system through feature extraction of the file attribute information corresponding to the first file in the (i-1)th access request; the directory data feature corresponding to the first file in the (i-1)th access request is obtained by the data storage system through feature extraction of the directory attribute information corresponding to the first file in the (i-1)th access request; and the file format data feature corresponding to the first file in the (i-1)th access request is obtained by the data storage system through feature extraction of the request data information carried in the (i-1)th access request. The first file's access frequency at the (i-1)th access request is obtained by feature extraction of the file format attribute information corresponding to the first file. The access frequency of the first file at the (i-1)th access request is determined by the data storage system based on the total number of accesses of the first file within a certain period (e.g., a sliding time window before the access time of the (i-1)th access request, for example, the length of the sliding time window is 1 hour). The complete read frequency of the first file at the (i-1)th access request is determined by the data storage system based on the request length carried in multiple access requests of the first file within a certain period (e.g., a sliding time window before the access time of the (i-1)th access request).

[0065] For example, see Table 1, which presents the access data characteristics of any file at different information granularities (such as access request, file, directory, and file format).

[0066] Table 1

[0067]

[0068]

[0069] It should be noted that Table 1 is only a simple example, used to illustrate the technical solutions in the embodiments of this application, and does not constitute a limitation on the technical solutions in the embodiments of this application.

[0070] Optionally, the request information may include, but is not limited to, at least one of the following: request operation, request offset, or request length; file attribute information may include, but is not limited to, at least one of the following: file identifier, file size, file creation time, last access time, last update time, most recent open times, or access popularity; directory attribute information may include, but is not limited to, at least one of the following: directory identifier, total number of different files included in the directory, total access frequency of the multiple files included in the directory, or percentage of access patterns of the multiple files included in the directory; file format attribute information may include, but is not limited to, at least one of the following: file format identifier, total number of different files belonging to the file format, total access frequency of the multiple files belonging to the file format, or percentage of access patterns of the multiple files belonging to the file format.

[0071] For example, the access patterns of a file may include, but are not limited to, at least one of the following: time pattern, spatial pattern, length pattern, or frequency pattern. Among them, the time pattern includes time-dense pattern and time-sparse pattern; the spatial pattern includes spatially ordered pattern and spatially random pattern; the length pattern includes uniform length pattern and variable length pattern; and the frequency pattern includes file access frequency pattern and file full read frequency pattern.

[0072] The time pattern refers to the frequency with which a file (e.g., file 1) is requested for access within a certain period (e.g., a sliding time window before the access time of the current access request for file 1). Optionally, the access frequency of a file may be dense or sparse, thus the time pattern can be divided into time-intensive and time-sparse patterns. For example, to identify the time pattern of any file, it is necessary to obtain the access times of the last n access requests for that file. This requires the data storage system to store the access times of the last n access requests for that file. For instance, for a file (e.g., file 1), the data storage system can calculate the difference between any two consecutive access times among the last n access requests for file 1, obtaining multiple first differences. If all of these first differences are less than or equal to a preset threshold, the data storage system can determine that file 1 has a time-intensive access pattern. If at least one of these first differences is greater than the preset threshold, the data storage system can determine that file 1 has a time-sparse access pattern.

[0073] Spatial patterns represent the location of data (such as file blocks or file pages) being accessed within a file (i.e., the request offset carried in the access request submitted by the user to request access to the file). Optionally, access requests for data contained in a file can be sequential or non-sequential, thus spatial patterns can be divided into spatially sequential patterns and spatially random patterns. Spatially sequential patterns refer to continuous access to data contained in a file, with no gaps between requests. Spatially random patterns refer to non-strictly sequential access requests, following a certain ordering rule. For example, to identify the spatial pattern of any file (e.g., file 1), it is necessary to obtain multiple access requests for file 1 within a certain period (e.g., a sliding time window before the current access time of file 1), excluding the last access request for file 1. Then, the data storage system can calculate the difference between the request offsets carried by any two consecutive access requests within that period, obtaining multiple second differences. If multiple second differences are constant values ​​and equal to the requested length (or the requested data size), the data storage system can determine that file 1 has a spatial sequential mode. If multiple second differences are constant values ​​and equal to the sum of the requested length and a set constant, the data storage system can determine that file 1 has a spatial step mode. If at least one of the multiple second differences is not a constant value, the data storage system can determine that file 1 has a spatial random mode.

[0074] It's important to note that a time window consists of a set of time slices (i.e., multiple segments of the same size). For example, assuming a time window is 1 hour long and each time slice is 10 seconds long, such as a time slice from 8:00:10 to 9:00:10, then the time window comprises 6 time slices. Each time slice stores the status information of files, directories, and file formats accessed within a consecutive time interval. Because there can be significant time intervals between access requests (e.g., a request generated on Monday morning may have a large time interval with a request generated in the evening), the time intervals are continuous within a time slice, but there may be gaps between time slices. A time window only retains time slices within its specified duration; any time slices exceeding this duration are discarded. The time window and time slices are configurable parameters. By default, time slices retain information for 10 seconds, and the time window is set to 1 hour.

[0075] To optimize performance, a time window has a special time slice called an aggregate time slice, which stores aggregated information across time slices within the time window. For example, taking file 1 as the first file, a time window has 60 time slices. File 1 has a different number of accesses in each time slice. The data storage system will aggregate the accesses of file 1 across the 60 time slices to obtain the total number of accesses of file 1 within the time window. The aggregate time slice is updated each time a time slice is created or updated. Similarly, the aggregate time slice is updated when a time slice is discarded. Therefore, the aggregate time slice always maintains the latest information and uses it to extract features when needed. In this way, time windows and time slices provide the following advantages: (1) They bind the amount of stored information about files, directories, and file formats accessed only within the time window; (2) They can efficiently calculate the aggregate state as time moves forward without recalculating any values; (3) Old information is discarded from the data storage system without affecting the future decisions of the data storage system. This property can be used to observe changes and evolutions in access patterns over time.

[0076] Length patterns represent variations in the length of data requested to be read or written in file access requests. Optionally, the size of the data requested to be read or written in a file access request may be uniform or variable in length. Thus, length patterns can be categorized into uniform length patterns and variable length patterns. For example, to identify the length pattern of any file (e.g., file 1), it is necessary to obtain multiple access requests for file 1 over a period of time (e.g., a sliding time window before the access time of the current access request for file 1). If any two consecutive access requests within this time period carry the same request length, the data storage system can determine that file 1 has a uniform length pattern. If any two consecutive access requests within this time period carry different request lengths, the data storage system can determine that file 1 has a variable length pattern.

[0077] The frequency pattern represents the number of times a file (e.g., file 1) is accessed within a certain period (e.g., a sliding time window before the access time of the current access request for file 1). Optionally, there are two forms of frequency patterns. The first form is the file access frequency, represented by a number between 0 and 1, which can be expressed using a logarithmic function. For each file, a counter is configured to count the number of times the file is accessed. Then, the data storage system can calculate the access frequency of the file based on the number of accesses counted by the counter, i.e., (1-1 / log(c+2)), where c represents the total number of times the file is accessed within that period. The second form is the file complete read frequency, which is an approximation of the number of times any file is completely read. In practice, the complete read frequency of a file can be represented by the ratio between the total number of bytes accessed by the file within that period and the file size. The total number of bytes accessed by the file within that period can be obtained from the request length carried in multiple access requests for the file within that period.

[0078] Optionally, once the data storage system identifies the access patterns of any file, it can aggregate these access patterns into the access patterns corresponding to the directory to which the file belongs, or it can aggregate them into the access patterns corresponding to the file format to which the file belongs. In one example, the access patterns corresponding to the directory to which the file belongs and the access patterns corresponding to the file format to which the file belongs can be the same; for example, both the access patterns corresponding to the directory to which the file belongs and the access patterns corresponding to the file format to which the file belongs may include time mode, spatial mode, length mode, and frequency mode. In another example, the access patterns corresponding to the directory to which the file belongs and the access patterns corresponding to the file format to which the file belongs can be different; for example, the access patterns corresponding to the directory to which the file belongs may include time mode, spatial mode, and length mode, while the access patterns corresponding to the file format to which the file belongs may include time mode and spatial mode.

[0079] Optionally, for files located in the same directory, these files are generally considered to have the same or similar access patterns. For example, taking files 1 and 2 located in the same directory, if the data storage system identifies file 1 as having a time-intensive pattern and a spatial pattern as having a spatial order pattern, and if the data storage system determines that files 1 and 2 are in the same directory, then the data storage system can also determine that file 2 has a time-intensive pattern and a spatial order pattern. Optionally, for files belonging to the same file format, these files are generally considered to have the same or similar access patterns. For example, taking files 1 and 2 belonging to the same file format, if the data storage system identifies file 1 as having a time-intensive pattern and a spatial pattern as having a spatial order pattern, and if the data storage system determines that files 1 and 2 belong to the same file format, then the data storage system can also determine that file 2 has a time-intensive pattern and a spatial order pattern.

[0080] For example, based on the above, the following are some possible examples illustrating the implementation process of a data storage system determining the proportion of access modes corresponding to a directory.

[0081] Optionally, before obtaining the directory data features corresponding to the first file in the (i-1)th access request, the data storage system needs to identify the access patterns of multiple files contained in the directory to which the first file belongs. Then, the data storage system aggregates the identified access patterns of the multiple files to obtain the percentage of access patterns corresponding to that directory.

[0082] Example 1: Suppose a directory (e.g., directory a) contains three files: file 1, file 2, and file 3. Assume the (i-1)th access request received by the data storage system is for file 1. Taking the example that all three files have time, space, and length patterns, when the data storage system receives the (i-1)th access request, it determines that this request is for file 1 and identifies the directory to which file 1 belongs as directory a. Then, based on directory a, the data storage system can determine that directory a contains file 1, file 2, and file 3. The data storage system can then perform access pattern recognition on file 1, file 2, and file 3 respectively. Optionally, the data storage system first obtains the access request information for file 1, file 2, and file 3 within a sliding time window before the access time of the (i-1)th access request. For example, assuming the sliding time window is one hour long and the access time of the (i-1)th access request is t0, the data storage system obtains multiple access requests for file 1, file 2, and file 3 within the time period (t0-1 to t0).

[0083] For file 1, in one example, the data storage system can calculate the difference between any two consecutive access times of the last n access requests within the time period (t0-1 to t0) of file 1, obtaining multiple first differences. When all of the multiple first differences are less than or equal to a preset threshold, the data storage system can determine that file 1 has a time-dense pattern. In another example, the data storage system can calculate the difference between the request offsets carried by any two consecutive access requests of file 1 within the time period (t0-1 to t0), obtaining multiple second differences. When all of the multiple second differences are constant values ​​and equal to the request length, the data storage system can determine that file 1 has a spatial pattern of spatial order. In yet another example, when the request lengths carried by any two consecutive access requests of file 1 within the time period (t0-1 to t0) are the same, the data storage system can determine that file 1 has a length pattern of uniform length. Optionally, when file 1 also has a frequency pattern, the data storage system can calculate the sum of the request lengths carried in multiple access requests for file 1 within the time period (t0-1 to t0), and use this sum as the total number of bytes accessed by file 1 within the time period (t0-1 to t0). Then, the data storage system can calculate the ratio of the total number of bytes accessed by file 1 within the time period (t0-1 to t0) to the file size of file 1, and use this ratio as the complete read frequency of file 1. The complete read frequency of file 1 is used to characterize the complete read frequency pattern of file 1. Furthermore, the data storage system can also calculate the total number of accesses to file 1 within the time period (t0-1 to t0), and calculate the access frequency of file 1 based on the total number of accesses within the time period (t0-1 to t0). The access frequency of file 1 is used to characterize the file access frequency pattern of file 1.

[0084] For file 2, the data storage system can calculate the difference between any two consecutive access times among the last n access requests of file 2 within the time period (t0-1 to t0), obtaining multiple first differences. When all of the multiple first differences are less than or equal to a preset threshold, the data storage system can determine that file 2 has a time-dense pattern. In another example, the data storage system can calculate the difference between the request offsets carried by any two consecutive access requests of file 2 within the time period (t0-1 to t0), obtaining multiple second differences. When at least one of the multiple second differences is not a constant value, the data storage system can determine that file 2 has a spatial pattern of spatial randomness. In yet another example, when the request lengths carried by any two consecutive access requests of file 2 within the time period (t0-1 to t0) are the same, the data storage system can determine that file 2 has a length pattern of uniform length.

[0085] For file 3, the data storage system can calculate the difference between any two consecutive access times of the last n access requests within the time period (t0-1 to t0) of file 3, obtaining multiple first differences. When all of the multiple first differences are less than or equal to a preset threshold, the data storage system can determine that file 3 has a time-intensive pattern. In another example, the data storage system can calculate the difference between the request offsets carried by any two consecutive access requests of file 3 within the time period (t0-1 to t0), obtaining multiple second differences. When at least one of the multiple second differences is not a constant value, the data storage system can determine that file 3 has a spatial random pattern. In yet another example, when the request lengths carried by any two consecutive access requests of file 3 within the time period (t0-1 to t0) are not the same, the data storage system can determine that file 3 has a length variable pattern.

[0086] Then, the data storage system can aggregate the access patterns of the three identified files to determine the proportion of each access pattern corresponding to directory a. For example, the proportion of time-intensive mode corresponding to directory a is 3 / 3 = 1, the proportion of spatially ordered mode corresponding to directory a is 1 / (1+2) = 1 / 3, the proportion of spatially random mode corresponding to directory a is 2 / (1+2) = 2 / 3, the proportion of uniform length mode corresponding to directory a is 2 / (1+2) = 2 / 3, and the proportion of variable length mode corresponding to directory a is 1 / (1+2) = 1 / 3.

[0087] Example 2: Continuing with the example of directory a containing files 1, 2, and 3, and the (i-1)th access request obtained by the data storage system requesting access to file 1, assume that all three files have a time pattern, a spatial pattern, and a length pattern. Optionally, if the data storage system identifies that file 1 has a time pattern of temporal sparseness, a spatial pattern of spatial order, and a length pattern of variable length, then when determining that files 1, 2, and 3 belong to the same directory a, the data storage system can determine, based on the fact that multiple files belonging to the same directory have the same or similar access patterns, that file 2 also has a time pattern of temporal sparseness, a spatial pattern of spatial order, and a length pattern of variable length; file 3 also has a time pattern of temporal sparseness, a spatial pattern of spatial order, and a length pattern of variable length. Next, the data storage system can aggregate the access patterns of the three identified files, that is, determine the proportion of each access pattern corresponding to directory a. For example, the proportion of the temporally sparse pattern corresponding to directory a is 3 / (1+1+1) = 1, the proportion of the spatially ordered pattern corresponding to directory a is 3 / (1+1+1) = 1, and the proportion of the uniform length pattern corresponding to directory a is 3 / (1+1+1) = 1. It should be noted that the process of identifying the access patterns of the files in Example 2 can refer to the process of identifying the access patterns of the files in Example 1 above, and will not be repeated here.

[0088] Example 3: Continuing with the example that directory a includes files 1, 2, and 3, and the (i-1)th access request from the data storage system is for file 1, assume file 1 has a time pattern and a spatial pattern, file 2 has a time pattern and a length pattern, and file 3 has a spatial pattern and a length pattern. When the data storage system identifies file 1's time pattern as time-dense, its spatial pattern as spatially ordered, file 2's time pattern as time-sparse, its length pattern as uniform, and file 3's spatial pattern as spatially ordered, and file 3 has... After the length pattern is set to variable length, the access patterns of the three identified files can be aggregated. This means determining the proportion of each access pattern corresponding to directory 'a'. For example, the proportion of time-dense pattern for directory 'a' is 1 / (1+1+0) = 1 / 2, the proportion of time-sparse pattern is 1 / (1+1+0) = 1 / 2, the proportion of spatially ordered pattern is 2 / (1+0+1) = 1, the proportion of uniform length pattern is 1 / (0+1+1) = 1 / 2, and the proportion of variable length pattern is 1 / (0+1+1) = 1 / 2. It should be noted that the file access pattern identification process in Example 3 can refer to the file access pattern identification process in Example 1 above, and will not be repeated here.

[0089] Optionally, when the data storage system obtains the directory data characteristics corresponding to the first file at the (i-1)th access request, it can determine the access frequency (or access count) of each file contained in the directory to which the first file belongs, and aggregate the determined access frequencies of each file to obtain the total access frequency of the files contained in the directory. For example, continuing with the example that directory a contains 3 files, assume that the (i-1)th access request obtained by the data storage system is a request to access file 1. When the data storage system obtains the (i-1)th access request, this (i-1)th access request is used to request access to file 1, and determines that the directory to which file 1 belongs is directory a. Then, based on directory a, the data storage system can determine that directory a includes file 1, file 2, and file 3. Then, the data storage system can determine the access frequency of file 1 as 5 within a certain period (e.g., a sliding time window before the access time of the (i-1)th access request of file 1), the access frequency of file 2 as 3 within the same period, and the access frequency of file 3 as 8 within the same period. Afterward, the data storage system aggregates the access frequencies of the three files to obtain the total access frequency for directory a = (5 + 3 + 8) = 16.

[0090] Optionally, the data storage system can also determine the total number of different files contained in the directory to which the first file belongs when it obtains the directory data characteristics corresponding to the (i-1)th access request for the first file. For example, continuing with the example that directory a contains 3 files, assuming that the (i-1)th access request obtained by the data storage system is a request to access file 1, when the data storage system obtains the (i-1)th access request, it can determine that the total number of files contained in the directory a is 3.

[0091] For example, the following examples illustrate the implementation process of a data storage system determining the proportion of access modes corresponding to file formats.

[0092] Optionally, before obtaining the file format data characteristics of the first file at the (i-1)th access request, the data storage system needs to identify the access patterns of multiple files belonging to the same file format as the first file. Then, the data storage system aggregates the identified access patterns of the multiple files to obtain the proportion of access patterns corresponding to that file format.

[0093] Example 1: Suppose there are three files belonging to a certain file format (e.g., file format a): file 1, file 2, and file 3. Assume the (i-1)th access request received by the data storage system is for file 1. Taking the example that all three files have time mode, space mode, and length mode, when the data storage system receives the (i-1)th access request, it determines that this request is for file 1 and identifies file 1 as file format a. Then, based on file format a, the data storage system can determine that the files belonging to file format a are file 1, file 2, and file 3. Then, the data storage system can perform access pattern recognition on file 1, file 2 and file 3 respectively, and identify that file 1 has a time pattern of time sparsity, a spatial pattern of spatial order, and a length pattern of variable length; file 2 has a time pattern of time density, a spatial pattern of spatial order, and a length pattern of uniform length; file 3 has a time pattern of time density, a spatial pattern of spatial randomness, and a length pattern of variable length. Then, the data storage system can aggregate the access patterns of the three identified files to determine the proportion of each access pattern corresponding to file format 'a'. For example, the proportion of time-dense pattern corresponding to file format 'a' is 2 / (1+2) = 2 / 3, the proportion of time-sparse pattern is 1 / (1+2) = 1 / 3, the proportion of spatially ordered pattern is 2 / (1+2) = 2 / 3, the proportion of spatially random pattern is 1 / (1+2) = 1 / 3, the proportion of uniform length pattern is 1 / (1+2) = 1 / 3, and the proportion of variable length pattern is 2 / (1+2) = 2 / 3. It should be noted that the process of identifying file access patterns in Example 1 can refer to the process of identifying file access patterns in Example 1 in the implementation of determining the proportion of access patterns corresponding to directories, and will not be repeated here.

[0094] Example 2: Continuing with the example of files 1, 2, and 3 belonging to file format a, and the (i-1)th access request received by the data storage system requesting access to file 1, assume that all three files have a time pattern, a spatial pattern, and a length pattern. Optionally, if the data storage system identifies that file 1 has a time pattern of temporal sparseness, a spatial pattern of spatial order, and a length pattern of variable length, then when determining that files 1, 2, and 3 belong to the same file format a, the data storage system can, based on the fact that multiple files belonging to the same file format have the same or similar access patterns, determine that file 2 also has a time pattern of temporal sparseness, a spatial pattern of spatial order, and a length pattern of variable length; file 3 also has a time pattern of temporal sparseness, a spatial pattern of spatial order, and a length pattern of variable length. Subsequently, the data storage system can aggregate the access patterns of the three identified files, that is, determine the proportion of each access pattern corresponding to file format 'a'. For example, the proportion of the temporally sparse pattern corresponding to file format 'a' is 3 / (1+1+1) = 1, the proportion of the spatially ordered pattern corresponding to file format 'a' is 3 / (1+1+1) = 1, and the proportion of the uniformly long pattern corresponding to file format 'a' is 3 / (1+1+1) = 1. It should be noted that the process of identifying file access patterns in Example 2 can refer to the process of identifying file access patterns in Example 1 during the implementation of determining the proportion of access patterns corresponding to directories, and will not be repeated here.

[0095] In one possible implementation, when the data storage system receives the (i-1)th access request for the first file, it can parse the (i-1)th access request to obtain the request operation, request timestamp, filename, request offset, request length, file size, process ID, and duration of the request operation carried in the (i-1)th access request. Then, the data storage system can extract the request data features corresponding to the (i-1)th access request. These request data features may include, but are not limited to, the data features of the request operation, the data features of the request offset, and the data features of the request length. Optionally, these request data features may also include the data features of the total access frequency of the first file within a certain period (e.g., a sliding time window before the access time of the (i-1)th access request for the first file), the data features of the number of times the first file is opened within that period, or the data features of the complete read frequency of the first file, etc.

[0096] Optionally, for the request operation carried in the (i-1)th access request, the data storage system can convert the request operation into a class with a specific number, represented by a number between 1 and 5. Here, number 1 represents an open operation, number 2 represents a write operation, number 3 represents a read operation, number 4 represents a close operation, and number 5 represents a delete operation. The converted number is then standardized. For example, if the request operation carried in the (i-1)th access request is a read operation, the data storage system converts the read operation into the number 3, divides the number 3 by 5, and obtains the value 0.6. This value 0.6 is used to characterize the data characteristics of the request operation. For the request offset (e.g., request offset a) carried in the (i-1)th access request, the data storage system can standardize the request offset a, obtaining the standardized value (a / M)^0.2. This value (a / M)^0.2 is used to characterize the data characteristics of the request offset. For the request length (e.g., request length L) carried in the (i-1)th access request, the data storage system can standardize the request length L to obtain a standardized value of (L / M)^0.2. This value (L / M)^0.2 is used to characterize the data feature of the request length. Here, M represents the largest file size among the file sizes carried in multiple access requests acquired by the data storage system within a certain period (e.g., a sliding time window before the access time of the (i-1)th access request of the first file).

[0097] The process ID is used to identify the application that submitted the (i-1)th access request; the request timestamp is used to indicate the submission time of the (i-1)th access request; the request operation includes file operations such as opening, closing, reading, or writing; the filename is used to indicate the filename of the first file to be accessed; the request offset is used to indicate the file position (in bytes) at which processing begins when reading or writing the file; the request length is used to indicate the amount of data (in bytes) that the request operation needs to process; the file size is used to indicate the current size of the first file; and the duration of the request operation is used to indicate the total time required to process the (i-1)th access request.

[0098] In another possible implementation, after the data storage system obtains the filename carried in the (i-1)th access request, it can query the metadata of the first file corresponding to that filename and extract file data features from the metadata. These file data features may include, but are not limited to, file identifier data features, file size data features, file creation time data features, last access time data features, last update time data features, most recent open times data features, and current access popularity data features. Specifically, the current size of the first file can be used to characterize the file size, the creation time of the first file can be used to characterize the file creation time, the access time corresponding to the (i-2)th access request for the first file can be used to characterize the last access time, and the most recent update time of the first file can be used to characterize the last update time.

[0099] Optionally, for the data characteristics of the file identifier (e.g., identifier 'a') of the first file, the data storage system can normalize identifier 'a' to obtain a normalized value of |hash value of identifier 'a'|%M / M. This value, |hash value of identifier 'a'|%M / M, is used to characterize the data characteristics of the file identifier 'a' of the first file. Alternatively, the data storage system can also normalize the filename (e.g., filename 'd') of the first file to obtain a normalized value of |hash value of filename 'd'|%M / M. This value, |hash value of filename 'd'|%M / M, can also be used to characterize the data characteristics of the file identifier 'a' of the first file.

[0100] For the data characteristics of the most recent multiple opening times of the first file, the data storage system can use the most recent multiple opening times before the (i-1)th access request to characterize the data characteristics of the most recent multiple opening times of the first file. Optionally, the data storage system can also calculate the time difference between any consecutive opening times among the most recent multiple opening times before the (i-1)th access request, obtain multiple time differences, and normalize these multiple time differences to obtain multiple normalized values. These normalized values ​​are used to characterize the data characteristics of the most recent multiple opening times of the first file.

[0101] To determine the current access popularity of the first file, the data storage system can first obtain the amount of data read at each access time within a certain period (e.g., a sliding time window, where the sliding time window is 1 hour). Then, the data storage system can weight the amount of data read at each access time and use the ratio of the weighted data read to the current size of the first file as the current access popularity of the first file. That is, the current access popularity of the first file = the amount of data read at each access time t within a certain period * weight w(t) / the current size of the first file. Here, the weight w(t) of the amount of data read at each access time t is related to the current access time t. 当前 The access frequency is inversely proportional to the difference between the access times t and t. For example, taking file 1 as the first file, assume that file 1 contains four access times within a certain period, namely time t1, time t2, time t3, and time t4, where time t4 is the current access time of file 1. Assume that the current size of file 1 is B, and that the amount of data read by file 1 at time t1 is A1, the amount of data read by file 1 at time t2 is A2, and the amount of data read by file 1 at time t3 is A3. Then, the data storage system can calculate the current access frequency of file 1 as (A1 / (t4-t1)+A2 / (t4-t2)+A3 / (t4-t3)) / B based on the four access times of file 1, the corresponding data read amounts of file 1 at the four access times, and the current size of file 1. Where 1 / (t4-t1) represents the weight of the amount of data read A1 at time t1, 1 / (t4-t2) represents the weight of the amount of data read A2 at time t2, and 1 / (t4-t3) represents the weight of the amount of data read A3 at time t3.

[0102] In another possible implementation, after the data storage system obtains the filename carried in the (i-1)th access request, it can query the directory to which the first file belongs, and query the directory attribute information of the directory to which the first file belongs. Then, the data storage system can extract directory data features from the directory attribute information of the directory to which the first file belongs. These directory data features may include, but are not limited to, data features of directory identifiers, data features of the total number of different files included in the directory, data features of the total access frequency of the multiple files included in the directory, or data features of the proportion of access patterns of the multiple files included in the directory.

[0103] Optionally, for the data characteristics of the directory identifier (e.g., identifier b) of the directory to which the first file belongs, the data storage system can normalize identifier b to obtain a normalized value of |hash value of identifier b|%N / N. This value |hash value of identifier b|%N / N is used to characterize the data characteristics of the directory identifier b of the directory to which the first file belongs. Optionally, the data storage system can also normalize the directory name (e.g., directory name c) of the directory to which the first file belongs to, to obtain a normalized value of |hash value of directory name c|%N / N. This value |hash value of directory name c|%N / N can also be used to characterize the data characteristics of the directory identifier b of the directory to which the first file belongs. Here, N represents the maximum number of files included in the directory to which the first file belongs within a certain period (e.g., a sliding time window before the access time of the (i-1)th access request of the first file).

[0104] For the data characteristic of the total number of multiple different files included in the directory to which the first file belongs, the data storage system can calculate the total number of multiple different files included in the directory to which the first file belongs over a period of time (e.g., a sliding time window before the access time of the (i-1)th access request of the first file), and this total number is used to characterize the data characteristic of the total number of multiple different files included in the directory to which the first file belongs.

[0105] For the data characteristic of the total access frequency of multiple files included in the directory to which the first file belongs, the data storage system can calculate the total number of accesses (i.e., the total access frequency) of multiple files included in the directory to which the first file belongs, within a certain period of time (e.g., a sliding time window before the access time of the (i-1)th access request of the first file), based on at least one access request of each file in the multiple files included in the directory to which the first file belongs. This total number of accesses is used to characterize the data characteristic of the total access frequency of multiple files included in the directory to which the first file belongs.

[0106] For the data characteristics of the access pattern proportions of multiple files included in the directory to which the first file belongs, the data storage system can refer to the above-described implementation process for determining the access pattern proportions corresponding to the directory to which the first file belongs, and determine the access pattern proportions corresponding to the directory to which the first file belongs, such as at least one of the following: time-dense pattern proportion, time-sparse pattern proportion, spatially ordered pattern proportion, spatially random pattern proportion, uniform-length pattern proportion, or variable-length pattern proportion. The access pattern proportions corresponding to the directory to which the first file belongs are used to characterize the data characteristics of the access pattern proportions of multiple files included in the directory to which the first file belongs.

[0107] In another possible implementation, after the data storage system obtains the filename carried in the (i-1)th access request, it can query the file format of the first file corresponding to that filename, and can also query the file format attribute information of the first file's file format. Then, the data storage system can extract file format data features from the file format attribute information of the first file's file format. These file format data features may include, but are not limited to, data features identifying the file format, data features indicating the total number of different files belonging to that file format, data features indicating the total access frequency of multiple files belonging to that file format, or data features indicating the proportion of access patterns possessed by multiple files belonging to that file format.

[0108] Optionally, for the data characteristics of the file format identifier (e.g., identifier f) of the file format to which the first file belongs, the data storage system can normalize identifier f to obtain a normalized value of |hash value of identifier f|%N / N. This value |hash value of identifier f|%N / N is used to characterize the data characteristics of the file format identifier f of the file format to which the first file belongs. Optionally, the data storage system can also normalize the file format name (e.g., file format name g) of the file format to which the first file belongs, to obtain a normalized value of |hash value of file format name g|%N / N. This value |hash value of file format name g|%N / N can also be used to characterize the data characteristics of the file format identifier f of the file format to which the first file belongs. Here, N represents the maximum number of files included in the file format to which the first file belongs within a certain period (e.g., a sliding time window before the access time of the (i-1)th access request of the first file).

[0109] For the data characteristic of the total number of multiple different files belonging to the file format, the data storage system can calculate the total number of multiple different files belonging to the file format over a period of time (e.g., a sliding time window before the access time of the (i-1)th access request of the first file), and this total number is used to characterize the data characteristic of the total number of multiple different files belonging to the file format.

[0110] For the data characteristic of the total access frequency of multiple files belonging to the file format, the data storage system can calculate the total number of accesses (i.e., the total access frequency) of multiple files belonging to the file format over a period of time (e.g., a sliding time window before the access time of the (i-1)th access request of the first file) based on at least one access request of each file belonging to the file format. This total number of accesses is used to characterize the data characteristic of the total access frequency of multiple files belonging to the file format.

[0111] For the data characteristics of the access mode proportions of multiple files belonging to the same file format, the data storage system can refer to the above-described implementation process for determining the access mode proportions corresponding to the file format to which the first file belongs, and determine the access mode proportions corresponding to the file format to which the first file belongs, such as at least one of the following: time-dense mode proportion, time-sparse mode proportion, spatially ordered mode proportion, spatially random mode proportion, uniform-length mode proportion, or variable-length mode proportion. The access mode proportion corresponding to the file format to which the first file belongs is used to characterize the data characteristics of the access mode proportions of multiple files belonging to the same file format.

[0112] Step 302: The data storage system trains the first streaming model based on multiple first access data features and the first access attribute parameters of the first file.

[0113] Optionally, the first access attribute parameter is used to represent the actual access attribute parameters that the first file has at the time of the i-th access request. The first access attribute parameter may include, but is not limited to, actual request offset, actual request offset category, actual access popularity, or actual access popularity category.

[0114] For example, the following describes the implementation process of training the first streaming model in a data storage system by taking the prediction of the next request offset of the first file (e.g., the predicted request offset of the first file at the (i+1)th access request) as an example.

[0115] Implementation Method 1: The data storage system can refer to the implementation method in step 301 above to determine multiple fourth access data features corresponding to the first file in the (i-1)th access request. Then, the data storage system can select multiple first access data features from the multiple fourth access data features and input the multiple first access data features and the actual request offset carried in the i-th access request of the first file into the first streaming model for training. In this way, by implementing Method 1, the data storage system can delete some unrepresentative or highly correlated access data features to select sufficiently good access data features for training the first streaming model, effectively reducing the number of access data features. This reduces the memory pressure on the data storage system and reduces potential noise during the training process of the first streaming model, thereby effectively improving the training efficiency of the first streaming model. Furthermore, by implementing Method 1, the data storage system can reduce the number of access data features, ensuring that, with limited storage and computing resources, the data storage system can effectively use the first streaming model to predict the predicted access attribute parameters of the first file in the (i+1)th access request.

[0116] Based on the above implementation method one, the following two possible implementation methods will be used to describe the implementation process of a data storage system selecting multiple first access data features from multiple fourth access data features.

[0117] Method 1: The data storage system determines the P-value for each of the multiple fourth-access data features using the P-value test, the chi-square value using the chi-square test, and the Gini value using the Gini coefficient. Then, the data storage system weights the P-value, chi-square value, and Gini value for each of the multiple fourth-access data features to determine the weighted value for each feature. The data storage system can then select the fourth-access data feature whose weighted value is greater than or equal to a first set threshold as the first access data feature, or it can sort the weighted values ​​of the multiple fourth-access data features in descending order and select the first m features as the first access data feature. In this way, by removing or deleting some unrepresentative access data features, the data storage system can retain only the access data features that have a good impact on the prediction performance of the first-stream model, thereby improving the training effect of the first-stream model and reducing the storage and computing resources consumed in training the first-stream model.

[0118] Optionally, after selecting multiple first access data features, the data storage system can also determine the correlation between any two first access data features using a correlation algorithm (such as cosine similarity algorithm or Pearson correlation coefficient algorithm) or a correlation matrix algorithm, thus obtaining multiple correlation scores. When any of the multiple correlation scores is greater than a second preset threshold, the data storage system removes or deletes one of the two first access data features corresponding to that correlation score. Afterwards, the data storage system can use the remaining first access data features (excluding the removed or deleted features) to train the first streaming model. Optionally, the data storage system can also sort the multiple correlation scores in descending order of correlation and select the top n correlation scores. Then, for any one of the top n correlation scores, the data storage system removes or deletes one of the two first access data features corresponding to that correlation score. Then, the data storage system can use the remaining first access data features (excluding those that have been removed or deleted) from multiple first access data features to train the first streaming model. For example, the remaining first access data features include all data features included in the request data features corresponding to the (i-1)th access request for the first file, all data features included in the directory data features, all data features included in the file data features, and data features including the file format identifier and the total number of different files belonging to that file format. In this way, by removing or deleting some highly correlated access data features, the data storage system can retain only the access data features that have a good impact on the prediction performance of the first streaming model, thereby improving the training effect of the first streaming model and reducing the storage and computing resources consumed in training it.

[0119] Method 2: The data storage system can determine the correlation between any two fourth access data features from multiple fourth access data features using a correlation algorithm or a correlation matrix algorithm, thus obtaining multiple correlation scores. When any of the multiple correlation scores is greater than a second set threshold, the data storage system removes or deletes one of the two fourth access data features corresponding to that correlation score. Then, the data storage system can use the remaining fourth access data features (excluding the removed or deleted features) as multiple first access data features. Optionally, the data storage system can also sort the multiple correlation scores in descending order of correlation and select the top n correlation scores. Then, for any one of the top n correlation scores, the data storage system removes or deletes one of the two fourth access data features corresponding to that correlation score. Finally, the data storage system can use the remaining fourth access data features (excluding the removed or deleted features) as multiple first access data features.

[0120] Implementation Method Two: The data storage system can refer to the implementation method in step 301 above to determine multiple fourth access data features corresponding to the first file in the (i-1)th access request. Then, the data storage system directly uses these multiple fourth access data features as the first access data features, and inputs these multiple fourth access data features and the actual request offset carried in the i-th access request of the first file into the first streaming model for training.

[0121] Optionally, first-stream models are a class of models that can provide high accuracy through online training, testing, and evaluation. For example, a first-stream model may support at least one of the following: (1) processing only one instance at a time; (2) using a limited amount of time to process each instance; (3) using limited memory; (4) adapting to changes over time; and (5) being ready to provide predictions at any time. In other words, first-stream models can capture changes in the underlying data distribution (such as changes in file access patterns), and they can naturally adjust and adapt to changes in workload over time with minimal overhead, thus significantly reducing the need for massive historical sample datasets.

[0122] In this first-stream model, each individual training sample is seen only once. Therefore, each individual training sample is used for both training and testing the model. This eliminates the need to split the input data as typically used in batch models (e.g., 80% for training and 20% for testing), allowing each individual training sample to be used for both training and testing. For example, when a current access request for a file is received, the data storage system can determine multiple access data features corresponding to the previous access request for file 1, and can use these features to train and test the first-stream model. Furthermore, prequential evaluation is used in the online training, testing, and evaluation of the first-stream model. In prequential evaluation, individual training samples are processed according to their submission order; each individual training sample is used to test the model before being used to train it. This ensures that the first-stream model is always tested on samples it has never seen before and forces the use of all available data. After training the first-stream model with each training sample, its accuracy increases, gradually increasing and stabilizing as more samples are processed.

[0123] Optionally, when the data storage system predicts the next request offset or the next access popularity of the first file, since predicting the request offset or the access popularity is a regression problem, the first streaming model that the data storage system can use may include, but is not limited to, the Adaptive Random Forest (ARF) model, the K-Nearest Neighbor (KNN) model, the Stochastic Gradient Descent (SGD) model, the Stochastic Gradient Descent-Squared Loss (SGD-SL) model, or the Adaptive Gradient-Squared Loss (AdaGrad-SL) model.

[0124] For example, taking the first streaming model as an ARF model, when the data storage system needs to predict the predicted request offset of the first file at the (i+1)th access request, the data storage system can use the ARF model to make the prediction. The ARF model offers a better balance between prediction accuracy and execution time. When the data storage system receives the i-th access request for the first file, it needs to obtain multiple fourth access data features (e.g., [data feature 1, data feature 2, data feature 3, ..., data feature n]) corresponding to the first file at the (i-1)-th access request. It then selects multiple first access data features from these fourth access data features. Afterward, the data storage system can use these multiple first access data features and the actual request offset of the first file at the i-th access request to train the ARF model online, obtaining the trained ARF model.

[0125] Optionally, when the data storage system predicts the next request offset category or the next access popularity category of the first file, since predicting the request offset category or predicting the access popularity category is a classification problem, the first streaming model that the data storage system can use may include the Hoeffding Tree (HT) algorithm, etc.

[0126] For example, taking the first streaming model as the HT model, when the data storage system needs to predict the predicted access popularity category of the first file in the (i+1)th access request, the data storage system can use the HT model to make this prediction. The HT model is an advanced, memory-efficient decision tree designed specifically for streaming data and capable of learning from large datasets. It leverages the fact that small samples are sufficient to select the optimal splitting attribute. It also has a unique feature: it guarantees performance because its output is asymptotically identical to non-incremental learning using an infinite number of examples. When the data storage system receives the i-th access request for the first file, it needs to obtain multiple fourth access data features (e.g., [data feature 1, data feature 2, data feature 3, ..., data feature n]) corresponding to the first file in the (i-1)-th access request, and select multiple first access data features from these fourth access data features. Then, the data storage system can use these multiple first access data features and the actual access popularity category of the first file in the i-th access request to train the HT model online, obtaining the trained HT model.

[0127] In one example, let's take the training of an ARF model using multiple first access data features corresponding to the first file in the (i-1)th access request as an example. First, the data storage system can refer to the implementation method of step 301 above to determine the [data feature 1, data feature 2, data feature 3, ..., data feature n] corresponding to the first file in the (i-1)th access request. Then, the data storage system can determine the value P1 corresponding to data feature 1, the value P2 corresponding to data feature 2, the value P3 corresponding to data feature 3, ..., the value Pn corresponding to data feature n according to the P-value test method. According to the chi-square test method, it can determine the chi-square value s1 corresponding to data feature 1, the chi-square value s2 corresponding to data feature 2, the chi-square value s3 corresponding to data feature 3, ..., the chi-square value sn corresponding to data feature n. According to the Gini coefficient method, it can determine the Gini coefficient q1 corresponding to data feature 1, the Gini coefficient q2 corresponding to data feature 2, the Gini coefficient q3 corresponding to data feature 3, ..., the Gini coefficient qn corresponding to data feature n. Next, the data storage system can weight the P-value, chi-square value, and Gini value corresponding to each data feature to obtain the weighted value for each data feature. For example, the weighted value for data feature 1 is (P1+s1+q1), the weighted value for data feature 2 is (P2+s2+q2), the weighted value for data feature 3 is (P3+s3+q3), ..., the weighted value for data feature n is (Pn+sn+qn). Then, the data storage system can select data features whose weighted values ​​are greater than or equal to a first set threshold from the n data features to train the ARF model. For example, if the weighted values ​​for data feature 1, data feature 2, data feature 3, data feature 5, data feature 7, and data feature 10 are greater than or equal to the first set threshold, then the data storage system can use data features 1, 2, 3, 5, 7, and 10 to train the ARF model.

[0128] Step 303: The data storage system obtains multiple second access data features corresponding to the first file at the i-th access request, inputs the multiple second access data features into the first streaming model, and obtains the second access attribute parameters of the first file.

[0129] Optionally, the second access attribute parameter is used to represent the access attribute parameters that the predicted first file has at the (i+1)th access request. The second access attribute parameter may include, but is not limited to, the predicted request offset, the predicted request offset category, the predicted access popularity, or the predicted access popularity category.

[0130] Optionally, the second access data feature may include request data features, file data features, directory data features, or file format data features corresponding to the first file in the i-th access request. Optionally, the second access data feature may also include the access frequency or the complete read frequency of the first file corresponding to the first file in the i-th access request.

[0131] Specifically, the request data characteristics of the first file in the i-th access request are obtained by the data storage system through feature extraction of the request information carried in the i-th access request; the file data characteristics of the first file in the i-th access request are obtained by the data storage system through feature extraction of the file attribute information of the first file in the i-th access request; the directory data characteristics of the first file in the i-th access request are obtained by the data storage system through feature extraction of the directory attribute information of the first file in the i-th access request; the file format data characteristics of the first file in the i-th access request are obtained by the data storage system through feature extraction of the file format attribute information of the first file in the i-th access request; the access frequency of the first file in the i-th access request is determined by the data storage system based on the total number of accesses of the first file within a certain period (e.g., a sliding time window before the access time of the i-th access request); and the complete read frequency of the first file in the i-th access request is determined by the data storage system based on the request length carried in multiple access requests of the first file within a certain period (e.g., a sliding time window before the access time of the i-th access request).

[0132] For example, the following describes the process of a data storage system determining the predicted request offset of a first file at the (i+1)th access request by using a trained first streaming model, with two possible implementation methods.

[0133] Implementation Method 1: The data storage system can refer to the implementation method of step 301 above to determine multiple third access data features corresponding to the first file at the i-th access request. Then, the data storage system can select multiple second access data features from the multiple third access data features and input the multiple second access data features into the trained first streaming model to obtain the predicted request offset of the first file at the (i+1)-th access request.

[0134] Optionally, the process of selecting multiple second access data features from multiple third access data features in the data storage system can refer to the above-described process of selecting multiple first access data features from multiple fourth access data features in the data storage system, and will not be repeated here.

[0135] Implementation Method Two: The data storage system can refer to the implementation method in step 301 above to determine multiple third access data features corresponding to the first file at the i-th access request. Then, the data storage system directly inputs these multiple third access data features as second access data features into the first streaming model to obtain the predicted request offset of the first file at the (i+1)-th access request.

[0136] In one example, when the data storage system needs to predict the predicted request offset of the first file at the (i+1)th access request, taking the first streaming model as an ARF model and the first file as file 1 as an example, the data storage system can make the prediction using the trained ARF model.

[0137] Optionally, when the data storage system receives the i-th access request for file 1, before predicting the predicted request offset of file 1 in the (i+1)-th access request, the data storage system needs to obtain multiple third access data features (e.g., [data feature 1, data feature 2, data feature 3, ..., data feature 20]) corresponding to file 1 in the i-th access request. Then, the data storage system can determine the value P1 corresponding to data feature 1, the value P2 corresponding to data feature 2, the value P3 corresponding to data feature 3, ..., the value P20 corresponding to data feature 20 using the P-value test method; determine the chi-square value s1 corresponding to data feature 1, the chi-square value s2 corresponding to data feature 2, the chi-square value s3 corresponding to data feature 3, ..., the chi-square value s20 corresponding to data feature 20 using the chi-square test method; and determine the Gini value q1 corresponding to data feature 1, the Gini value q2 corresponding to data feature 2, the Gini value q3 corresponding to data feature 3, ..., the Gini value q20 corresponding to data feature 20 using the Gini metric method. Then, the data storage system can perform weighted processing on the P-value, chi-square value, and Gini value corresponding to each data feature to obtain the weighted value corresponding to each data feature. For example, the weighted value corresponding to data feature 1 is (P1+s1+q1), the weighted value corresponding to data feature 2 is (P2+s2+q2), the weighted value corresponding to data feature 3 is (P3+s3+q3), ..., the weighted value corresponding to data feature 20 is (P20+s20+q20).

[0138] Then, the data storage system can select data features from 20 data features whose weighted values ​​are greater than or equal to a first set threshold. For example, the weighted values ​​corresponding to data feature 1, data feature 2, data feature 3, data feature 5, data feature 7, data feature 8, data feature 9, data feature 10, data feature 16, and data feature 18 are greater than or equal to the first set threshold. Then, the data storage system can input data feature 1, data feature 2, data feature 3, data feature 5, data feature 7, data feature 8, data feature 9, data feature 10, data feature 16, and data feature 18 into the trained ARF model to obtain the predicted request offset of file 1 at the (i+1)th access request, for example, the predicted request offset is 3.

[0139] In another example, when a data storage system needs to predict the predicted access popularity category of the first file at the (i+1)th access request, taking the first streaming model as the HT model and the first file as file 1 as an example, the data storage system can make the prediction using the trained HT model.

[0140] Optionally, when the data storage system receives the i-th access request for file 1, before predicting the predicted access popularity category of file 1 in the (i+1)-th access request, the data storage system needs to obtain multiple third access data features (e.g., [data feature 1, data feature 2, data feature 3, ..., data feature 15]) corresponding to file 1 in the i-th access request. Then, the data storage system can determine the value P1 corresponding to data feature 1, the value P2 corresponding to data feature 2, the value P3 corresponding to data feature 3, ..., the value P15 corresponding to data feature 15 using the P-value test method; determine the chi-square value s1 corresponding to data feature 1, the chi-square value s2 corresponding to data feature 2, the chi-square value s3 corresponding to data feature 3, ..., the chi-square value s15 corresponding to data feature 15 using the chi-square test method; and determine the Gini value q1 corresponding to data feature 1, the Gini value q2 corresponding to data feature 2, the Gini value q3 corresponding to data feature 3, ..., the Gini value q15 corresponding to data feature 15 using the Gini metric method. Then, the data storage system can perform weighted processing on the P-value, chi-square value, and Gini value corresponding to each data feature to obtain the weighted value corresponding to each data feature. For example, the weighted value corresponding to data feature 1 is (P1+s1+q1), the weighted value corresponding to data feature 2 is (P2+s2+q2), the weighted value corresponding to data feature 3 is (P3+s3+q3), ..., the weighted value corresponding to data feature 20 is (P15+s15+q15).

[0141] Then, the data storage system can select data features from 15 data features whose weighted values ​​are greater than or equal to a first set threshold. For example, the weighted values ​​corresponding to data feature 1, data feature 2, data feature 3, data feature 5, data feature 7, data feature 8, data feature 16, and data feature 18 are greater than or equal to the first set threshold. Then, the data storage system can input data feature 1, data feature 2, data feature 3, data feature 5, data feature 7, data feature 8, data feature 16, and data feature 18 into the trained HT model to obtain the predicted access popularity category of file 1 at the (i+1)th access request. For example, the predicted access popularity category is HOTNESS_SIX (indicating that file 1 has the highest access popularity).

[0142] For example, the access popularity categories include the following categories: HOTNESS_ONE, HOTNESS_TWO, HOTNESS_THREE, HOTNESS_FOUR, HOTNESS_FIVE, and HOTNESS_SIX (indicating that the file's access popularity is the highest). The numbers ONE, TWO, THREE, FOUR, FIVE, and SIX in the category represent the degree of "hot" (or "cold") access popularity of the file, with HOTNESS_ONE indicating the lowest access popularity and HOTNESS_SIX indicating the highest access popularity.

[0143] Step 304: The data storage system prefetches or migrates the first file according to the second access attribute parameters.

[0144] Optionally, in one example, when the second access attribute parameter is the predicted request offset corresponding to the (i+1)th access request of the first file (e.g., the predicted request offset is 3), the data storage system can prefetch the first file based on this predicted request offset. For example, taking the first file as file 1, assuming file 1 includes 3 file blocks, namely file block 1, file block 2, and file block 3, the predicted request offset 3 corresponds to file block 3 in the first file. That is to say, the (i+1)th access request is very likely to request access to file block 3 in the first file, and file block 3 is stored in a low-performance medium (e.g., HDD medium). Then, the data storage system can prefetch a copy of file block 3 to a high-performance storage medium (e.g., main memory medium). In this way, when the (i+1)th access request of the first file arrives, the data storage system can quickly query file block 3 in the main memory medium, thereby effectively shortening the access latency of the (i+1)th access request.

[0145] In another example, when the second access attribute parameter is the predicted request offset category corresponding to the first file at the (i+1)th access request, the data storage system can prefetch the first file based on the predicted request offset category.

[0146] For example, the request offset categories include the following categories: (1) OFFSET_SEQ, which indicates that the next request offset is adjacent to the current request offset; (2) OFFSET_RANDOM, which indicates that the next request offset is not adjacent to the current request offset; and (3) OFFSET_NONE, which indicates that the file will not be read again in the next access request.

[0147] For example, let's assume the first file is file 1, and the predicted request offset class for the (i+1)th access request is OFFSET_SEQ. Assume file 1 comprises four blocks: block 1, block 2, block 3, and block 4. Also, assume the request offset for the i-th access request is 3, which corresponds to block 3 within file 1. The data storage system can determine, based on the predicted request offset class being OFFSET_SEQ, that the (i+1)th access request for file 1 is highly likely to request access to block 4. If block 4 is stored on high-performance media (e.g., main memory), the data storage system does not need to prefetch it. If block 4 is stored on low-performance media (e.g., HDD), the data storage system needs to prefetch a copy of block 4 into a high-performance storage medium (e.g., cache). This way, when the (i+1)th access request for the first file arrives, the data storage system can quickly retrieve block 4 from the cache, effectively reducing the access latency of the (i+1)th access request.

[0148] In another example, when the second access attribute parameter is the predicted access popularity of the first file at the (i+1)th access request, the data storage system can migrate the first file based on the predicted access popularity.

[0149] For example, the popularity of a file can be represented by a value between 0 and 1. A value of 0 indicates that the file is the least popular, and a value of 1 indicates that the file is the most popular.

[0150] For example, let's say the first file is file 1, and file 1 has a predicted access popularity of 0.9 at the (i+1)th access request. Assume file 1 comprises three blocks: block 1, block 2, and block 3. Based on the predicted access popularity of 0.9 at the (i+1)th access request, the data storage system can determine that file 1 will be relatively popular in the future. Therefore, when the data storage system determines that all three blocks of file 1 are stored in high-performance storage media, it does not need to migrate them. However, when the data storage system determines that all three blocks of file 1 are stored in low-performance storage media (such as optical disc media), it needs to migrate them to high-performance storage media in advance. For example, the data storage system can migrate all three blocks to main memory, or it can migrate block 1 to cache media and migrate blocks 2 and 3 to main memory. When the data storage system determines that file block 1 is stored in a high-performance storage medium, while file blocks 2 and 3 are stored in a low-performance storage medium, the data storage system does not need to migrate file block 1. Instead, it needs to migrate file blocks 2 and 3 from the low-performance storage medium to the high-performance storage medium. This way, when the (i+1)th access request for file 1 arrives, the data storage system can quickly locate the file block corresponding to the request offset carried by the (i+1)th access request in the high-performance storage medium, thereby effectively shortening the access latency of the (i+1)th access request.

[0151] In another example, when the second access attribute parameter is the predicted access popularity category of the first file at the (i+1)th access request, the data storage system can migrate the first file according to the predicted access popularity category.

[0152] For example, taking file 1 as the first file, and the predicted access popularity category of file 1 at the (i+1)th access request being HOTNESS_FIVE, assuming file 1 includes three file blocks: file block 1, file block 2, and file block 3. Based on the predicted access popularity category HOTNESS_FIVE for file 1 at the (i+1)th access request, the data storage system can determine that file 1 will have relatively high access popularity in the future. Therefore, when the data storage system determines that all three file blocks of file 1 are stored on high-performance storage media, the data storage system does not need to migrate the three file blocks of file 1. When the data storage system determines that all three file blocks of file 1 are stored on low-performance storage media (such as optical disc media), the data storage system needs to migrate the three file blocks of file 1 to high-performance storage media in advance. When the data storage system determines that file block 1 is stored on high-performance storage media, while file blocks 2 and 3 are stored on low-performance storage media, the data storage system does not need to migrate file block 1, but it needs to migrate file blocks 2 and 3 from low-performance storage media to high-performance storage media. Alternatively, when the data storage system determines that a portion of data (such as file block 2) in a high-performance storage medium (such as main memory medium) needs to be discarded, if the data storage system determines that file 1 will be frequently accessed in the future, the data storage system can store file block 2 in another high-performance storage medium (such as SSD medium). If the data storage system determines that file 1 will be less frequently accessed in the future, the data storage system can directly discard file block 2.

[0153] For example, taking file 1 as the first file, and the predicted access popularity category of file 1 at the (i+1)th access request being HOTNESS_TWO, assume file 1 includes three file blocks: file block 1, file block 2, and file block 3. Based on the predicted access popularity category HOTNESS_TWO for file 1 at the (i+1)th access request, the data storage system can determine that file 1 will have relatively low access popularity in the future. Therefore, when the data storage system determines that all three file blocks of file 1 are stored on low-performance storage media, the data storage system does not need to migrate the three file blocks of file 1. When the data storage system determines that all three file blocks of file 1 are stored on high-performance storage media (such as main memory), the data storage system needs to migrate the three file blocks of file 1 to low-performance storage media in advance. For example, the data storage system can migrate file blocks 1, 2, and 3 to HDD media, or it can migrate file block 1 to HDD media and file blocks 2 and 3 to optical disc media. When the data storage system determines that file block 1 is stored in a low-performance storage medium and file blocks 2 and 3 are stored in a high-performance storage medium, the data storage system does not need to migrate file block 1, but needs to migrate file blocks 2 and 3 from the high-performance storage medium to the low-performance storage medium.

[0154] It should be noted that in the description of this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, "at least one of A, B, and C" includes A, B, C, AB, AC, BC, or ABC. Furthermore, unless otherwise specified, the ordinal numbers "first," "second," "third," etc., mentioned in the embodiments of this application are used to distinguish multiple objects and are not used to limit the order, sequence, priority, or importance of multiple objects. In addition, the terms "including," "comprising," "having," and their variations appearing in this application all mean "including but not limited to," unless otherwise specifically emphasized.

[0155] Furthermore, it should be noted that each step in the above embodiments can be executed by the corresponding device, or by components such as chips, processors, or chip systems within that device. This application does not limit the scope of these steps. The above embodiments are only illustrated by examples of execution by the corresponding device.

[0156] It should be noted that in the above embodiments, some steps may be selected for implementation, and the order of the steps in the figures may be adjusted. This application does not limit this. It should be understood that performing some of the steps in the figures, adjusting the order of the steps, or combining them in a specific implementation all fall within the protection scope of this application.

[0157] It is understood that, in order to achieve the functions described in the above embodiments, each device involved in the above embodiments includes a hardware structure and / or software module corresponding to perform each function. Those skilled in the art should readily recognize that, based on the units and method steps of the various examples described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0158] It should be noted that the "steps" in the embodiments of this application are merely illustrative and are intended to better understand one method of presentation used in the embodiments. They do not constitute a substantial limitation on the execution of the solution of this application. For example, the "step" can also be understood as a "feature". Furthermore, the steps do not constitute any limitation on the execution order of the solution of this application. Any changes to the order of steps, or the merging or splitting of steps made on this basis without affecting the overall solution implementation, resulting in a new technical solution, are also within the scope of disclosure of this application.

[0159] Based on the same concept, embodiments of this application also provide a possible data storage system, which is suitable for... Figure 1 In the illustrated application scenario, the data storage system is used to implement the data processing method provided in the above embodiments, or the modules (such as chips) of the data storage system are used to implement the data processing method provided in the above embodiments, thus achieving the beneficial effects of the above embodiments. In the embodiments of this application, the data storage system may be as follows: Figure 1 The data storage system 200 shown.

[0160] See Figure 4The data storage system 400 includes an acquisition module 401 and a processing module 402. The acquisition module 401 is used to acquire multiple first access data features corresponding to the first file at the (i-1)th access request. The processing module 402 is used to train a first streaming model based on the multiple first access data features and the first access attribute parameters of the first file. The first access attribute parameters are the actual access attribute parameters that the first file has at the i-th access request. The acquisition module 401 is also used to acquire multiple second access data features corresponding to the first file at the i-th access request. The processing module 402 is also used to input the multiple second access data features into the first streaming model to obtain the second access attribute parameters of the first file. The second access attribute parameters are the predicted access attribute parameters that the first file has at the (i+1)-th access request. The processing module 402 is also used to prefetch or migrate the first file based on the second access attribute parameters.

[0161] For a more detailed description of the acquisition module 401 and the processing module 402, please refer to the relevant description of the data storage system in the above method embodiments, which will not be repeated here.

[0162] It should be understood that the acquisition module 401 in the embodiments of this application can be implemented by a communication interface or communication interface-related circuit components, and the processing module 402 can be implemented by a processor or processor-related circuit components.

[0163] It should be noted that the module division in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical entities, or have two or more units integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0164] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, or a server, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0165] Based on the same concept, embodiments of this application also provide a possible data storage system, which is suitable for... Figure 1 In the illustrated application scenario, the data storage system is used to implement the technical solutions involved in the data storage systems in the above embodiments, and therefore can also achieve the beneficial effects of the data storage systems in the above method embodiments. See also Figure 5 The data storage system 500 includes a communication interface 501 and a processor 502. Optionally, the data storage system 500 further includes a memory 503. The communication interface 501, the processor 502, and the memory 503 are interconnected. When the data storage system 500 is used to implement the technical solutions related to the data storage system provided in the above embodiments, the communication interface 501 can be used to implement the function of the acquisition module 401, and the processor 502 can be used to implement the function of the processing module 402.

[0166] Optionally, the communication interface 501, the processor 502, and the memory 503 are interconnected via a bus 504. The bus 504 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0167] The communication interface 501 is used to receive and send data, for example, when the data storage system is as follows: Figure 1The data storage system 200 shown herein, wherein the communication interface 501 enables communication with... Figure 1 It can communicate with the terminal 100 shown, or it can also achieve communication with... Figure 1 The system can communicate with devices other than those shown in the intended application scenario (such as cloud or service devices). Optionally, the communication interface 501 can be an input / output interface. For example, the data storage system can use the communication interface to communicate with cloud or service devices.

[0168] The functions of the processor 502 can be referred to the descriptions of the corresponding functions involved in the data storage system in the above embodiments, and will not be repeated here. The processor 502 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP, etc. The processor 502 may further include hardware chips. The aforementioned hardware chips can be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLDs can be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof. When implementing the above functions, the processor 502 can be implemented in hardware, or it can be implemented by hardware executing corresponding software.

[0169] The memory 503 is used to store program instructions, etc. Specifically, the program instructions may include program code, which includes computer operation instructions. The memory 503 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. The processor 502 executes the program instructions stored in the memory 503 to implement the above functions, thereby implementing the method steps required by the data storage system in the above embodiment.

[0170] Based on the same concept, this application also provides a computer program product, which includes a computer program or instructions that, when run on a computer, cause the computer to perform the methods provided in the above embodiments.

[0171] Based on the same concept, embodiments of this application also provide a computer-readable storage medium storing a computer program or instructions, which, when executed by a computer, causes the computer to perform the methods provided in the above embodiments.

[0172] The storage medium can be any available medium that a computer can access. For example, but not limited to, a computer-readable medium can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0173] Based on the same concept, this application also provides a chip coupled to a memory, which is used to read a computer program stored in the memory to implement the method provided in the above embodiments.

[0174] Based on the same concept, embodiments of this application also provide a chip system including a processor for supporting a computer device in implementing the functions involved in the data storage system described in the above embodiments. In one possible design, the chip system further includes a memory for storing necessary programs and data for the computer device. This chip system may be composed of chips or may include chips and other discrete devices.

[0175] The methods provided in this application can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0176] The steps of the methods described in the embodiments of this application can be directly embedded in hardware, a software unit executed by a processor, or a combination of both. The software unit can be stored in RAM, ROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be housed in an ASIC.

[0177] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0178] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0179] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized in that, Applied to a data storage system, the method includes: Obtain multiple first access data features corresponding to the (i-1)th access request for the first file; The first streaming model is trained based on the plurality of first access data features and the first access attribute parameters of the first file; wherein, the first access attribute parameters are the actual access attribute parameters that the first file has at the i-th access request; Obtain multiple second access data features corresponding to the first file at the i-th access request, input the multiple second access data features into the first streaming model, and obtain the second access attribute parameters of the first file; wherein, the second access attribute parameters are the predicted access attribute parameters of the first file at the (i+1)-th access request; Based on the second access attribute parameter, the first file is prefetched or migrated.

2. The method as described in claim 1, characterized in that, Obtain multiple first access data features corresponding to the (i-1)th access request for the first file, including: The first file is obtained by acquiring multiple first access data features corresponding to the (i-1)th access request through at least one of the following: request information, file attribute information, directory attribute information of the directory to which the first file belongs, or file format attribute information of the file format to which the first file belongs during the (i-1)th access request. Obtaining multiple second access data features corresponding to the first file at the time of the i-th access request, including: The first file is obtained by acquiring multiple second access data features corresponding to the first file at the time of the i-th access request through at least one of the following: request information, file attribute information, directory attribute information of the directory to which the first file belongs, or file format attribute information of the file format to which the first file belongs at the time of the i-th access request.

3. The method as described in claim 2, characterized in that, The request information includes at least one of the following: request operation, request offset, or request length; The file attribute information includes at least one of the following: file identifier, file size, file creation time, last access time, last update time, most recent opening time, or access popularity; The directory attribute information includes at least one of the following: directory identifier, total number of multiple different files included in the directory, total access frequency of multiple files included in the directory, or percentage of access patterns of multiple files included in the directory; The file format attribute information includes at least one of the following: file format identifier, total number of multiple different files belonging to the file format, total access frequency of multiple files belonging to the file format, or percentage of access modes of multiple files belonging to the file format.

4. The method as described in claim 3, characterized in that, The access mode includes at least one of the following: time-intensive mode, time-sparse mode, spatially ordered mode, spatially random mode, uniform length mode, variable length mode, file access frequency mode, or file full read frequency mode.

5. The method according to any one of claims 1-4, characterized in that, Obtaining multiple second access data features corresponding to the first file at the time of the i-th access request, including: Determine multiple third access data features corresponding to the first file at the time of the i-th access request; Select the plurality of second access data features from the plurality of third access data features.

6. The method as described in claim 5, characterized in that, Selecting the plurality of second access data features from the plurality of third access data features includes: According to the P-value test method, the P-value corresponding to each of the plurality of third access data features is determined; according to the chi-square test method, the chi-square value corresponding to each of the plurality of third access data features is determined; according to the Gini metric method, the Gini value corresponding to each of the plurality of third access data features is determined. The P-value, chi-square value, and Gini value corresponding to each of the plurality of third access data features are weighted to determine the weighted value corresponding to each of the plurality of third access data features; Select the plurality of second access data features whose weighted value is greater than or equal to a first set threshold from the plurality of third access data features.

7. The method as described in claim 5, characterized in that, Selecting the plurality of second access data features from the plurality of third access data features includes: Determine the correlation between any two third access data features among the plurality of third access data features; When any relevance exceeds the second set threshold, one of the two third access data features corresponding to that relevance is removed; The remaining third access data features, excluding the removed third access data features, are taken as the multiple second access data features.

8. The method according to any one of claims 1-4, characterized in that, The first access attribute parameter is one of the following: actual request offset, actual request offset category, actual access popularity, or actual access popularity category; the second access attribute parameter is one of the following: predicted request offset, predicted request offset category, predicted access popularity, or predicted access popularity category.

9. A data storage system, characterized in that, Includes a module for performing the method as described in any one of claims 1-8.

10. A data storage system, characterized in that, include A communication interface used to receive and send data; Memory is used to store computer program instructions and data; A processor for executing computer program instructions and data that invoke the memory to cause the data storage system to perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1-8.

13. A chip, characterized in that, The chip is coupled to a memory, and the chip reads a computer program stored in the memory to execute the method as described in any one of claims 1-8.

Images