Data processing method and apparatus

Abstract

This application discloses a data processing method and apparatus, belonging to the field of storage technology. The method includes: acquiring indication information of all data read by a data reader from a first storage medium and a second storage medium during a historical period; based on the indication information and historical prediction results of multiple prediction algorithms for the historical period, obtaining future prediction results of multiple prediction algorithms for future periods, wherein the prediction results indicate one or more of the following: data that needs to be prefetched from the second storage medium to the first storage medium, and data that needs to be evicted from the first storage medium; selecting a target prediction algorithm from multiple prediction algorithms based on the indication information and historical prediction results; and executing the operation indicated by the future prediction results based on the future prediction results of the target prediction algorithm. This application can improve the performance of prediction and eviction algorithms.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a data processing method and apparatus. Background Technology

[0002] Data prefetching and eviction techniques have wide applications, covering scenarios such as data movement and sharing between processors, hardware, software, data centers, and across network nodes. For example, at the hardware level, data prefetching and eviction can occur between different levels of cache in the central processing unit (CPU), and between storage media and their upper-level caches. At the operating system level, data prefetching and eviction can occur between pages managed by the operating system (OS) and memory. At the application level, metadata and data in databases and distributed file systems can be prefetched and evicted in memory caches.

[0003] Due to the significant demand for caching across various scenarios, in order to meet these demands, and considering both performance and cost, in addition to directly increasing the cache capacity, it is also necessary to have a high-performance prediction algorithm. Based on this, this application provides a data processing method. Summary of the Invention

[0004] This application provides a data processing method and apparatus. This application can improve the performance of prediction and elimination algorithms. The technical solution provided by this application is as follows:

[0005] In a first aspect, this application provides a data processing method. The method includes: acquiring indication information of all data read by a data reader from a first storage medium and a second storage medium during a historical period; based on the indication information and historical prediction results of multiple prediction algorithms for the historical period, obtaining future prediction results of multiple prediction algorithms for a future period, wherein the prediction results indicate one or more of the following: data that needs to be prefetched from the second storage medium to the first storage medium, and data that needs to be evicted from the first storage medium; selecting a target prediction algorithm from among the multiple prediction algorithms based on the indication information and historical prediction results; and executing the operation indicated by the future prediction results based on the future prediction results of the target prediction algorithm.

[0006] In this way, the data processing method is equivalent to running multiple prediction algorithms simultaneously before performing data prefetching. Based on the prediction results of these algorithms for historical periods and the actual data read by the data reader during those periods, a prediction algorithm is selected for the current prefetching. After selecting the algorithm, data prefetching is performed based on the predicted results for future periods. Similarly, this data processing method is equivalent to running multiple prediction algorithms simultaneously before performing data eviction. Based on the prediction results of these algorithms for historical periods and the actual data read by the data reader during those periods, a prediction algorithm is selected for the current prefetching. After selecting the algorithm, data eviction is performed based on the predicted results for future periods. Therefore, this data processing method can be considered as selecting a prediction algorithm during the process and performing data prefetching and / or data eviction based on the selected algorithm. Compared to mechanisms that select prediction algorithms before or after the process, this method can predict the data to be prefetched and evictioned based on the data requirements of the data reading scenario, thus improving the accuracy of data prefetching and data eviction.

[0007] In one implementation, the indication information includes one or more of the following: data location information, data length, and a timestamp indicating when the data was read by a data reader. The data location information indicates the location of the data in the storage medium. The timestamp indicating when the data was read by the data reader indicates the time when the data was read from the storage medium. For example, for prefetched data, the timestamp indicates the time when the data was read from a first memory. Optionally, when the prediction algorithm can serve multiple data readers, the indication information also includes the identifier of the data reader to distinguish data read by different data readers.

[0008] Among them, the selection of a target prediction algorithm from multiple prediction algorithms based on indication information and historical prediction results includes: obtaining the scores of multiple prediction algorithms for one or more measurement indicators based on indication information and historical prediction results; and selecting a target prediction algorithm from multiple prediction algorithms based on the scores of multiple prediction algorithms for one or more measurement indicators.

[0009] In one implementation, a target prediction algorithm is selected from multiple prediction algorithms based on the scores of multiple prediction algorithms for one or more metrics, including: determining the weights of multiple metrics based on a configuration strategy; and selecting the target prediction algorithm from multiple prediction algorithms based on the scores of multiple prediction algorithms for one or more metrics and the weights of the multiple metrics.

[0010] The metrics include: the probability that data is read by the data reader; and / or, the ratio of the capacity of data stored in the first storage medium to the total capacity of the first storage medium; and / or, the computing power required to execute the operation indicated by the future prediction result. When the prediction result indicates that data needs to be prefetched from the second storage medium to the first storage medium, the probability that the data is read by the data reader includes at least the probability that the data stored in the second storage medium is read by the data reader. Generally, when this probability is high, the data needs to be prefetched into the first storage medium. When the prediction result indicates that data needs to be evicted from the first storage medium, the probability that the data is read by the data reader is actually the probability that the data stored in the first storage medium is read by the data reader without distinguishing how the data stored in the first storage medium was read into the first storage medium. Generally, when this probability is low, the data needs to be evicted from the first storage medium. When the prediction result indicates that data needs to be prefetched from the second storage medium to the first storage medium, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium includes at least the ratio of the capacity of the data prefetched from the second storage medium to the total capacity of the first storage medium. When the prediction result indicates that data needs to be evicted from the first storage medium, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium is actually the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium without distinguishing how the data stored in the first storage medium was read from the first storage medium. When the prediction result indicates that data needs to be prefetched from the second storage medium to the first storage medium, the computing power required to execute the operation indicated by the future prediction result includes at least the computing power required to prefetch the data from the second storage medium to the first storage medium. When the prediction result indicates that data needs to be evicted from the first storage medium, the computing power required to execute the operation indicated by the future prediction result includes at least the computing power required to delete the data from the first storage medium.

[0011] Optionally, the method further includes: adjusting the parameters of some or all of the multiple prediction algorithms based on their scores for one or more metrics. For example, if some prediction algorithms score low, indicating poor performance, the parameters of those algorithms are adjusted to improve their performance.

[0012] Secondly, this application provides a data processing apparatus. The apparatus includes: an acquisition module, configured to acquire indication information of all data read by a data reader from a first storage medium and a second storage medium during a historical period; a processing module, configured to, based on the indication information and historical prediction results of multiple prediction algorithms for the historical period, obtain future prediction results of multiple prediction algorithms for a future period, wherein the prediction results indicate one or more of the following: data that needs to be prefetched from the second storage medium to the first storage medium, or data that needs to be evicted from the first storage medium; the processing module is further configured to, based on the indication information and historical prediction results, select a target prediction algorithm from among the multiple prediction algorithms; and a reading module, configured to, based on the future prediction results of the target prediction algorithm, execute the operation indicated by the future prediction results.

[0013] Optionally, the indication information may include one or more of the following: the location information of the data, the length of the data, and the timestamp of the data being read by the data reader.

[0014] Optionally, the processing module is specifically used to: obtain scores of multiple prediction algorithms for one or more metrics based on indication information and historical prediction results; and select a target prediction algorithm from among the multiple prediction algorithms based on the scores of multiple prediction algorithms for one or more metrics.

[0015] Optionally, the processing module is specifically used to: determine the weights of multiple metrics based on a configuration strategy; and select a target prediction algorithm from multiple prediction algorithms based on the scores of one or more metrics and the weights of the multiple metrics.

[0016] Optionally, the metrics include: the probability that the data is read by the data reader; and / or, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium; and / or, the computing power required to execute the operation indicating the future prediction result.

[0017] Optionally, the processing module is also used to: adjust the parameters of some or all of the multiple prediction algorithms based on the scores of multiple prediction algorithms for one or more metrics.

[0018] Thirdly, this application provides a computing device including a memory and a processor, the memory storing program instructions, and the processor executing the program instructions to perform the methods provided in the first aspect of this application and any possible implementation thereof.

[0019] Fourthly, this application provides a computer-readable storage medium that is a non-volatile computer-readable storage medium, which includes program instructions that, when executed on a computing device, cause the computing device to perform the methods provided in the first aspect of this application and any of its possible implementations.

[0020] Fifthly, this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods provided in the first aspect of this application and any possible implementation thereof. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating the usage process of a data prediction algorithm provided in an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of the implementation environment involved in a data processing method provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of the implementation environment of another data processing method provided in this application embodiment;

[0024] Figure 4 This is a flowchart of a data processing method provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram illustrating a data processing method provided in an embodiment of this application, implemented through multiple functional units;

[0026] Figure 6 This is a schematic diagram of a bitmap provided in an embodiment of this application;

[0027] Figure 7 This is a flowchart illustrating how to select a target prediction algorithm from multiple prediction algorithms based on indication information and historical prediction results, as provided in an embodiment of this application.

[0028] Figure 8 This is a schematic diagram of a prediction algorithm based on hit rate, occupancy rate of the first storage medium, and computing power consumption, provided in an embodiment of this application.

[0029] Figure 9 This is a flowchart provided in an embodiment of the present application, which describes the selection of a target prediction algorithm from multiple prediction algorithms based on the scores of one or more measurement indicators.

[0030] Figure 10 This is a schematic diagram illustrating how an AI algorithm selects a target prediction algorithm from multiple prediction algorithms, as provided in an embodiment of this application.

[0031] Figure 11 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application;

[0032] Figure 12 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0034] To facilitate understanding, the technologies and background involved in the embodiments of this application will be explained below.

[0035] Typically, the process of a data reader retrieving data from secondary memory (L2) is as follows: the data reader issues a read request; primary memory (L1) first reads the data indicated by the read request from L2; then, the data reader reads the data from primary memory. The data access speed of L1 is greater than that of L2. However, the data processing speed of the data reader is usually faster than the data access speed. Therefore, the industry has proposed a data prefetching mechanism. Data prefetching refers to predicting the data that the data reader might need to read before actually reading it. Then, based on the prediction result, primary memory prefetches the data from L2 into primary memory so that the data reader can directly read the data from primary memory when needed. In this way, when the data reader performs a data read operation, if the required data has already been prefetched into the primary memory, the data reader can directly read the data from that primary memory without initiating a read operation to transfer data from secondary memory to primary memory. This hides the time spent transferring data from secondary memory to primary memory within the data reader's processing of other data, effectively shortening the data read time and achieving the goal of accelerating data reading speed. The algorithm used to predict what data needs to be prefetched into primary memory is called the data prefetch algorithm.

[0036] Since memory capacity is limited, when the capacity of the primary memory reaches a certain level, some data stored in the primary memory needs to be evicted to free up space for more valuable data. In some implementations, algorithms can be used to predict which data needs to be evicted; these algorithms are called data eviction algorithms. Data eviction algorithms and data prediction algorithms are combined and called data prediction algorithms. It should be noted that data eviction is not limited to data prefetching scenarios. For example, during normal data reading, the capacity of the primary memory may also become large, in which case the data stored in the primary memory also needs to be evicted.

[0037] Data prefetching and eviction techniques have wide applications, covering scenarios such as data movement and sharing between processors, hardware, software, data centers, and across network nodes. For example, at the hardware level, data prefetching and eviction can occur between different levels of cache in the central processing unit (CPU), and between storage media and their upper-level caches. At the operating system level, data prefetching and eviction can occur between pages managed by the operating system (OS) and memory. At the application level, metadata and data in databases and distributed file systems can be prefetched and evicted in memory caches.

[0038] Due to the high demand for caching across various scenarios, to meet these needs, in addition to directly increasing cache capacity, and considering both performance and cost, data prediction algorithms require superior performance. However, currently, no single data prediction algorithm can adapt to multiple application scenarios. This is because current data prediction algorithms tightly couple the data prefetching system with the prefetch eviction algorithm, making it impossible to run multiple algorithm instances simultaneously, perform performance statistics and evaluation concurrently, or dynamically enable, disable, and select the data prediction algorithm based on the results of simultaneous execution of multiple algorithm instances. For example, in current data prefetching systems, only one instance of a specific data prefetching algorithm with a certain set of parameters is running at any given time. Furthermore, current methods either select prediction algorithms based on pre-emptive or post-emptive mechanisms. However, pre-emptive selection often struggles to capture the patterns in data flow in real-world scenarios, leading to significant uncertainty in the actual performance gains of the prediction algorithm. Post-emptive selection typically switches prediction algorithms only when their actual hit rate is poor. Since performance loss has already occurred by this point, and the actual performance gains of the switched prediction algorithm can only be evaluated after actual execution, the effectiveness of post-emptive selection is also unsatisfactory.

[0039] For example, such as Figure 1 As shown in the top left figure, prediction algorithm A is used in scenario X. Based on the prediction results of prediction algorithm A, data is prefetched from main storage to the cache. The prefetched data includes all the data that the data reader needs to read. It can be seen that the prediction algorithm achieves good performance. When the scenario changes to... Figure 1 In scenario Y (top right of the diagram), prediction algorithm A is still used for prefetching. Based on the prediction results of algorithm A, the cache prefetches data from main storage to the cache. However, the prefetched data does not include all the data that the data reader needs to read. The data reader still needs to initiate a read operation from main storage to the cache. Therefore, it is evident that the performance of this prediction algorithm is poor in scenario Y. Figure 1As shown in the lower right image, the user, considering the degraded performance of the prediction algorithm, configured to stop using it. When the scenario changes to scenario Z in the lower left image, because the prediction algorithm is not enabled, the data reader needs to initiate a read operation from the main storage to the cache every time it needs to read data, resulting in slower data reading. Furthermore, since the prediction algorithm is not enabled in scenario Z, its performance cannot be evaluated.

[0040] Based on this, embodiments of this application provide a data processing method. In this data processing method, a computing device can run multiple prediction algorithms to obtain prediction results for future time periods. Furthermore, based on the prediction results of multiple prediction algorithms for historical time periods and the data actually read by the data reader during those historical time periods, a target prediction algorithm is selected from among the multiple prediction algorithms. Then, based on the prediction result of the target prediction algorithm for future time periods, the operation indicated by the future prediction result is executed. For example, when the prediction result indicates that data needs to be prefetched from the second storage medium to the first storage medium, based on the future prediction result of the target prediction algorithm, the data indicated by the future prediction result is prefetched from the second storage medium to the first storage medium so that the data reader can read data from the first storage medium in the future time period. When the prediction result indicates that data needs to be evicted from the first storage medium, based on the future prediction result of the target prediction algorithm, the data indicated by the future prediction result is evicted from the first storage medium. Wherein, the data access speed of the first storage medium is greater than that of the second storage medium. For example, the second storage medium is the primary storage, and the first storage medium is the cache.

[0041] In this way, the data processing method is equivalent to running multiple prediction algorithms simultaneously before performing data prefetching. Based on the prediction results of these algorithms for historical periods and the actual data read by the data reader during those periods, a prediction algorithm is selected for the current prefetching. After selecting the algorithm, data prefetching is performed based on the predicted results for future periods. Similarly, this data processing method is equivalent to running multiple prediction algorithms simultaneously before performing data eviction. Based on the prediction results of these algorithms for historical periods and the actual data read by the data reader during those periods, a prediction algorithm is selected for the current prefetching. After selecting the algorithm, data eviction is performed based on the predicted results for future periods. Therefore, this data processing method can be considered as selecting a prediction algorithm during the process and performing data prefetching and / or data eviction based on the selected algorithm. Compared to mechanisms that select prediction algorithms before or after the process, this method can predict the data to be prefetched and evictioned based on the data requirements of the data reading scenario, thus improving the accuracy of data prefetching and data eviction.

[0042] The implementation environment of a data processing method provided in this application embodiment will be described below.

[0043] Figure 2 This is a schematic diagram illustrating the implementation environment of a data processing method provided in an embodiment of this application. For example... Figure 2 As shown, the implementation environment includes a computing device 10. The computing device 10 can acquire indication information of data read by the data reader from the first storage medium and the second storage medium during a historical period, as well as historical prediction results of various prediction algorithms for the historical period. Based on the indication information and the historical prediction results, it can obtain future prediction results of various prediction algorithms for future periods, select a target prediction algorithm from among the multiple prediction algorithms, and then perform corresponding operations based on the future prediction results of the target prediction algorithm.

[0044] In one implementation, the data processing method provided in this application embodiment can be implemented by running an executable program on a computing device 10. For example, the executable program for this data processing method is presented in the form of an application installation package. After the application installation package is installed on the computing device 10, the data processing method can be implemented by running the executable program. In this case, the computing device 10 can be a terminal. The terminal can be a computer, personal computer, portable mobile terminal, multimedia player, e-book reader, or wearable device, etc.

[0045] Figure 3 This is a schematic diagram illustrating the implementation environment of another data processing method provided in this application embodiment. For example... Figure 3 As shown, the implementation environment may include a client 20 and a computing device 10. The client 20 is capable of establishing a communication connection with the computing device 10. For example, the client 20 and the computing device 10 can establish a communication connection via a network. Optionally, the network can be a local area network, the Internet, or other networks; this embodiment does not limit the scope of the application.

[0046] Client 20 is used for user interaction with computing device 10. In one implementation, client 20 is used to send instructions to computing device 10 according to user instructions. For example, client 20 is used to send configuration instructions to computing device 10 according to user instructions, the configuration instructions indicating configuration requirements for the prediction algorithm. Computing device 10 is used to receive the instructions sent by client 20 and perform related operations according to the instructions. For example, computing device 10 is used to receive the configuration instructions and select a prediction algorithm during the execution of the data processing method provided in the embodiments of this application according to the configuration requirements indicated by the configuration instructions.

[0047] In one possible implementation, the client 20 can be a desktop computer, laptop computer, mobile phone, smartphone, tablet computer, cloud server, portable mobile terminal, multimedia player, smart home appliance, artificial intelligence device, smart wearable device, e-reader, smart in-vehicle device, or IoT device, etc. The computing device 10 can be a server (such as a cloud server). Typically, the computing device 10 can be a single server, a server cluster consisting of several servers, or a cloud computing service center. The cloud computing service center deploys a large amount of basic resources owned by the cloud service provider. For example, the cloud computing service center deploys computing resources, storage resources, and network resources. The cloud computing service center can utilize these large amounts of basic resources to implement the data extraction method provided in this embodiment.

[0048] When computing device 10 is implemented through a cloud computing service center, users can access the cloud platform through client 20 to send configuration commands to computing device 10 via the cloud platform. At this time, the functions implemented by the data processing method provided in this embodiment can be abstracted into a data processing cloud service by the cloud service provider on the cloud platform. The cloud platform can utilize resources in the cloud computing center to provide this data processing cloud service to users. After purchasing the data processing cloud service on the cloud platform, users can perform data reading operations according to the data processing cloud service. Furthermore, this data processing cloud service can be provided as an independent cloud service or as an add-on service to other cloud services. Optionally, the cloud platform can be a central cloud platform, an edge cloud platform, or a cloud platform including both a central cloud and an edge cloud; this embodiment does not specifically limit its scope. It should be noted that... Figure 3 In the implementation environment shown, the computing device 10 can also be implemented through other resource platforms besides the cloud platform, and this application embodiment does not specifically limit it.

[0049] When performing a data prefetch operation, this application needs to prefetch the data recorded on the second storage medium into the first storage medium. When performing a data eviction operation, it needs to delete the specified data stored in the first storage medium. The first and second storage media can optionally both be deployed in the computing device 10, or neither can be deployed in the computing device 10, or one of the first and second storage media can be deployed in the computing device 10 while the other is not. For example, when both the first and second storage media are deployed in the computing device 10, the implementation scenario can be to prefetch data from external storage into memory and delete data stored in memory.

[0050] It should be understood that the above content is an exemplary description of the application scenarios of the data processing method provided in the embodiments of this application, and does not constitute a limitation on the application scenarios of the data processing method. As those skilled in the art know, as business needs change, the application scenarios can be adjusted according to the application needs, and the embodiments of this application do not list them one by one.

[0051] The implementation process of the data processing method provided in the embodiments of this application will be described below. For example... Figure 4 As shown, the data processing method includes the following steps:

[0052] Step 401: Obtain indication information of all data read by the data reader from the first storage medium and the second storage medium during the historical period.

[0053] Data prefetched from the second storage medium to the first storage medium can be considered data that the data reader is highly likely to read. Data evicted from the first storage medium can be considered data that the data reader is highly unlikely to read. Considering the temporal correlation of the data reader's reading behavior, when predicting the data to be prefetched and evicted, relevant information about all data read by the data reader from the first and second storage media in historical time periods can be referenced. Each time the data reader reads data, a record is kept, and the computing device can obtain relevant information about all data read by the data reader from the first and second storage media in historical time periods based on this record. For example, the data processing method provided in this application embodiment can be implemented through the collaborative cooperation of multiple functional units. Figure 5 This diagram illustrates a data processing method implemented by multiple functional units, as provided in an embodiment of this application. Figure 5 As shown, the multiple functional units include: a read processing unit, a preprocessing unit, an algorithm execution unit, a storage unit, a statistics unit, and a decision-making unit. The read processing unit can receive feedback from the data reader and record relevant information about the data read by the data reader based on this feedback. Therefore, the computing device can obtain relevant information about all data read by the data reader from the first and second storage media during a historical period based on the records of the read processing unit.

[0054] The relevant information about the data can be indicative information. This indicative information is used to indicate the data. In one implementation, the indicative information includes one or more of the following: data location information, data length, and a timestamp indicating when the data was read by a data reader. The data location information indicates the location of the data in the storage medium. The timestamp indicating when the data was read by the data reader indicates the time when the data was read from the storage medium. For example, for prefetched data, the timestamp indicates the time when the data was read from a first memory by the data reader. Optionally, when the prediction algorithm can serve multiple data readers, the indicative information also includes the identity (ID) of the data reader to distinguish the data read by different data readers.

[0055] Step 402: Based on the indication information and the historical prediction results of various prediction algorithms for historical periods, obtain the future prediction results of various prediction algorithms for future periods. The prediction results are used to indicate one or more of the following: data that needs to be prefetched from the second storage medium to the first storage medium, and data that needs to be evicted from the first storage medium.

[0056] After obtaining indication information of all data read by the data reader from the first and second storage media during a historical period, the system can predict the data the data reader might read in a future period based on this indication information and historical prediction results from various prediction algorithms for that historical period, thus obtaining future prediction results for the future period. In one implementation, the computing device analyzes the temporal correlation of the data read by the data reader based on the indication information, and obtains the deviation between the historical prediction results and the data reader's actual reading intention based on the historical prediction results and the indication information. Then, based on this correlation and the deviation, it predicts the data the data reader might read in the future, thus obtaining future prediction results for the future period. It should be noted that the embodiments of this application do not limit the prediction algorithms used in the prediction process.

[0057] The prediction algorithm indicates one or more of the following: prefetched data that needs to be prefetched from the second storage medium to the first storage medium, and obsolete data that needs to be deleted from the first storage medium. Since the first storage medium has a limited capacity, its usage may exceed a specified limit during data prefetching. In this case, deleting some data stored in the first storage medium allows the storage space originally used to store the prefetched data to be reused. In one implementation, a capacity waterline can be determined based on the actual usage of the first storage medium, and then compared with a threshold waterline. When the capacity waterline reaches the threshold waterline, the prediction algorithm indicates obsolete data. Here, the capacity waterline is the ratio of the actual usage of the first storage medium to the actual usage of the second storage medium. The threshold waterline is the ratio of the total capacity of the first storage medium to the total capacity of the second storage medium. For example, assuming the total capacity of the second storage medium is 256 gigabytes (GB) and the total capacity of the first storage medium is 8GB, then the threshold waterline is 8GB / 256GB = 3.125%.

[0058] In this embodiment, the computing device runs multiple prediction algorithm instances. These multiple prediction algorithm instances can be distinguished based on scenarios, algorithm parameters, and data streams. Distinguishing prediction algorithm instances based on scenarios means that the same prediction algorithm applied to different scenarios will result in different prediction algorithm instances. Distinguishing prediction algorithm instances based on algorithm parameters means that the same prediction algorithm with different parameter configurations will result in different prediction algorithm instances. Distinguishing prediction algorithm instances based on data streams means that the same prediction algorithm applied to different data streams will result in different prediction algorithm instances. Different data streams indicate different services to which the data belongs. After obtaining the indication information and historical prediction results, the computing device can determine the prediction algorithm instance suitable for the data reader. Based on the indication information and historical prediction results, the suitable prediction algorithm instance can obtain the future prediction result of that suitable prediction algorithm instance. For example, such as... Figure 5 As shown, after acquiring the indication information and historical prediction results, the preprocessing unit can determine multiple prediction algorithm instances suitable for the data reader. These multiple suitable prediction algorithm instances can then obtain future prediction results based on the indication information and historical prediction results. For example, when the indication information includes the data reader's ID, the preprocessing unit can convert the data reader's ID into a data stream ID, then determine multiple prediction algorithms that can be used with that data stream ID, and correspondingly identify the multiple prediction algorithm instances used to execute those algorithms as multiple prediction algorithm instances suitable for the data reader.

[0059] In one implementation, the location and length of the data indicated by the instruction information and historical prediction information can be recorded using a bitmap. Furthermore, as... Figure 5 As shown, the bitmap can be maintained by the storage unit. This application configures four physical bitmaps and two logical bitmaps for each prediction algorithm instance. Each bit of the bitmap is used to record information about the data recorded in a data block in the second storage medium. For example, when a bit of the bitmap records the key of the data block, it indicates that the data block satisfies the conditions corresponding to the bitmap. Before using the bitmap to record information, the size of the bitmap can be optionally instantiated based on the total capacity of the second storage medium. For example, assuming each data block occupies 8 kilobytes (KB), and the total capacity of the second storage medium is 256 GB, then the second storage medium can store 32 megabytes (MB) of data blocks, and the size of the bitmap is at least 32 megabits (Mb), or 4 MB.

[0060] The four entity bitmaps are: ioBitMap (read operation hit bitmap), ioHitBitMap (repeated IO hit bitmap), prefetchBitMap (prefetch bitmap), and prefetchHitBitMap (prefetch hit bitmap). ioBitMap records information about data not hit in the first storage medium, i.e., data not prefetched but read by the data reader. For example, if the data needed by the data reader is not prefetched to the first storage medium, the data reader can initiate a read operation from the second storage medium to the first storage medium. In this case, ioBitMap records information about the data read through the read operation sent by the data reader. ioHitBitMap records information about data hit in the first storage medium due to repeated IO. For example, when memory manages data according to access frequency, if data has high access frequency, it is retained in memory for future use. In this case, ioHitBitMap records information about data cached in memory due to access frequency and read by the data reader. prefetchBitMap records information about data prefetched based on the prediction results of the current prediction algorithm instance. The `prefetchHitBitMap` is used to record information about data prefetched based on the prediction results of the current prediction algorithm instance and read by the data reader. In this application, "hit" refers to data being read by the data reader.

[0061] Logical bitmaps do not occupy actual memory space and can be implemented by logically combining entity locations. For example, a logical bitmap is obtained by performing a logical OR operation on an entity bitmap. Two logical bitmaps are used: a cache bitmap (cacheBitMap) and a cache hit bitmap (cacheHitBitMap). The cacheBitMap is composed of entity bitmaps ioBitMap and premiumBitMap. It is evident that cacheBitMap indicates prefetched data and data read through read operations sent by the data reader. The cacheHitBitMap is also composed of entity bitmaps ioHitBitMap and premiumHitBitMap. It is evident that cacheHitBitMap indicates data read by the data reader due to prefetching and data read by the data reader due to repeated I / O. Figure 6 This is a schematic diagram of a bitmap provided in an embodiment of this application. According to... Figure 6 As can be seen, cacheBitMap is obtained by performing a logical OR operation between the entity bitmap ioBitMap and premiumBitMap. cacheHitBitMap is obtained by performing a logical OR operation between the entity bitmap ioHitBitMap and premiumHitBitMap.

[0062] Step 403: Based on the indication information and historical prediction results, select the target prediction algorithm from multiple prediction algorithms.

[0063] After acquiring indication information and historical prediction results, the computing device can optionally evaluate multiple prefetching algorithms based on the indication information and historical prediction results, and select a target prediction algorithm from among the multiple prediction algorithms according to the evaluation results. This allows the target prediction algorithm to be used to perform the operation indicated by the future prediction result on the data. In one implementation, such as... Figure 7 As shown, the implementation process of step 403 includes:

[0064] Step 4031: Based on the indication information and historical prediction results, obtain the scores of multiple prediction algorithms for one or more measurement indicators.

[0065] Metrics are indicators used to measure the performance of prediction algorithms. Multiple metrics can measure the performance of prediction algorithms from multiple dimensions. In one implementation, the metrics include: the probability that data is read by a data reader; and / or, the ratio of the capacity of data stored in the first storage medium to the total capacity of the first storage medium; and / or, the computing power required to execute the operation indicating the future prediction result. When the prediction result indicates that data needs to be prefetched from the second storage medium to the first storage medium, the probability that the data is read by the data reader includes at least the probability that the data stored in the second storage medium is read by the data reader. Generally, when this probability is high, the data needs to be prefetched into the first storage medium. When the prediction result indicates that data needs to be evicted from the first storage medium, the probability that the data is read by the data reader is actually the probability that the data stored in the first storage medium is read by the data reader without distinguishing how the data was read from the first storage medium. Generally, when this probability is low, the data needs to be evicted from the first storage medium. When the prediction result indicates that data needs to be prefetched from the second storage medium to the first storage medium, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium includes at least the ratio of the capacity of the data prefetched from the second storage medium to the total capacity of the first storage medium. When the prediction result indicates that data needs to be evicted from the first storage medium, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium is actually the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium without distinguishing how the data stored in the first storage medium was read from the first storage medium. When the prediction result indicates that data needs to be prefetched from the second storage medium to the first storage medium, the computing power required to execute the operation indicated by the future prediction result includes at least the computing power required to prefetch the data from the second storage medium to the first storage medium. When the prediction result indicates that data needs to be evicted from the first storage medium, the computing power required to execute the operation indicated by the future prediction result includes at least the computing power required to delete the data from the first storage medium.

[0066] The three metrics above are used to measure the hit rate of the prediction algorithm, the utilization rate of the first storage medium, and the computing power consumption, respectively. Figure 8 This application provides a method based on hit rate for the first storage medium ( Figure 8 The diagram illustrates the algorithm for predicting cache occupancy and computing power consumption. Furthermore, as shown... Figure 5 As shown, the metrics for obtaining the prediction algorithm can be performed by the statistical unit.

[0067] Optionally, the types of metrics can be adjusted according to application requirements, such as increasing or decreasing the types of metrics. The calculation method of the metrics can also be adjusted according to application requirements. For example, the metric for measuring computing power consumption can be expanded to: the ratio of computing power consumed in prefetching data from the second storage medium to the first storage medium to the computing power consumed in reading data through a read operation sent by the data reader. For example, assuming that each time data prefetching can store ten data from the second storage medium into the first storage medium, and each time the data reader sends a read operation can read one data into the first storage medium, the computing power consumed by prefetching and reading data through read operations is X, and the computing power consumed by running the prediction algorithm is a, then the computing power consumed by prefetching one data is a+X / 10, and the computing power consumed by reading one data through a read operation is X. Therefore, the ratio of the computing power consumed by prefetching one data from the second storage medium to the first storage medium to the computing power consumed by reading one data through a read operation sent by the data reader is (a+X / 10) / X.

[0068] It should be understood that the data used to calculate each metric can be adjusted according to application requirements. For example, each metric can be determined based on data readings within a specified time period before the current time, or based on data readings between the start of the prediction algorithm and the current time, or based on a specified time period before the current time, etc. This application embodiment does not specifically limit it.

[0069] In one implementation, this application embodiment configures multiple counters for each prediction algorithm to statistically measure the performance of the prediction algorithm. As shown in Table 1, the uses and calculation methods of the various counters are as follows:

[0070] Table 1

[0071]

[0072]

[0073] When the aforementioned entity bitmap and logic bitmap are used to record information related to the prediction algorithm, the counters in Table 1 can count according to the data recorded in the bitmap. The implementation methods are explained below using the following examples.

[0074] Case 1: If the key of the current IO data is not in the prefetchBitMap, it means that the data meets the conditions of the ioBitMap. In this case, the key of the data is inserted into the corresponding bit of the ioBitMap.

[0075] Case 2: If the key of the current IO data is in ioBitMap but not in ioHitBitMap, then insert the corresponding bit into ioHitBitMap and increment the ioHit count.

[0076] Statistics Case 3: If the key of the current IO data is in ioBitMap and ioHitBitMap, then only the ioHit count is incremented.

[0077] Case 4: If the key of the current IO data is in prefetchBitMap but not in prefetchHitBitMap, then insert the corresponding bit in prefetchHitBitMap and increment the prefetchHit count.

[0078] Statistics for Case 5: If the key of the current IO data is in cacheBitMap and also in prefetchBitMap and prefetchHitBitMap, then only the prefetchHit count will be incremented.

[0079] Prefetch Case 1: If the key of the data to be prefetched is already in ioBitMap or prefetchBitMap, then no update or counting is required.

[0080] Prefetch Case 2: If the key of the data to be prefetched is not in ioBitMap or prefetchBitMap, then insert the corresponding bit into prefetchBitMap and increment the non-piggybacked computing power count. If the key of the data to be prefetched predicted by the prediction algorithm is not in the same piggybackable range as the key actually read by the current IO (e.g., the same page), then increment the computing power count.

[0081] Step 4032: Based on the scores of one or more measurement indicators from multiple prediction algorithms, select the target prediction algorithm from among the multiple prediction algorithms.

[0082] In one implementation, such as Figure 9 As shown, the implementation process of step 4032 includes:

[0083] Step 4032a: Determine the weights of various metrics based on the configuration strategy.

[0084] Configuration strategies can be user-configured strategies for prediction algorithms or default strategies configured for computing device administrators, reflecting different needs such as performance priority, cost priority, and a comprehensive consideration of performance and cost. Optionally, configuration strategies may include one or more combinations of the following: prioritizing the probability that data prefetched from the second storage medium to the first storage medium is read by the data reader; prioritizing reducing the ratio of the capacity of data prefetched from the second storage medium to the capacity of the first storage medium; and prioritizing saving the computing power required to execute operations indicating future prediction results. For example, prioritizing saving the computing power consumed in prefetching data from the second storage medium to the first storage medium, and prioritizing saving the computing power consumed in deleting data from the first storage medium. These three correspond to hit rate priority, cache saving priority, and computing power saving priority, respectively. Furthermore, configuration strategies can also be performance-priority strategies and weighted combinations of various corresponding strategies. After determining the configuration strategy, the degree of influence of different indicators on the prediction results can be obtained, and this degree of influence is expressed as the weight of various metrics. For example, when the configuration strategy prioritizes hit rate over cache saving, the weights of hit rate, cache utilization, and computing power consumption are 0.5, 0.3, and 0.2, respectively.

[0085] Step 4032b: Based on the scores of one or more metrics and the weights of the metrics from multiple prediction algorithms, select the target prediction algorithm from the multiple prediction algorithms.

[0086] After acquiring scores and weights for one or more metrics from various prediction algorithms, the computing device calculates the scores of each algorithm based on these scores and weights. Then, it selects a target prediction algorithm from among these algorithms based on their scores. For example, when selecting a target prediction algorithm from multiple algorithms, the device can calculate a weighted sum of the scores and weights of each algorithm's metrics, and then determine the algorithm with the largest weighted sum as the target prediction algorithm. Figure 5 As shown, both steps 4031a and 40312b can be executed by the decision-making unit. Furthermore, as... Figure 8 As shown, the decision-making unit may include a first decision subunit and multiple second decision subunits. Each of the multiple second decision subunits corresponds to a multiple prediction algorithm instance. The second decision subunit corresponding to any prediction algorithm instance is used to make a decision for that prediction algorithm instance, and the first decision subunit is used to select the target prediction algorithm instance from the multiple prediction algorithm instances. For example, any prediction algorithm instance may generate multiple sets of outputs based on its multiple sets of algorithm parameters for each input. The second decision subunit corresponding to that prediction algorithm instance is used to select the set of algorithm parameters with the best performance from the multiple sets of algorithm parameters based on the multiple sets of outputs.

[0087] Optionally, such as Figure 8 As shown, the computing device can also optionally adjust the parameters of some or all of the multiple prediction algorithms based on their scores for one or more metrics. For example, if some of the prediction algorithms score low, indicating poor performance, the parameters of those algorithms can be adjusted to improve their performance.

[0088] Furthermore, the computing device can also optionally determine whether to execute a prediction algorithm based on the scores of multiple prediction algorithms. For example, if the weighted sum of the scores of multiple prediction algorithms is less than the execution score threshold, it can be determined that performing data prefetching operations based on the prediction results of these multiple prediction algorithms will not yield good prediction performance, then the computing device determines that data prefetching operations are unnecessary.

[0089] In one implementation, step 403 may be implemented using an artificial intelligence (AI) algorithm. For example, the decision-making unit may be configured with an AI algorithm to decide whether to perform prefetching and culling, and which prediction algorithm to select. Figure 10 This is a schematic diagram illustrating step 403 implemented using an AI algorithm, as provided in an embodiment of this application. Figure 10 Compared to Figure 8 A decision learning unit was added, and the Figure 10 The decision learning unit, the first decision subunit, and the second decision subunit are all implemented using AI algorithms. The input of the decision learning unit is connected to the outputs of multiple prediction algorithm instances, allowing it to learn from the outputs of these instances and provide guidance for the decision-making of the second decision subunit. Furthermore, compared to... Figure 8 The second decision subunit shown, for any prediction algorithm instance, is also used to make decisions on adjusting parameters for its corresponding prediction algorithm instance. When implementing step 403 using an AI algorithm, since the AI ​​algorithm can continuously learn the running results and corresponding metrics of each algorithm instance, it can obtain better algorithm and parameter selection results under the same configuration strategy.

[0090] Step 404: Based on the future prediction results of the target prediction algorithm, execute the operation indicated by the future prediction results.

[0091] Data recorded on the second storage medium is prefetched into the first storage medium so that the data reader can read the data from the first storage medium at a future time.

[0092] After determining a target prediction algorithm from multiple prediction algorithms, the computing device can obtain the future prediction result of the target prediction algorithm and execute the operation indicated by the future prediction result. For example, when the prediction result indicates that data needs to be prefetched from the second storage medium to the first storage medium, the data indicated by the future prediction result is prefetched from the second storage medium to the first storage medium based on the future prediction result of the target prediction algorithm, so that the data reader can read the data from the first storage medium in a future time period. When the prediction result indicates that data needs to be evicted from the first storage medium, the data indicated by the future prediction result is evicted from the first storage medium based on the future prediction result of the target prediction algorithm.

[0093] In the data processing method provided in this application embodiment, it is equivalent to simultaneously executing multiple prediction algorithms based on multiple inputs during each process of obtaining a prediction result based on inputs, selecting a target prediction algorithm for this prefetching based on the performance of these multiple prediction algorithms in historical time periods, and then performing corresponding operations based on the future prediction results of the target prediction algorithm. Therefore, this data processing method can be considered as selecting a target prediction algorithm during the process and performing operations based on the target prediction algorithm. Compared with mechanisms that select prediction algorithms before or after the process, this data processing method can predict the data to be prefetched and discarded based on the data requirements of the data reading scenario, which helps to improve the accuracy of data prefetching and discarding. Furthermore, this data processing method enables the data reading system to be decoupled from the prediction algorithm instance, which helps to achieve flexible configuration and dynamic online operation of the prediction algorithm and parameters.

[0094] It should be noted that the approach of running multiple algorithms simultaneously and making decisions in real time based on multiple indicators in this application is not only applicable to prediction algorithms, but also to algorithms in other fields. For example, according to the data flow classification, the applicable data flow for the prefetching algorithm and the eviction algorithm is shown in Table 2.

[0095] Table 2

[0096]

[0097]

[0098] For example, according to the application scenario, the scalable application scenarios of this application for prefetching algorithm and eviction algorithm are shown in Table 3.

[0099] Table 3

[0100]

[0101] In summary, this data processing method is equivalent to simultaneously running multiple prediction algorithms before performing data prefetching. Based on the prediction results of these algorithms for historical periods and the actual data read by the data reader during those periods, a prediction algorithm is selected for the current prefetching. After selecting the algorithm, data prefetching is performed based on the predicted results for future periods. Similarly, this data processing method is equivalent to simultaneously running multiple prediction algorithms before performing data eviction. Based on the prediction results of these algorithms for historical periods and the actual data read by the data reader during those periods, a prediction algorithm is selected for the current prefetching. After selecting the algorithm, data eviction is performed based on the predicted results for future periods. Therefore, this data processing method can be considered as selecting a prediction algorithm during the process and performing data prefetching and / or data eviction based on the selected algorithm. Compared to mechanisms that select prediction algorithms before or after the process, this method can predict the data to be prefetched and evictioned based on the data requirements of the data reading scenario, thus improving the accuracy of data prefetching and data eviction.

[0102] It should be noted that the order of steps in the data processing method provided in this application embodiment can be appropriately adjusted, and steps can also be added or removed as appropriate. Any variations that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application, and therefore will not be elaborated further.

[0103] The data processing method of this application embodiment has been described above. Corresponding to the above method, this application embodiment also provides a data processing apparatus. A schematic diagram of the structure of the data processing apparatus is shown below. Figure 11 As shown. It should be understood that the data processing apparatus may include more additional modules than the structure shown, or may omit some of the modules shown, and the embodiments of this application do not impose limitations in this regard. Figure 11 As shown, the data processing device 110 includes:

[0104] The acquisition module 1101 is used to acquire indication information of all data read by the data reader from the first storage medium and the second storage medium during a historical period.

[0105] Processing module 1102 is used to obtain future prediction results for future periods based on indication information and historical prediction results of multiple prediction algorithms for historical periods. The prediction results indicate one or more of the following: data that needs to be prefetched from the second storage medium to the first storage medium, and data that needs to be evicted from the first storage medium.

[0106] The processing module 1102 is also used to select a target prediction algorithm from a variety of prediction algorithms based on the indication information and historical prediction results.

[0107] The reading module 1103 is used to perform operations indicated by the future prediction results based on the future prediction results of the target prediction algorithm.

[0108] Optionally, the indication information may include one or more of the following: the location information of the data, the length of the data, and the timestamp of the data being read by the data reader.

[0109] Optionally, the processing module 1102 is specifically used to: obtain the scores of multiple prediction algorithms for one or more metrics based on the indication information and historical prediction results; and select the target prediction algorithm from the multiple prediction algorithms based on the scores of multiple prediction algorithms for one or more metrics.

[0110] Optionally, the processing module 1102 is specifically used for: determining the weights of multiple metrics based on a configuration strategy; and selecting a target prediction algorithm from multiple prediction algorithms based on the scores of one or more metrics and the weights of the multiple metrics.

[0111] Optionally, the metrics include: the probability that the data is read by the data reader; and / or, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium; and / or, the computing power required to execute the operation indicating the future prediction result.

[0112] Optionally, the processing module 1102 is also used to: adjust the parameters of some or all of the multiple prediction algorithms based on the scores of multiple prediction algorithms for one or more metrics.

[0113] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the modules described above can be referred to the corresponding content in the foregoing method embodiments, and will not be repeated here.

[0114] In summary, this data processing device is equivalent to simultaneously running multiple prediction algorithms before performing data prefetching. Based on the prediction results of these algorithms for historical periods and the actual data read by the data reader during those periods, it selects the prediction algorithm to use for the current prefetch. After selecting the algorithm, it performs data prefetching based on the predicted results for future periods. Similarly, this data processing device is equivalent to simultaneously running multiple prediction algorithms before performing data eviction. Based on the prediction results of these algorithms for historical periods and the actual data read by the data reader during those periods, it selects the prediction algorithm to use for the current prefetch. After selecting the algorithm, it performs data eviction based on the predicted results for future periods. Therefore, this data processing device can be considered as selecting a prediction algorithm during the process and performing data prefetching and / or data eviction based on the selected algorithm. Compared to mechanisms that select prediction algorithms before or after the process, this data processing device can predict the data to be prefetched and evictioned based on the data requirements of the data reading scenario, thus improving the accuracy of data prefetching and data eviction.

[0115] This application provides a computing device. This computing device is used to implement some or all of the functions in the data processing method provided in this application. Figure 12 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 12 As shown, the computing device 1200 includes a processor 1201, a memory 1202, a communication interface 1203, and a bus 1204. The processor 1201, memory 1202, and communication interface 1203 are interconnected via the bus 1204.

[0116] Processor 1201 may include a general-purpose processor and / or a dedicated hardware chip. The general-purpose processor may include a central processing unit (CPU), a microprocessor, or a graphics processing unit (GPU). The CPU may be a single-core processor or a multi-core processor. The dedicated hardware chip is a high-performance processing hardware module. The dedicated hardware chip includes at least one of a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a network processor (NP). Processor 1201 may also be an integrated circuit chip with signal processing capabilities. In implementation, some or all of the functions of the data processing method of this application may be implemented through the integrated logic circuits in the hardware of processor 1201 or through software instructions.

[0117] Memory 1202 is used to store computer programs, including an operating system 1202a and executable code (i.e., program instructions) 1202b. Memory 1202 may be, for example, a read-only memory or other type of static storage device capable of storing static information and instructions; or a random access memory or other type of dynamic storage device capable of storing information and instructions; or an electrically erasable programmable read-only memory, a read-only optical disc or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), a magnetic disk storage medium, or other magnetic storage device; or any other medium capable of carrying or storing desired executable code in the form of instructions or data structures and accessible by a computer, but not limited thereto. For example, memory 1202 may be used to store output port queues, etc. Memory 1202 may exist independently and be connected to processor 1201 via bus 1204. Alternatively, memory 1202 and processor 1201 may be integrated together. The memory 1202 can store executable code. When the executable code stored in the memory 1202 is executed by the processor 1201, the processor 1201 performs some or all of the functions of the data processing method provided in the embodiments of this application. Please refer to the relevant descriptions in the foregoing embodiments for the implementation method of the processor 1201 executing this process. The memory 1202 may also include other software modules and data required by the running process, such as the operating system.

[0118] The communication interface 1203 uses a transceiver module, such as, but not limited to, a transceiver, to enable communication with other devices or communication networks. For example, the communication interface 1203 can be any one or any combination of the following devices: a network interface (such as an Ethernet interface), a wireless network card, or other devices with network access capabilities.

[0119] Bus 1204 can be of any type, used for interconnecting internal devices (e.g., memory 1202, processor 1201, communication interface 1203) of a computing device. For example, a system bus. This embodiment illustrates the interconnection of the aforementioned devices within a computing device via bus 1204. Optionally, the devices within computing device 1200 can also communicate with each other using other connection methods besides bus 1204. For example, the devices within computing device 1200 can be interconnected via internal logic interfaces.

[0120] It should be noted that the aforementioned devices can be disposed on separate chips, or at least partially or entirely on the same chip. Whether to dispose of the devices independently on different chips or integrate them on one or more chips often depends on the needs of the product design. This application does not limit the specific implementation of the aforementioned devices. Furthermore, the descriptions of the processes corresponding to the various figures above each have their own emphasis; for parts of a process not described in detail in one figure, please refer to the relevant descriptions of other processes.

[0121] In the above embodiments, the methods can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented entirely or partially in the form of a computer program product. The computer program product providing the program development platform includes one or more computer instructions. When these computer program instructions are loaded and executed on a computing device, they implement some or all of the functions of the data processing method provided in the embodiments of this application.

[0122] Furthermore, computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide a program development platform.

[0123] This application also provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium, and includes program instructions that, when executed on a computing device, cause the computing device to implement the data processing method provided in this application.

[0124] This application also provides a computer program product containing instructions, which, when run on a computer, enables the computer to implement the data processing method provided in this application.

[0125] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0126] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the image information and executable code involved in this application were obtained with full authorization.

[0127] In the embodiments of this application, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "at least one" refers to one or more, and the term "multiple" refers to multiples, unless otherwise expressly defined.

[0128] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0129] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0130] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0131] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the concept and principles of this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain indication information of all data read by the data reader from the first and second storage media during a historical period; Based on the indicated information and the historical prediction results of multiple prediction algorithms for the historical period, multiple prediction algorithms are run to predict the data to be prefetched and / or evicted in this period, and the future prediction results of the multiple prediction algorithms for future periods are obtained respectively. The prediction results indicate one or more of the following: data that needs to be prefetched from the second storage medium to the first storage medium, and data that needs to be evicted from the first storage medium. Based on the indication information and the historical prediction results, select the target prediction algorithm used for this prefetching and / or elimination from the multiple prediction algorithms; Based on the future prediction results of the target prediction algorithm, perform the current prefetching and / or elimination operations indicated by the future prediction results.

2. The method as described in claim 1, characterized in that, The indication information includes one or more of the following: the location information of the data, the length of the data, and the timestamp of the data being read by the data reader.

3. The method as described in claim 1, characterized in that, The step of selecting the target prediction algorithm for this prefetching and / or elimination based on the indication information and the historical prediction results from the multiple prediction algorithms includes: Based on the indicated information and the historical prediction results, obtain the scores of the various prediction algorithms for one or more measurement indicators; Based on the scores of the various prediction algorithms for the one or more metrics, the target prediction algorithm is selected from the various prediction algorithms.

4. The method as described in claim 3, characterized in that, The step of selecting the target prediction algorithm from the multiple prediction algorithms based on the scores of the one or more measurement indicators includes: The weights of the various metrics are determined based on the configuration strategy; Based on the scores of the various prediction algorithms for one or more metrics and the weights of the various metrics, the target prediction algorithm is selected from the various prediction algorithms.

5. The method as described in claim 3, characterized in that, The metrics include: The probability that the data is read by the data reader; And / or, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium; And / or, the computing power required to perform the operation indicated by the future prediction result.

6. The method as described in claim 1, characterized in that, The method further includes: Based on the scores of the various prediction algorithms for one or more metrics, the parameters of some or all of the various prediction algorithms are adjusted.

7. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire indication information of all data read by the data reader from the first storage medium and the second storage medium during a historical period; The processing module is configured to run multiple prediction algorithms to predict the data to be prefetched and / or evicted in this instance, based on the indication information and the historical prediction results of multiple prediction algorithms for the historical period, and to obtain the future prediction results of the multiple prediction algorithms for the future period, wherein the prediction results indicate one or more of the following: data that needs to be prefetched from the second storage medium to the first storage medium, and data that needs to be evicted from the first storage medium; The processing module is also used to select the target prediction algorithm used for this prefetching and / or elimination from the multiple prediction algorithms based on the indication information and the historical prediction results; The reading module is used to perform the current prefetching and / or elimination operation indicated by the future prediction result based on the future prediction result of the target prediction algorithm.

8. The apparatus as claimed in claim 7, characterized in that, The indication information includes one or more of the following: the location information of the data, the length of the data, and the timestamp of the data being read by the data reader.

9. The apparatus as claimed in claim 7, characterized in that, The processing module is specifically used for: Based on the indicated information and the historical prediction results, obtain the scores of the various prediction algorithms for one or more measurement indicators; Based on the scores of the various prediction algorithms for the one or more metrics, the target prediction algorithm is selected from the various prediction algorithms.

10. The apparatus as claimed in claim 9, characterized in that, The processing module is specifically used for: The weights of the various metrics are determined based on the configuration strategy; Based on the scores of the various prediction algorithms for one or more metrics and the weights of the various metrics, the target prediction algorithm is selected from the various prediction algorithms.

11. The apparatus as claimed in claim 9, characterized in that, The metrics include: The probability that the data is read by the data reader; And / or, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium; And / or, the computing power required to perform the operation indicated by the future prediction result.

12. The apparatus as claimed in claim 7, characterized in that, The processing module is further configured to: adjust the parameters of some or all of the multiple prediction algorithms based on the scores of the multiple prediction algorithms for one or more measurement indicators.

13. A computing device, characterized in that, The computing device includes a processor and a memory, wherein the processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1 to 6.

14. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the method as described in any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1 to 6.

Images