Data processing method and device

Abstract

The invention discloses a data processing method and device, and belongs to the technical field of storage. The method comprises the following steps: acquiring indication information of all data read by a data reader from a first storage medium and a second storage medium in a historical period; on the basis of the indication information and historical prediction results of the multiple prediction algorithms for the historical time period, future prediction results of the multiple prediction algorithms for the future time period are obtained, and the prediction results indicate one or more of data needing to be pre-fetched from the second storage medium to the first storage medium and data needing to be eliminated from the first storage medium; selecting a target prediction algorithm from a plurality of prediction algorithms based on the indication information and the historical prediction result; and based on the future prediction result of the target prediction algorithm, executing an operation indicated by the future prediction result. The performance of the prediction and elimination algorithm can be improved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data processing method and device. Background Art

[0002] Data pre-fetching and elimination technologies are widely used, covering scenarios such as data movement and sharing between processors, hardware, software, data centers, and cross-network nodes. For example, from the hardware media level, data can be pre-fetched and eliminated between caches at all levels of the central processing unit (CPU), and data can be pre-fetched and eliminated between the storage medium and its upper cache. From the operating system level, data can be pre-fetched and eliminated between pages and memory managed by the operating system (OS). From the application level, metadata and data in databases and distributed file systems can be pre-fetched and eliminated in memory caches.

[0003] Since various scenarios have a great demand for cache, in order to meet the cache demand, in addition to directly increasing the cache capacity, the prediction algorithm needs to have higher performance while comprehensively considering performance and cost. Based on this, the present application provides a data processing method. Summary of the invention

[0004] This application provides a data processing method and device. This application can improve the performance of prediction and elimination algorithms. The technical solutions provided by this application are as follows:

[0005] In a first aspect, the present application provides a data processing method. The method includes: obtaining indication information of all data read by a data reader from a first storage medium and a second storage medium in a historical period; based on the indication information and the historical prediction results of multiple prediction algorithms for the historical period, respectively obtaining future prediction results of multiple prediction algorithms for future periods, the prediction results indicating one or more of the following: data that needs to be pre-fetched from the second storage medium to the first storage medium, data that needs to be eliminated from the first storage medium; based on the indication information and the historical prediction results, selecting a target prediction algorithm from multiple prediction algorithms; based on the future prediction results of the target prediction algorithm, performing operations indicated by the future prediction results.

[0006] In this way, the data processing method is equivalent to running multiple prediction algorithms at the same time before executing the data pre-fetching operation, and selecting the prediction algorithm used for this pre-fetching according to the prediction results of the multiple prediction algorithms for the historical period and the actual situation of the data reader reading the data in the historical period, and after selecting the prediction algorithm, performing the data pre-fetching operation on the prediction results of the future period according to the selected prediction algorithm. Similarly, the data processing method is equivalent to running multiple prediction algorithms at the same time before executing the data elimination operation, and selecting the prediction algorithm used for this pre-fetching according to the prediction results of the multiple prediction algorithms for the historical period and the actual situation of the data reader reading the data in the historical period, and after selecting the prediction algorithm, performing the data elimination operation on the prediction results of the future period according to the selected prediction algorithm. Therefore, the data processing method can be considered to select the prediction algorithm in the event, and perform the data pre-fetching operation and / or data elimination operation according to the selected prediction algorithm. Compared with the mechanism of selecting the prediction algorithm in advance and the mechanism of selecting the prediction algorithm after the event, the data processing method can predict the data that needs to be pre-fetched and eliminated according to the data demand characteristics of the data reading scene, which helps to improve the accuracy of data pre-fetching and data elimination.

[0007] In one implementation, the indication information includes one or more of the following: location information of the data, length of the data, and a timestamp when the data is read by the data reader. The location information of the data is used to indicate the location of the data in the storage medium. The timestamp when the data is read by the data reader indicates the time when the data is read from the storage medium by the data reader. For example, for pre-fetched data, the timestamp indicates the time when the data is read from the first memory by the data reader. Optionally, when the prediction algorithm can serve multiple data readers, the indication information also includes an identifier of the data reader to distinguish data read by different data readers.

[0008] Among them, based on the indication information and historical prediction results, a target prediction algorithm is selected from multiple prediction algorithms, including: based on the indication information and historical prediction results, obtaining scores of multiple prediction algorithms for one or more measurement indicators; based on the scores of multiple prediction algorithms for one or more measurement indicators, selecting a target prediction algorithm from multiple prediction algorithms.

[0009] In one implementation, a target prediction algorithm is selected from a plurality of prediction algorithms based on scores of the plurality of prediction algorithms for one or more measurement indicators, including: determining weights of the plurality of measurement indicators based on a configuration strategy; and selecting a target prediction algorithm from a plurality of prediction algorithms based on scores of the plurality of prediction algorithms for one or more measurement indicators and weights of the plurality of measurement indicators.

[0010] The measurement indicators include: the probability of the data being read by the data reader; and / or the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium; and / or the computing power required to execute the operation indicated by the future prediction result. When the prediction result indicates that data needs to be pre-fetched from the second storage medium to the first storage medium, the probability of the data being read by the data reader includes at least the probability that the data stored in the second storage medium is read by the data reader. Generally speaking, when the probability of the data is large, the data needs to be pre-fetched to the first storage medium. When the prediction result indicates that the data needs to be eliminated from the first storage medium, the probability of the data being read by the data reader is actually the probability that the data stored in the first storage medium is read by the data reader without distinguishing how the data stored in the first storage medium is read to the first storage medium. Generally speaking, when the probability of the data is small, the data needs to be eliminated from the first storage medium. When the prediction result indicates that data needs to be pre-fetched from the second storage medium to the first storage medium, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium includes at least the ratio of the capacity of the data pre-fetched from the second storage medium to the capacity of the first storage medium. When the prediction result indicates that data needs to be eliminated from the first storage medium, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium is actually the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium without distinguishing how the data stored in the first storage medium is read to the first storage medium. When the prediction result indicates that data needs to be pre-fetched from the second storage medium to the first storage medium, the computing power required to perform the operation indicated by the future prediction result includes at least the computing power consumed to pre-fetch data from the second storage medium to the first storage medium. When the prediction result indicates that data needs to be eliminated from the first storage medium, the computing power required to perform the operation indicated by the future prediction result includes at least the computing power required to delete the data from the first storage medium.

[0011] Optionally, the method further includes: adjusting parameters of some or all of the multiple prediction algorithms based on the scores of the multiple prediction algorithms for one or more measurement indicators. For example, when the scores of some prediction algorithms among the multiple prediction algorithms are low, indicating that the performance of the prediction algorithm is poor, the parameters of the prediction algorithm are adjusted to improve the performance of the prediction algorithm.

[0012] In a second aspect, the present application provides a data processing device. The device includes: an acquisition module, which is used to acquire indication information of all data read by a data reader from a first storage medium and a second storage medium in a historical period; a processing module, which is used to obtain future prediction results of multiple prediction algorithms for future periods based on the indication information and historical prediction results of multiple prediction algorithms for the historical period, respectively, and the prediction results indicate one or more of the following: data that needs to be pre-fetched from the second storage medium to the first storage medium, and data that needs to be eliminated from the first storage medium; the processing module is also used to select a target prediction algorithm from multiple prediction algorithms based on the indication information and the historical prediction results; and a reading module is used to perform operations indicated by the future prediction results based on the future prediction results of the target prediction algorithm.

[0013] Optionally, the indication information includes one or more of the following: location information of the data, length of the data, and a timestamp when the data is read by the data reader.

[0014] Optionally, the processing module is specifically used to: obtain scores of multiple prediction algorithms for one or more measurement indicators based on the indication information and historical prediction results; and select a target prediction algorithm from multiple prediction algorithms based on the scores of multiple prediction algorithms for one or more measurement indicators.

[0015] Optionally, the processing module is specifically used to: determine weights of multiple measurement indicators based on a configuration strategy; and select a target prediction algorithm from multiple prediction algorithms based on scores of the multiple prediction algorithms for one or more measurement indicators and the weights of the multiple measurement indicators.

[0016] Optionally, the measurement indicators include: the probability of the data being read by the data reader; and / or, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium; and / or, the computing power required to execute the operation indicated by the future prediction result.

[0017] Optionally, the processing module is further used to: adjust parameters of some or all of the multiple prediction algorithms based on the scores of the multiple prediction algorithms for one or more measurement indicators.

[0018] In a third aspect, the present application provides a computing device including a memory and a processor, wherein the memory stores program instructions, and the processor runs the program instructions to execute the method provided in the first aspect of the present application and any possible implementation thereof.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium, and the computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device executes the method provided in the first aspect of the present application and any possible implementation thereof.

[0020] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method provided in the first aspect of the present application and any possible implementation thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the use process of a data prediction algorithm provided in an embodiment of the present application;

[0022] Figure 2 It is a schematic diagram of an implementation environment involved in a data processing method provided in an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an implementation environment involved in another data processing method provided in an embodiment of the present application;

[0024] Figure 4 is a flow chart of a data processing method provided in an embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of a data processing method provided in an embodiment of the present application implemented by multiple functional units;

[0026] Figure 6 is a schematic diagram of a bitmap provided in an embodiment of the present application;

[0027] Figure 7 It is a flowchart of selecting a target prediction algorithm from multiple prediction algorithms based on indication information and historical prediction results provided by an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of a prediction algorithm based on hit rate, occupancy rate of a first storage medium and computing power consumption measurement provided by an embodiment of the present application;

[0029] Fig. 9 It is a flowchart of selecting a target prediction algorithm from multiple prediction algorithms based on scores of multiple prediction algorithms for one or more measurement indicators provided by an embodiment of the present application;

[0030] Fig.10 It is a schematic diagram of selecting a target prediction algorithm from multiple prediction algorithms through an AI algorithm provided in an embodiment of the present application;

[0031] Fig.11 is a schematic diagram of a data processing device provided in an embodiment of the present application;

[0032] Fig.12 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0034] To facilitate understanding, the technology and background involved in the embodiments of the present application are first explained below.

[0035] Typically, the process of a data reader reading data from a secondary memory is as follows: the data reader issues a read request, the primary memory first reads the data indicated by the read request from the secondary memory according to the read request, and then the data reader reads the data from the primary memory. Among them, the data access speed of the primary memory is greater than the data access speed of the secondary memory. However, the data reader's processing speed for data is usually faster than the data access speed, so the industry has proposed a data prefetch mechanism. Data prefetching means predicting the data that the data reader may read before the data reader actually reads the data, and then the primary memory reads the data from the secondary memory to the primary memory in advance based on the prediction result, so that the data reader can directly read the data from the primary memory when it needs to read the data. In this way, when the data reader performs a data reading operation, if the data it needs has been pre-fetched into the primary memory, the data reader can directly read the data from the primary memory without initiating a read operation to read the data from the secondary memory to the primary memory. In this way, the time required to read the data from the secondary memory to the primary memory can be hidden in the process of the data reader processing other data, effectively shortening the data reading time of the data reader and achieving the purpose of speeding up the data reading speed. Among them, the algorithm used to predict the need to pre-fetch into the primary memory is called a data pre-fetch algorithm.

[0036] Since the storage capacity of the memory is limited, when the capacity of the primary memory reaches a certain value, some of the data stored in the primary memory needs to be eliminated so that the capacity occupied by the eliminated data can be used to store more valuable data. In some implementations, an algorithm can also be used to predict which data needs to be eliminated, and the algorithm can be called a data elimination algorithm. The data elimination algorithm and the data prediction algorithm are combined and referred to as a data prediction algorithm. It should be noted that data elimination is not limited to data pre-fetching scenarios. For example, in the normal data reading process, the capacity of the primary memory may be relatively large, and the data stored in the primary memory also needs to be eliminated at this time.

[0037] Data pre-fetching and elimination technologies are widely used, covering scenarios such as data movement and sharing between processors, hardware, software, data centers, and cross-network nodes. For example, from the hardware media level, data can be pre-fetched and eliminated between caches at all levels of the central processing unit (CPU), and data can be pre-fetched and eliminated between the storage medium and its upper cache. From the operating system level, data can be pre-fetched and eliminated between pages managed by the operating system (OS) and memory. From the application level, metadata and data in databases and distributed file systems can be pre-fetched and eliminated in memory caches.

[0038] Since various scenarios have great demands for cache, in order to meet the cache demand, in addition to directly increasing the cache capacity, the data prediction algorithm needs to have better performance in the case of comprehensive consideration of performance and cost. However, there is no data prediction algorithm that can adapt to multiple application scenarios. This is because the current data prediction algorithm strongly couples the data prefetching system with the prefetching elimination algorithm, and it is impossible to run multiple algorithm instances at the same time during the system operation, perform performance statistics and evaluation at the same time, and it is impossible to dynamically open, close and select the data prediction algorithm to be used according to the results of the simultaneous operation of multiple algorithm instances. For example, in the current data prefetching system, only an instance of a certain data prefetching algorithm under a certain set of parameters is running at the same time. In addition, the prediction algorithm is currently selected either based on the ex ante mechanism or based on the ex post mechanism. However, the ex ante selection method is usually difficult to obtain the law of data flow in the actual scenario, resulting in greater uncertainty in the actual performance benefits of the prediction algorithm. The ex post selection method usually switches the prediction algorithm only when the actual hit rate of the prediction algorithm is poor. Since performance loss has already occurred at this time, and the actual performance benefits of the switched prediction algorithm also need to be evaluated after the actual execution of the prediction algorithm, the effect of the ex post selection method is also unsatisfactory.

[0039] For example, Figure 1 As shown in the upper left figure, prediction algorithm A is used in scenario X. Based on the prediction results of prediction algorithm A, data is pre-fetched from the main storage to the cache. The pre-fetched data includes all the data that the data reader needs to read. It can be seen that the prediction algorithm has achieved good performance. Figure 1 In scenario Y in the upper right figure, the prediction algorithm A is continued to be used for prefetching. Based on the prediction result of the prediction algorithm A, the cache prefetches data from the main storage to the cache. The prefetched data does not include all the data that the data reader needs to read. The data reader still needs to initiate a read operation from the main storage to the cache. It can be seen that the performance of the prediction algorithm in scenario Y is poor. Figure 1As shown in the lower right figure, the user considers that the performance of the prediction algorithm has deteriorated, so he configures it to stop using the prediction algorithm. When the scene is converted to scene Z in the lower left figure, since the prediction algorithm is not enabled, the data reader needs to initiate a read operation from the main storage to the cache every time it needs to read data, resulting in slower data reading. In addition, since the prediction algorithm is not enabled in scene Z, it is impossible to evaluate the performance of the prediction algorithm.

[0040] Based on this, an embodiment of the present application provides a data processing method. In the data processing method, the computing device can run multiple prediction algorithms on the one hand to obtain prediction results of multiple prediction algorithms for future time periods. On the other hand, it can select a target prediction algorithm from multiple prediction algorithms based on the prediction results of multiple prediction algorithms for historical moments and the data actually read by the data reader in the historical time period. Then, based on the prediction result of the target prediction algorithm for the future time period, the operation indicated by the future prediction result is performed. For example, when the prediction result indicates that data needs to be pre-fetched from the second storage medium to the first storage medium, based on the future prediction result of the target prediction algorithm, the data indicated by the future prediction result is pre-fetched from the second storage medium to the first storage medium, so that the data reader can read the data from the first storage medium in the future time period. When the prediction result indicates that data needs to be eliminated from the first storage medium, based on the future prediction result of the target prediction algorithm, the data indicated by the future prediction result is eliminated from the first storage medium. Wherein, the data access speed of the first storage medium is greater than the data access speed of the second storage medium. For example, the second storage medium is the primary storage and the first storage medium is the cache.

[0041] In this way, the data processing method is equivalent to running multiple prediction algorithms at the same time before executing the data pre-fetching operation, and selecting the prediction algorithm used for this pre-fetching according to the prediction results of the multiple prediction algorithms for the historical period and the actual situation of the data reader reading the data in the historical period, and after selecting the prediction algorithm, performing the data pre-fetching operation on the prediction results of the future period according to the selected prediction algorithm. Similarly, the data processing method is equivalent to running multiple prediction algorithms at the same time before executing the data elimination operation, and selecting the prediction algorithm used for this pre-fetching according to the prediction results of the multiple prediction algorithms for the historical period and the actual situation of the data reader reading the data in the historical period, and after selecting the prediction algorithm, performing the data elimination operation on the prediction results of the future period according to the selected prediction algorithm. Therefore, the data processing method can be considered to select the prediction algorithm in the event, and perform the data pre-fetching operation and / or data elimination operation according to the selected prediction algorithm. Compared with the mechanism of selecting the prediction algorithm in advance and the mechanism of selecting the prediction algorithm after the event, the data processing method can predict the data that needs to be pre-fetched and eliminated according to the data demand characteristics of the data reading scene, which helps to improve the accuracy of data pre-fetching and data elimination.

[0042] The following first describes an implementation environment involved in a data processing method provided in an embodiment of the present application.

[0043] Figure 2 Schematic diagram of an implementation environment involved in a data processing method provided in an embodiment of the present application. Figure 2 As shown, the implementation environment includes: a computing device 10. The computing device 10 can obtain indication information of data read by a data reader from the first storage medium and the second storage medium in a historical period, and historical prediction results of multiple prediction algorithms for the historical period, and obtain future prediction results of multiple prediction algorithms for future periods based on the indication information and the historical prediction results, respectively, and select a target prediction algorithm from multiple prediction algorithms, and then perform corresponding operations based on the future prediction results of the target prediction algorithm.

[0044] In one implementation, the data processing method provided in the embodiment of the present application can be implemented by running an executable program on the computing device 10. For example, the executable program of the data processing method is presented in the form of an application installation package. After the application installation package is installed in the computing device 10, the data processing method can be implemented by running the executable program. In this case, the computing device 10 can be a terminal. The terminal can be a computer, a personal computer, a portable mobile terminal, a multimedia player, an e-book reader, or a wearable device, etc.

[0045] Figure 3 FIG. 1 is a schematic diagram of an implementation environment involved in another data processing method provided in an embodiment of the present application. Figure 3 As shown, the implementation environment may include: a client 20 and a computing device 10. The client 20 can establish a communication connection with the computing device 10. For example, a communication connection may be established between the client 20 and the computing device 10 via a network. Optionally, the network may be a local area network, the Internet, or other networks, which are not limited in the embodiments of the present application.

[0046] The client 20 is used for the user to interact with the computing device 10. In one implementation, the client 20 is used to send instructions to the computing device 10 according to the user's instructions. For example, the client 20 is used to send a configuration instruction to the computing device 10 according to the user's instructions, and the configuration instruction is used to indicate the configuration requirements for the prediction algorithm. The computing device 10 is used to receive the instruction sent by the client 20 and to indicate related operations according to the instruction. For example, the computing device 10 is used to receive the configuration instruction and select the prediction algorithm in the process of executing the data processing method provided in the embodiment of the present application according to the configuration requirements indicated by the configuration instruction.

[0047] In one implementable manner, the client 20 may be a desktop computer, a laptop computer, a mobile phone, a smart phone, a tablet computer, a cloud host, a portable mobile terminal, a multimedia player, a smart home appliance, an artificial intelligence device, a smart wearable device, an e-reader, a smart vehicle-mounted device, or an Internet of Things device. The computing device 10 may be a server (such as a cloud server). Typically, the computing device 10 may be a server, or a server cluster consisting of several servers, or a cloud computing service center. Among them, a large number of basic resources owned by a cloud service provider are deployed in the cloud computing service center. For example, computing resources, storage resources, and network resources are deployed in the cloud computing service center. The cloud computing service center can utilize this large amount of basic resources to implement the data extraction method provided in the embodiment of the present application.

[0048] When the computing device 10 is implemented through a cloud computing service center. The user can access the cloud platform through the client 20 to send configuration instructions to the computing device 10 through the cloud platform. At this time, the functions implemented by the data processing method provided in the embodiment of the present application can be abstracted by the cloud service provider into a data processing cloud service on the cloud platform, and the cloud platform can use the resources in the cloud computing center to provide the data processing cloud service to users. After the user purchases the data processing cloud service on the cloud platform, he can perform data reading operations based on the data processing cloud service. In addition, the data processing cloud service can be provided as an independent cloud service or as an additional service to other cloud services. Optionally, the cloud platform can be a cloud platform of a central cloud, a cloud platform of an edge cloud, or a cloud platform including a central cloud and an edge cloud, which is not specifically limited in the embodiment of the present application. It should be noted that in Figure 3 In the implementation environment shown, the computing device 10 can also be implemented through other resource platforms besides the cloud platform, and the embodiments of the present application do not specifically limit it.

[0049] When the present application performs a data pre-fetch operation, it is necessary to pre-fetch the data recorded in the second storage medium into the first storage medium. When performing a data elimination operation again, it is necessary to delete the specified data stored in the first memory. The first storage medium and the second storage medium may be optionally both deployed in the computing device 10, or neither may be deployed in the computing device 10, or one of the first storage medium and the second storage medium may be deployed in the computing device 10 and the other may not be deployed in the computing device 10. For example, when the first storage medium and the second storage medium are both deployed in the computing device 10, the implementation scenario may be to pre-fetch data from an external memory into the memory, and to delete the data stored in the memory.

[0050] It should be understood that the above content is an illustrative description of the application scenarios of the data processing method provided in the embodiments of the present application, and does not constitute a limitation on the application scenarios of the data processing method. A person of ordinary skill in the art can know that as business needs change, the application scenarios can be adjusted according to application needs, and the embodiments of the present application do not list them one by one.

[0051] The implementation process of the data processing method provided in the embodiment of the present application is described below. Figure 4 As shown, the data processing method includes the following steps:

[0052] Step 401: Obtain indication information of all data read by a data reader from a first storage medium and a second storage medium in a historical period.

[0053] The data pre-fetched from the second storage medium to the first storage medium can be considered as the data that the data reader will read with a high probability. The data eliminated from the first storage medium can be considered as the data that the data reader will not read with a high probability. Taking into account that the reading behavior of the data reader is temporally correlated, when predicting the data that needs to be pre-fetched and eliminated, the relevant information of all the data read by the data reader from the first storage medium and the second storage medium in the historical period can be referred to. Each time the data is read by the data reader, there will be a record, and the computing device can obtain the relevant information of all the data read by the data reader from the first storage medium and the second storage medium in the historical period based on the record. By way of example, the data processing method provided in the embodiment of the present application can be implemented through the collaborative cooperation of multiple functional units. Figure 5 The following is a schematic diagram of a data processing method provided in an embodiment of the present application implemented by multiple functional units. Figure 5 As shown, the multiple functional units include: a reading processing unit, a preprocessing unit, an algorithm running unit, a storage unit, a statistical unit, and a decision unit. The reading processing unit can receive feedback from the data reader and record relevant information of the data read by the data reader based on the feedback. Therefore, the computing device can obtain relevant information of all data read by the data reader from the first storage medium and the second storage medium in the historical period according to the record of the reading processing unit.

[0054] The relevant information of the data may be indication information of the data. The indication information is used to indicate the data. In one implementation, the indication information includes one or more of the following: location information of the data, length of the data, and a timestamp when the data is read by the data reader. The location information of the data is used to indicate the location of the data in the storage medium. The timestamp when the data is read by the data reader indicates the time when the data is read from the storage medium by the data reader. For example, for pre-fetched data, the timestamp indicates the time when the data is read from the first memory by the data reader. Optionally, when the prediction algorithm can serve multiple data readers, the indication information also includes an identification (identity, ID) of the data reader to distinguish data read by different data readers.

[0055] Step 402: Based on the indication information and the historical prediction results of the multiple prediction algorithms for the historical time periods, the future prediction results of the multiple prediction algorithms for the future time periods are obtained respectively, and the prediction results are used to indicate one or more of the following: data that needs to be pre-fetched from the second storage medium to the first storage medium, and data that needs to be eliminated from the first storage medium.

[0056] After obtaining the indication information of all the data read by the data reader from the first storage medium and the second storage medium in the historical period, the data that the data reader may read in the future period can be predicted based on the indication information and the historical prediction results of multiple prediction algorithms for the historical period, and the future prediction results of multiple prediction algorithms for the future period can be obtained. In one implementation, the computing device analyzes the temporal correlation of the data read by the data reader according to the indication information, and obtains the deviation of the historical prediction results relative to the actual reading intention of the data reader according to the historical prediction results and the indication information, and then predicts the data that the data reader may read in the future time according to the correlation and the deviation, and obtains the future prediction results for the future period. It should be noted that the embodiment of the present application does not limit the prediction algorithm used in the prediction process.

[0057] The prediction result of the prediction algorithm indicates one or more of the following: pre-fetched data that needs to be pre-fetched from the second storage medium to the first storage medium, and obsolete data that needs to be deleted from the first storage medium. Since the capacity of the first storage medium is limited, during the process of pre-fetching data, the usage of the first storage medium may exceed the specified usage. At this time, part of the data stored in the first storage medium is deleted, and the storage space originally storing this part of the data can be used to store the pre-fetched data. In one implementation, the capacity waterline of the first storage medium can be determined according to the actual usage of the first storage medium, and then the capacity waterline is compared with the threshold waterline. When the capacity waterline reaches the threshold waterline, the prediction result of the prediction algorithm needs to indicate obsolete data. Among them, the capacity waterline is the ratio of the actual usage of the first storage medium to the actual usage of the second storage medium. The threshold waterline is the ratio of the total capacity of the first storage medium to the total capacity of the second storage medium. For example, assuming that the total capacity of the second storage medium is 256 Gigabytes (GB) and the total capacity of the first storage medium is 8GB, then the threshold waterline is 8GB / 256GB=3.125%.

[0058] In an embodiment of the present application, a computing device runs multiple prediction algorithm instances. Multiple prediction algorithm instances can be distinguished according to scenarios, algorithm parameters, data streams, etc. Prediction algorithm instances are distinguished according to scenarios, which means that when the same prediction algorithm is applied to different scenarios, different prediction algorithm instances can be obtained. Prediction algorithm instances are distinguished according to algorithm parameters, which means that when the parameter configuration of the same prediction algorithm is different, different prediction algorithm instances can be obtained. Prediction algorithm instances are distinguished according to data streams, which means that when the same prediction algorithm is applied to different data streams, different prediction algorithm instances can be obtained. Among them, different data streams indicate that the data belongs to different businesses. After the computing device obtains the indication information and historical prediction results, it can determine the prediction algorithm instance adapted by the data reader, and the adapted prediction algorithm instance can obtain the future prediction results of the adapted prediction algorithm instance based on the indication information and historical prediction results. For example, such as Figure 5 As shown, after obtaining the indication information and the historical prediction results, the preprocessing unit can determine multiple prediction algorithm instances adapted to the data reader, and the multiple adapted prediction algorithm instances can obtain the future prediction results of the adapted prediction algorithm instance based on the indication information and the historical prediction results. For example, when the indication information includes the ID of the data reader, the preprocessing unit can convert the data stream ID based on the ID of the data reader, and then determine multiple prediction algorithms that can be used for the data stream ID, and correspondingly determine the multiple prediction algorithm instances used to execute the prediction algorithm as the multiple prediction algorithm instances adapted to the data reader.

[0059] In one implementation, the location information and length of the data indicated by the indication information and the historical prediction information may be recorded via a bitmap. Figure 5 As shown, the bitmap can be maintained by the storage unit. The present application configures four physical bitmaps and two logical bitmaps for each prediction algorithm instance. Each bit of the bitmap is used to record the information of the data recorded in a data block in the second storage medium. For example, when the bit of the bitmap records the key of the data block, it means that the data block meets the conditions corresponding to the bitmap. Among them, before using the bitmap to record information, the size of the bitmap can be optionally instantiated according to the total capacity of the second storage medium. For example, assuming that each data block occupies 8 kilobytes (kilobyte, KB), the total capacity of the second storage medium is 256GB, and the second storage medium can store 32 megabytes (megabyte, MB) of data blocks, then the size of the bitmap is at least 32 megabits (megabit, Mb), that is, 4MB in size.

[0060] The four entity bitmaps are: read operation hit bitmap ioBitMap, repeated IO hit bitmap ioHitBitMap, prefetch bitmap prefetechBitMap and prefetch hit bitmap prefetchHitBitMap. ioBitMap is used to record the information of data that is not hit in the first storage medium, that is, the information of data that is not prefetched but read by the data reader. For example, when the data required by the data reader is not prefetched to the first memory, the data reader can initiate a read operation from the second storage medium to the first storage medium. At this time, ioBitMap is used to record the information of the data read by the read operation sent by the data reader. ioHitBitMap is used to record the information of the data hit in the first storage medium caused by repeated IO. For example, when the memory manages data according to the access heat of the data, if the access heat of the data is high, the data is retained in the memory to facilitate the next use of the data. At this time, ioHitBitMap is used to record the information of the data cached in the memory due to the access heat and read by the data reader. prefetechBitMap is used to record the information of the data prefetched according to the prediction result of the current prediction algorithm instance. prefetchHitBitMap is used to record information about data that is prefetched according to the prediction result of the current prediction algorithm instance and read by the data reader. In this application, hits refer to data being read by the data reader.

[0061] The logical bitmap does not occupy actual memory space and can be implemented by logically combining the physical locations. For example, the logical bitmap is obtained by performing a logical OR operation on the physical bitmap. The two logical bitmaps are: cache bitmap cacheBitMap and cache hit bitmap cacheHitBitMap. cacheBitMap is composed of the physical bitmaps ioBitMap and prefetechBitMap. It can be seen that cacheBitMap is used to indicate pre-fetched data and data read by the read operation sent by the data reader. cacheHitBitMap is composed of the physical bitmaps ioHitBitMap and prefetechHitBitMap. It can be seen that cacheHitBitMap is used to indicate data read by the data reader due to pre-fetching and data read by the data reader due to repeated IO. Figure 6 is a schematic diagram of a bitmap provided in an embodiment of the present application. Figure 6 It can be seen that cacheBitMap is obtained by performing a logical OR operation on the entity bitmap ioBitMap and prefetechBitMap. cacheHitBitMap is obtained by performing a logical OR operation on the entity bitmap ioHitBitMap and prefetechHitBitMap.

[0062] Step 403: Select a target prediction algorithm from a plurality of prediction algorithms based on the indication information and historical prediction results.

[0063] After the computing device obtains the indication information and the historical prediction results, it can optionally evaluate multiple pre-fetching algorithms based on the indication information and the historical prediction results, and select a target prediction algorithm from the multiple prediction algorithms according to the evaluation results, so as to use the future prediction results of the target prediction algorithm to perform the operation indicated by the future prediction results on the data. In one implementation, Figure 7 As shown, the implementation process of step 403 includes:

[0064] Step 4031: Based on the indication information and historical prediction results, obtain the scores of multiple prediction algorithms for one or more measurement indicators.

[0065] The measurement index is an index for measuring the performance of the prediction algorithm. A variety of measurement indexes can measure the performance of the prediction algorithm from multiple dimensions. In one implementation, the measurement index includes: the probability of data being read by the data reader; and / or, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium; and / or, the computing power required to execute the operation indicated by the future prediction result. When the prediction result indicates that data needs to be pre-fetched from the second storage medium to the first storage medium, the probability of the data being read by the data reader includes at least the probability that the data stored in the second storage medium is read by the data reader. Generally speaking, when the probability of the data is large, the data needs to be pre-fetched to the first storage medium. When the prediction result indicates that data needs to be eliminated from the first storage medium, the probability of the data being read by the data reader is actually the probability that the data stored in the first storage medium is read by the data reader without distinguishing how the data stored in the first storage medium is read to the first storage medium. Generally speaking, when the probability of the data is small, the data needs to be eliminated from the first storage medium. When the prediction result indicates that data needs to be pre-fetched from the second storage medium to the first storage medium, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium includes at least the ratio of the capacity of the data pre-fetched from the second storage medium to the capacity of the first storage medium. When the prediction result indicates that data needs to be eliminated from the first storage medium, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium is actually the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium without distinguishing how the data stored in the first storage medium is read to the first storage medium. When the prediction result indicates that data needs to be pre-fetched from the second storage medium to the first storage medium, the computing power required to perform the operation indicated by the future prediction result includes at least the computing power consumed to pre-fetch data from the second storage medium to the first storage medium. When the prediction result indicates that data needs to be eliminated from the first storage medium, the computing power required to perform the operation indicated by the future prediction result includes at least the computing power required to delete the data from the first storage medium.

[0066] The above three measurement indicators are used to measure the hit rate of the prediction algorithm, the occupancy rate of the first storage medium, and the computing power consumption respectively. Figure 8 The embodiment of the present application provides a method based on the hit rate and the first storage medium ( Figure 8 Schematic diagram of the prediction algorithm for measuring the occupancy rate and computing power consumption of the cache. Figure 5 As shown, obtaining indicators of the prediction algorithm can be performed by the statistical unit.

[0067] Optionally, the type of measurement indicator can also be adjusted according to application requirements, such as increasing or decreasing the type of measurement indicator according to application requirements. The calculation method of the measurement indicator can also be adjusted according to application requirements. For example, the indicator for measuring computing power consumption can be expanded to: the ratio of the computing power consumed by pre-fetching data from the second storage medium to the first storage medium to the computing power consumed by the data reader sending a read operation to read the data. For example, assuming that ten data in the second storage medium can be stored in the first storage medium each time through data pre-fetching, and the data reader can read one data to the first storage medium each time it sends a read operation, the pre-fetching method and the computing power consumed by reading the data through the read operation are both X, and the computing power consumed by running the prediction algorithm is a, then the computing power consumed by pre-fetching one data is a+X / 10, and the computing power consumed by the read operation to read one data is X, then the computing power consumed by pre-fetching one data from the second storage medium to the first storage medium, and the ratio of the computing power consumed by the data reader sending a read operation to read one data is (a+X / 10) / X.

[0068] It should be understood that the data referenced for calculating each measurement indicator can also be adjusted according to application requirements. For example, each measurement indicator can be optionally determined based on the data reading situation within a specified time period before the current moment, or based on the data reading situation between the moment when the prediction algorithm starts to execute and the current moment, or based on a specified moment before the current moment, etc., which is not specifically limited in the embodiments of the present application.

[0069] In one implementation, the embodiment of the present application configures a variety of counters for each prediction algorithm to count the measurement indicators of the prediction algorithm. As shown in Table 1, the purpose and calculation method of various counters are as follows:

[0070] Table 1

[0071]

[0072]

[0073] When the aforementioned physical bitmap and logical bitmap are used to record the relevant information of the prediction algorithm, the counter in Table 1 can count according to the data recorded in the bitmap. The following uses the following cases as examples to illustrate the implementation method.

[0074] Statistics Case 1: The key of the current IO data is not in the prefetchBitMap, which means that the data meets the corresponding conditions of ioBitMap, so the corresponding bit of ioBitMap is inserted, and the corresponding bit of ioBitMap records the key of the data.

[0075] Statistics Case 2: If the key of the current IO data is in ioBitMap but not in ioHitBitMap, insert the corresponding bit in ioHitBitMap and increase the ioHit count.

[0076] Statistics Case 3: If the key of the current IO data is in ioBitMap and ioHitBitMap, only the ioHit count is increased.

[0077] Statistics Case 4: If the key of the current IO data is in the prefetchBitMap but not in the prefetchHitBitMap, insert the corresponding bit in the prefetchHitBitMap and increase the prefetchHit count.

[0078] Statistics Case 5: If the key of the current IO data is in cacheBitMap and in prefetchBitMap and prefetchHitBitMap, only the prefetchHit count is increased.

[0079] Prefetch Case 1: If the key of the data to be prefetched is already in ioBitMap or prefetchBitMap, no updating or counting is required.

[0080] Prefetch Case 2: If the key of the currently prefetched data is not in ioBitMap and prefetchBitMap, the corresponding bit of prefetchBitMap is inserted, and the non-piggybacked computing power count is increased by one; if the key of the "to be" prefetched data predicted by the prediction algorithm is not in the same piggybacked range (such as the same page) as the key actually read by the current IO, the consumed computing power count is increased by one.

[0081] Step 4032: Select a target prediction algorithm from among the multiple prediction algorithms based on the scores of the multiple prediction algorithms for one or more measurement indicators.

[0082] In one implementation, Fig. 9 As shown, the implementation process of step 4032 includes:

[0083] Step 4032a: Determine the weights of multiple measurement indicators based on the configuration strategy.

[0084] The configuration strategy can be a strategy configured by the user for the prediction algorithm, or a default strategy configured by the administrator of the computing device, so as to reflect different requirements such as performance priority, cost priority, and comprehensive consideration of performance and cost through the configuration strategy. Optionally, the configuration strategy includes one or more of the following combinations: giving priority to ensuring the probability that the data pre-fetched from the second storage medium to the first storage medium is read by the data reader; giving priority to reducing the ratio of the capacity of the data pre-fetched from the second storage medium to the capacity of the first storage medium; and giving priority to saving the computing power required to execute the operation indicated by the future prediction result. For example, giving priority to saving the computing power consumed by pre-fetching data from the second storage medium to the first storage medium, and giving priority to saving the computing power consumed by deleting data from the first storage medium. The three correspond to hit rate priority, cache saving priority, and computing power saving priority, respectively. Further, the configuration strategy can also be a strategy such as performance priority and a weighted combination of various corresponding strategies. After determining the configuration strategy, the degree of influence of different indicators on the prediction result can be obtained, and the degree of influence is expressed as the weight of multiple measurement indicators. For example, when the configuration strategy indicates that the hit rate is prioritized first and cache saving is secondly guaranteed, the weights of the hit rate, cache occupancy, and computing power consumption are 0.5, 0.3, and 0.2, respectively.

[0085] Step 4032b: Select a target prediction algorithm from the multiple prediction algorithms based on the scores of the multiple prediction algorithms for one or more measurement indicators and the weights of the multiple measurement indicators.

[0086] After the computing device obtains the scores and weights of the multiple prediction algorithms for one or more measurement indicators, it obtains the scores of the multiple prediction algorithms based on the scores and weights, and selects from the multiple prediction algorithms according to the scores of the multiple prediction algorithms to obtain the target prediction algorithm. For example, when it is necessary to select a target prediction algorithm from multiple prediction algorithms, it is possible to obtain the weighted sum of the prediction algorithms according to the scores and weights of the measurement indicators of any one of the prediction algorithms, and then determine the prediction algorithm with the largest weighted sum among the multiple prediction algorithms as the target prediction algorithm. Figure 5 As shown, both step 4031a and step 40312b can be performed by the decision unit. Figure 8 As shown, the decision unit may include a first decision subunit and multiple second decision subunits, the multiple second decision subunits respectively correspond to multiple prediction algorithm instances, the second decision subunit corresponding to any prediction algorithm instance is used to make a decision for the prediction algorithm instance, and the first decision subunit is used to select a target prediction algorithm instance from the multiple prediction algorithm instances. For example, any prediction algorithm instance can generate multiple sets of outputs for inputs based on its multiple sets of algorithm parameters, and the second decision subunit corresponding to the prediction algorithm instance is used to select a set of algorithm parameters with the best performance from the multiple sets of algorithm parameters based on the multiple sets of outputs.

[0087] Alternatively, if Figure 8 As shown, the computing device may also optionally adjust the parameters of some or all of the multiple prediction algorithms based on the scores of the multiple prediction algorithms for one or more measurement indicators. For example, when the scores of some prediction algorithms among the multiple prediction algorithms are low, indicating that the performance of the prediction algorithm is poor, the parameters of the prediction algorithm are adjusted to improve the performance of the prediction algorithm.

[0088] Furthermore, the computing device may also optionally determine whether to execute the prediction algorithm based on the scores of the multiple prediction algorithms. For example, when the weighted sum of the multiple prediction algorithms is less than the execution score threshold, it can be obtained that executing the data prefetch operation based on the prediction results of the multiple prediction algorithms cannot obtain good prediction performance, and the computing device determines that there is no need to execute the data prefetch operation.

[0089] In one implementation, step 403 may be implemented by an artificial intelligence (AI) algorithm. For example, the decision unit may configure the AI ​​algorithm and use the AI ​​algorithm to decide whether to pre-fetch or eliminate, and which prediction algorithm to choose. Fig.10 4 is a schematic diagram of a step 403 implemented by an AI algorithm provided in an embodiment of the present application. Fig.10 Compared to Figure 8 Added decision learning unit, and the Fig.10 The decision learning unit, the first decision subunit and the second decision subunit in the decision learning unit are all implemented by AI algorithms. The input of the decision learning unit is connected to the output ends of multiple prediction algorithm instances respectively, and is used to learn based on the outputs of multiple prediction algorithm instances to provide guidance information for the decision of the second decision subunit. Moreover, compared with Figure 8 The second decision subunit shown, the second decision subunit corresponding to any prediction algorithm instance is also used to make a decision on adjusting parameters for the corresponding prediction algorithm instance. When the AI ​​algorithm is used to implement step 403, since the AI ​​algorithm can continuously learn the operation results and corresponding measurement indicators of each algorithm instance, under the same configuration strategy, a better algorithm and parameter selection result can be obtained.

[0090] Step 404: Based on the future prediction result of the target prediction algorithm, perform the operation indicated by the future prediction result.

[0091] The data recorded in the second storage medium is pre-fetched into the first storage medium, so that a data reader can read the data from the first storage medium in a future period.

[0092] After determining the target prediction algorithm among multiple prediction algorithms, the computing device can obtain the future prediction result of the target prediction algorithm, and perform the operation indicated by the future prediction result according to the future prediction result. For example, when the prediction result indicates that data needs to be pre-fetched from the second storage medium to the first storage medium, based on the future prediction result of the target prediction algorithm, the data indicated by the future prediction result is pre-fetched from the second storage medium to the first storage medium, so that the data reader can read the data from the first storage medium in the future period. When the prediction result indicates that data needs to be eliminated from the first storage medium, based on the future prediction result of the target prediction algorithm, the data indicated by the future prediction result is eliminated from the first storage medium.

[0093] In the data processing method provided in the embodiment of the present application, it is equivalent to executing multiple prediction algorithms according to multiple inputs at the same time in the process of obtaining prediction results according to inputs each time, and selecting the target prediction algorithm used for this pre-fetching according to the performance of the multiple prediction algorithms in the historical period, and then performing corresponding operations according to the future prediction results of the target prediction algorithm. Therefore, the data processing method can be considered to select the target prediction algorithm in the event and perform operations according to the target prediction algorithm. Compared with the mechanism of selecting the prediction algorithm in advance and the mechanism of selecting the prediction algorithm after the event, the data processing method can predict the data that needs to be pre-fetched and eliminated according to the data demand characteristics of the data reading scenario, which helps to improve the accuracy of data pre-fetching and elimination. In addition, the data processing method enables the data reading system to be decoupled from the prediction algorithm instance, which helps to realize the flexible configuration and dynamic online operation of the prediction algorithm and parameters.

[0094] It should be noted that the idea of ​​running multiple algorithms simultaneously and making decisions based on multiple indicators in real time in this application is not only applicable to prediction algorithms, but also to algorithms in other fields. For example, according to the classification of data flow, for the pre-fetching algorithm and the elimination algorithm, the data flow applicable to this application is shown in Table 2.

[0095] Table 2

[0096]

[0097]

[0098] For another example, classified according to application scenarios, for the pre-fetching algorithm and the elimination algorithm, the scalable application scenarios of the present application are shown in Table 3.

[0099] Table 3

[0100]

[0101] In summary, the data processing method is equivalent to running multiple prediction algorithms at the same time before executing the data pre-fetching operation, and selecting the prediction algorithm used for this pre-fetching according to the prediction results of the multiple prediction algorithms for the historical period and the actual situation of the data reader reading the data in the historical period, and after selecting the prediction algorithm, performing the data pre-fetching operation on the prediction results of the future period according to the selected prediction algorithm. Similarly, the data processing method is equivalent to running multiple prediction algorithms at the same time before executing the data elimination operation, and selecting the prediction algorithm used for this pre-fetching according to the prediction results of the multiple prediction algorithms for the historical period and the actual situation of the data reader reading the data in the historical period, and after selecting the prediction algorithm, performing the data elimination operation on the prediction results of the future period according to the selected prediction algorithm. Therefore, the data processing method can be considered to select the prediction algorithm in the event, and perform the data pre-fetching operation and / or data elimination operation according to the selected prediction algorithm. Compared with the mechanism of selecting the prediction algorithm in advance and the mechanism of selecting the prediction algorithm after the event, the data processing method can predict the data that needs to be pre-fetched and eliminated according to the data demand characteristics of the data reading scene, which helps to improve the accuracy of data pre-fetching and data elimination.

[0102] It should be noted that the order of the steps of the data processing method provided in the embodiment of the present application can be adjusted appropriately, and the steps can be increased or decreased accordingly according to the situation. Any technician familiar with the technical field can easily think of a method of change within the technical scope disclosed in this application, which should be included in the protection scope of this application, so it will not be repeated.

[0103] The above describes the data processing method of the embodiment of the present application. Corresponding to the above method, the embodiment of the present application also provides a data processing device. The structural diagram of the data processing device is shown in FIG. Fig.11 It should be understood that the data processing device may include more additional modules than the structure shown or omit some of the modules shown therein, and the present application embodiment does not limit this. Fig.11 As shown, the data processing device 110 includes:

[0104] The acquisition module 1101 is used to acquire indication information of all data read by a data reader from a first storage medium and a second storage medium in a historical period.

[0105] Processing module 1102 is used to obtain future prediction results of multiple prediction algorithms for future time periods based on the indication information and the historical prediction results of multiple prediction algorithms for historical time periods, and the prediction results indicate one or more of the following: data that needs to be pre-fetched from the second storage medium to the first storage medium, and data that needs to be eliminated from the first storage medium.

[0106] The processing module 1102 is further configured to select a target prediction algorithm from a plurality of prediction algorithms based on the indication information and the historical prediction results.

[0107] The reading module 1103 is used to perform the operation indicated by the future prediction result based on the future prediction result of the target prediction algorithm.

[0108] Optionally, the indication information includes one or more of the following: location information of the data, length of the data, and a timestamp when the data is read by the data reader.

[0109] Optionally, the processing module 1102 is specifically used to: obtain scores of multiple prediction algorithms for one or more measurement indicators based on the indication information and historical prediction results; and select a target prediction algorithm from multiple prediction algorithms based on the scores of multiple prediction algorithms for one or more measurement indicators.

[0110] Optionally, the processing module 1102 is specifically used to: determine weights of multiple measurement indicators based on the configuration strategy; and select a target prediction algorithm from multiple prediction algorithms based on scores of the multiple prediction algorithms for one or more measurement indicators and the weights of the multiple measurement indicators.

[0111] Optionally, the measurement indicators include: the probability of the data being read by the data reader; and / or, the ratio of the capacity of the data stored in the first storage medium to the total capacity of the first storage medium; and / or, the computing power required to execute the operation indicated by the future prediction result.

[0112] Optionally, the processing module 1102 is further configured to adjust parameters of some or all of the multiple prediction algorithms based on the scores of the multiple prediction algorithms for one or more measurement indicators.

[0113] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the modules described above can refer to the corresponding content in the aforementioned method embodiment, and will not be repeated here.

[0114] In summary, the data processing device is equivalent to running multiple prediction algorithms at the same time before executing the data pre-fetching operation, and selecting the prediction algorithm used for this pre-fetching according to the prediction results of the multiple prediction algorithms for the historical period and the actual situation of the data reader reading the data in the historical period, and after selecting the prediction algorithm, performing the data pre-fetching operation on the prediction results of the future period according to the selected prediction algorithm. Similarly, the data processing device is equivalent to running multiple prediction algorithms at the same time before executing the data elimination operation, and selecting the prediction algorithm used for this pre-fetching according to the prediction results of the multiple prediction algorithms for the historical period and the actual situation of the data reader reading the data in the historical period, and after selecting the prediction algorithm, performing the data elimination operation on the prediction results of the future period according to the selected prediction algorithm. Therefore, the data processing device can be considered to select the prediction algorithm in the event, and perform the data pre-fetching operation and / or data elimination operation according to the selected prediction algorithm. Compared with the mechanism of selecting the prediction algorithm in advance and the mechanism of selecting the prediction algorithm after the event, the data processing device can predict the data that needs to be pre-fetched and eliminated according to the data demand characteristics of the data reading scene, which helps to improve the accuracy of data pre-fetching and data elimination.

[0115] The embodiment of the present application provides a computing device, which is used to implement part or all of the functions of the data processing method provided in the embodiment of the present application. Fig.12 Schematic diagram of a computing device provided in an embodiment of the present application. Fig.12 As shown, the computing device 1200 includes a processor 1201, a memory 1202, a communication interface 1203 and a bus 1204. The processor 1201, the memory 1202 and the communication interface 1203 are connected to each other through the bus 1204.

[0116] Processor 1201 may include a general processor and / or a dedicated hardware chip. A general processor may include: a central processing unit (CPU), a microprocessor or a graphics processing unit (GPU). The CPU is, for example, a single-core processor (single-CPU), or a multi-core processor (multi-CPU). A dedicated hardware chip is a hardware module for high-performance processing. A dedicated hardware chip includes at least one of a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or a network processor (NP). Processor 1201 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, some or all of the functions of the data processing method of the present application may be completed by the hardware integrated logic circuit in the processor 1201 or instructions in the form of software.

[0117] The memory 1202 is used to store computer programs, and the computer programs include an operating system 1202a and executable codes (i.e., program instructions) 1202b. The memory 1202 is, for example, a read-only memory or other types of static storage devices that can store static information and instructions, or a random access memory or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory, a read-only optical disc or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a blue-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired executable code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. For example, the memory 1202 is used to store an outbound port queue, etc. The memory 1202 is, for example, independent and connected to the processor 1201 via a bus 1204. Or the memory 1202 and the processor 1201 are integrated together. The memory 1202 can store executable code. When the executable code stored in the memory 1202 is executed by the processor 1201, the processor 1201 is used to perform part or all of the functions of the data processing method provided in the embodiment of the present application. For the implementation of the processor 1201 to perform the process, please refer to the relevant description in the above embodiment. The memory 1202 may also include software modules and data required for other running processes such as the operating system.

[0118] The communication interface 1203 uses a transceiver module such as, but not limited to, a transceiver to achieve communication with other devices or communication networks. For example, the communication interface 1203 can be any one or any combination of the following devices: a network interface (such as an Ethernet interface), a wireless network card, and other devices with network access functions.

[0119] The bus 1204 is any type of communication bus for interconnecting the internal devices of the computing device (e.g., the memory 1202, the processor 1201, and the communication interface 1203). For example, a system bus. The embodiment of the present application takes the interconnection of the above-mentioned devices inside the computing device through the bus 1204 as an example. Optionally, the above-mentioned devices inside the computing device 1200 can also be connected to each other in communication with each other using other connection methods other than the bus 1204. For example, the above-mentioned devices inside the computing device 1200 are interconnected through an internal logical interface.

[0120] It should be noted that the above-mentioned multiple devices can be respectively arranged on independent chips, or at least partially or completely arranged on the same chip. Whether to independently arrange each device on different chips or to integrate and arrange it on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above-mentioned devices. The descriptions of the processes corresponding to the above-mentioned figures have different focuses. For the parts not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0121] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product providing a program development platform includes one or more computer instructions, and when these computer program instructions are loaded and executed on a computing device, all or part of the functions of the data processing method provided in the embodiments of the present application are implemented.

[0122] Furthermore, computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide a program development platform.

[0123] An embodiment of the present application also provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium. The computer-readable storage medium includes program instructions. When the program instructions are executed on a computing device, the computing device implements a data processing method as provided in the embodiment of the present application.

[0124] The embodiment of the present application also provides a computer program product including instructions. When the computer program product is executed on a computer, the computer implements the data processing method provided by the embodiment of the present application.

[0125] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0126] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the image information and executable code involved in this application are all obtained with full authorization.

[0127] In the embodiments of the present application, the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "at least one" means one or more, and the term "plurality" means a plurality, unless otherwise expressly defined.

[0128] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0129] It should be understood that the terms used in the description of the various examples herein are only for describing specific examples and are not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0130] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0131] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the concept and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire indication information of all data read by the data reader from the first storage medium and the second storage medium in a historical period; Based on the indication information and the historical prediction results of the multiple prediction algorithms for the historical period, respectively obtaining future prediction results of the multiple prediction algorithms for the future period, the prediction results indicating one or more of the following: data that needs to be pre-fetched from the second storage medium to the first storage medium, and data that needs to be eliminated from the first storage medium; Selecting a target prediction algorithm from among the multiple prediction algorithms based on the indication information and the historical prediction results; Based on the future prediction result of the target prediction algorithm, the operation indicated by the future prediction result is performed.

2. The method according to claim 1, characterized in that The indication information includes one or more of the following: location information of the data, length of the data, and a timestamp when the data is read by the data reader.

3. The method according to claim 1, characterized in that The selecting a target prediction algorithm from the multiple prediction algorithms based on the indication information and the historical prediction results includes: Based on the indication information and the historical prediction results, obtaining scores of the multiple prediction algorithms for one or more measurement indicators; A target prediction algorithm is selected from the multiple prediction algorithms based on the scores of the multiple prediction algorithms for the one or more measurement indicators.

4. The method according to claim 3, characterized in that The selecting a target prediction algorithm from the multiple prediction algorithms based on the scores of the multiple prediction algorithms for the one or more measurement indicators includes: Determining weights of the multiple metrics based on the configuration policy; A target prediction algorithm is selected from the multiple prediction algorithms based on the scores of the multiple prediction algorithms for the one or more measurement indicators and the weights of the multiple measurement indicators.

5. The method according to claim 3, characterized in that The metrics include: The probability that the data is read by the data reader; and / or, a ratio of the capacity of data stored in the first storage medium to the total capacity of the first storage medium; and / or, the computing power required to execute the operation indicated by the future prediction result.

6. The method according to claim 1, characterized in that The method further comprises: Based on the scores of the multiple prediction algorithms for one or more measurement indicators, the parameters of some or all of the multiple prediction algorithms are adjusted.

7. A data processing device, characterized in that: The device comprises: An acquisition module, used to acquire indication information of all data read by a data reader from the first storage medium and the second storage medium in a historical period; a processing module, configured to obtain future prediction results of the multiple prediction algorithms for future time periods based on the indication information and the historical prediction results of the multiple prediction algorithms for the historical time periods, respectively, wherein the prediction results indicate one or more of the following: data that needs to be pre-fetched from the second storage medium to the first storage medium, and data that needs to be eliminated from the first storage medium; The processing module is further used to select a target prediction algorithm from the multiple prediction algorithms based on the indication information and the historical prediction results; The reading module is used to execute the operation indicated by the future prediction result based on the future prediction result of the target prediction algorithm.

8. The device according to claim 7, characterized in that The indication information includes one or more of the following: location information of the data, length of the data, and a timestamp when the data is read by the data reader.

9. The device according to claim 7, characterized in that The processing module is specifically used for: Based on the indication information and the historical prediction results, obtaining scores of the multiple prediction algorithms for one or more measurement indicators; A target prediction algorithm is selected from the multiple prediction algorithms based on the scores of the multiple prediction algorithms for the one or more measurement indicators.

10. The device according to claim 9, characterized in that The processing module is specifically used for: Determining weights of the multiple metrics based on the configuration policy; A target prediction algorithm is selected from the multiple prediction algorithms based on the scores of the multiple prediction algorithms for the one or more measurement indicators and the weights of the multiple measurement indicators.

11. The device according to claim 9, characterized in that The metrics include: The probability that the data is read by the data reader; and / or, a ratio of the capacity of data stored in the first storage medium to the total capacity of the first storage medium; and / or, the computing power required to execute the operation indicated by the future prediction result.

12. The device according to claim 7, characterized in that The processing module is further used to adjust parameters of some or all of the multiple prediction algorithms based on the scores of the multiple prediction algorithms for one or more measurement indicators.

13. A computing device, characterized in that: The computing device comprises a processor and a memory, wherein the processor of the computing device is used to execute instructions stored in the memory, so that the computing device executes the method according to any one of claims 1 to 6.

14. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device, the computing device is caused to perform the method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the method according to any one of claims 1 to 6.

Images