Data scheduling method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data scheduling method, electronic equipment and a storage medium. The data scheduling method comprises the following steps: acquiring file access information of target equipment to a storage system; according to the file access information, predicting a probability that the target device accesses a target file in the storage system in a subsequent time slice, and obtaining a first predicted access probability of the target file; according to the first predicted access probability of the target file, determining respective second predicted access probabilities of a plurality of data blocks in the target file; obtaining a physical topology sequence of the target device to each storage node in the storage system; determining a target storage node of the data block according to the second predicted access probability of the data block and the physical topological sequence of each storage node; and caching the data block from the target storage node to the corresponding target equipment so as to improve the pre-caching hit rate of the file in the storage and calculation separation scene and improve the access efficiency of the file in the storage and calculation separation scene.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data scheduling method, electronic equipment and storage medium. Background Art

[0002] At present, the separation of data storage and data computing is the mainstream architecture mode of big data. Among them, the separation of data storage and data computing can be referred to as storage and computing separation. While improving the utilization rate of server resources and reducing costs, storage and computing separation has brought new technical challenges to data-intensive applications.

[0003] However, with the demand for storage and computing separation of massive data, such as the storage and computing separation of massive files, it is easy for the pre-cache hit rate of files to be low in the storage and computing separation scenario, resulting in low file access efficiency in the storage and computing separation scenario. Summary of the invention

[0004] The main purpose of the present invention is to provide a file caching method and electronic device, aiming to improve the pre-cache hit rate of files in a storage and computing separation scenario, and to improve the access efficiency of files in a storage and computing separation scenario.

[0005] In a first aspect, an embodiment of the present invention provides a data scheduling method, the data scheduling method comprising:

[0006] Obtain the target device's file access information to the storage system;

[0007] Predicting, based on the file access information, a probability that the target device will access a target file in the storage system in a subsequent time slice, and obtaining a first predicted access probability of the target file;

[0008] Determining, according to the first predicted access probability of the target file, a second predicted access probability of each of the plurality of data blocks in the target file;

[0009] Obtaining a physical topological order of the target device for each storage node in the storage system;

[0010] Determining a target storage node for the data block according to the second predicted access probability of the data block and the physical topological order of each of the storage nodes;

[0011] The data block is cached from the target storage node to the corresponding target device.

[0012] In a second aspect, an embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor;

[0013] The memory is used to store computer programs;

[0014] The processor is used to execute the computer program and implement any one of the data scheduling methods provided in the specification of the present invention when executing the computer program.

[0015] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed by a processor, and the processor implements any one of the data scheduling methods provided in the specification of the present invention.

[0016] An embodiment of the present invention provides a data scheduling method, an electronic device and a storage medium. The embodiment of the present invention obtains file access information of a target device to a storage system; predicts the probability of the target device accessing a target file in the storage system in a subsequent time slice based on the file access information, and obtains a first predicted access probability of the target file; determines the second predicted access probability of each of multiple data blocks in the target file based on the first predicted access probability of the target file; obtains the physical topological order of each storage node in the storage system from the target device; determines the target storage node of the data block based on the second predicted access probability of the data block and the physical topological order of each storage node; and caches the data block from the target storage node to the corresponding target device, which is beneficial to improving the pre-cache hit rate of files in a storage-computing separation scenario and improving the access efficiency of files in a storage-computing separation scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0018] Figure 1 A flow chart of a data scheduling method provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of file access information involved in an embodiment of the present invention;

[0020] Figure 3 A schematic block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0023] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0024] The embodiment of the present invention provides a data scheduling method, an electronic device and a storage medium. The communication method can be applied to an electronic device.

[0025] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0026] See also Figure 1 , Figure 1 A flow chart of a data scheduling method provided for an embodiment of the present invention. Exemplarily, the data scheduling method is applied to an electronic device. The electronic device may be a mobile phone, a tablet computer, a laptop computer, a desktop computer, and the like. The electronic device may also be a server, and the server may be a separate server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The server may be a separate server or a server cluster, which is not limited here. Electronic devices are collectively referred to as computing systems hereinafter.

[0027] like Figure 1 As shown, the data scheduling method includes steps S101 to S106.

[0028] Step S101: Obtain file access information of a target device to a storage system.

[0029] Exemplarily, the computing system may abstract the target device's file access process to the storage system as an uninterrupted continuous time window file access with the merged access within a single time window as the minimum unit. For example, the time may be divided into a preceding time slice, a current time slice, and a succeeding time slice. Each time slice may maintain and update one or more time windows. Of course, this is not limited to this and is not limited here.

[0030] In some implementations, the file access information of the target device to the storage system may include at least one of the file access information of the target device to the storage system in a previous time slice and the file access information of the target device to the storage system in a current time slice.

[0031] For example, taking the storage system as a distributed storage system, the computing system can obtain file access information of the target device to the distributed storage system, such as obtaining files in the distributed storage system accessed by the target device in the previous time slice, files in the distributed storage system accessed by the target device in the current time slice, etc., without limitation here.

[0032] In some embodiments, the file access information of the target device to the storage system obtained can be used as a basis for predicting the probability of the target device accessing the corresponding file in the storage system in a subsequent time slice, so as to schedule data corresponding to the corresponding file in the storage system based on the corresponding predicted access probability, so that the target device can access the corresponding file in the storage system in a subsequent time slice, thereby facilitating improving the pre-cache hit rate of files in the storage and computing separation scenario, and improving the access efficiency to files in the storage and computing separation scenario.

[0033] Step S102: predicting the probability of a target device accessing a target file in a storage system in a subsequent time slice based on the file access information, and obtaining a first predicted access probability of the target file.

[0034] For example, for each preset file stored in the storage system, such as the first preset file and the second preset file, if it is determined based on the file access information that the target device has accessed the first preset file in the previous time slice, and the target device has not accessed the second preset file in the previous time slice, then the computing system can predict that the probability of the target device accessing the first preset file in the subsequent time slice is higher than the probability of the target device accessing the second preset file in the subsequent time slice. Of course, this is not limited to this. For example, if it is determined based on the file access information that the target device has accessed the first preset file in the current time slice, and the target device has not accessed the second preset file in the current time slice, then the computing system can predict that the probability of the target device accessing the first preset file in the subsequent time slice is higher than the probability of the target device accessing the second preset file in the subsequent time slice, and this is not limited here. Based on this, the computing system can predict the probability of the target device accessing the target file in the storage system in the subsequent time slice based on the file access information, and obtain the first predicted access probability of the target file.

[0035] In some implementations, the file access information may include file access information in the current time slice and file access information of a plurality of consecutive previous time slices before the current time slice. For example, the file access information may be input into a file access prediction model, and the file access model may predict the probability of the target device accessing the target file in the storage system in the subsequent time slice based on the file access information, and obtain a first predicted access probability of the target file. Of course, this is not limited to this, and is not limited here.

[0036] In this way, by predicting the probability of the target device accessing the target file in the storage system in a subsequent time slice based on the file access information, a first predicted access probability of the target file is obtained, which can be used as a basis for data scheduling of the data corresponding to the target file. For example, the probability of the target device accessing a target file with a higher first predicted access probability in the storage system in a subsequent time slice is greater than the probability of the target device accessing a target file with a lower first predicted access probability in the storage system in a subsequent time slice. In this way, the data corresponding to the target file with a higher first predicted access probability in the storage system can be preferentially scheduled, such as pre-cached, so that the target device can access the target file in the storage system in a subsequent time slice, which is beneficial to improve the pre-cache hit rate of files in the storage and computing separation scenario.

[0037] In some embodiments, a target file is determined from a preset file of a storage system based on file access information; based on a file access prediction model, a probability of a target device accessing the target file in a subsequent time slice is predicted based on access feature information corresponding to the target file to obtain a first predicted access probability of the target file; the access feature information includes at least one of the file attributes of the target file, the access time of the target file in a previous time slice, the creation time of the target file, and the access frequency of the target file in the previous time slice.

[0038] Exemplarily, the separation of storage and computing can be applied to distributed scenarios such as big data and cloud computing. In distributed scenarios such as big data and cloud computing, due to the large number of files, if the probability of each file in distributed scenarios such as big data and cloud computing being accessed by the target device in the subsequent time slice is predicted, the computing system needs to consume a lot of computing resources and prediction time. Based on this, in order to save computing resources and prediction time, the computing system can determine the target file from the preset files of the storage system based on the file access information. For example, the preset files accessed by the target device in the previous time slice are filtered out from the preset files of the storage system as the target files, and the probability of the target device accessing the target file in the subsequent time slice is predicted. For another example, the preset files that the target device did not access in the previous time slice are removed from the preset files of the storage system. Of course, it is not limited to this and is not limited here.

[0039] In some embodiments, when the target file is determined from the preset file of the storage system, the computing system can predict the probability of the target device accessing the target file in the subsequent time slice through the file access prediction model to obtain the first preset access probability of the target file. For example, the computing system can determine the access feature information corresponding to the target file based on the file access information corresponding to the target file. For example, based on the file access information corresponding to the target file, the file attribute of the target file, the creation time of the target file, the access time of the target file, the access frequency of the target file, etc. can be determined. Based on this, the access feature information corresponding to the target file can be combined with the file access information corresponding to the target file to determine the access feature information corresponding to the target file, such as at least one of the file attribute of the target file, the access time of the target file in the previous time slice, the creation time of the target file, and the access frequency of the target file in the previous time slice. Of course, it is not limited to this. For example, the access feature information corresponding to the target file can be determined in combination with the file access information of the target file in the most recent multiple time slices, which is not limited here.

[0040] For example, when the access feature information corresponding to the target file is determined, the computing system can input the access feature information corresponding to the target file into the file access prediction model, and the file access prediction model can predict the probability of the target device accessing the target file in the subsequent time slice according to at least one of the file attribute of the target file, the access time of the target file in the previous time slice, the creation time of the target file, and the access frequency of the target file in the previous time slice, and output the first predicted access probability of the target file. For example, the file access prediction model can include an XGBoost machine learning model, but is certainly not limited to this and is not limited here.

[0041] In some embodiments, the access feature information may be pre-set, may be set by the user, or may be dynamically updated. For example, when predicting the probability of a target device accessing a target file in a subsequent time slice based on the access feature information corresponding to the target file based on a file access prediction model, the access time interval of the target file in the previous time slice may also affect the probability of the target device accessing the target file in the subsequent time slice. The computing system may update the access feature information corresponding to the target file including at least one of the file attributes of the target file, the access time of the target file in the previous time slice, the creation time of the target file, the access frequency of the target file in the previous time slice, and the access time interval of the target file in the previous time slice, so that the probability of a target device accessing a target file in a subsequent time slice may be predicted based on the file access prediction model according to one or more access feature information corresponding to the target file. Of course, this is not limited to this and is not limited here.

[0042] For example, the computing system may predict the first predicted access probability of each of the multiple target files based on the file access prediction model. In some embodiments, the order in which the file access model predicts the first predicted access probability of each of the multiple target files may be determined based on the access feature information corresponding to the target files. For example, the order in which the file access prediction model predicts the first predicted access probability of each of the multiple target files may be determined based on the time when the target device last accessed the target file. Taking the example that the target files include a first target file and a second target file, when the time when the target device last accessed the first target file is later than the time when the target device last accessed the second target file, the first predicted access probability of the first target file may be predicted first, and then the first predicted access probability of the second target file may be predicted. Of course, this is not limited to this, and no limitation is made here.

[0043] In an exemplary embodiment, in the process of predicting the first predicted access probability of each of multiple target files according to the file access prediction model, the computing system can count the total prediction time of the file access model predicting the first predicted access probability of the multiple target files. When the total prediction time is greater than or equal to the total prediction time threshold, the prediction of the first predicted access probability of the multiple target files is terminated. For target files that are not predicted by the file access prediction model, such as target files with a large time difference between the time of the most recent access and the current time slice, target files with a low access frequency in the previous time slice, etc., the probability of the target device accessing the target file in the subsequent time slice can be set to 0. Of course, it is not limited to this and is not limited here.

[0044] In this way, based on the file access prediction model, according to the access feature information corresponding to the target file, the probability of the target device accessing the target file in a subsequent time slice is predicted to obtain the first predicted access probability of the target file. The file access prediction model can output the first predicted access probability of the target file based on one or more access feature information corresponding to the target file, which is conducive to improving the flexibility and accuracy of determining the first predicted access probability of the target file.

[0045] Exemplarily, the computing system may determine the target file from preset files of the storage system according to file access information of the target device to the storage system in a previous time slice.

[0046] For example, the computing system may determine, based on the file access information, that the target device has accessed a preset file in the storage system within a plurality of preceding time slices. Figure 2 As shown, take multiple preceding time slices including time slice 0 to time slice k, and preset files including preset file 0 to preset file 4 as an example. Among them, time slice 0 is earlier than time slice 1, time slice 1 is earlier than time slice 2, and so on. File access information, for example, includes that the target device accessed preset file 1, preset file 2, preset file 4 and preset file 3 in the preceding time slice 3; the target device accessed preset file 0 in the preceding time slice 4; the target device accessed preset file 3 and preset file 1 in the preceding time slice k. Based on this, the mapping relationship between the preceding time slice and the preset file can be determined, and an inverted index of the preceding time slice and the preset file can be constructed, as shown in Table 1. The mapping relationship between the preceding time slice and the preset file, and the inverted index of the preceding time slice and the preset file can be established and maintained based on a hash table, for example, but it is certainly not limited to this and is not limited here.

[0047] Time Slice Preset Files Time Slice 3 Preset file 1, Preset file 2, Preset file 4, Preset file 3 Time Slice 4 Default file 0, ... …… …… Time slice k Preset file 3, Preset file 1

[0048] In some embodiments, when it is determined based on file access information that the target device accesses a first preset file of the storage system within a first preceding time slice, and the time difference between the first preceding time slice and the current time slice is less than a preset time difference threshold, the first preset file is determined to be the target file.

[0049] For example, please refer to Table 1, taking the first preceding time slice including time slice 0 to time slice k as an example, when it is determined according to the file access information that the target device accesses preset files 1 to preset files 4 of the storage system within time slice 3, if the time difference between time slice 3 and the current time slice is less than the preset time difference threshold, then preset files 1 to preset files 4 can be determined as target files. Correspondingly, when it is determined according to the file access information that the target device accesses preset file 0 of the storage system within time slice 4, if the time difference between time slice 4 and the current time slice is less than the preset time difference threshold, then preset file 0 can be determined as the target file. Of course, it is not limited to this, and no limitation is made here.

[0050] For example, please refer to Table 1, taking the first preceding time slice including time slice 0 to time slice k as an example, when it is determined according to the file access information that the target device accesses preset files 1 to preset files 4 of the storage system within time slice 3, if the time difference between time slice 3 and the current time slice is greater than or equal to the preset time difference threshold, then it can be combined with time slice 4 to time slice k to determine whether any one of preset files 1 to preset files 4 is the target file. For example, when it is determined according to the file access information that the target device accesses preset files 1 and preset files 3 of the storage system within time slice k, if the time difference between time slice k and the current time is less than the preset time difference threshold, it can be determined that preset files 1 and preset files 3 are the target files. Of course, it is not limited to this, and no limitation is made here.

[0051] In this way, by determining that the target device accesses the first preset file of the storage system within the first previous time slice based on the file access information, and the time difference between the first previous time slice and the current time slice is less than the preset time difference threshold, the first preset file is determined to be the target file, which is conducive to improving the convenience of determining the target file from the preset files of the storage system.

[0052] In some implementations, when it is determined according to the file access information that the target device accesses a second preset file of the storage system in the current time slice, the second preset file is determined as the target file.

[0053] For example, please refer to Table 1, taking the current time slice as time slice 4, and the target files including preset files 1 to preset files 3 as an example, according to the file access information, it is determined that the target device accesses the preset file 0 of the storage system within time slice 4, and then the preset file 0 can be determined as the target file. Taking the current time slice as time slice k, and the target files including preset file 0 as an example, according to the file access information, it is determined that the target device accesses the preset file 1 and preset file 3 of the storage system within time slice k, and then the preset file 1 and preset file 3 can be determined as the target files. Of course, it is not limited to this, and no limitation is made here.

[0054] In this way, by determining that the target device accesses the second preset file of the storage system within the current time slice based on the file access information, the second preset file is determined to be the target file. The target file can be dynamically updated based on the file access information of the target device in the current time slice, which is beneficial to improving the convenience and comprehensiveness of determining the target file from the preset files of the storage system.

[0055] In some embodiments, when it is determined based on file access information that the time when the target device last accessed a target file in the storage system is within a second preceding time slice, and the time difference between the second preceding time slice and the current time slice is greater than or equal to a preset time difference threshold, the target file is determined to be a non-target file.

[0056] For example, please refer to Table 1, taking the second preceding time slice as time slice 0 to time slice k, and the target files including preset files 0 to preset files 4 as an example, when it is determined according to the file access information that the target device last accessed the preset files 2 and 4 of the storage system in time slice 3, if the time difference between time slice 3 and the current time slice is greater than or equal to the preset time difference threshold, it can be determined that preset files 2 and preset files 4 are non-target files. When it is determined according to the file access information that the target device last accessed the preset file 0 in time slice 4, if the time difference between time slice 4 and the current time slice is greater than or equal to the preset time difference threshold, it is determined that preset file 0 is a non-target file. Of course, it is not limited to this, and no limitation is made here.

[0057] In this way, by determining that the target file is a non-target file when the time when the target device last accessed the target file of the storage system is within the second preceding time slice based on the file access information, and the time difference between the second preceding time slice and the current time slice is greater than or equal to the preset time difference threshold, the target file is determined to be a non-target file. Then, as the time slice passes, the non-target files can be promptly removed from the target files, which is beneficial to maintaining the timeliness of the target file, so as to avoid the adverse effect on the subsequent predicted probability of the target device accessing the target file in the subsequent time slice due to the target device last accessing the target file of the storage system too long ago, thereby helping to improve the accuracy of determining the first predicted access probability of the target file.

[0058] Step S103: Determine the second predicted access probability of each of the plurality of data blocks in the target file according to the first predicted access probability of the target file.

[0059] For example, in distributed scenarios such as big data and cloud computing, the data volume of the target file can reach hundreds of trillions or even terabytes. If data scheduling is performed at the file granularity for the target file in distributed scenarios such as big data and cloud computing, due to the large data volume of the target file, it is easy to cause the target device to have low access efficiency to the target file in the subsequent time slice.

[0060] In some embodiments, in order to improve the access efficiency of the target device to the target file in the subsequent time slice, the target file can be scheduled according to the granularity of the data block. For example, the computing system can determine the second predicted access probability of each of the multiple data blocks in the target file based on the first predicted access probability of the target file, so as to subsequently schedule the multiple data blocks in the target file based on the second predicted access probability. For example, the first predicted access probability of the target file can be weighted mapped to obtain the second predicted access probability of each of the multiple data blocks in the target file.

[0061] In this way, according to the first predicted access probability of the target file, the second predicted access probability of each of the multiple data blocks in the target file is determined, which can be used for subsequent data scheduling of the target file at the data block granularity. In addition, when the target device accesses the target file in a subsequent time slice, it can obtain the data block corresponding to the target file at the data block granularity, which is conducive to improving the access efficiency of the target file in the storage and computing separation scenario.

[0062] In some embodiments, weight coefficients of each of the multiple data blocks in the target file are determined based on the position order of the multiple data blocks in the target file, wherein the weight coefficient of the data block near the head of the target file is greater than the weight coefficient of the data block near the tail of the target file; and the second predicted access probability of the data block is determined based on the first predicted access probability of the target file and the product of the weight coefficient of the data block.

[0063] For example, if the target file includes i data blocks, the weight coefficients of the i data blocks in the target file can be determined according to the position order of the i data blocks in the target file. When the target device accesses the target file, the data blocks near the head of the target file will be accessed by the target device earlier than the data blocks near the tail of the target file. The weight coefficient is, for example, 0.98. i, of course, it is not limited to this and is not limited here. The weight coefficient of the data block near the head in the target file is greater than the weight coefficient of the data block near the tail in the target file, then the weight coefficient of the data block near the head in the target file is larger, and as the data block gets closer and closer to the tail of the target file, the weight coefficient continues to decay. Based on the fact that the weight coefficient of the data block near the head in the target file is greater than the weight coefficient of the data block near the tail in the target file, the second predicted access probability of the data block can be determined based on the first predicted access probability of the target file and the product of the weight coefficient of the data block. Among them, the second predicted access probability of the data block near the head in the target file is greater than the second predicted access probability of the data block near the tail in the target file, so that the second predicted access probability can be used as the basis for subsequent data scheduling of multiple data blocks in the target file.

[0064] In this way, the weight coefficients of the multiple data blocks in the target file are determined according to the position order of the multiple data blocks in the target file; the second predicted access probability of the data block is determined according to the product of the first preset access probability of the target file and the weight coefficient of the data block, which is conducive to improving the convenience of determining the second predicted access probability of the data block.

[0065] Step S104: Obtain the physical topological order of the target device for each storage node in the storage system.

[0066] For example, when data scheduling is performed on a target file at a data block granularity, for example, multiple data blocks in the target file need to be pre-stored in a target storage node in a storage system, or a target storage node for the data block needs to be selected from multiple storage nodes storing the same data block, so that the target device can obtain the corresponding data block from the target cache node in the storage system when accessing the target file. In some embodiments, taking the storage nodes of the storage system including a first storage node and a second storage node as an example, when the physical topological order of the first storage node is better than that of the second storage node, the efficiency of the target device in obtaining the data block of the target file from the first storage node when accessing the target file is higher than the efficiency of obtaining the data block of the target file from the second storage node. Based on this, the physical topological order of each storage node in the storage system by the target device can be obtained, so as to determine the target storage node of the data block according to the physical topological order of each storage node in the storage node by the target device, thereby performing data scheduling on the data block.

[0067] In this way, by obtaining the physical topological order of the target device to each storage node in the storage system, it is helpful to improve the convenience of subsequently determining the target storage node of the data block.

[0068] In some embodiments, the physical topological order of the target device for each storage node in the storage system is determined based on the first location information of the target device and the second location information of each storage node in the storage system, wherein the physical topological order of the storage nodes whose second location information is close to the first location information takes precedence over the physical topological order of the storage nodes whose second location information is far from the first location information.

[0069] For example, based on the closest location principle, the physical topological order of the target device to each storage node in the storage system may be determined according to the first location information of the target device and the second location information of each storage node in the storage system. For example, taking the storage nodes including the storage nodes in the same data center as the target device, the storage nodes in the same rack as the target device, and the storage nodes in the same network segment as the target device as an example, when the position difference between the second position information of the storage nodes in the same network segment and the first position information of the target device is less than the position difference between the second position information of the storage nodes in the same rack and the first position information of the target device, and the position difference between the second position information of the storage nodes in the same rack and the first position information of the target device is less than the position difference between the second position information of the storage nodes in the same data center and the first position information of the target device, it can be determined that the second position information of the storage nodes in the same data center, the second position information of the storage nodes in the same rack, and the second position information of the storage nodes in the same network segment gradually approach the first position information of the target device, and therefore, the physical topological order of the target device to the storage nodes in the same network segment takes precedence over the physical topological order of the target device to the storage nodes in the same rack, and the physical topological order of the target device to the storage nodes in the same rack takes precedence over the physical topological order of the target device to the storage nodes in the same data center. Of course, it is not limited to this, and is not limited here.

[0070] Taking the storage nodes including storage nodes 1 to storage nodes 3 as an example, where storage node 1 and the target device are in the same network segment, storage node 2 and the target device are in the same rack, and storage node 3 and the target device are in the same data center, it can be determined that the position difference between the second position information of storage node 1 and the first position information of the target device is small, the position difference between the second position information of storage node 2 and the first position information of the target device is medium, and the position difference between the second position information of storage node 3 and the first position information of the target device is large. Based on this, it can be determined that the physical topology order of storage node 1 takes precedence over storage node 2, and the physical topology order of storage node 2 takes precedence over storage node 3. Of course, it is not limited to this, and no limitation is made here.

[0071] In this way, the physical topological order of the target device to each storage node in the storage system is determined based on the first position information of the target device and the second position information of each storage node in the storage system, which is conducive to improving the convenience of determining the physical topological order of the target device to each storage node in the storage system, thereby facilitating improving the convenience of subsequently determining the target storage node of the data block. In addition, based on the fact that the physical topological order of each storage node in the storage system is determined based on the first position information of the target device and the second position information of each storage node in the storage system, when the target device accesses the target file in a subsequent time slice, it can preferentially obtain the data block corresponding to the target file from the storage node whose second position information is close to the first position information, which is conducive to subsequently improving the access efficiency to the file in the storage and computing separation scenario.

[0072] Step S105 : determining a target storage node for the data block according to the second predicted access probability of the data block and the physical topological order of each storage node.

[0073] For example, the computing system may preferentially store data blocks with a higher second predicted access probability in storage nodes with an earlier physical topological order, or preferentially obtain data blocks with a higher second predicted access probability from storage nodes with an earlier physical topological order. Therefore, the target storage node of the data block can be determined based on the second predicted access probability of the data block and the physical topological order of each storage node.

[0074] In some embodiments, when the second predicted access probability of the first data block is greater than the second predicted access probability of the second data block, and the physical topological order of the first storage node takes precedence over the physical topological order of the second storage node, the target storage node of the first data block is determined to be the first storage node.

[0075] For example, taking the example that the physical topological order of the target device to the storage nodes in the same network segment takes precedence over the physical topological order of the target device to the storage nodes in the same rack, and the physical topological order of the target device to the storage nodes in the same rack takes precedence over the physical topological order of the target device to the storage nodes in the same data center, when there is a second predicted access probability of the first data block greater than the second predicted access probability of the second data block, the storage node in the same network segment as the target device is determined to be the target storage node of the first data block. Of course, this is not limited to this and is not limited here.

[0076] Taking the example that the storage nodes include storage nodes 1 to 3, the physical topological order of storage node 1 takes precedence over the physical topological order of storage node 2, and the physical topological order of storage node 2 takes precedence over the physical topological order of storage node 3, if the first data block and the second data block are both stored in storage nodes 1 to 3, and the second predicted access probability of the first data block is greater than the second predicted access probability of the second data block, then storage node 1 can be preferentially determined as the target storage node for the first data block. Of course, this is not limited to this, and no limitation is made here.

[0077] In this way, the target storage node of the data block is determined according to the second predicted access probability of the data block and the physical topological order of each storage node, which is conducive to improving the convenience of determining the target storage node of the data block. In addition, when the target device accesses the target file in a subsequent time slice, it can preferentially obtain the data block corresponding to the target file from the storage node with a higher physical topological order. For example, it preferentially obtains the data block with a higher second predicted access probability from the storage node with a higher physical topological order, which is conducive to improving the access efficiency of the file in the storage and computing separation scenario.

[0078] Exemplarily, when the cache size information of a data block is larger than the remaining cache size information corresponding to the cache space of the target device, the target storage node of the data block is determined based on the second predicted access probability of the data block and the physical topological order of each storage node; when the cache size information of the data block is less than or equal to the remaining cache size information corresponding to the cache space of the target device, the target storage node of the data block is determined to be the cache space of the target device.

[0079] In some embodiments, for data blocks with a higher second predicted access probability, in order to subsequently improve the access efficiency to files in a storage and computing separation scenario, the computing system may preferentially store the data blocks with a higher second predicted access probability in the cache space of the target device. When the target device subsequently accesses the target file, the data blocks of the target file can be directly obtained from the cache space of the target device, which is conducive to improving the access probability to the file in a storage and computing separation scenario.

[0080] For example, when the cache size information of the data block is less than or equal to the remaining cache size information corresponding to the cache space of the target device, the target storage node of the data block is determined to be the cache space of the target device, so that the data block is stored in the cache space of the target device. When the cache size information of the data block is greater than the remaining cache size information corresponding to the cache space of the target device, the target storage node of the data block is determined based on the second predicted access probability of the data block and the physical topological order of each storage node.

[0081] In this way, by comparing the cache size information of the data block with the remaining cache size information corresponding to the cache space of the target device, the target storage node of the data block can be flexibly determined based on the comparison result, which is conducive to improving the flexibility of determining the target storage node of the data block, and thus is conducive to subsequently improving the probability of accessing files in the storage and computing separation scenario.

[0082] Step S106: Cache the data block from the target storage node to the corresponding target device.

[0083] Exemplarily, after determining the target storage node of the data block, the computing system can obtain and store the association relationship between the target storage node and the data block. When it is detected that the target device accesses the target file, the computing center can cache the data block from the target storage node to the corresponding target device according to the association relationship between the target storage node and the data block.

[0084] For example, when it is detected that the target device accesses the target file, the second location information corresponding to the target storage node of the data block in the target file is sent to the target device so that the target device obtains the data block from the target storage node; or, when it is detected that the target device accesses the target file, the data block is obtained from the target storage node of the data block in the target file.

[0085] For example, when it is detected that the target device accesses the target file, the computing system may send the second location information corresponding to the target storage node of the data block in the target file to the target device based on the association relationship between the target storage node and the data block, so that the target device obtains the data block from the target storage node. For another example, when it is detected that the target device accesses the target file, the computing system may obtain the data block from the target storage node of the data block in the target file based on the association relationship between the target storage node and the data block, so as to send the data block to the target device. Of course, this is not limited to this, and no limitation is made here.

[0086] In this way, by caching data blocks from the target storage node to the corresponding target device, the computing system or target device can communicate directly with the slave nodes of the distributed storage system, such as the target storage node, without the help of a storage system, such as a master node of the distributed storage system, and obtain the data blocks stored in the target storage node, which is conducive to improving the access efficiency to data blocks in the storage and computing separation scenario, and thus is conducive to improving the access efficiency to files in the storage and computing separation scenario.

[0087] The data scheduling method provided in the above embodiment obtains file access information of the target device to the storage system; predicts the probability of the target device accessing the target file in the storage system in a subsequent time slice based on the file access information, and obtains a first predicted access probability of the target file; determines the second predicted access probability of each of multiple data blocks in the target file based on the first predicted access probability of the target file; obtains the physical topological order of each storage node in the storage system from the target device; determines the target storage node of the data block based on the second predicted access probability of the data block and the physical topological order of each storage node; and caches the data block from the target storage node to the corresponding target device, thereby improving the pre-cache hit rate of the file in the storage and computing separation scenario, and improving the access efficiency of the file in the storage and computing separation scenario.

[0088] See also Figure 3 , Figure 3 A schematic block diagram of the structure of an electronic device provided by an embodiment of the present invention.

[0089] like Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302, and the processor 301 and the memory 302 are connected via a bus 303, such as an I2C (Inter-integrated Circuit) bus.

[0090] Specifically, the processor 301 is used to provide computing and control capabilities to support the operation of the entire electronic device. The processor 301 can be a central processing unit (CPU), and the processor 301 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0091] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0092] Those skilled in the art will understand that Figure 3The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the electronic device to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0093] The processor is used to execute a computer program stored in a memory, and implement any one of the data scheduling methods provided by the embodiments of the present invention when executing the computer program.

[0094] In one embodiment, the processor is used to execute a computer program stored in the memory, and implements the following steps when executing the computer program:

[0095] Obtain the target device's file access information to the storage system;

[0096] Predicting, based on the file access information, a probability that the target device will access a target file in the storage system in a subsequent time slice, and obtaining a first predicted access probability of the target file;

[0097] Determining, according to the first predicted access probability of the target file, a second predicted access probability of each of the plurality of data blocks in the target file;

[0098] Obtaining a physical topological order of the target device for each storage node in the storage system;

[0099] Determining a target storage node for the data block according to the second predicted access probability of the data block and the physical topological order of each of the storage nodes;

[0100] The data block is cached from the target storage node to the corresponding target device.

[0101] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the electronic device described above can refer to the corresponding process in the aforementioned data scheduling method embodiment, and will not be repeated here.

[0102] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by a processor, and the processor implements any data scheduling method provided in the description of the embodiment of the present invention.

[0103] The storage medium may be an internal storage unit of the electronic device described in the above embodiment, such as a hard disk or memory of the electronic device. The storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0104] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0105] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0106] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data scheduling method, characterized in that: The data scheduling method comprises: Obtain the target device's file access information to the storage system; Predicting, based on the file access information, a probability that the target device will access a target file in the storage system in a subsequent time slice, and obtaining a first predicted access probability of the target file; Determining, according to the first predicted access probability of the target file, a second predicted access probability of each of the plurality of data blocks in the target file; Obtaining a physical topological order of the target device for each storage node in the storage system; Determining a target storage node for the data block according to the second predicted access probability of the data block and the physical topological order of each of the storage nodes; The data block is cached from the target storage node to the corresponding target device.

2. The data scheduling method according to claim 1, characterized in that: The step of predicting, based on the file access information, the probability of the target device accessing the target file in the storage system in a subsequent time slice to obtain a first predicted access probability of the target file includes: Determining a target file from preset files in the storage system according to the file access information; Based on the file access prediction model, according to the access feature information corresponding to the target file, the probability of the target device accessing the target file in a subsequent time slice is predicted to obtain a first predicted access probability of the target file; the access feature information includes at least one of the file attributes of the target file, the access time of the target file in the previous time slice, the creation time of the target file, and the access frequency of the target file in the previous time slice.

3. The data scheduling method according to claim 2, characterized in that: The determining the target file from the preset files of the storage system according to the file access information includes at least one of the following: When it is determined according to the file access information that the target device accesses a first preset file of the storage system within a first preceding time slice, and a time difference between the first preceding time slice and the current time slice is less than a preset time difference threshold, determining that the first preset file is a target file; In a case where it is determined according to the file access information that the target device accesses a second preset file of the storage system in the current time slice, determining the second preset file as the target file; When it is determined based on the file access information that the time when the target device last accessed the target file of the storage system is within the second preceding time slice, and the time difference between the second preceding time slice and the current time slice is greater than or equal to a preset time difference threshold, the target file is determined to be a non-target file.

4. The data scheduling method according to claim 1, characterized in that: Determining the second predicted access probability of each of the plurality of data blocks in the target file according to the first predicted access probability of the target file comprises: Determine, according to the position order of the multiple data blocks in the target file, the weight coefficients of the multiple data blocks in the target file, wherein the weight coefficient of the data block close to the head of the target file is greater than the weight coefficient of the data block close to the tail of the target file; A second predicted access probability of the data block is determined according to the product of the first predicted access probability of the target file and the weight coefficient of the data block.

5. The data scheduling method according to any one of claims 1 to 4, characterized in that: The obtaining of the physical topological order of the target device for each storage node in the storage system includes: Based on the first location information of the target device and the second location information of each storage node in the storage system, the physical topological order of the target device to each storage node in the storage system is determined, wherein the physical topological order of the storage nodes whose second location information is close to the first location information takes precedence over the physical topological order of the storage nodes whose second location information is far from the first location information.

6. The data scheduling method according to claim 5, characterized in that: The step of determining a target storage node for the data block according to the second predicted access probability of the data block and the physical topological order of each of the storage nodes comprises: When the second predicted access probability of the first data block is greater than the second predicted access probability of the second data block, and the physical topological order of the first storage node takes precedence over the physical topological order of the second storage node, the target storage node of the first data block is determined to be the first storage node.

7. The data scheduling method according to claim 5, characterized in that: The step of determining a target storage node for the data block according to the second predicted access probability of the data block and the physical topological order of each of the storage nodes comprises: In a case where the cache size information of the data block is greater than the remaining cache size information corresponding to the cache space of the target device, determining a target storage node for the data block according to the second predicted access probability of the data block and the physical topological order of each of the storage nodes; When the cache size information of the data block is less than or equal to the remaining cache size information corresponding to the cache space of the target device, the target storage node of the data block is determined to be the cache space of the target device.

8. The data scheduling method according to any one of claims 1 to 4, characterized in that: The caching the data block from the target storage node to the corresponding target device includes: In the case where it is detected that the target device accesses the target file, second location information corresponding to the target storage node of the data block in the target file is sent to the target device so that the target device obtains the data block from the target storage node; or In a case where it is detected that the target device accesses the target file, the data block is obtained from a target storage node of the data block in the target file.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the data scheduling method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data scheduling method according to any one of claims 1 to 8 is implemented.

Images