Data reading methods, devices, and computer storage media

Abstract

This application provides a data reading method, apparatus, and computer storage medium. Upon receiving a data reading request from a terminal device, if the target data is stored in a cache hard disk cluster, the current network latency of the cache hard disk cluster is determined, and it is judged whether the current network latency is greater than a preset first latency threshold. If the current network latency is less than or equal to the first latency threshold, the target data is read from the cache hard disk cluster through the cache hard disk server and forwarded to the terminal device. Within the allowable network latency range, this solution directly reads the target data from the cache hard disk cluster for all data reading requests, thereby reducing the number of times the mechanical hard disk cluster is accessed and extending the lifespan of the mechanical hard disks.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data reading method, apparatus, and computer storage medium. Background Technology

[0002] With the development of internet technology, a network-based distributed file system has emerged. This distributed file system consists of a cache disk cluster (including multiple solid-state drives) and corresponding cache disk servers, as well as a hard disk cluster (including multiple hard disks) and corresponding hard disk servers.

[0003] When a client needs to read data, it sends a data read request to the data module. The data module compares the current data transfer rate of the cache hard disk server with the current data transfer rate of the mechanical hard disk server, and reads the data from the corresponding hard disk cluster through the server with the lower data transfer rate and sends it to the client.

[0004] Based on existing methods, the access frequency of cache disk clusters is basically the same as that of mechanical hard disk clusters. The maximum number of accesses allowed for mechanical hard disks is far less than that allowed for solid-state drives (SSDs). Therefore, in existing distributed file systems, mechanical hard disks have a shorter lifespan and require frequent replacement. Summary of the Invention

[0005] Based on the shortcomings of the prior art, this application provides a data reading method to solve the problem of short lifespan of mechanical hard drives in distributed file systems caused by existing reading methods.

[0006] The first aspect of this application provides a method for reading data, including:

[0007] Receive data read requests from terminal devices;

[0008] If the target data is stored in a cache disk cluster, determine the current network latency of the cache disk cluster; wherein, the target data refers to the data that satisfies the data read request;

[0009] Determine whether the current network latency is greater than a preset first latency threshold;

[0010] If the current network latency is less than or equal to the first latency threshold, the target data is read from the cache disk cluster via the cache disk server;

[0011] The target data is fed back to the terminal device.

[0012] Optionally, after determining whether the current network latency is greater than a preset first latency threshold, the method further includes:

[0013] If the current network latency is greater than the first latency threshold, the theoretical reading frequency corresponding to the current network latency is determined according to the preset correspondence between network latency and reading frequency.

[0014] The historical read frequency is obtained by calculating the proportion of historical data read requests that read target data from the cache disk cluster in the historical read record; wherein, the historical read record includes K historical data read requests received before the current data read request and the disk cluster that provides target data for each historical data read request; K is a preset positive integer;

[0015] Determine whether the historical read frequency is greater than the theoretical read frequency;

[0016] If the historical read frequency is greater than the theoretical read frequency, the target data is read from the hard disk cluster via the hard disk server.

[0017] If the historical read frequency is less than or equal to the theoretical read frequency, the target data is read from the cache disk cluster through the cache disk server.

[0018] Optionally, determining the current network latency of the cache disk cluster includes:

[0019] Based on the time sequence of received requests, starting from the current data read request, the historical network latency corresponding to the previous N data read requests is sequentially obtained; wherein, each data read request corresponds to a historical network latency, the historical network latency refers to the network latency of the cache disk cluster when reading part or all of the data that satisfies the corresponding data read request from the cache disk cluster; N is a preset positive integer;

[0020] The current network latency of the cache disk cluster is obtained by calculating the average of the N historical network latencies.

[0021] Optionally, determining the current network latency of the cache disk cluster includes:

[0022] Based on the time sequence of received requests, starting from the current data read request, the historical network latency corresponding to the previous N data read requests is sequentially obtained; wherein, each data read request corresponds to a historical network latency, the historical network latency refers to the network latency of the cache disk cluster when reading part or all of the data that satisfies the corresponding data read request from the cache disk cluster; N is a preset positive integer;

[0023] Compare the N historical network delays obtained to determine the maximum historical network delay;

[0024] Calculate the average of N-1 historical network delays other than the maximum historical network delay to obtain the current network delay of the cache disk cluster.

[0025] Optionally, if the target data is stored in a cache disk cluster, before determining the current network latency of the cache disk cluster, the method further includes:

[0026] If the target data is not stored in the cache hard disk cluster, the target data is read from the hard disk cluster through the hard disk server.

[0027] Optionally, reading the target data from the cache disk cluster via the cache disk server includes:

[0028] Send the data read request to the cache disk server;

[0029] The system receives the address information of the target data fed back by the cache hard disk server; wherein the address information of the target data carries the address of the solid-state drive storing the target data, and the address of the target data in the solid-state drive;

[0030] Based on the address information of the target data, the target data is read from the solid-state drive storing the target data.

[0031] A second aspect of this application provides a data reading device, comprising:

[0032] The receiving unit is used to receive data read requests from terminal devices;

[0033] The determining unit, if the target data is stored in a cache disk cluster, is used to determine the current network latency of the cache disk cluster; wherein, the target data refers to the data that satisfies the data read request;

[0034] The judgment unit is used to determine whether the current network latency is greater than a preset first latency threshold;

[0035] The reading unit, if the current network latency is less than or equal to the first latency threshold, is used to read the target data from the cache disk cluster through the cache disk server;

[0036] A sending unit is used to send the target data back to the terminal device.

[0037] Optionally, the determining unit is further configured to:

[0038] If the current network latency is greater than the first latency threshold, the theoretical reading frequency corresponding to the current network latency is determined according to the preset correspondence between network latency and reading frequency.

[0039] The historical read frequency is obtained by calculating the proportion of historical data read requests that read target data from the cache disk cluster in the historical read record; wherein, the historical read record includes K historical data read requests received before the current data read request and the disk cluster that provides target data for each historical data read request; K is a preset positive integer;

[0040] Determine whether the historical read frequency is greater than the theoretical read frequency;

[0041] The reading unit is used for:

[0042] If the historical read frequency is greater than the theoretical read frequency, the target data is read from the hard disk cluster via the hard disk server.

[0043] If the historical read frequency is less than or equal to the theoretical read frequency, the target data is read from the cache disk cluster through the cache disk server.

[0044] Optionally, when determining the current network latency of the cache disk cluster, the determining unit is specifically used for:

[0045] Based on the time sequence of received requests, starting from the current data read request, the historical network latency corresponding to the previous N data read requests is sequentially obtained; wherein, each data read request corresponds to a historical network latency, the historical network latency refers to the network latency of the cache disk cluster when reading part or all of the data that satisfies the corresponding data read request from the cache disk cluster; N is a preset positive integer;

[0046] Compare the N historical network delays obtained to determine the maximum historical network delay;

[0047] Calculate the average of N-1 historical network delays other than the maximum historical network delay to obtain the current network delay of the cache disk cluster.

[0048] A third aspect of this application provides a computer storage medium for storing a program, which, when executed, implements a data reading method as provided in any of the first aspects of this application.

[0049] This application provides a data reading method, apparatus, and computer storage medium. Upon receiving a data reading request from a terminal device, if the target data is stored in a cache hard disk cluster, the current network latency of the cache hard disk cluster is determined, and it is judged whether the current network latency is greater than a preset first latency threshold. If the current network latency is less than or equal to the first latency threshold, the target data is read from the cache hard disk cluster through the cache hard disk server and forwarded to the terminal device. Within the allowable network latency range, this solution directly reads the target data from the cache hard disk cluster for all data reading requests, thereby reducing the number of times the mechanical hard disk cluster is accessed and extending the lifespan of the mechanical hard disks. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of a distributed file system architecture;

[0052] Figure 2 A flowchart illustrating a data reading method provided in an embodiment of this application;

[0053] Figure 3 A flowchart illustrating a method for determining a theoretical read frequency based on current network latency, provided in an embodiment of this application;

[0054] Figure 4 A flowchart illustrating a method for determining the current network latency of a cache disk cluster based on historical network latency, provided in an embodiment of this application;

[0055] Figure 5 A flowchart illustrating a method for a data server to read data from a hard disk cluster, provided in an embodiment of this application;

[0056] Figure 6 This is a schematic diagram of a data reading device provided in an embodiment of this application. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] The data reading method provided in any embodiment of this application is mainly applicable to distributed file systems. To better understand the method provided in this application, let's first combine it with... Figure 1 Describe the architecture of a distributed file system.

[0059] refer to Figure 1 The distributed file system includes two hard disk clusters for storing data: a cache hard disk cluster 100 and a mechanical hard disk cluster 200. The cache hard disk cluster includes multiple solid-state drives (SSDs) located in different locations, all of which are connected to the cache hard disk server 101 via a network. Similarly, the mechanical hard disk cluster includes multiple mechanical hard disks located in different locations, all of which are connected to the mechanical hard disk server 201 via a network.

[0060] Meanwhile, both the mechanical hard drive server and the cache hard drive server are connected to the data server 300 via the network, and the data server is connected to several terminal devices 400 (or client machines) via the network. The data server is also connected to the mechanical hard drive cluster and the cache hard drive cluster via the network.

[0061] The cache disk server is used to record what data each solid-state drive in the cache disk cluster stores, and the storage address of this data on the corresponding solid-state drive.

[0062] Similarly, the hard disk drive server is used to record what data each hard disk in the hard disk cluster stores, and the storage address of this data on the corresponding hard disk.

[0063] In the aforementioned distributed file system, both the cache disk cluster and the hard disk cluster are used to store files. A file can be stored entirely on one hard disk in the disk cluster, or it can be split into multiple sub-files and stored on multiple hard disks in the disk cluster. On the other hand, typically all files stored in the distributed file system have at least one copy in the hard disk cluster. Furthermore, based on the frequency of file usage, the file system will also copy at least one copy of most frequently accessed (equivalent to being read by terminal devices) files stored in the hard disk cluster, and store these copies in the cache disk cluster.

[0064] When any terminal device needs to read data from the distributed file system (which could be all the data in a complete file or a part of the data in a file), it first sends a data read request to the data server. If both the mechanical hard disk cluster and the cache hard disk cluster store the data requested by the terminal device, the data server selects to read the data requested by the terminal device from the mechanical hard disk cluster or the cache hard disk cluster according to a certain scheduling strategy, and then sends the read data to the terminal device, thus completing a data read process.

[0065] In existing distributed file systems, in order to ensure that the load on the hard disk cluster and the cache disk cluster is as balanced as possible in terms of data transmission, if the target data (referring to the data requested by the terminal device, or the data that satisfies the data reading request of the terminal device) is stored in both the hard disk cluster and the cache disk cluster, the data server will compare the current network latency of the hard disk cluster and the current network latency of the cache disk cluster, and then read the target data from the hard disk cluster with the lower current network latency.

[0066] However, this scheduling strategy has the following drawbacks:

[0067] Based on this scheduling strategy, the access frequency of mechanical hard drives in a mechanical hard drive cluster is basically the same as that of solid-state drives in a cache hard drive cluster. However, hard drives have an access limit (or maximum number of accesses). Once the number of accesses exceeds the limit, the hard drive will fail. Furthermore, the maximum number of accesses for mechanical hard drives is generally less than that for solid-state drives. Therefore, the existing scheduling strategy of accessing mechanical hard drives and solid-state drives at the same frequency will result in a significantly shorter lifespan for mechanical hard drives in a distributed file system compared to that of solid-state drives, thus requiring frequent replacement of mechanical hard drives.

[0068] To address the shortcomings of the existing technology, this application provides a data reading method that extends the lifespan of the hard disk drive in a distributed file system by improving the scheduling strategy when the data server reads data. Please refer to [link / reference]. Figure 2 The method provided in this embodiment includes the following steps:

[0069] S201, Receive data read request from terminal device.

[0070] First, it should be noted that this embodiment uses the data server in the distributed file system as the execution entity for the entire data reading process.

[0071] Specifically, a data read request can carry the following information:

[0072] The file identifier of the file to which the target data belongs, the offset address of the target data, and the length of the target data.

[0073] The file identifier mentioned above can be a filename or a file identification code generated by a specific algorithm to uniquely identify each file in the distributed file system.

[0074] If the target data is a complete file, then the offset address of the target data is the starting position of the file, and the length of the target data is the length of the file.

[0075] If the target data is a segment of data in a file, then the offset address of the target data is the position of the starting character of this segment of data in the file, and the length of the target data is the length of this segment of data.

[0076] Of course, a terminal device can request multiple data segments from multiple files at the same time. In this case, the data reading request carries the file identifier of the file to which each data segment belongs, the offset address of each data segment, and the length of each data segment.

[0077] The target data refers to the data that satisfies the data reading request of the terminal device, that is, the data requested by the terminal device.

[0078] Optionally, each time the distributed file system acquires a new file, it can record the file identifier of these files in the data server. Therefore, the terminal device can learn from the data server what files the distributed file system currently has, and then determine the target data according to the needs of the user operating the terminal device.

[0079] S202. Determine if the target data exists in the cache disk cluster.

[0080] If the target data is not stored in the cache disk cluster, proceed to step S209.

[0081] If the target data is stored in the cache disk cluster, then proceed to step S203.

[0082] As mentioned above, each file in the distributed file system has at least one copy stored in the mechanical hard disk cluster. Meanwhile, the files that are used more frequently have at least one copy stored in the cache hard disk cluster. If the target data is not stored in the cache hard disk cluster, it means that the file corresponding to the target data is used less frequently. In this case, step S209 needs to be executed directly to read the target data from the mechanical hard disk cluster.

[0083] Specifically, when executing step S202, a data read request can be sent to the cache disk server. After receiving the data read request, the cache disk server can determine whether the target data is stored in the cache disk cluster based on the data stored in each solid-state drive it records, and then send the result back to the data server.

[0084] S203. Determine the current network latency of the cache disk cluster.

[0085] The current network latency of the cache disk cluster can be defined as the time required for the current data server to read a certain amount of data from the cache disk cluster. In this application, the time required to read 100KB of data is taken as the network latency of the cache disk cluster.

[0086] Specifically, on the one hand, the current network latency of the cache disk cluster can be determined by real-time detection, which means sending a short message to the cache disk cluster and receiving feedback from the cache disk cluster, and calculating the current network latency of the cache disk cluster based on the time required from sending the message to receiving the feedback.

[0087] Of course, the above method will increase the burden on the cache disk cluster. Therefore, a second method can be used, which is to estimate the current network latency of the cache disk cluster based on the network latency of the most recent times data was read from the cache disk cluster.

[0088] For specific calculation methods, please refer to the embodiments in the following embodiments of this application.

[0089] S204. Determine whether the current network latency of the cache disk cluster is greater than the first latency threshold.

[0090] If the current network latency of the cache disk cluster is greater than the first latency threshold, then proceed to step S205;

[0091] If the current network latency of the cache disk cluster is less than or equal to the first latency threshold, then proceed to step S208.

[0092] Specifically, the first latency threshold can be set based on the maximum data transfer rate allowed for each solid-state drive (SSD) in the cache disk cluster. For example, if the maximum data transfer rate supported by the SSDs in the cache disk cluster is 3.6 GB / s, then the first latency threshold can be set to 400 ms. If the maximum data transfer rate of the SSDs in the cache disk cluster is increased, then the first latency threshold can be increased accordingly.

[0093] S205. Based on the preset correspondence between network latency and reading frequency, determine the theoretical reading frequency corresponding to the current network latency.

[0094] Optionally, the correspondence between network latency and reading frequency described in step S205 can be represented as a table recording multiple network latency values ​​and corresponding reading frequency values; or it can be represented as a formula for calculating the reading frequency based on the network latency values.

[0095] If the above correspondence is recorded in a table, then step S205 can directly look up the table to determine the reading frequency corresponding to the current network latency, which is the theoretical reading frequency. If the above correspondence is recorded in the form of a formula, then step S205 can use the formula to calculate the current network latency and obtain the corresponding theoretical reading frequency.

[0096] It should be noted that the read frequency mentioned in this application refers to the frequency at which the data server reads the target data from the cache hard disk cluster during a series of data reads. Correspondingly, the theoretical read frequency calculated in step S205 is equivalent to the optimal read frequency determined according to the above correspondence under the current network latency. In simpler terms, under the current network latency, in order to ensure that users can obtain the target data in a timely manner and to extend the lifespan of the mechanical hard disk cluster as much as possible, the data server should control the frequency of reading the target data from the cache hard disk cluster to be as close as possible to the corresponding theoretical read frequency.

[0097] Specifically, assuming that the theoretical read frequency is determined to be 90% in step S205, it means that the data server should read the target data from the cache hard disk cluster at a frequency of 90%. In other words, based on the current network latency, for every 100 data read processes performed by the data server, it should ensure that 90 of the data read processes are read from the cache hard disk cluster, while the other 10 data read processes are read from the mechanical hard disk cluster.

[0098] S206. Calculate the proportion of historical data read requests that read target data from the cache disk cluster in the historical read records to obtain the historical read frequency.

[0099] The historical read record includes K historical data read requests received before the current data read request, as well as a cluster of hard disks that provide the target data for each historical data read request.

[0100] K is a preset positive integer. Optionally, considering that the reading frequency is generally expressed as a percentage, K can be set to 100. That is, when executing step S206, the 100 most recently executed data reading processes are considered in chronological order, and the number of times the target data is read from the cache disk cluster during these 100 data reading processes is calculated. The result obtained is the aforementioned historical reading frequency.

[0101] In conjunction with step S205, after determining the theoretical read frequency, the data server should read the target data from the cache hard disk cluster at the frequency specified by the theoretical read frequency. In this case, the specific hard disk cluster from which to read the target data during this data read needs to be considered based on the data read process that has been performed before. The hard disk cluster to be read this time is determined based on the relationship between the theoretical read frequency and the historical read frequency, so as to control the frequency of reading data from the cache hard disk cluster.

[0102] It should be noted that the execution order of steps S206 and S205 can be interchanged. That is, after step S204 determines that the current network latency of the cache disk cluster is greater than the first latency threshold, step S206 can be executed first, followed by step S205.

[0103] S207. Determine whether the historical reading frequency is greater than the theoretical reading frequency.

[0104] If the historical reading frequency is less than or equal to the theoretical reading frequency, proceed to step S208.

[0105] If the historical read frequency is greater than the theoretical read frequency, proceed to step S209.

[0106] Understandably, in order to make the frequency of reading target data from the cache hard disk cluster close to the theoretical read frequency of step S205, when the historical read frequency is less than or equal to the theoretical read frequency, the frequency of reading target data from the cache hard disk cluster should be increased, so step S208 should be executed to read target data from the cache hard disk cluster; when the historical read frequency is greater than the theoretical read frequency, the frequency of reading target data from the cache hard disk cluster should be decreased, so step S209 should be executed to read target data from the mechanical hard disk cluster.

[0107] S208: Read target data directly from the cache disk cluster.

[0108] S209: Read target data directly from the mechanical hard disk cluster.

[0109] For details on how a data server reads specific data from any hard drive cluster, please see below. Figure 5 The corresponding embodiments are described below.

[0110] S210, Send the target data to the terminal device.

[0111] This application provides a data reading method. Upon receiving a data read request from a terminal device, if the target data is stored in a cache hard disk cluster, the current network latency of the cache hard disk cluster is determined, and it is judged whether the current network latency is greater than a preset first latency threshold. If the current network latency is less than or equal to the first latency threshold, the target data is read from the cache hard disk cluster through the cache hard disk server and forwarded to the terminal device. Within the allowable network latency range, this solution directly reads the target data from the cache hard disk cluster for all data read requests, thereby reducing the number of times the mechanical hard disk cluster is accessed and extending the lifespan of the mechanical hard disks.

[0112] Please refer to Figure 3 The following describes another specific implementation of step S205, based on the current network latency and the correspondence between network latency and reading frequency. It should be understood that the method described in this embodiment can be used as a specific implementation to completely replace... Figure 2 In the corresponding embodiment, step S205, thereby... Figure 2 The other steps in the corresponding method are combined to form a new data reading method.

[0113] S301. In order of network latency from low to high, compare the current network latency with the network latency recorded in the latency gradient table one by one.

[0114] Based on the example above, which supports a maximum data transfer rate of 3.6 GB / s for solid-state drives, a latency gradient table as shown in Table 1 below can be configured.

[0115] Table 1

[0116] Network latency Corresponding interval Read frequency 400ms Less than or equal to 400ms 100％ 420ms Greater than 400ms and less than or equal to 420ms 95％ 440ms Greater than 420ms and less than or equal to 440ms 90％ 460ms Greater than 440ms and less than or equal to 460ms 85％ 480ms Greater than 460ms and less than or equal to 480ms 80％ 500ms Greater than 480ms and less than or equal to 500ms 75％ 520ms Greater than 500ms and less than or equal to 520ms 70％ 540ms Greater than 520ms and less than or equal to 540ms 65％ 560ms Greater than 540ms and less than or equal to 560ms 60％ 580ms Greater than 560ms and less than or equal to 580ms 55％ 600ms Greater than 580ms and less than or equal to 600ms 50％ Greater than 600ms 50％

[0117] The latency gradient table is a table configured based on the maximum data transfer rate of the solid-state drives in the cache disk cluster. It records multiple network latencies at certain intervals, with each pair of adjacent network latencies forming an interval. By comparing the current network latency with the network latencies recorded in the latency gradient table, it can be determined which network latency the current network latency is greater than, less than, or equal to, thus determining the position of the current network latency in the latency gradient table.

[0118] The latency gradient table records the number of network latencies, the interval between network latencies, and the value of each network latency, which can be set according to the maximum data transfer rate supported by the solid-state drive.

[0119] It should be noted that if the current network latency corresponds to the last row of Table 1, which is greater than 600ms, the reading frequency can be further reduced according to the specific situation.

[0120] S302. Determine the position of the current network latency in the latency gradient table based on the comparison results, thereby determining the theoretical read frequency corresponding to the current network latency.

[0121] After executing step S301, the position of the current network latency in the latency gradient table can be determined, and then the reading frequency at that position can be found from Table 1. This reading frequency is the theoretical reading frequency corresponding to the current network latency.

[0122] The following example illustrates the method for determining the theoretical readout frequency described in this embodiment, and how this method is implemented. Figure 2 The execution process of the data reading method in the corresponding embodiment:

[0123] Assuming the current network latency is determined to be 516ms, using the method for determining the theoretical read frequency provided in this embodiment, and the latency gradient table shown in Table 1, the theoretical read frequency corresponding to the current network latency is determined to be 70%. Then, consider the number of times the target data was read from the cache disk cluster during the 100 data read processes that have been performed before this data read request.

[0124] Assuming that in the previous 100 data reads, 80 of them read the target data from the cache hard disk cluster, then the historical read frequency is determined to be 80%. In other words, the current frequency of reading the target data from the cache hard disk cluster is 80%. This frequency is greater than the theoretical read frequency corresponding to the current network latency. Therefore, for the current data read request, the data server reads the target data from the mechanical hard disk cluster.

[0125] Assuming that in the previous 100 data reads, 60 of them read the target data from the cache disk cluster, then the historical read frequency is determined to be 60%. In other words, the current frequency of reading the target data from the cache disk cluster is less than the theoretical read frequency corresponding to the current network latency. Therefore, the target data is read from the cache disk cluster.

[0126] Based on the data reading method implemented in this embodiment, when the current network latency of the cache hard disk cluster is too high, the data server will schedule some data reading requests to the mechanical hard disk server. Therefore, on the one hand, it can ensure that the terminal device will not have to wait for a long time to obtain the target data due to the excessive network latency of the cache hard disk cluster. On the other hand, it can also avoid significantly shortening the service life of the mechanical hard disk by reading data from the mechanical hard disk cluster only when necessary.

[0127] Figure 2 In the corresponding method, step S203 mentions that the current network latency of the cache disk cluster can be estimated based on the historical network latency of the cache disk cluster during the most recent data reads. The following section combines this with... Figure 4 This paper provides a method for estimating current network latency based on historical network latency as a reference.

[0128] S401, a cache hard disk cluster that records the network latency of each data read process in real time.

[0129] Specifically, for each data read process, the data server can record the time required from the start of reading data from the cache disk cluster to the completion of reading all the data required for this data read process from the cache disk cluster. Then, this time is divided by the amount of data read from the cache disk cluster in this process, and the result is the network latency of the cache disk cluster for this data read process.

[0130] For example, if the time required to read 100KB of data is used as a measure of network latency, and the time required for a data server to read 200KB from the cache disk cluster is approximately 600ms, then the network latency of the cache disk cluster during this data read process is 300ms.

[0131] S402. When it is necessary to calculate the current network latency, select the network latency of the most recent N data reading processes in chronological order.

[0132] Optionally, N can be set to 20. That is, when executing step S402, starting from the currently received data read request, the network latency of the data read process corresponding to the 1st to the 20th data read requests received in chronological order from the most recent to the oldest needs needs to be selected to obtain the network latency of the cache disk cluster for each data read in the 20 most recent data read processes.

[0133] Corresponding to the value of N mentioned above, when executing step S401, an array containing 20 elements can be created. When the array is empty, the first element is first determined as the current element. Then, each time a data reading process occurs, the current element is set as the network latency of the cache disk cluster during this data reading process, and then the next element is determined as the current element to record the network latency of the next data reading process.

[0134] In the above loop, if the current element is the last element of the array, that is, the 20th element, then the next element is returned to the first element of the array. This is equivalent to replacing the network latency of the earliest data read process recorded in the array with the network latency of the current data read process when the array is full.

[0135] S403. Select some or all of the aforementioned N network delays as the network delays to be calculated.

[0136] Firstly, in step S403, all of the aforementioned N network delays can be used as network delays to be calculated, and step S404 can be executed directly.

[0137] Secondly, in step S403, the M largest network delays among the aforementioned N network delays can be deleted, and the remaining NM network delays can be determined as the network delays to be calculated. M is a preset positive integer less than N, which can be set to 1 or other positive integers less than N.

[0138] When step S403 is executed in the second implementation method, it is equivalent to sorting the aforementioned N network delays by numerical value, with the larger ones first and the smaller ones last, and then selecting NM network delays one by one from the last one to the front as the network delays to be calculated.

[0139] The execution of step S403 based on the implementation of the second aspect mentioned above mainly takes into account the potential impact of solid-state drive failures and network fluctuations on network latency. The method provided in this application mainly considers the impact of the access frequency of the cache hard disk cluster on network latency. Therefore, it is necessary to delete records with excessive network latency in order to eliminate the interference of solid-state drive failures and network fluctuations.

[0140] S404. Calculate the average of all network delays to be calculated to obtain the current network delay.

[0141] Specifically, when executing step S404, first sum all the network delays to be calculated, and then divide the calculated sum by the number of network delays to be calculated. The result is the current network delay.

[0142] Please refer to Figure 5 The process of reading data from a cache hard disk cluster or a mechanical hard disk cluster involved in any embodiment of this application can be performed according to the method provided in the following embodiments:

[0143] S501, the data server sends a read request to the hard disk server.

[0144] The aforementioned hard disk server can be either the cache hard disk server mentioned earlier, or a mechanical hard disk server.

[0145] When the data server determines that it needs to read the complete target data requested by the terminal device from the cache disk cluster, the disk server involved in step S501 is the cache disk server, and the read request is the data read request sent by the terminal device to the data server.

[0146] When the data server determines that it needs to read the target data requested by the terminal device from the hard disk cluster, the hard disk server involved in step S501 is the hard disk server, and the read request is still the data read request sent by the terminal device to the data server.

[0147] S502, The data server receives the address information fed back by the hard disk server.

[0148] The address information mentioned in step S502 includes the following two aspects:

[0149] On one hand, this includes the network address of the hard drive that stores the data corresponding to the read request. For example, if the data server sends a read request to the mechanical hard drive server, then the address information returned by the mechanical hard drive server includes the network address of the mechanical hard drive in the mechanical hard drive cluster that stores the target data.

[0150] On the other hand, it includes the storage address of the data corresponding to the read request on the aforementioned hard drive. For example, the data server sends a read request to the cache hard drive server. After the cache hard drive server searches and determines that the target data is stored on solid-state drive H in the cache hard drive cluster, in order to read the target data from solid-state drive H, the aforementioned address information should carry the location of the target data stored on solid-state drive H, that is, the storage address of the data on the hard drive.

[0151] Optionally, in a distributed file system, each hard disk is typically divided into multiple data blocks, and data is stored in these data blocks. In this case, the aforementioned storage address can be represented by the block identifier of the data block that contains the target data.

[0152] S503: The data server reads data from the corresponding hard drive based on the address information.

[0153] In conjunction with step S502, the data server can first establish a connection with the hard drive (mechanical hard drive or solid-state drive) that stores the data to be read, based on the network address of the hard drive recorded in the address information. Then, it can use the storage address to find the data to be read in the hard drive, thereby completing the data reading operation.

[0154] In conjunction with the data reading method provided in any embodiment of this application, embodiments of this application also provide a data reading device, please refer to... Figure 6 The device includes:

[0155] The receiving unit 601 is used to receive data reading requests from the terminal device.

[0156] Determining unit 602, if the target data is stored in a cache disk cluster, is used to determine the current network latency of the cache disk cluster.

[0157] Here, target data refers to data that satisfies the data reading request.

[0158] The judgment unit 603 is used to determine whether the current network latency is greater than a preset first latency threshold.

[0159] Reading unit 604, if the current network latency is less than or equal to the first latency threshold, is used to read target data from the cache disk cluster through the cache disk server.

[0160] If the determining unit 602 determines that the target data is not stored in the mechanical hard disk cluster, the reading unit 604 reads the target data from the mechanical hard disk cluster through the mechanical hard disk server.

[0161] The sending unit 605 is used to send target data back to the terminal device.

[0162] The judgment unit 603 is also used for:

[0163] Based on the preset correspondence between network latency and reading frequency, determine the theoretical reading frequency corresponding to the current network latency;

[0164] The historical read frequency is obtained by calculating the proportion of historical data read requests that read target data from the cache disk cluster in the historical read records;

[0165] The historical read record includes K historical data read requests received before the current data read request and a hard disk cluster that provides target data for each historical data read request. K is a preset positive integer, which is optional and can be set to 100.

[0166] Determine whether the historical read frequency is greater than the theoretical read frequency.

[0167] Reading unit 604 is used for:

[0168] If the historical read frequency is greater than the theoretical read frequency, the target data is read from the hard disk cluster via the hard disk server.

[0169] If the historical read frequency is less than or equal to the theoretical read frequency, the target data is read from the cache disk cluster through the cache disk server.

[0170] Specifically, the relationship between network latency and read frequency can be expressed as follows: Figure 3 The corresponding embodiment shows the delay gradient table in Table 1. The judgment unit 603 can be based on... Figure 3 The corresponding implementation example determines the theoretical read frequency.

[0171] Optionally, when determining the current network latency of the cache disk cluster, the determining unit 602 is specifically used for:

[0172] Based on the time sequence of the received requests, retrieve the historical network latency corresponding to the previous N data read requests, starting from the current data read request.

[0173] Each data read request corresponds to a historical network latency, which refers to the network latency of the cache disk cluster when reading part or all of the data that satisfies the corresponding data read request from the cache disk cluster; N is a preset positive integer.

[0174] By comparing the N historical network delays obtained, the maximum historical network delay can be determined.

[0175] Calculate the average of N-1 historical network delays excluding the highest historical network delay to obtain the current network delay of the cache disk cluster.

[0176] Optionally, when determining the current network latency of the cache disk cluster, the determining unit 602 can also be used for:

[0177] Based on the time sequence of the received requests, retrieve the historical network latency corresponding to the previous N data read requests, starting from the current data read request.

[0178] Each data read request corresponds to a historical network latency, which refers to the network latency of the cache disk cluster when reading part or all of the data that satisfies the corresponding data read request from the cache disk cluster; N is a preset positive integer.

[0179] The current network latency of the cache disk cluster is obtained by calculating the average of the N historical network latencies.

[0180] When the reading unit 604 reads data from the corresponding hard disk cluster through the hard disk server, it is specifically used for:

[0181] Send a read request to the hard drive server;

[0182] Receive address information from the hard drive server; the address information carries the network address of the hard drive that stores the requested data, and the storage address of the requested data on the hard drive.

[0183] Data is read from the hard drive based on the address information.

[0184] The working principle of the data reading device provided in this embodiment can be referred to the data reading method provided in any embodiment of this application, and will not be repeated here.

[0185] This application also provides a storage medium for storing a program. When the stored program is executed, it is used to implement the data reading method provided in any embodiment of this application.

[0186] Those skilled in the art will be able to implement or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for reading data, characterized in that, include: Receive data read requests from terminal devices; If the target data is stored in a cache disk cluster, determine the current network latency of the cache disk cluster; wherein, the target data refers to the data that satisfies the data read request; Determine whether the current network latency is greater than a preset first latency threshold; If the current network latency is less than or equal to the first latency threshold, the target data is read from the cache disk cluster via the cache disk server; The target data is fed back to the terminal device; If the current network latency is greater than the first latency threshold, the theoretical reading frequency corresponding to the current network latency is determined according to the preset correspondence between network latency and reading frequency. The historical read frequency is obtained by calculating the proportion of historical data read requests that read target data from the cache disk cluster in the historical read record; wherein, the historical read record includes K historical data read requests received before the current data read request and the disk cluster that provides target data for each historical data read request; K is a preset positive integer; Determine whether the historical reading frequency is greater than the theoretical reading frequency; If the historical read frequency is greater than the theoretical read frequency, the target data is read from the hard disk cluster via the hard disk server. If the historical read frequency is less than or equal to the theoretical read frequency, the target data is read from the cache disk cluster through the cache disk server.

2. The reading method according to claim 1, characterized in that, Determining the current network latency of the cache disk cluster includes: Based on the time sequence of received requests, starting from the current data read request, the historical network latency corresponding to the previous N data read requests is sequentially obtained; wherein, each data read request corresponds to a historical network latency, the historical network latency refers to the network latency of the cache disk cluster when reading part or all of the data that satisfies the corresponding data read request from the cache disk cluster; N is a preset positive integer; The current network latency of the cache disk cluster is obtained by calculating the average of the N historical network latencies.

3. The reading method according to claim 1, characterized in that, Determining the current network latency of the cache disk cluster includes: Based on the time sequence of received requests, starting from the current data read request, the historical network latency corresponding to the previous N data read requests is sequentially obtained; wherein, each data read request corresponds to a historical network latency, the historical network latency refers to the network latency of the cache disk cluster when reading part or all of the data that satisfies the corresponding data read request from the cache disk cluster; N is a preset positive integer; Compare the N historical network delays obtained to determine the maximum historical network delay; Calculate the average of N-1 historical network delays other than the maximum historical network delay to obtain the current network delay of the cache disk cluster.

4. The reading method according to claim 1, characterized in that, If the target data is stored in a cache disk cluster, before determining the current network latency of the cache disk cluster, the method further includes: If the target data is not stored in the cache hard disk cluster, the target data is read from the hard disk cluster through the hard disk server.

5. The reading method according to claim 1, characterized in that, The step of reading the target data from the cache disk cluster via the cache disk server includes: Send the data read request to the cache disk server; The system receives the address information of the target data fed back by the cache hard disk server; wherein the address information of the target data carries the address of the solid-state drive storing the target data, and the address of the target data in the solid-state drive; Based on the address information of the target data, the target data is read from the solid-state drive storing the target data.

6. A data reading device, characterized in that, include: The receiving unit is used to receive data read requests from the terminal device; The determining unit, if the target data is stored in a cache disk cluster, is used to determine the current network latency of the cache disk cluster; wherein, the target data refers to the data that satisfies the data read request; The judgment unit is used to determine whether the current network latency is greater than a preset first latency threshold; The reading unit, if the current network latency is less than or equal to the first latency threshold, is used to read the target data from the cache disk cluster via the cache disk server; A sending unit is used to send the target data back to the terminal device; The determination unit is also used for: If the current network latency is greater than the first latency threshold, the theoretical reading frequency corresponding to the current network latency is determined according to the preset correspondence between network latency and reading frequency. The historical read frequency is obtained by calculating the proportion of historical data read requests that read target data from the cache disk cluster in the historical read record; wherein, the historical read record includes K historical data read requests received before the current data read request and the disk cluster that provides target data for each historical data read request; K is a preset positive integer; Determine whether the historical reading frequency is greater than the theoretical reading frequency; The reading unit is used for: If the historical read frequency is greater than the theoretical read frequency, the target data is read from the hard disk cluster via the hard disk server. If the historical read frequency is less than or equal to the theoretical read frequency, the target data is read from the cache disk cluster through the cache disk server.

7. The reading device according to claim 6, characterized in that, When determining the current network latency of the cache disk cluster, the determining unit is specifically used for: Based on the time sequence of received requests, starting from the current data read request, the historical network latency corresponding to the previous N data read requests is sequentially obtained; wherein, each data read request corresponds to a historical network latency, the historical network latency refers to the network latency of the cache disk cluster when reading part or all of the data that satisfies the corresponding data read request from the cache disk cluster; N is a preset positive integer; Compare the N historical network delays obtained to determine the maximum historical network delay; Calculate the average of N-1 historical network delays other than the maximum historical network delay to obtain the current network delay of the cache disk cluster.

8. A computer storage medium, characterized in that, Used to store a program, which, when executed, is used to implement the data reading method as described in any one of claims 1 to 5.

Images