Data processing method, device, data processing equipment and storage medium

Abstract

The present application discloses a data processing method, apparatus, data processing device, and storage medium, wherein the method includes: in response to a read event of replica data of target data, determining a set of candidate replica data from multiple replica data corresponding to the target data; if the storage system where multiple candidate replica data in the candidate replica data set are located is a storage system of a preset type, calculating the input / output load of the storage system where each candidate replica data is located; determining target replica data based on the input / output load of the storage system where each candidate replica data is located, and outputting the target replica data. Multiple replica data of the target data can be evenly scheduled, improving the resource utilization of each storage system, thereby improving the overall system performance.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, a data processing device, a data processing equipment and a computer storage medium. Background Art

[0002] Impala is an open source query system developed by Cloudera. It can provide Structured Query Language (SQL) and query data stored in the Hadoop Distributed File System (HDFS).

[0003] Data in HDFS is stored in blocks, and HDFS uses a replication mechanism to ensure high availability of data blocks. That is, an HDFS file can consist of multiple blocks, each of which can have multiple replicas. When Impala schedules and selects multiple replicas of a block, it can lead to resource imbalance and low resource utilization. Therefore, how to schedule and select replicas of blocks is a pressing issue. Summary of the Invention

[0004] Embodiments of the present invention provide a data processing method, apparatus, data processing device, and storage medium, which can evenly schedule multiple copies of target data, improve resource utilization of each storage system, and thus enhance overall system performance.

[0005] In one aspect, an embodiment of the present invention provides a data processing method, the data processing method comprising:

[0006] In response to a read event of the replica data of the target data, determining a candidate replica data set from the plurality of replica data corresponding to the target data;

[0007] If the storage systems where multiple candidate replica data in the candidate replica data set are located are storage systems of a preset type, calculating the input / output load of the storage system where each candidate replica data is located;

[0008] Target copy data is determined based on the input / output load of the storage system where each candidate copy data is located, and the target copy data is output.

[0009] In another aspect, an embodiment of the present invention provides a data processing device, comprising:

[0010] a determining unit, configured to determine, in response to a read event of a replica data of a target data, a candidate replica data set from a plurality of replica data corresponding to the target data;

[0011] a calculation unit, configured to calculate the input / output load of the storage system where each candidate replica data in the candidate replica data set is located, if the storage system where the plurality of candidate replica data in the candidate replica data set are located is a storage system of a preset type;

[0012] The output unit is configured to determine target replica data based on the input / output load of the storage system where each candidate replica data is located, and output the target replica data.

[0013] In another aspect, an embodiment of the present invention provides a data processing device, the data processing device comprising:

[0014] a processor adapted to implement one or more instructions; and

[0015] A computer storage medium storing one or more instructions adapted to be loaded by the processor and executed by the processor:

[0016] In response to a read event of the replica data of the target data, determining a candidate replica data set from the plurality of replica data corresponding to the target data;

[0017] If the storage systems where multiple candidate replica data in the candidate replica data set are located are storage systems of a preset type, calculating the input / output load of the storage system where each candidate replica data is located;

[0018] Target copy data is determined based on the input / output load of the storage system where each candidate copy data is located, and the target copy data is output.

[0019] In another aspect, an embodiment of the present invention provides a computer storage medium, wherein the computer storage medium stores computer program instructions, and the computer program instructions are loaded by a processor and execute the following steps:

[0020] In response to a read event of the replica data of the target data, determining a candidate replica data set from the plurality of replica data corresponding to the target data;

[0021] If the storage systems where multiple candidate replica data in the candidate replica data set are located are storage systems of a preset type, calculating the input / output load of the storage system where each candidate replica data is located;

[0022] Target copy data is determined based on the input / output load of the storage system where each candidate copy data is located, and the target copy data is output.

[0023] In an embodiment of the present invention, a data processing device determines a set of candidate replica data from a plurality of replica data corresponding to the target data in response to a read event of the replica data of the target data; if the storage system where the plurality of candidate replica data in the candidate replica data set are located is a storage system of a preset type, the input / output load of the storage system where each candidate replica data is located is calculated, and the target replica data is determined based on the input / output load of the storage system where each candidate replica data is located; if the storage system where the plurality of candidate replica data in the candidate replica data set are located is a storage system other than the preset type, the target replica data is determined based on the data reading information of the storage system where each candidate replica data is located. Since the embodiment of the present invention introduces the input / output load of the storage system where the candidate replica data is located, the selection weight of the replica data of the target data is determined not only by the data reading information of the storage system where the candidate replica data is located, but also by the input / output load of the storage system where the candidate replica data is located. Compared with the method of determining the selection weight of the copy data of the target data only by the data reading information of the storage system where the candidate copy data is located, the embodiment of the present invention combines the data reading information of the storage system where the candidate copy data is located and the data reading information of the storage system where the candidate copy data is located to determine the selection weight of the copy data of the target data. This can take into account the input / output performance of the local storage system, can evenly schedule multiple copies of the target data, improve the resource utilization of each storage system, and thus improve the overall system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 is a structural diagram of a data processing system provided by an embodiment of the present invention;

[0026] Figure 2 This is a flow chart of a data processing method provided by an embodiment of the present invention;

[0027] Figure 3 is another flow chart of a data processing method provided by an embodiment of the present invention;

[0028] Figure 4 is a flow chart of another data processing method provided by an embodiment of the present invention;

[0029] Figure 5is another flow chart of another data processing method provided by an embodiment of the present invention;

[0030] Figure 6 is a structural diagram of a data processing device provided by an embodiment of the present invention;

[0031] Figure 7 It is a structural diagram of a data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0033] With the advent of the big data era, vast amounts of data are generated daily in fields such as e-commerce, social networking, and scientific research. Traditional standalone systems are unable to address storage and data analysis challenges. To improve the storage efficiency of large amounts of data, distributed storage systems are often adopted. Current distributed storage systems generally utilize Hadoop storage technology, an open-source distributed system infrastructure. The distributed file system HDFS, based on Hadoop storage technology, is well-suited for storing large files, such as files reaching hundreds of megabytes, gigabytes, or even terabytes. To store large files, HDFS divides files into data blocks. Furthermore, to improve HDFS reliability, a replication mechanism allows each data block in HDFS to have multiple replicas. HDFS consists of a name node and multiple data nodes. When storing a file, the name node divides the file into multiple data blocks. Data nodes use the replication mechanism to generate replicas for each data block and store these replicas on the data node. When subsequently reading data from HDFS, multiple replicas corresponding to the target data can be scheduled and selected to determine the target replica.

[0034] To improve the efficiency of reading replica data, an embodiment of the present invention provides a data processing method. In this data processing method, when a data processing device needs to schedule and select multiple replica data corresponding to target data, the data processing device can determine a set of candidate replica data from the multiple replica data corresponding to the target data, and then execute a scheduling policy based on the types of storage systems where the multiple candidate replica data in the candidate replica data set are located, thereby determining the target replica data of the target data. This method can evenly schedule the multiple replica data of the target data, improve resource utilization of the storage system where each candidate replica data is located, and thus enhance the overall system performance of HDFS.

[0035] In one embodiment, the data processing method of the embodiment of the present invention can be used to schedule and select multiple copies of target data (i.e., data blocks) in the HDFS file system, thereby determining the target data copy corresponding to the target data. When the data processing method is used to schedule and select multiple copies of target data, the data processing method can be applied to the following situations: Figure 1 In the data processing system shown in FIG. Figure 1 As shown, the data processing system may include at least: a data processing device 11, a name node 12 and a data node 13. The data processing device 11 may be a terminal device or a server. The terminal devices include but are not limited to smart phones, tablet computers, laptops, wearable devices, desktop computers, and the like. The server may be an independent physical server, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, content delivery networks (CDN), middleware services, domain name services, security services, and big data and artificial intelligence platforms, and the like. The name node 12 may be a terminal device or a server for managing metadata (i.e., data blocks) in HDFS, which is not limited in the present invention. The data node 13 may include one or more servers, which may be, for example Figure 1 One server or a combination of multiple servers such as server 13a, server 13b or server 13c shown is used to manage replica data in HDFS.

[0036] It is understandable that the data processing device 11 can be a name node 12 or a data node 13 in HDFS, or a device independent of HDFS, such as Figure 1 The embodiment of the present invention does not limit this.

[0037] This data processing method can be applied to the Impala query system to query the replica data of each data block in HDFS. However, it is understood that as new business scenarios emerge, the data processing method provided by the embodiments of the present invention can also be applied to other scenarios, such as schedulers that use similar scheduling options for replica data, and the embodiments of the present invention are not limited to this.

[0038] See Figure 2 , is a flow chart of a data processing method proposed in an embodiment of the present invention. Figure 2 As shown, the data processing method includes steps S201-S203:

[0039] S201 : In response to a read event of replica data of target data, determine a candidate replica data set from a plurality of replica data corresponding to the target data.

[0040] The target data can refer to any of the data blocks mentioned above. Because HDFS uses a replication mechanism, each target data item has multiple replicas. For example, a target data item can have three replicas, five replicas, and so on.

[0041] Since each data node may include one or more storage systems, the multiple replicas corresponding to the target data may be located on one data node or multiple data nodes. Specifically, when the number of storage systems within a data node is greater than or equal to the number of replicas corresponding to the target data, the multiple replicas corresponding to the target data may be located on one data node. For example, assuming that a target data item has three replicas, and when data node A includes three storage systems, the three replicas corresponding to the target data may all be stored on data node A. When the number of storage systems within a data node is less than the number of replicas corresponding to the target data, the multiple replicas corresponding to the target data may be located on multiple data nodes. For example, assuming that a target data item has three replicas, and when data node A includes one storage system, data node B includes one storage system, and data node C includes one storage system, the three replicas corresponding to the target data may be stored on data node A, data node B, and data node C, respectively. For another example, when data node A includes one storage system and data node B includes two storage systems, the three replicas corresponding to the target data may be stored on data node A and data node B.

[0042] Among them, the reading event of the replica data of the target data can be expressed in various ways. In one embodiment, in a data processing device, the target data can be associated with a corresponding target page. When the data processing device receives a user operation on the target page, it determines that a reading event of the replica data of the target data is detected, so that in response to the reading event of the replica data of the target data, multiple replica data corresponding to the target data can be found from the data node, and a candidate replica data set can be determined from the multiple replica data corresponding to the target data. Among them, the target page includes multiple types. For example, the target page can be a user page within an application; for example, the target page can also be an embedded h5 page, and so on. User operations can include one or more operations such as clicking, sliding, long pressing, and double-clicking.

[0043] In another embodiment, the read event of the replica data of the target data may be that the data processing device receives a data read request, and the data read request carries the identification information of the target data. Specifically, the data processing device may determine that a read event of the replica data of the target data has been detected when receiving the above-mentioned data read request carrying the identification information of the target data, so that in response to the read event of the replica data of the target data, the data processing device may find multiple replica data corresponding to the target data from the data node according to the identification information of the target data, and determine a set of candidate replica data from the multiple replica data corresponding to the target data. Optionally, the identification information of the target data may be the unique identification information of the target data, which is used to uniquely identify the target data. For example, the identification information of the target data may be the name information of the target data, or it may be a hash value obtained by performing a hash operation on the name information of the target data. When the identification information of the target data is the unique identification information, the replica data of the target data can be accurately read. Optionally, the identification information of the target data may also be other information different from the unique identification information, such as date, category, etc.

[0044] S202 : If the storage systems where multiple candidate replica data in the candidate replica data set are located are storage systems of a preset type, then the input / output load of the storage system where each candidate replica data is located is calculated.

[0045] Since different storage systems have different loads, after the data processing device determines the candidate copy data, it needs to select the target copy data in the storage system with the smallest load from multiple candidate copy data based on the load of the storage system where the candidate copy data is located, so as to achieve system load balancing and improve the overall performance of the system.

[0046] In a specific implementation, if the storage system where multiple candidate copy data in the candidate copy data set are located is a storage system of a preset type, and the load of the storage system where the candidate copy data are located is related to the input / output load, the target copy data can be determined through steps S202 and S203; if the storage system where multiple candidate copy data in the candidate copy data set are located is a storage system other than the preset type, the target copy data can be directly determined from the candidate copy data set based on the data reading information of the storage system where the candidate copy data are located. Specifically, if the storage system where multiple candidate copy data in the candidate copy data set are located is a storage system other than the preset type, the candidate copy data in the storage system with the smallest data reading information can be used as the target copy data. Among them, the preset type of storage system may include but is not limited to the local storage system.

[0047] Since the load of the local storage system is not only related to the data reading information of the storage system, but also to the throughput information of the storage system. When the data reading information and / or throughput information of the local storage system are different, the load of the local storage system is different. Therefore, when the storage system where the candidate copy data in the candidate copy data set is located is a local storage system, the data processing device can determine the target copy data from the multiple candidate copy data in the candidate copy data set based on the data reading information and throughput information. Specifically, if the storage systems where the multiple candidate copy data in the candidate copy data set are located are storage systems of preset types, the data reading information and throughput information of the storage system where any candidate copy data is located are obtained, and then the input / output load of the storage system where any candidate copy data is located is calculated based on the data reading information and throughput information, so as to determine the target copy data from the candidate copy data set based on the input / output load of the storage system where each candidate copy data is located.

[0048] The data read information of any storage system is used to indicate the ratio between the first amount of data read and the total amount of data stored in any storage system. Optionally, when the total amount of data stored in each storage system is the same, the data read information of the storage system may include the first amount of data read in the storage system, and the first amount of data read in the storage system may be used to directly indicate the ratio between the first amount of data read and the total amount of data stored in the storage system. Optionally, when the total amount of data stored in each storage system is different, the data read information of the storage system may include the first amount of data read in the storage system and the total amount of data stored in the storage system, and the ratio between the first amount of data read and the total amount of data stored may be calculated based on the first amount of data read in the data read information and the total amount of data stored in the storage system. The total amount of data stored in the storage system may be in an order of magnitude of any unit, for example, the total amount of data stored in the storage system may be in kilobytes (KB), megabytes (MB), gigabytes (GB), or terabytes (TB), etc. The first amount of data read in the storage system is similar to the total amount of data stored in the storage system, and the first amount of data read in the storage system may also be in an order of magnitude of any unit.

[0049] The throughput information of each storage system is related to the category of the storage system. For the same type of storage system, different categories of storage systems may have different throughput information. For example, a local storage system may include a local solid-state drive (SSD), a local hard disk drive (HDD), and a cloud block storage (CBS). The throughput information of the local SSD is better than the throughput information of the CBS, and the throughput information of the CBS is better than the local HDD. The throughput information can refer to the maximum throughput of each category of storage system or the average throughput of each category of storage system.

[0050] The throughput information of each storage system category can be set based on experience or business needs. For example, the throughput information of a local SSD can be configured to 500 megabytes, the throughput information of a CBS can be configured to 350 megabytes, and the throughput information of a local HDD can be configured to 200 megabytes. The data processing device can directly obtain the throughput information of each storage system from the configuration information of each storage system, or the throughput information can be obtained by testing each storage system using third-party testing software.

[0051] In one embodiment, a data processing device can calculate the input / output load of a storage system where any candidate replica data among multiple candidate replica data in a candidate replica data set is located in the following manner. The data read information and throughput information of the storage system where any candidate replica data is located are obtained, and then the ratio of the candidate data read information to the throughput information is calculated. A constant factor is obtained, and the constant factor is multiplied by the ratio to obtain the input / output load of the storage system where any candidate replica data is located. The input / output load of the storage system where any candidate replica data is located can be calculated using the following expression:

[0052] Input / output load = (data read information * K) / T

[0053] K is used to represent a constant factor for adjusting the input / output load; T is used to represent the throughput information of each category of storage system.

[0054] After calculating the input / output load of the storage system where each candidate copy data in the candidate copy data set is located, the data processing device can determine the target copy data based on the input / output load of the storage system where each candidate copy data is located, that is, turn to execute step S203.

[0055] S203: Determine target replica data based on the input / output load of the storage system where each candidate replica data is located, and output the target replica data.

[0056] Specifically, the input / output loads of the storage systems where the candidate replica data are located are compared, and the candidate replica data in the storage system with the smallest input / output load is used as the target replica data.

[0057] In order to better describe the embodiments of the present invention, Figure 3 The data processing method of the present invention is further described. Figure 3 As shown, a data processing device receives a read event for a replica of target data and, in response to the read event for the replica of target data, determines a set of candidate replica data from the multiple replicas corresponding to the target data. The device then determines the preset types of the multiple candidate replicas in the candidate replica set. If the storage systems where the multiple candidate replicas in the candidate replica set reside are of the preset type, the target replica data is determined based on the input / output load of the storage systems where the respective candidate replicas reside. If the storage systems where the multiple candidate replicas in the candidate replica set reside are other than the preset type, the target replica data is determined from the candidate replica set based on data read information from the storage systems where the respective candidate replicas reside. Finally, the data processing device outputs the target replica data.

[0058] In an embodiment of the present invention, a data processing device determines a set of candidate replica data from a plurality of replica data corresponding to the target data in response to a read event of the replica data of the target data; if the storage system where the plurality of candidate replica data in the candidate replica data set are located is a storage system of a preset type, the input / output load of the storage system where each candidate replica data is located is calculated, and the target replica data is determined based on the input / output load of the storage system where each candidate replica data is located; if the storage system where the plurality of candidate replica data in the candidate replica data set are located is a storage system other than the preset type, the target replica data is determined based on the data reading information of the storage system where each candidate replica data is located. Since the embodiment of the present invention introduces the input / output load of the storage system where the candidate replica data is located, the selection weight of the replica data of the target data is determined not only by the data reading information of the storage system where the candidate replica data is located, but also by the input / output load of the storage system where the candidate replica data is located. Compared with the method of determining the selection weight of the copy data of the target data only by the data reading information of the storage system where the candidate copy data is located, the embodiment of the present invention combines the data reading information of the storage system where the candidate copy data is located and the data reading information of the storage system where the candidate copy data is located to determine the selection weight of the copy data of the target data. This can take into account the input / output performance of the local storage system, can evenly schedule multiple copies of the target data, improve the resource utilization of each storage system, and thus improve the overall system performance.

[0059] See above Figure 2 It can be seen from the description of the method embodiment shown that Figure 2 The data processing method shown can schedule and select multiple replicas of target data. However, in order to further ensure the speed at which the data processing device reads the target replica data, the data processing device can first determine a set of candidate replica data based on the read speed of the storage system where the replica data is located, and then determine the target replica data based on the load of the storage system where the candidate replica data in the candidate replica data set is located. Based on this, an embodiment of the present invention also provides another data processing method, see Figure 4 , the data processing method includes S401-S405:

[0060] S401 , in response to a read event of replica data of target data, obtaining a plurality of replica data corresponding to the target data.

[0061] In one embodiment, some feasible implementations included in step S401 can be found in Figure 2 The description of the relevant steps in the embodiment will not be repeated here.

[0062] S402 : Determine a candidate copy data set according to a reading speed of a storage system where each copy data among a plurality of copy data corresponding to the target data is located.

[0063] Since the speed at which a data processing device reads any replica data corresponding to the target data is related to the read speed of the storage system where the replica data resides, in order to ensure that the data processing device reads the target replica data quickly enough, in one embodiment, after obtaining multiple replica data corresponding to the target data, the data processing device needs to compare the read speeds of the storage systems where each of the multiple replica data corresponding to the target data resides, and add the replica data in the storage system with the fastest read speed to the candidate replica data set, so that the target replica data subsequently determined from the candidate replica data set has a faster read speed. For example, the multiple replica data corresponding to the target data are replica data A, replica data B, replica data C, replica data D, and replica data E. The storage systems where replica data A, replica data B, and replica data C reside are all high-speed storage systems (cache), the storage system where replica data D resides is a local storage system (local), and the storage system where replica data E resides is a remote storage system (remote). The relationship between cache, local, and remote in terms of read speed from fastest to slowest is: cache > local > remote. Therefore, replica data A, replica data B, and replica data C can be added to the candidate replica data set. It should be understood that the storage systems where the multiple replica data corresponding to the target data are located, and the relationship between the reading speeds of the storage systems where the respective replica data are located, are merely exemplary. In other implementations, the storage systems where the multiple replica data corresponding to the target data are located may also be other situations. For example, the storage system where replica data A and replica data B are located is cache1, the storage system where replica data C is located is cache2, the storage system where replica data D is located is local, and the storage system where replica data E is located is remote. The relationship among cache1, cache2, local, and remote in terms of reading speed from fastest to slowest is: cache1>cache2>local>remote; therefore, replica data A and replica data B can be added to the candidate replica data set.

[0064] In another embodiment, after obtaining multiple copy data corresponding to the target data, the data processing device compares the reading speed of the storage system where each copy data in the multiple copy data corresponding to the target data is located with a first threshold, and adds the copy data in the storage system with a reading speed greater than the first threshold to the candidate copy data set, so that the speed of the target copy data subsequently determined from the candidate copy data set is faster.

[0065] In one embodiment, the candidate replica data set determined by the data processing device from the multiple replica data corresponding to the target data may include one or more candidate replica data. When the candidate replica data set includes one candidate replica data, the candidate replica data may be directly used as the target replica data. When the candidate replica data set includes multiple candidate replica data, step S403 is executed.

[0066] S403: If the storage systems where multiple candidate replica data in the candidate replica data set are located are storage systems of a preset type, then obtain data reading information and throughput information of the storage systems where each candidate replica data is located.

[0067] S404 , calculating the input / output load of the storage system where each candidate copy data is located according to the data reading information and throughput information of the storage system where each candidate copy data is located.

[0068] In one embodiment, some feasible implementations of step S403 and step S404 can be found in Figure 2 The description of the relevant steps in the embodiment will not be repeated here.

[0069] S405 : Determine the target replica data based on the input / output load of the storage system where each candidate replica data is located.

[0070] The data processing device can determine the candidate replica data of the storage system with the smallest input / output load as the target replica data. In one embodiment, if there is only one storage system with the smallest input / output load, the candidate replica data in the storage system with the smallest input / output load is used as the target replica data. For example, the candidate replica data set includes three candidate replica data, namely candidate replica data A, candidate replica data B, and candidate replica data C, and the storage system where candidate replica data A is located is storage system A, the storage system where candidate replica data B is located is storage system B, and the storage system where candidate replica data C is located is storage system C. Based on the data read information and throughput information, it is calculated that: the input / output load of storage system A is greater than the input / output load of storage system B and the input / output load of storage system C, and the input / output load of storage system C is greater than the input / output load of storage system B. Then, it can be determined that storage system B is the storage system with the smallest input / output load. Therefore, candidate replica data B can be determined as the target replica data. In another embodiment, if there are multiple storage systems with the smallest input / output load, a polling strategy can be used to select any one of the storage systems with the smallest input / output load from the multiple storage systems with the smallest input / output load, and the candidate copy data in any one of the storage systems with the smallest input / output load is used as the target copy data, that is, a polling strategy is used to select any one of the target data copies from the multiple target copy data, and the selected one of the target data copies is output. Continuing with the above example, if the input / output load of storage system A is greater than the input / output load of storage system B and the input / output load of storage system C, and the input / output load of storage system B is equal to the input / output load of storage system C, a polling strategy can be used to output the candidate copy data B in storage system B or the candidate copy data C in storage system C.

[0071] In order to better describe the embodiments of the present invention, Figure 5 The data processing method of the present invention is further described. Figure 5As shown, the data processing device receives a read event for the replica data of the target data, and the data processing device obtains multiple replica data corresponding to the target data in response to the read event for the replica data of the target data. Then, the candidate replica data set is determined based on the reading speed of the storage system where each replica data in the multiple replica data corresponding to the target data is located. Then, the type of the multiple candidate replica data in the candidate replica data set is determined. If the storage system where the multiple candidate replica data in the candidate replica data set is located is a storage system of a preset type (for example, a local storage system), the target replica data is determined based on the input / output load of the storage system where each candidate replica data is located; if the storage system where the multiple candidate replica data in the candidate replica data set is located is a storage system other than the preset type, the target replica data is determined from the candidate replica data set based on the data reading information of the storage system where each candidate replica data is located. Finally, the data processing device outputs the target replica data according to the polling strategy.

[0072] In an embodiment of the present invention, a data processing device obtains multiple replica data corresponding to the target data in response to a read event of the replica data of the target data, and determines a set of candidate replica data based on the read speed of the storage system where each replica data in the multiple replica data corresponding to the target data is located; and then executes different strategies to determine the target replica data based on the type of the storage system where each replica data in the candidate replica data set is located. Since the embodiment of the present invention takes into account the read speed of the storage system where the multiple replica data of the target data are located when scheduling and selecting the multiple replica data of the target data, the replica data with a faster read speed can be scheduled and selected to ensure user experience. In addition, the embodiment of the present invention also determines the selection weight of the replica data of the target data in combination with the data read information of the storage system where the candidate replica data is located and the data read information of the storage system where the candidate replica data is located, which can evenly schedule the multiple replica data of the target data, improve the resource utilization of each storage system, and improve the overall system performance.

[0073] Based on the description of the above data processing method embodiment, the embodiment of the present invention further discloses a data processing device, which can be a computer program (including program code) running on the above-mentioned data processing device. The data processing device can execute Figure 2 or Figure 4 See the method shown in Figure 6 , the data processing device can run the following units:

[0074] A determining unit 601 is configured to determine, in response to a read event of a replica data of a target data, a candidate replica data set from a plurality of replica data corresponding to the target data;

[0075] A calculation unit 602 is configured to calculate the input / output load of the storage system where each candidate replica data in the candidate replica data set is located, if the storage system where the candidate replica data in the candidate replica data set is located is a storage system of a preset type;

[0076] The output unit 603 is configured to determine target replica data based on the input / output load of the storage system where each candidate replica data is located, and output the target replica data.

[0077] In one embodiment, the determining unit 601 determines a candidate replica data set from a plurality of replica data corresponding to the target data, including:

[0078] Obtaining a read speed of a storage system where each of the plurality of replica data corresponding to the target data is located;

[0079] The replica data in the storage system with the fastest reading speed is added to the candidate replica data set.

[0080] In another embodiment, the computing unit 602 is also used to determine the target copy data from the candidate copy data set based on the data reading information of the storage system where each candidate copy data is located if the storage system where multiple candidate copy data in the candidate copy data set is located is a storage system other than the preset type of storage system; the data reading information of any storage system is used to indicate the ratio between the first amount of data that has been read and the total amount of data stored in any storage system.

[0081] In another embodiment, the calculation unit 602 is configured to calculate the input / output load of the storage system where each candidate replica data is located, including:

[0082] Obtain data read information and throughput information of the storage system where any candidate replica data is located;

[0083] The input / output load of the storage system where any candidate copy data is located is calculated according to the data reading information and the throughput information.

[0084] In another embodiment, the calculation unit 602 is configured to calculate the input / output load of the storage system where any candidate replica data is located based on the data reading information and the throughput information, including:

[0085] calculating a ratio of the data reading information to the throughput information;

[0086] A constant factor is obtained, and the constant factor is multiplied by the ratio to obtain the input / output load of the storage system where any candidate copy data is located.

[0087] In another embodiment, the output unit 603 is configured to determine the target replica data based on the input / output load of the storage system where each candidate replica data is located, including:

[0088] Comparing the input / output loads of the storage systems where the candidate replica data are located;

[0089] The candidate copy data in the storage system with the smallest input / output load is used as the target copy data.

[0090] In another embodiment, the number of the target copy data includes multiple ones, and the output unit 603 is configured to output the target copy data including:

[0091] Use a polling strategy to select any target data copy from multiple target copy data;

[0092] Output any selected target data copy.

[0093] According to one embodiment of the present invention, Figure 2 or Figure 4 Each step involved in the method shown can be performed by Figure 6 The data processing apparatus shown in FIG. Figure 2 Step S201 shown is performed by Figure 6 The determination unit 601 shown in FIG is executed, and step S202 is performed by Figure 6 The calculation unit 602 shown in FIG is executed, and step S203 is performed by Figure 6 , and the output unit 603 shown in FIG. Figure 4 Steps S401 and S402 are shown by Figure 6 The determination unit 601 shown in FIG is executed, and steps S403 and S404 are performed by Figure 6 The calculation unit 602 shown in FIG is executed, and step S405 is performed by Figure 6 Executed by the output unit 603 shown in FIG.

[0094] In an embodiment of the present invention, a data processing device determines a set of candidate replica data from a plurality of replica data corresponding to the target data in response to a read event of replica data of the target data; if the storage system where the plurality of candidate replica data in the candidate replica data set are located is a storage system of a preset type, the input / output load of the storage system where each candidate replica data is located is calculated, and the target replica data is determined based on the input / output load of the storage system where each candidate replica data is located; if the storage system where the plurality of candidate replica data in the candidate replica data set are located is a storage system other than the preset type, the target replica data is determined based on the data read information of the storage system where each candidate replica data is located. Since the embodiment of the present invention introduces the input / output load of the storage system where the candidate replica data is located, the selection weight of the replica data of the target data is determined not only by the data read information of the storage system where the candidate replica data is located, but also by the input / output load of the storage system where the candidate replica data is located. Compared with the method of determining the selection weight of the copy data of the target data only by the data reading information of the storage system where the candidate copy data is located, the embodiment of the present invention combines the data reading information of the storage system where the candidate copy data is located and the data reading information of the storage system where the candidate copy data is located to determine the selection weight of the copy data of the target data. This can take into account the input / output performance of the local storage system, can evenly schedule multiple copies of the target data, improve the resource utilization of each storage system, and thus improve the overall system performance.

[0095] Based on the description of the above data processing method embodiment, the present invention also discloses a data processing device. Figure 7 The data processing device at least includes a processor 701, an input interface 702, an output interface 703 and a computer storage medium 704 which can be connected via a bus or other means.

[0096] The computer storage medium 704 is a memory device in a data processing device, which is used to store programs and data. It is understandable that the computer storage medium 704 here can include both the built-in storage medium of the data processing device and, of course, the extended storage medium supported by the data processing device. The computer storage medium 704 provides a storage space, which stores the operating system of the data processing device. In addition, one or more instructions suitable for being loaded and executed by the processor 701 are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed RAM memory; optionally, it can also be at least one computer storage medium away from the aforementioned processor. The processor can be called a central processing unit (CPU), which is the core and control center of the data processing device, suitable for implementing one or more instructions, specifically loading and executing one or more instructions to realize the corresponding method flow or function.

[0097] In one embodiment, the processor 701 may load and execute one or more instructions stored in the computer storage medium 704 to implement execution. Figure 2 or Figure 4 In the specific implementation of the steps involved in the corresponding method shown in FIG, one or more instructions in the computer storage medium 704 are loaded by the processor 701 and the following steps are executed:

[0098] In response to a read event of the replica data of the target data, determining a candidate replica data set from the plurality of replica data corresponding to the target data;

[0099] If the storage systems where multiple candidate replica data in the candidate replica data set are located are storage systems of a preset type, calculating the input / output load of the storage system where each candidate replica data is located;

[0100] Target copy data is determined based on the input / output load of the storage system where each candidate copy data is located, and the target copy data is output.

[0101] In one embodiment, the processor 701 determines a candidate replica data set from a plurality of replica data corresponding to the target data, including:

[0102] Obtaining a read speed of a storage system where each of the plurality of replica data corresponding to the target data is located;

[0103] The replica data in the storage system with the fastest reading speed is added to the candidate replica data set.

[0104] In another embodiment, the processor 701 is also used to determine the target copy data from the candidate copy data set based on the data reading information of the storage system where each candidate copy data is located if the storage system where multiple candidate copy data in the candidate copy data set is located is a storage system other than the preset type of storage system; the data reading information of any storage system is used to indicate the ratio between the first amount of data that has been read and the total amount of data stored in any storage system.

[0105] In another embodiment, the processor 701 is configured to calculate the input / output load of the storage system where each candidate replica data is located, including:

[0106] Obtain data read information and throughput information of the storage system where any candidate replica data is located;

[0107] The input / output load of the storage system where any candidate copy data is located is calculated according to the data reading information and the throughput information.

[0108] In another embodiment, the processor 701 calculates the input / output load of the storage system where any candidate replica data is located according to the data reading information and the throughput information, including:

[0109] calculating a ratio of the data reading information to the throughput information;

[0110] A constant factor is obtained, and the constant factor is multiplied by the ratio to obtain the input / output load of the storage system where any candidate copy data is located.

[0111] In another embodiment, the processor 701 determines the target replica data based on the input / output load of the storage system where each candidate replica data is located, including:

[0112] Comparing the input / output loads of the storage systems where the candidate replica data are located;

[0113] The candidate copy data in the storage system with the smallest input / output load is used as the target copy data.

[0114] In another embodiment, the number of the target copy data includes multiple, and the processor 701 is configured to output the target copy data including:

[0115] Use a polling strategy to select any target data copy from multiple target copy data;

[0116] Output any selected target data copy.

[0117] In an embodiment of the present invention, a data processing device determines a set of candidate replica data from a plurality of replica data corresponding to the target data in response to a read event of the replica data of the target data; if the storage system where the plurality of candidate replica data in the candidate replica data set are located is a storage system of a preset type, the input / output load of the storage system where each candidate replica data is located is calculated, and the target replica data is determined based on the input / output load of the storage system where each candidate replica data is located; if the storage system where the plurality of candidate replica data in the candidate replica data set are located is a storage system other than the preset type, the target replica data is determined based on the data reading information of the storage system where each candidate replica data is located. Since the embodiment of the present invention introduces the input / output load of the storage system where the candidate replica data is located, the selection weight of the replica data of the target data is determined not only by the data reading information of the storage system where the candidate replica data is located, but also by the input / output load of the storage system where the candidate replica data is located. Compared with the method of determining the selection weight of the copy data of the target data only by the data reading information of the storage system where the candidate copy data is located, the embodiment of the present invention combines the data reading information of the storage system where the candidate copy data is located and the data reading information of the storage system where the candidate copy data is located to determine the selection weight of the copy data of the target data. This can take into account the input / output performance of the local storage system, can evenly schedule multiple copies of the target data, improve the resource utilization of each storage system, and thus improve the overall system performance.

[0118] It should be noted that an embodiment of the present invention further provides a computer program product or computer program, which includes computer program instructions stored in a computer-readable storage medium. The computer storage medium stores the computer program instructions, and the processor of the data processing device reads the computer program instructions from the computer-readable storage medium and executes the computer program instructions, so that the data processing device performs the above-mentioned data processing method embodiment. Figure 2 or Figure 4 The steps performed in .

[0119] The above disclosure is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A data processing method, characterized in that: include: In response to a read event of the replica data of the target data, determining a candidate replica data set from the plurality of replica data corresponding to the target data; If the storage systems where multiple candidate replica data in the candidate replica data set are located are storage systems of a preset type, calculating the input / output load of the storage system where each candidate replica data is located; Determine target replica data based on the input / output load of the storage system where each candidate replica data is located, and output the target replica data; The step of calculating the input / output load of the storage system where each candidate replica data is located includes: Obtaining data read information and throughput information of a storage system where any candidate replica data is located; wherein the data read information of the storage system is used to indicate a ratio between a first amount of data that has been read and a total amount of data stored in the storage system; The input / output load of the storage system where any candidate copy data is located is calculated according to the data reading information and the throughput information.

2. The method according to claim 1, wherein The determining of a candidate replica data set from a plurality of replica data corresponding to the target data includes: Obtaining a read speed of a storage system where each of the plurality of replica data corresponding to the target data is located; The replica data in the storage system with the fastest reading speed is added to the candidate replica data set.

3. The method according to claim 1, wherein The method further comprises: If the storage systems where multiple candidate copy data in the candidate copy data set are located are storage systems other than the preset type of storage system, the target copy data is determined from the candidate copy data set based on data reading information of the storage systems where the respective candidate copy data are located.

4. The method according to claim 1, wherein The calculating the input / output load of the storage system where any candidate replica data is located according to the data reading information and the throughput information includes: calculating a ratio of the data reading information to the throughput information; A constant factor is obtained, and the constant factor is multiplied by the ratio to obtain the input / output load of the storage system where any candidate copy data is located.

5. The method according to claim 1, wherein The determining the target replica data based on the input / output load of the storage system where each candidate replica data is located includes: Comparing the input / output loads of the storage systems where the candidate replica data are located; The candidate copy data in the storage system with the smallest input / output load is used as the target copy data.

6. The method according to any one of claims 2 to 5, characterized in that The number of the target copy data includes multiple ones, and the outputting the target copy data includes: Use a polling strategy to select any target data copy from multiple target copy data; Output any selected target data copy.

7. A data processing device, characterized in that: include: a determining unit, configured to determine, in response to a read event of a replica data of a target data, a candidate replica data set from a plurality of replica data corresponding to the target data; a calculation unit, configured to calculate the input / output load of the storage system where each candidate replica data in the candidate replica data set is located, if the storage system where the plurality of candidate replica data in the candidate replica data set are located is a storage system of a preset type; an output unit, configured to determine target replica data based on the input / output load of the storage system where each candidate replica data is located, and output the target replica data; Wherein, the calculation unit is used to obtain data reading information and throughput information of the storage system where any candidate copy data is located; wherein, the data reading information of the storage system is used to indicate the ratio between the first amount of data read and the total amount of data stored in the storage system; and the input / output load of the storage system where any candidate copy data is located is calculated based on the data reading information and the throughput information.

8. A data processing device comprising an input interface and an output interface, characterized in that: Also includes: a processor adapted to implement one or more instructions; as well as, A computer storage medium storing one or more instructions, wherein the one or more instructions are suitable for being loaded by the processor and executing the data processing method according to any one of claims 1 to 6.

9. A computer storage medium, characterized in that The computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the data processing method according to any one of claims 1 to 6.

10. A computer program product comprising computer program instructions, characterized in that When the computer program instructions are executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

Images