Data processing method, device, equipment and computer readable storage medium

Abstract

The present disclosure relates to a data processing method, apparatus, device and computer-readable storage medium. The present disclosure determines a first shard replica set whose machine performance utilization rate is greater than a first threshold and a second shard replica set whose machine performance utilization rate is less than a second threshold from multiple shard replica sets through the performance value of each shard replica set in the multiple shard replica sets included in the shard cluster and the number of data units stored in each shard replica set. Furthermore, part of the data units stored in the first shard replica set are moved to the second shard replica set for storage, so that the resource utilization rate of multiple shard replica sets is more balanced, thereby avoiding resource shortage of some shard replica sets and waste of resources of some shard replica sets.

Description

Technical Field

[0001] The present disclosure relates to the field of information technology, and in particular to a data processing method, device, equipment, and computer-readable storage medium. Background Art

[0002] With the continuous development of information technology, people generate a large amount of data in their work and life, and usually store this large amount of data in a database, for example, a database based on distributed file storage (MongoDB).

[0003] For example, a MongoDB shard cluster is composed of shard replica sets of different specifications. However, shard replica sets of different specifications have different resource utilization rates in the MongoDB shard cluster, which results in resource shortages in some shard replica sets and waste of resources in some shard replica sets. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a data processing method, apparatus, device and computer-readable storage medium to make the resource utilization of multiple shard replica sets more balanced, thereby avoiding resource shortages in some shard replica sets while wasting resources in some shard replica sets.

[0005] In a first aspect, an embodiment of the present disclosure provides a data processing method, including:

[0006] Obtaining the performance value of each shard replica set in the plurality of shard replica sets included in the shard cluster and the number of data units stored in each shard replica set;

[0007] Determine a first shard replica set and a second shard replica set from the multiple shard replica sets according to the performance value of each shard replica set and the number of data units stored in each shard replica set, wherein the machine performance utilization rate of the first shard replica set is greater than a first threshold, and the machine performance utilization rate of the second shard replica set is less than a second threshold;

[0008] Move part of the data units stored in the first shard replica set to the second shard replica set for storage.

[0009] In a second aspect, an embodiment of the present disclosure provides a data processing device, including:

[0010] An acquisition module, used to acquire the performance value of each shard replica set in a plurality of shard replica sets included in the shard cluster, and the number of data units stored in each shard replica set;

[0011] a determination module, configured to determine a first shard replica set and a second shard replica set from the multiple shard replica sets according to the performance value of each shard replica set and the number of data units stored in each shard replica set, wherein the machine performance utilization rate of the first shard replica set is greater than a first threshold, and the machine performance utilization rate of the second shard replica set is less than a second threshold;

[0012] A moving module is used to move part of the data units stored in the first shard replica set to the second shard replica set for storage.

[0013] In a third aspect, an embodiment of the present disclosure provides a data processing device, including:

[0014] Memory;

[0015] Processor; and

[0016] Computer programs;

[0017] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method described in the first aspect.

[0019] The data processing method, device, equipment and computer-readable storage medium provided by the embodiments of the present disclosure determine the first shard replica set whose machine performance utilization rate is greater than the first threshold and the second shard replica set whose machine performance utilization rate is less than the second threshold from the multiple shard replica sets through the performance value of each shard replica set in the multiple shard replica sets included in the shard cluster and the number of data units stored in each shard replica set. Further, some data units stored in the first shard replica set are moved to the second shard replica set for storage, so that the resource utilization rate of the multiple shard replica sets is more balanced, thereby avoiding the shortage of resources in some shard replica sets and the waste of resources in some shard replica sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 A flow chart of a data processing method provided by an embodiment of the present disclosure;

[0023] Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0024] Figure 3 A flow chart of a data processing method provided by another embodiment of the present disclosure;

[0025] Figure 4 A schematic diagram of an application scenario provided by another embodiment of the present disclosure;

[0026] Figure 5 A schematic diagram of an application scenario provided by another embodiment of the present disclosure;

[0027] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure;

[0028] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0030] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0031] Generally, a shard cluster of a database (MongoDB) based on distributed file storage is composed of shard replica sets of different specifications. However, shard replica sets of different specifications have different resource utilization rates in a MongoDB shard cluster, which results in resource shortages in some shard replica sets and waste of resources in some shard replica sets. To address this problem, an embodiment of the present disclosure provides a data processing method, which is described below in conjunction with a specific embodiment.

[0032] Figure 1 This is a flow chart of a data processing method provided by an embodiment of the present disclosure. The specific steps of the method are as follows:

[0033] S101. Obtain the performance value of each shard replica set in a plurality of shard replica sets included in a shard cluster, and the number of data units stored in each shard replica set.

[0034] A MongoDB shard cluster can be composed of multiple shard replica sets of different specifications. Figure 2 As shown, physical machines 21, 22 and 23 can constitute a shard replica set, for example, recorded as shard replica set A. Physical machines 31, 32 and 33 can constitute a shard replica set, for example, recorded as shard replica set B. Physical machines 41, 42 and 43 can constitute a shard replica set, for example, recorded as shard replica set C. Taking shard replica set A as an example, physical machines 21, 22 and 23 can be three different physical devices, or can be three virtual machines in the same physical device. In some embodiments, the shard replica set can also be recorded as a shard replica cluster. It can be understood that the number of physical machines included in different shard replica sets can be the same or different.

[0035] In addition, the data stored in the MongoDB shard cluster can be recorded as the overall data, and multiple shard replica sets of different specifications in the MongoDB shard cluster can respectively store part of the overall data. For example, if the overall data is 0-300, shard replica set A can store 0-100 of the overall data, shard replica set B can store 101-200 of the overall data, and shard replica set C can store 201-300 of the overall data. The data stored on different physical machines in the same shard replica set can be the same. For example, taking shard replica set A as an example, the data stored in physical machine 21, physical machine 22, and physical machine 23 are the same. For example, physical machine 21 stores 0-100 of the overall data, and physical machine 22 and physical machine 23 store copies of 0-100 respectively.

[0036] In addition, the performance of different physical machines in the same shard replica set can be the same, for example, the performance of physical machine 21, physical machine 22 and physical machine 23 is the same. The performance here can specifically include but is not limited to central processing unit (CPU), memory, network bandwidth, disk, etc.

[0037] In addition, shard replica set A, shard replica set B, and shard replica set C can be shard replica sets of different specifications, that is, the performance of physical machines from different shard replica sets may be different, that is, the resource configuration of different shard replica sets can be different. For example, the resource configuration of shard replica set A is higher, that is, shard replica set A is a high-configuration shard replica set, and the performance of the physical machines included in shard replica set A is higher. The resource configuration of shard replica set B is lower, that is, shard replica set B is a low-configuration shard replica set, and the performance of the physical machines included in shard replica set B is lower.

[0038] In some embodiments, the MongoDB shard cluster may further include a configuration server, which may obtain the performance values ​​of shard replica set A, shard replica set B, and shard replica set C, respectively. The performance value may be recorded as Score, which may be an integer type (int), and Score is used to identify machine capabilities, machine performance, etc. The machine capabilities and machine performance here may be the performance described above. The stronger the machine capabilities and machine performance, the greater the value of Score.

[0039] Specifically, the performance values ​​of different physical machines in the same shard replica set can be the same. For example, physical machine 21, physical machine 22, and physical machine 23 have the same performance value. In addition, the performance values ​​of physical machines in different shard replica sets may be different. For example, the performance value of the physical machines in the low-configuration shard replica set is lower, and the performance value of the physical machines in the high-configuration shard replica set is higher. Among them, the performance value of the same shard replica set can be the sum of the performance values ​​of each physical machine in the multiple physical machines included in the shard replica set. For example, the performance value of shard replica set A can be the sum of the performance values ​​corresponding to physical machine 21, physical machine 22, and physical machine 23, respectively.

[0040] In addition, the physical machines in shard replica set A, shard replica set B, and shard replica set C can store data in the form of data units. The data unit can specifically be a data block (chunk). The sizes of data blocks stored in different shard replica sets can be the same or different. For example, the data blocks stored by the physical machines in the low-configuration shard replica set are smaller, and the data blocks stored by the physical machines in the high-configuration shard replica set are larger. Specifically, the configuration server can also obtain the number of data blocks stored in shard replica set A, shard replica set B, and shard replica set C respectively. Among them, the number of data units stored in the same shard replica set can be the sum of the number of data units stored in each of the multiple physical machines included in the shard replica set. For example, the number of data units stored in shard replica set A can be the sum of the number of data units stored in physical machine 21, physical machine 22, and physical machine 23 respectively.

[0041] S102. Determine a first shard replica set and a second shard replica set from the multiple shard replica sets according to the performance value of each shard replica set and the number of data units stored in each shard replica set, wherein the machine performance utilization rate of the first shard replica set is greater than a first threshold, and the machine performance utilization rate of the second shard replica set is less than a second threshold.

[0042] For example, the configuration server may determine the first shard replica set and the second shard replica set from shard replica set A, shard replica set B, and shard replica set C according to the performance values ​​corresponding to shard replica set A, shard replica set B, and shard replica set C, respectively, and the number of data units stored in shard replica set A, shard replica set B, and shard replica set C, respectively. The first shard replica set may be a shard replica set whose machine performance utilization rate is greater than a first threshold among shard replica set A, shard replica set B, and shard replica set C. The second shard replica set may be a shard replica set whose machine performance utilization rate is less than a second threshold among shard replica set A, shard replica set B, and shard replica set C. The machine performance utilization rate may be a ratio of the amount of data actually stored to the performance value. If the machine performance utilization rate of the first shard replica set is greater than the first threshold, it means that the first shard replica set is not capable of accommodating more data blocks. If the machine performance utilization rate of the second shard replica set is less than the second threshold, it means that the second shard replica set is still capable of accommodating more data blocks. Therefore, the first shard replica set can be the shard replica set that needs to migrate out data blocks among shard replica set A, shard replica set B, and shard replica set C, and the second shard replica set can be the shard replica set that can migrate in more data blocks among shard replica set A, shard replica set B, and shard replica set C.

[0043] S103: Move part of the data units stored in the first shard replica set to the second shard replica set for storage.

[0044] When the first shard replica set and the second shard replica set are determined, some or all of the data blocks stored in the first shard replica set can be migrated to the second shard replica set for storage. For example, when shard replica set A is the first shard replica set and shard replica set B is the second shard replica set, some or all of the data blocks stored in physical machine 21 can be migrated to physical machine 31 for storage. Furthermore, the data stored in physical machine 22 and physical machine 23 are kept synchronized with the remaining data stored in physical machine 21, and the data stored in physical machine 32 and physical machine 33 are kept synchronized with the remaining data stored in physical machine 31.

[0045] The disclosed embodiment determines a first shard replica set whose machine performance utilization rate is greater than a first threshold and a second shard replica set whose machine performance utilization rate is less than a second threshold from multiple shard replica sets through the performance value of each shard replica set in the multiple shard replica sets included in the shard cluster and the number of data units stored in each shard replica set. Further, some data units stored in the first shard replica set are moved to the second shard replica set for storage, so that the resource utilization rate of multiple shard replica sets is more balanced, thereby avoiding resource shortage of some shard replica sets and waste of resources of other shard replica sets.

[0046] On the basis of the above embodiment, according to the performance value of each shard replica set and the number of data units stored in each shard replica set, a first shard replica set and a second shard replica set are determined from the multiple shard replica sets, including: according to a first ratio of the performance value of each shard replica set to the total performance value of the multiple shard replica sets, and a second ratio of the number of data units stored in each shard replica set to the total number of data units stored in the multiple shard replica sets, the first shard replica set and the second shard replica set are determined from the multiple shard replica sets.

[0047] For example, the performance value of shard replica set A, i.e., Score, is recorded as s1, the performance value of shard replica set B is recorded as s2, and the performance value of shard replica set C is recorded as s3. The total performance value of shard replica set A, shard replica set B, and shard replica set C is recorded as S, where S = s1 + s2 + s3. The number of data blocks stored in shard replica set A is recorded as n1, the number of data blocks stored in shard replica set B is recorded as n2, and the number of data blocks stored in shard replica set C is recorded as n3. The total number of data blocks stored in shard replica set A, shard replica set B, and shard replica set C is recorded as N, where N = n1 + n2 + n3.

[0048] For shard replica set A, the ratio of s1 to S, i.e., s1 / S, and the ratio of n1 to N, i.e., n1 / N, can be calculated respectively. For shard replica set B, the ratio of s2 to S, i.e., s2 / S, and the ratio of n2 to N, i.e., n2 / N, can be calculated respectively. For shard replica set C, the ratio of s3 to S, i.e., s3 / S, and the ratio of n3 to N, i.e., n3 / N, can be calculated respectively. Among them, s1 / S, s2 / S, and s3 / S can be respectively recorded as the first ratio. n1 / N, n2 / N, and n3 / N can be respectively recorded as the second ratio. Specifically, the first shard replica set and the second shard replica set can be determined from shard replica set A, shard replica set B, and shard replica set C according to the first ratio and the second ratio.

[0049] Optionally, a second ratio of the number of data units stored in the first shard replica set to the total number of data units stored in the multiple shard replica sets is greater than a first ratio of the performance value of the first shard replica set to the total performance value of the multiple shard replica sets; and a second ratio of the number of data units stored in the second shard replica set to the total number of data units stored in the multiple shard replica sets is less than the first ratio of the performance value of the second shard replica set to the total performance value of the multiple shard replica sets.

[0050] For example, for the same shard replica set in shard replica set A, shard replica set B, and shard replica set C, if the second ratio corresponding to the shard replica set is greater than the first ratio corresponding to the shard replica set, the shard replica set may be the first shard replica set. If the second ratio corresponding to the shard replica set is less than the first ratio corresponding to the shard replica set, the shard replica set may be the second shard replica set.

[0051] For example, if n1 / N is greater than s1 / S, it means that there are many data blocks currently stored in shard replica set A, and some or all of the data blocks in shard replica set A need to be migrated to other shard replica sets. If n2 / N is less than s2 / S, it means that shard replica set B can still accommodate more data blocks. In this case, shard replica set A can be determined as the first shard replica set, and shard replica set B can be determined as the second shard replica set. Furthermore, some or all of the data blocks in shard replica set A are migrated to shard replica set B for storage.

[0052] The disclosed embodiment determines the first shard replica set and the second shard replica set from multiple shard replica sets by a first ratio of the performance value of each shard replica set to the total performance value of multiple shard replica sets, and a second ratio of the number of data units stored in each shard replica set to the total number of data units stored in multiple shard replica sets. This allows the distribution ratio of data units to be automatically balanced between multiple shard replica sets in a shard cluster, achieving the purpose of targeting the distribution ratio of data units in a specified shard replica set. This allows the resources of shard replica sets of different specifications in a shard cluster to be reasonably utilized.

[0053] Figure 3 This is a flow chart of a data processing method provided by another embodiment of the present disclosure. The specific steps of the method are as follows:

[0054] S301. Obtain the performance value of each shard replica set in a plurality of shard replica sets included in a shard cluster, and the number of data units stored in each shard replica set.

[0055] Specifically, the implementation method and specific principle of S301 and S101 are consistent, and will not be repeated here.

[0056] S302. Determine a first shard replica set and a second shard replica set from the multiple shard replica sets according to the performance value of each shard replica set and the number of data units stored in each shard replica set, wherein the machine performance utilization rate of the first shard replica set is greater than a first threshold, and the machine performance utilization rate of the second shard replica set is less than a second threshold.

[0057] Specifically, the implementation method and specific principle of S302 and S102 are consistent, and will not be repeated here.

[0058] S303. Determine the portion of data units to be moved in the first shard replica set based on a first ratio of the performance value of the first shard replica set to the total performance value of the multiple shard replica sets, and a second ratio of the number of data units stored in the first shard replica set to the total number of data units stored in the multiple shard replica sets.

[0059] For example, when shard replica set A is determined as the first shard replica set and shard replica set B is determined as the second shard replica set, the part of the data blocks or all of the data blocks to be moved in shard replica set A can be determined based on the first ratio s1 / S of the performance value s1 of shard replica set A to the total performance value S of multiple shard replica sets, and the second ratio n1 / N of the number of data blocks n1 stored in shard replica set A to the total number of data blocks N stored in multiple shard replica sets, so that after the part of the data blocks or all of the data blocks to be moved are migrated out of shard replica set A, the second ratio n1' / N of the number of remaining data blocks in shard replica set A to the total number of data blocks N stored in multiple shard replica sets is infinitely close to the first ratio s1 / S corresponding to shard replica set A.

[0060] S304: Move the part of the data units to be moved in the first shard replica set to the second shard replica set for storage.

[0061] Further, the part of the data blocks or all of the data blocks to be moved in the shard replica set A determined above are moved to the shard replica set B for storage.

[0062] Optionally, after the part of the data units to be moved in the first shard replica set is moved to the second shard replica set for storage, the difference between a second ratio of the number of data units remaining in the first shard replica set to the total number of data units stored in the multiple shard replica sets and a first ratio corresponding to the first shard replica set is less than or equal to a preset difference.

[0063] For example, after migrating some data blocks in shard replica set A to shard replica set B for storage, the number of data blocks remaining in shard replica set A is recorded as n1', and the second ratio of n1' to the total number N of data blocks stored in multiple shard replica sets is n1' / N. The first ratio corresponding to shard replica set A is s1 / S, and the difference between n1' / N and s1 / S is less than or equal to the preset difference.

[0064] Optionally, a second ratio of the number of data units stored in each shard replica set to the total number of data units stored in the multiple shard replica sets changes as the number of data units stored in each shard replica set changes.

[0065] For example, after migrating some data blocks in shard replica set A to shard replica set B for storage, the number of data blocks in shard replica set B increases to (n1-n1'), and accordingly, the number of data blocks stored in shard replica set B changes from the original n2 to (n2+n1-n1'), and the second ratio of the number of data blocks stored in shard replica set B to the total number of data blocks stored in multiple shard replica sets changes from the original n2 / N to (n2+n1-n1') / N. In other words, the second ratio of the number of data blocks stored in each shard replica set to the total number of data blocks stored in multiple shard replica sets changes with the change of the number of data blocks stored in each shard replica set.

[0066] In the embodiment of the present disclosure, the second ratio of the number of data units stored in each shard replica set to the total number of data units stored in multiple shard replica sets changes as the number of data units stored in each shard replica set changes. This allows the number of data units stored in each shard replica set to be automatically adjusted, thereby automatically balancing the distribution ratio of data units in each shard replica set. This improves the resource utilization of each shard replica set.

[0067] Based on the above embodiment, the shard cluster also includes a configuration server, which is used to store relevant information of each shard replica set, and the relevant information of each shard replica set includes a performance value of the shard replica set.

[0068] like Figure 4 As shown, the shard cluster also includes configuration servers 41, which can store relevant information of each shard replica set in shard replica set A, shard replica set B, and shard replica set C, wherein the relevant information of each shard replica set includes the performance value of the shard replica set. The relevant information of each shard replica set can specifically be the member information of each shard replica set. The member information of each shard replica set can be stored in the config.Shards table. Specifically, the relevant information of each shard replica set includes an int field, which is the Score as described above. The value range corresponding to the value of the int field can be 1-100. By default, the Score of each shard replica set is 1.

[0069] Optionally, the shard cluster further includes a router, which is used to connect the terminal and the configuration server, and the router is used to forward the modification instructions of the terminal to the configuration server, and the modification instructions are used to modify the performance value of at least one shard replica set among the multiple shard replica sets.

[0070] like Figure 5 As shown, the shard cluster may also include a router (mongos) 42, and the router 42 may connect the terminal 43 and the configuration server 41, wherein the terminal 43 specifically includes but is not limited to a smart phone, a PDA, a tablet computer, a wearable device with a display screen, a desktop computer, a laptop computer, an all-in-one machine, a smart home device, etc. The terminal 43 may send a modification instruction to the configuration server 41 through the router 42, and the modification instruction is specifically used to modify the performance value of at least one shard replica set among the multiple shard replica sets stored in the configuration server 41. Specifically, when the router 42 receives the modification instruction from the terminal 43, the router 42 may send the modification instruction to the configuration server 41, so that the configuration server 41 may modify the score corresponding to at least one shard replica set among shard replica set A, shard replica set B, and shard replica set C, for example, modify the score corresponding to the shard replica set specified by the terminal 43. Specifically, the modification instruction may be:

[0071] sh.updateScore({"shardId":scoreValue})

[0072] It is understandable that, when the score corresponding to at least one of shard replica sets A, shard replica set B, and shard replica set C changes, the first ratios s1 / S, s2 / S, and s3 / S described above may also change. Furthermore, in the process of performing migration and balancing of data blocks in each shard replica set according to the first ratio, the second ratios n1 / N, n2 / N, and n3 / N may also change, so that the distribution of data blocks between multiple shard replica sets can be automatically adjusted according to a preset ratio.

[0073] Specifically, a balanced migration mechanism may be provided in the configuration server 41, and the configuration server 41 may compare the distribution of data blocks between multiple shard replica sets and the performance values ​​of multiple shard replica sets through the balanced migration mechanism, for example, comparing the first ratio and the second ratio corresponding to the same shard replica set. Further, the shard replica set with the lowest machine performance utilization rate and the shard replica set with the highest machine performance utilization rate are determined based on the comparison result. The shard replica set with the lowest machine performance utilization rate is taken as the target, and the data blocks in the shard replica set with the highest machine performance utilization rate are moved to the target, that is, a data block migration (movechunk) operation is performed on the data blocks in the shard replica set with the highest machine performance utilization rate.

[0074] The disclosed embodiment stores the relevant information of each shard replica set in the configuration server, and adds the performance value of the shard replica set to the relevant information of each shard replica set. Furthermore, according to the performance value of each shard replica set and the number of data units stored in each shard replica set, dynamic adjustment of the data units stored in each shard replica set can be achieved. In addition, by modifying the performance value of at least one shard replica set in the multiple shard replica sets through modification instructions, the shard cluster can support the distribution ratio of data blocks in each shard replica set, so that the resources of multiple shard replica sets can be used evenly, avoiding the shortage of resources of low-configuration shard replica sets and the waste of resources of high-configuration shard replica sets. The shard cluster can support the distribution ratio of customized data units between shard replica sets, avoiding unreasonable resource utilization of shard replica sets of different specifications, thereby improving resource utilization.

[0075] Figure 6 The data processing device provided in the embodiment of the present disclosure can execute the processing flow provided in the data processing method embodiment, such as Figure 6 As shown, the data processing device 60 includes:

[0076] An acquisition module 61 is used to acquire a performance value of each shard replica set in a plurality of shard replica sets included in the shard cluster, and the number of data units stored in each shard replica set;

[0077] A determination module 62 is configured to determine a first shard replica set and a second shard replica set from the multiple shard replica sets according to the performance value of each shard replica set and the number of data units stored in each shard replica set, wherein the machine performance utilization rate of the first shard replica set is greater than a first threshold, and the machine performance utilization rate of the second shard replica set is less than a second threshold;

[0078] The moving module 63 is used to move part of the data units stored in the first shard replica set to the second shard replica set for storage.

[0079] Optionally, the determination module 62 is specifically configured to:

[0080] A first shard replica set and a second shard replica set are determined from the multiple shard replica sets based on a first ratio of the performance value of each shard replica set to the total performance value of the multiple shard replica sets, and a second ratio of the number of data units stored in each shard replica set to the total number of data units stored in the multiple shard replica sets.

[0081] Optionally, a second ratio of the number of data units stored in the first shard replica set to the total number of data units stored in the multiple shard replica sets is greater than a first ratio of the performance value of the first shard replica set to the total performance value of the multiple shard replica sets;

[0082] A second ratio of the number of data units stored in the second shard replica set to the total number of data units stored in the multiple shard replica sets is less than a first ratio of the performance value of the second shard replica set to the total performance value of the multiple shard replica sets.

[0083] Optionally, the determination module 62 is also used to: determine the part of the data units to be moved in the first shard replica set based on a first ratio of the performance value of the first shard replica set to the total performance value of the multiple shard replica sets, and a second ratio of the number of data units stored in the first shard replica set to the total number of data units stored in the multiple shard replica sets; the moving module 63 is specifically used to: move the part of the data units to be moved in the first shard replica set to the second shard replica set for storage.

[0084] Optionally, after the part of the data units to be moved in the first shard replica set is moved to the second shard replica set for storage, the difference between a second ratio of the number of data units remaining in the first shard replica set to the total number of data units stored in the multiple shard replica sets and a first ratio corresponding to the first shard replica set is less than or equal to a preset difference.

[0085] Optionally, a second ratio of the number of data units stored in each shard replica set to the total number of data units stored in the multiple shard replica sets changes as the number of data units stored in each shard replica set changes.

[0086] Optionally, the shard cluster further includes a configuration server, which is used to store relevant information of each shard replica set, and the relevant information of each shard replica set includes a performance value of the shard replica set.

[0087] Optionally, the shard cluster further includes a router, which is used to connect the terminal and the configuration server, and the router is used to forward the modification instructions of the terminal to the configuration server, and the modification instructions are used to modify the performance value of at least one shard replica set among the multiple shard replica sets.

[0088] Figure 6 The data processing device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0089] Figure 7 The data processing device provided in the embodiment of the present disclosure can execute the processing flow provided in the data processing method embodiment, such as Figure 7 As shown, the data processing device 70 includes: a memory 71, a processor 72, a computer program and a communication interface 73; wherein the computer program is stored in the memory 71 and is configured so that the processor 72 executes the data processing method as described above.

[0090] In addition, an embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. The computer program is executed by a processor to implement the data processing method described in the above embodiment.

[0091] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0092] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining the performance value of each shard replica set in the plurality of shard replica sets included in the shard cluster and the number of data units stored in each shard replica set; Determine a first shard replica set and a second shard replica set from the multiple shard replica sets according to the performance value of each shard replica set and the number of data units stored in each shard replica set, wherein the machine performance utilization rate of the first shard replica set is greater than a first threshold, and the machine performance utilization rate of the second shard replica set is less than a second threshold; Moving part of the data units stored in the first shard replica set to the second shard replica set for storage; Determining a first shard replica set and a second shard replica set from the multiple shard replica sets according to the performance value of each shard replica set and the number of data units stored in each shard replica set includes: A first shard replica set and a second shard replica set are determined from the multiple shard replica sets based on a first ratio of the performance value of each shard replica set to the total performance value of the multiple shard replica sets, and a second ratio of the number of data units stored in each shard replica set to the total number of data units stored in the multiple shard replica sets.

2. The method according to claim 1, characterized in that A second ratio of the number of data units stored in the first shard replica set to the total number of data units stored in the multiple shard replica sets is greater than a first ratio of the performance value of the first shard replica set to the total performance value of the multiple shard replica sets; A second ratio of the number of data units stored in the second shard replica set to the total number of data units stored in the multiple shard replica sets is less than a first ratio of the performance value of the second shard replica set to the total performance value of the multiple shard replica sets.

3. The method according to claim 1, characterized in that Moving part of the data units stored in the first shard replica set to the second shard replica set for storage includes: Determine the part of data units to be moved in the first shard replica set according to a first ratio of the performance value of the first shard replica set to the total performance value of the multiple shard replica sets, and a second ratio of the number of data units stored in the first shard replica set to the total number of data units stored in the multiple shard replica sets; Move the part of the data units to be moved in the first shard replica set to the second shard replica set for storage.

4. The method according to claim 3, characterized in that After moving the part of the data units to be moved in the first shard replica set to be stored in the second shard replica set, the difference between a second ratio of the number of data units remaining in the first shard replica set to the total number of data units stored in the multiple shard replica sets and a first ratio corresponding to the first shard replica set is less than or equal to a preset difference.

5. The method according to claim 1, characterized in that The second ratio of the number of data units stored in each shard replica set to the total number of data units stored in the multiple shard replica sets changes as the number of data units stored in each shard replica set changes.

6. The method according to claim 1, characterized in that The shard cluster also includes a configuration server, which is used to store relevant information of each shard replica set, and the relevant information of each shard replica set includes a performance value of the shard replica set.

7. The method according to claim 6, characterized in that The shard cluster also includes a router, which is used to connect the terminal and the configuration server, and the router is used to forward the modification instruction of the terminal to the configuration server, and the modification instruction is used to modify the performance value of at least one shard replica set among the multiple shard replica sets.

8. A data processing device, characterized in that: include: An acquisition module, used to acquire the performance value of each shard replica set in a plurality of shard replica sets included in the shard cluster, and the number of data units stored in each shard replica set; a determination module, configured to determine a first shard replica set and a second shard replica set from the multiple shard replica sets according to the performance value of each shard replica set and the number of data units stored in each shard replica set, wherein the machine performance utilization rate of the first shard replica set is greater than a first threshold, and the machine performance utilization rate of the second shard replica set is less than a second threshold; A moving module, used to move part of the data units stored in the first shard replica set to the second shard replica set for storage; The determination module is specifically used for: A first shard replica set and a second shard replica set are determined from the multiple shard replica sets based on a first ratio of the performance value of each shard replica set to the total performance value of the multiple shard replica sets, and a second ratio of the number of data units stored in each shard replica set to the total number of data units stored in the multiple shard replica sets.

9. The device according to claim 8, characterized in that A second ratio of the number of data units stored in the first shard replica set to the total number of data units stored in the multiple shard replica sets is greater than a first ratio of the performance value of the first shard replica set to the total performance value of the multiple shard replica sets; A second ratio of the number of data units stored in the second shard replica set to the total number of data units stored in the multiple shard replica sets is less than a first ratio of the performance value of the second shard replica set to the total performance value of the multiple shard replica sets.

10. The device according to claim 8, characterized in that The determination module is further used to: determine the part of the data units to be moved in the first shard replica set according to a first ratio of the performance value of the first shard replica set to the total performance value of the multiple shard replica sets, and a second ratio of the number of data units stored in the first shard replica set to the total number of data units stored in the multiple shard replica sets; The moving module is specifically used to move the part of the data units to be moved in the first shard replica set to the second shard replica set for storage.

11. The device according to claim 10, characterized in that After moving the part of the data units to be moved in the first shard replica set to be stored in the second shard replica set, the difference between a second ratio of the number of data units remaining in the first shard replica set to the total number of data units stored in the multiple shard replica sets and a first ratio corresponding to the first shard replica set is less than or equal to a preset difference.

12. The device according to claim 8, characterized in that The second ratio of the number of data units stored in each shard replica set to the total number of data units stored in the multiple shard replica sets changes as the number of data units stored in each shard replica set changes.

13. The device according to claim 8, characterized in that The shard cluster also includes a configuration server, which is used to store relevant information of each shard replica set, and the relevant information of each shard replica set includes a performance value of the shard replica set.

14. The device according to claim 13, characterized in that The shard cluster also includes a router, which is used to connect the terminal and the configuration server, and the router is used to forward the modification instruction of the terminal to the configuration server, and the modification instruction is used to modify the performance value of at least one shard replica set among the multiple shard replica sets.

15. A data processing device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images