A distributed data balancing method, device, equipment and storage medium

Abstract

The present application relates to a distributed data balancing method, device, equipment and storage medium. It includes counting the data files to be balanced distributed in all storage nodes, determining the file feature vector corresponding to the data files to be balanced according to the attributes of each data file to be balanced; classifying the data files to be balanced based on the file feature vector to obtain multiple data file categories; in response to the balancing trigger operation, calculating the target storage space consumed by the data files to be balanced under the same data file category in each storage node, and migrating the data files to be balanced of the same data file category between the storage nodes according to the difference in target storage space between each storage node, until the difference in storage space occupied by the data files to be balanced of the same data file category in each storage node is less than the preset difference threshold. The use of this method can ensure that the data files on each node in the distributed file data storage system can be evenly distributed according to the rules.

Description

Technical Field

[0001] The present application relates to the technical field of distributed storage systems, and in particular to a distributed data balancing method, apparatus, device and storage medium. Background Art

[0002] With the development of science and technology, the current storage of massive data is mostly distributed storage of large quantities of data files. In a distributed storage system, data consistency, reliability, and balance are all important features. For example, if multiple files with high access frequencies are stored in a storage node or storage medium, when business programs frequently access specified files, the bandwidth utilization rate of this node or storage medium will be much higher than that of other nodes, resulting in a performance bottleneck in the entire distributed data system, and then performance degradation.

[0003] Currently, there are many ways to balance massive data. One is to balance the storage system, that is, to weight the storage nodes and then balance the file data to the corresponding nodes. Another is to compare the usage of the storage unit with the average usage of the entire storage system. If the usage of some storage units is higher than the average, the data in them will be migrated to the storage units below the average. However, the former has a large overall limitation and cannot be used in general databases or distributed file systems. The latter may still lead to the aggregation of hot data, which will create bottlenecks in the system.

[0004] Currently, no effective solution has been proposed for the problem of low data balance in distributed systems. Summary of the invention

[0005] Based on this, it is necessary to provide a distributed data balancing method, device, equipment and storage medium to address the above technical problems.

[0006] In a first aspect, the present application provides a distributed data balancing method. The method comprises:

[0007] Counting the data files to be balanced distributed in all preset storage nodes, and determining the file feature vectors corresponding to the data files to be balanced according to the attributes of each data file to be balanced;

[0008] Classify the data files to be balanced based on the file feature vectors to obtain multiple data file categories;

[0009] In response to the balancing trigger operation, the target storage space consumed by the data files to be balanced under the same data file category in each storage node is calculated, and based on the difference in target storage space between the storage nodes, the data files to be balanced under the same data file category are migrated between the storage nodes until the difference in storage space occupied by the data files to be balanced under the same data file category in each storage node is less than a preset difference threshold.

[0010] In one embodiment, the data files to be balanced are classified based on the file feature vectors to obtain multiple data file categories, including:

[0011] Obtaining a preset attribute weight corresponding to the attribute;

[0012] The attributes are weighted based on the attribute weights, and the file feature vector is obtained through normalization calculation. The data files to be balanced are classified based on the file feature vector to obtain the data file category.

[0013] In one embodiment, the attributes further include static attributes and dynamic attributes; the data files to be balanced are classified based on the file feature vectors to obtain multiple data file categories, including:

[0014] Determine the corresponding dynamic attribute in the data file to be balanced, and update the dynamic attribute regularly through a preset update cycle; and adjust the attribute weight of the dynamic attribute according to a preset weight adjustment strategy to obtain an updated dynamic attribute weight;

[0015] Determining a preset static attribute weight corresponding to the static attribute;

[0016] Based on dynamic attributes and static attributes, combined with the corresponding dynamic attribute weights and static attribute weights, the data files to be balanced are reclassified to obtain updated data file categories, wherein the dynamic attribute represents that the corresponding attribute is a dynamically changing attribute, and the static attribute represents that the corresponding attribute is an attribute that has not changed dynamically within a preset time period threshold. The weight adjustment strategy is that the dynamic change frequency of the dynamic attribute is inversely proportional to the dynamic attribute weight.

[0017] In one embodiment, the triggering conditions of the balancing action include automatic triggering and manual triggering.

[0018] In one embodiment, the data files to be balanced are classified based on the file feature vectors to obtain multiple data file categories, including:

[0019] The similarity between each feature value in the file feature vector is calculated, and the file feature vector is classified according to the similarity calculation result to obtain the data file category, wherein the corresponding feature value is obtained according to the attribute.

[0020] In one embodiment, according to the difference in target storage space between each storage node, the to-be-balanced data files of the same data file category are migrated between the storage nodes until the difference in storage space occupied by the to-be-balanced data files of the same data file category in each storage node is less than a preset difference threshold, including:

[0021] Determine the distribution of the to-be-balanced data files of the same data file category in each storage node, and calculate the ideal distribution value of the to-be-balanced data files based on the distribution;

[0022] The target storage space consumed by the data files to be balanced in each storage node is compared with the ideal distribution value. After detecting that the difference between the target storage space consumed by the data files to be balanced in the storage node and the ideal distribution value is greater than the difference threshold, the target data files to be balanced that are higher than the ideal distribution value are migrated to the storage node with the smallest storage space. The above method is repeated until the difference in storage space occupied by the data files to be balanced of the same data category in each storage node is less than the difference threshold.

[0023] In a second aspect, the present application also provides a distributed data balancing device. The device includes:

[0024] An acquisition module is used to count the data files to be balanced distributed in all preset storage nodes, and determine the file feature vector corresponding to the data file to be balanced according to the attributes of each data file to be balanced;

[0025] A calculation module, used for classifying the data files to be balanced based on the file feature vectors to obtain multiple data file categories;

[0026] A generation module is used to respond to a balancing trigger operation, calculate the target storage space consumed by the data files to be balanced under the same data file category in each storage node, and migrate the data files to be balanced under the same data file category between the storage nodes according to the difference in target storage space between the storage nodes, until the difference in storage space occupied by the data files to be balanced under the same data file category in each storage node is less than a preset difference threshold.

[0027] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0028] Counting the data files to be balanced distributed in all preset storage nodes, and determining the file feature vectors corresponding to the data files to be balanced according to the attributes of each data file to be balanced;

[0029] Classify the data files to be balanced based on the file feature vectors to obtain multiple data file categories;

[0030] In response to the balancing trigger operation, the target storage space consumed by the data files to be balanced under the same data file category in each storage node is calculated, and based on the difference in target storage space between the storage nodes, the data files to be balanced under the same data file category are migrated between the storage nodes until the difference in storage space occupied by the data files to be balanced under the same data file category in each storage node is less than a preset difference threshold.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0032] Counting the data files to be balanced distributed in all preset storage nodes, and determining the file feature vectors corresponding to the data files to be balanced according to the attributes of each data file to be balanced;

[0033] Classify the data files to be balanced based on the file feature vectors to obtain multiple data file categories;

[0034] In response to the balancing trigger operation, the target storage space consumed by the data files to be balanced under the same data file category in each storage node is calculated, and based on the difference in target storage space between the storage nodes, the data files to be balanced under the same data file category are migrated between the storage nodes until the difference in storage space occupied by the data files to be balanced under the same data file category in each storage node is less than a preset difference threshold.

[0035] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0036] Counting the data files to be balanced distributed in all preset storage nodes, and determining the file feature vectors corresponding to the data files to be balanced according to the attributes of each data file to be balanced;

[0037] Classify the data files to be balanced based on the file feature vectors to obtain multiple data file categories;

[0038] In response to the balancing trigger operation, the target storage space consumed by the data files to be balanced under the same data file category in each storage node is calculated, and based on the difference in target storage space between the storage nodes, the data files to be balanced under the same data file category are migrated between the storage nodes until the difference in storage space occupied by the data files to be balanced under the same data file category in each storage node is less than a preset difference threshold.

[0039] The above-mentioned distributed data balancing method, device, equipment and storage medium first count the data files to be balanced distributed in all storage nodes, and determine the file feature vector corresponding to the data files to be balanced according to the attributes of each data file to be balanced. Then, the data files to be balanced are classified and processed based on the file feature vector to obtain multiple data file categories. Finally, in response to the balancing trigger operation, the target storage space consumed by the data files to be balanced under the same data file category in each storage node is calculated, and according to the difference in storage space between each storage node, the data files to be balanced of the same data category are migrated between the storage nodes until the difference in storage space occupied by the data files to be balanced of the same data file category in each storage node is less than the difference threshold. Through this application, data balancing between each underlying storage node can be performed according to the file category of the data file, and all data files are comprehensively considered, avoiding the migration of a single type of data block causing hot spots or the aggregation of similar data, and ensuring that the data files on each storage node in the distributed file data storage system can be evenly distributed according to the rules. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 An application environment diagram of a distributed data balancing method in one embodiment;

[0041] Figure 2 A schematic diagram of a process flow of a distributed data balancing method in one embodiment;

[0042] Figure 3 A schematic diagram of distributed data balancing results in one embodiment;

[0043] Figure 4 A structural block diagram of a distributed data balancing device in one embodiment;

[0044] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The distributed data balancing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. First, the data files to be balanced distributed in all storage nodes are counted, and the file feature vectors corresponding to the data files to be balanced are determined according to the attributes of each data file to be balanced, and then the data files to be balanced are classified and processed based on the file feature vectors to obtain multiple data file categories, and finally, in response to the balancing trigger operation, the target storage space consumed by the data files to be balanced under the same data file category in each storage node is calculated, and according to the difference in target storage space between each storage node, the data files to be balanced of the same data file category are migrated between the storage nodes until the difference in storage space occupied by the data files to be balanced of the same data file category in each storage node is less than the difference threshold. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 may be implemented as an independent distributed server or a distributed server cluster consisting of multiple servers.

[0047] In one embodiment, Figure 2 As shown, a distributed data balancing method is provided, which is applied to Figure 1 The distributed server in the example is used to illustrate, including the following steps:

[0048] Step S210 , counting the data files to be balanced distributed in all preset storage nodes, and determining the file feature vectors corresponding to the data files to be balanced according to the attributes of the data files to be balanced.

[0049] In practical applications, the data files to be balanced are usually distributed in multiple storage nodes, including but not limited to disks, various types of storage devices, etc. The data files to be balanced distributed in each storage node are counted, and the file feature vectors corresponding to the data files to be balanced are determined according to the attributes of each data file to be balanced, wherein the attributes of the above-mentioned data files to be balanced include but are not limited to file name, volume size, query frequency, creation time, average size of a single data, etc. It should be noted that in order to more reasonably achieve the balanced distribution of each file, in this embodiment, it is usually necessary to comprehensively determine the above-mentioned file feature vectors by combining multiple types of attributes, wherein the above-mentioned file feature vectors are composed of indicators of N dimensions, and in some preferred embodiments, the above-mentioned file feature vectors include multiple feature values.

[0050] Step S220: classify the data files to be balanced based on the file feature vectors to obtain multiple data file categories.

[0051] The data files to be balanced are classified based on the file feature vectors, and the basis of classification includes but is not limited to completing the classification based on the similarity of the file feature vectors; or performing clustering processing on the file feature vectors to complete the classification; or classifying the file feature vectors by training a complete classification model, etc. After completing the classification of the data files to be balanced, multiple file data categories are obtained, wherein it can be understood that the category results obtained by the machine learning algorithm are natural, and are not artificially specified or defined categories.

[0052] Step S230, in response to the balancing trigger operation, calculates the target storage space consumed by the data files to be balanced under the same data file category in each storage node, and migrates the data files to be balanced under the same data file category between the storage nodes according to the difference in target storage space between the storage nodes, until the difference in storage space occupied by the data files to be balanced under the same data file category in each storage node is less than a preset difference threshold.

[0053] Among them, when it is detected that the balancing action is triggered, the balancing operation of the data file is started. First, the target storage space consumed by files under the same data file category in each storage node is required, so that the difference in storage space consumed by files of the same category in different storage nodes can be obtained. It can be understood that due to the classification of data files in the above text, the volume sizes of data files under the same category are close to the same. In some preferred embodiments, the volume size difference of data files under the same category does not exceed 10GB. Therefore, according to the difference in target storage space between different storage nodes of the same data file category, the data files of the same category can be migrated between storage nodes until the difference in storage space occupied by the same category of data files to be balanced in each storage node is less than the difference threshold. Similarly, the balancing of data files is completed for each data file category through the above-mentioned balancing method, wherein the difference threshold is mostly 10% to 20% of the storage capacity. The above migration method includes but is not limited to, for the above same data file category, querying the storage node with the largest storage space occupied by files of this category and the node with the least storage space among all storage nodes, and calculating the difference in storage space of files of this category on these two nodes, migrating half of the difference in storage space from the storage node with the largest storage space to the node with the least storage space, repeating the above method until all storage nodes are traversed and the difference in storage space occupied by data files to be balanced of the same data file category in each storage node is less than the difference threshold; or, calculating the average value of storage space consumed by the same data file category on each storage node based on the number of storage nodes, and then migrating data files with excess average value on storage nodes with storage space occupied by this category higher than the average value to storage nodes with storage space lower than the average value, repeating the above method until the difference in storage space occupied by data files to be balanced of the same data file category in each storage node is less than the difference threshold, and so on. This embodiment does not impose too many restrictions on the migration method.

[0054] Through steps S210 to S230, the various attributes of the data files to be balanced are integrated to determine the corresponding file feature vectors, and the data files under the same category are migrated in each storage node until the difference in storage space occupied by the data files to be balanced of the same category in each storage node is close to the same, or less than the difference threshold. Through this application, real-time balancing can be achieved according to the category of the data files to be balanced and the usage of the storage nodes to ensure that the data files are evenly distributed on each storage node in the distributed file storage system, thereby improving the efficiency of data file query and use.

[0055] In some embodiments, the method further comprises:

[0056] Obtaining a preset attribute weight corresponding to the attribute;

[0057] The attributes are weighted based on the attribute weights, and the file feature vector is obtained through normalization calculation. The data files to be balanced are classified based on the file feature vector to obtain the data file category.

[0058] Specifically, the preset attribute weights are obtained. In this embodiment, most attributes have corresponding weights, such as the attribute weight of the file size is generally set to 2.0, and the attribute weight of the number of data items is generally set to 1.5. When calculating the file feature vector, it is necessary to comprehensively consider the above attribute weights, perform weighted processing on the data files to be balanced, and normalize them into the above file feature vector. Then, the data files to be balanced are classified according to the file feature vectors to obtain data file categories. In some preferred embodiments, a file category list can be obtained based on the data file category, as well as a file list containing the storage path of the data files to be balanced under each category. Through this embodiment, the attribute weights corresponding to the attributes are further defined, and a more accurate file feature vector can be calculated by comprehensively considering the attribute weights, thereby further improving the accuracy of subsequent file classification.

[0059] In some embodiments, the above attributes further include static attributes and dynamic attributes, and the above method further includes:

[0060] Determine the corresponding dynamic attribute in the data file to be balanced, and update the dynamic attribute regularly through a preset update cycle; and adjust the attribute weight of the dynamic attribute according to a preset weight adjustment strategy to obtain an updated dynamic attribute weight;

[0061] Determining a preset static attribute weight corresponding to the static attribute;

[0062] Based on dynamic attributes and static attributes, combined with the corresponding dynamic attribute weights and static attribute weights, the data files to be balanced are reclassified to obtain updated data file categories, wherein the dynamic attribute represents that the corresponding attribute is a dynamically changing attribute, and the static attribute represents that the corresponding attribute is an attribute that has not changed dynamically within a preset time period threshold. The weight adjustment strategy is that the dynamic change frequency of the dynamic attribute is inversely proportional to the dynamic attribute weight.

[0063] Specifically, every time a new file is created in the file system in this application, since some attributes of the file, such as file volume, last write time, access frequency, and total data volume, do not exist or will change frequently, this embodiment sets such unstable attribute indicators as dynamic attributes. Similarly, this embodiment sets relatively stable attribute indicators as static attributes. Further, for the above-mentioned dynamic attributes, since they will change frequently, it is necessary to preset an update cycle, and update the dynamic attributes regularly based on the update cycle, wherein the update cycle is mostly 24h, and then after the dynamic attributes are updated, according to the weight adjustment strategy, the attribute weight of the dynamic attribute is also updated to obtain the updated dynamic attribute weight. It can be understood that since the dynamic attributes will change frequently, the corresponding dynamic attribute weight also needs to be changed. In some preferred embodiments, the above-mentioned weight adjustment strategy is that for dynamic attributes that change more frequently or change to a greater extent, the corresponding dynamic attribute weight is reduced accordingly. In some extreme cases, the dynamic attribute weight can be discarded.

[0064] Similarly, a corresponding static attribute weight is set for a static attribute. The static attribute mostly refers to an attribute that has not changed within a preset time period threshold. The preset time period threshold is usually long and can be set to 72 hours. In summary, after the dynamic attribute and the dynamic attribute weight are updated, the data file to be balanced is reclassified in combination with the dynamic attribute weight and the static attribute weight to obtain an updated data file category, and the data file is balanced again according to the updated data file category. Through this embodiment, the dynamic attribute and the static attribute, and the adjustment strategy of the corresponding attribute weight are further defined, so that it can be more in line with the actual application scenario, and the accuracy of data classification can also be effectively improved.

[0065] In some of the embodiments, the triggering conditions of the above-mentioned balancing action include automatic triggering and manual triggering.

[0066] Specifically, the triggering of the balancing action in this application may include automatic triggering by the system or manual triggering by the user, wherein the timing of the automatic triggering includes, but is not limited to, adding a copy of the file (such as duplication), file recovery, excessive balance difference, etc. Similarly, manual triggering means that the user manually triggers the balancing of data files according to actual needs.

[0067] In some embodiments, the method further comprises:

[0068] The similarity between each feature value in the file feature vector is calculated, and the file feature vector is classified according to the similarity calculation result to obtain the data file category, wherein the corresponding feature value is obtained according to the attribute.

[0069] Specifically, the attributes of a data file include file name, a list of field names / types contained therein, the average size of a single piece of data, the total amount of data, query frequency and other information. After weighting the different attributes of each data file, a feature vector can be combined. The feature vector includes the feature value of the data file. Ultimately, the feature values ​​and file objects generated by each data file in the system are placed into a key-value pair collection.

[0070] In some embodiments, the method further comprises:

[0071] Determine the distribution of the to-be-balanced data files of the same data file category in each storage node, and calculate the ideal distribution value of the to-be-balanced data files based on the distribution;

[0072] The target storage space consumed by the data files to be balanced in each storage node is compared with the ideal distribution value. After detecting that the difference between the target storage space consumed by the data files to be balanced in the storage node and the ideal distribution value is greater than the difference threshold, the target data files to be balanced that are higher than the ideal distribution value are migrated to the storage node with the smallest storage space. The above method is repeated until the difference in storage space occupied by the data files to be balanced of the same data category in each storage node is less than the difference threshold.

[0073] Specifically, after the balancing action is triggered, the number and size of files on each storage node are first counted according to each data file category, and then the balance is determined for each category. In this embodiment, since the file size is a very important indicator, the files of the same category will be close to the same in size, so only the number of files needs to be counted. The ideal distribution value under each category is calculated, and the ideal distribution value can preferably be the average value of the distribution of each category in each storage unit. For each category that needs to be balanced, the files with a large number of files under this category in each storage node are migrated to the node with the least number. After the migration is completed, the number of files of this category in each node is counted again, and the files higher than the ideal distribution value are migrated to the node with the smallest space occupied by the target data file to be balanced this time. Repeat the above method until the difference in storage space occupied by the data files to be balanced of the same data category in each node is close to the same or less than the difference threshold.

[0074] The present application also provides a preferred embodiment of a distributed data balancing method.

[0075] First, regularly collect various attributes of each file, including file name, volume size, query frequency, creation time, last write time and other indicators, then combine these attribute information with preset attribute weights for weighted processing and normalize them into feature vectors. Among them, volume size is a very important indicator, especially the attribute of volume size needs to be weighted, so that the volume size of each file in the same category is close to the same (for example, the size difference does not exceed 10GB). Furthermore, if it is a structured data file, it is also necessary to collect information such as the name / type list of the contained fields, the average size of a single data item, and the total amount of data.

[0076] Furthermore, every time a new file is created in the file system, since the file size, last write time, total data volume and other information do not exist or change in real time, for such uncertain attribute indicators, the corresponding dynamic attribute weights are set to discard or reduce the weight, and a dynamic indicator dimension is added to indicate the attributes of the file. Correspondingly, for a stable file that has not had data written for a long time, a static indicator dimension can be added to indicate the attributes of the file.

[0077] Finally, when the balancing action is triggered, the number and size of files on each storage node are first counted for each category according to the file classification, and then the balance is determined for each category. Since file size is a very important indicator, files of the same category will be similar in size, so only the number of files needs to be counted, and then the files on the nodes with a large number of files under the same file category are migrated to the storage nodes with a small number of files until the number of files of the same category on each storage node is close to the same. After all categories of files are migrated, the current distributed file system will achieve a more comprehensive balance in terms of storage capacity, number of files, and access frequency. Figure 3 This is a schematic diagram of distributed data balancing results in one embodiment. It can be seen from the figure that different categories of data files, such as category A, category B, and category C, are evenly distributed in different storage devices / nodes, wherein different data files correspond to corresponding feature vectors.

[0078] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0079] Based on the same inventive concept, the embodiment of the present application also provides a distributed data balancing device for implementing the above-mentioned distributed data balancing method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more distributed data balancing device embodiments provided below can refer to the limitations of the distributed data balancing method above, and will not be repeated here.

[0080] In one embodiment, Figure 4 As shown, a distributed data balancing device is provided, including: an acquisition module, a calculation module and a generation module, wherein:

[0081] An acquisition module is used to count the data files to be balanced distributed in all preset storage nodes, and determine the file feature vector corresponding to the data file to be balanced according to the attributes of each data file to be balanced;

[0082] A calculation module, used for classifying the data files to be balanced based on the file feature vectors to obtain multiple data file categories;

[0083] A generation module is used to calculate the target storage space consumed by the data files to be balanced under the same data file category in each storage node in response to the balancing trigger operation, and migrate the data files to be balanced under the same data file category between the storage nodes according to the difference in target storage space between the storage nodes, until the difference in storage space occupied by the data files to be balanced under the same data file category in each storage node is less than a preset difference threshold.

[0084] Specifically, the acquisition module counts the data files to be balanced distributed in all storage nodes, and determines the file feature vectors corresponding to the data files to be balanced according to the attributes of each data file to be balanced. Then the calculation module classifies the data files to be balanced based on the file feature vectors to obtain multiple file categories. Finally, the generation module responds to the trigger operation, calculates the target storage space consumed by the data files to be balanced under the same data file category in each storage node, and migrates the data files to be balanced of the same data file category between the storage nodes according to the difference in target storage space between the storage nodes, until the difference in space occupied by the data files to be balanced of the same data file category in each storage node is less than the difference threshold.

[0085] Each module in the above-mentioned distributed data balancing device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0086] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to distributed data balancing. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a distributed data balancing method is implemented.

[0087] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0088] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0089] The data files to be balanced that are distributed in all preset storage nodes are counted, and the file feature vectors corresponding to the data files to be balanced are determined according to the attributes of each data file to be balanced; the data files to be balanced are classified based on the file feature vectors to obtain multiple data file categories; in response to the balancing trigger operation, the target storage space consumed by the data files to be balanced under the same data file category in each storage node is calculated, and according to the difference in target storage space between each storage node, the data files to be balanced of the same data file category are migrated between the storage nodes until the difference in storage space occupied by the data files to be balanced of the same data file category in each storage node is less than a preset difference threshold.

[0090] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0091] The data files to be balanced that are distributed in all preset storage nodes are counted, and the file feature vectors corresponding to the data files to be balanced are determined according to the attributes of each data file to be balanced; the data files to be balanced are classified based on the file feature vectors to obtain multiple data file categories; in response to the balancing trigger operation, the target storage space consumed by the data files to be balanced under the same data file category in each storage node is calculated, and according to the difference in target storage space between each storage node, the data files to be balanced of the same data file category are migrated between the storage nodes until the difference in storage space occupied by the data files to be balanced of the same data file category in each storage node is less than a preset difference threshold.

[0092] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0093] The data files to be balanced that are distributed in all preset storage nodes are counted, and the file feature vectors corresponding to the data files to be balanced are determined according to the attributes of each data file to be balanced; the data files to be balanced are classified based on the file feature vectors to obtain multiple data file categories; in response to the balancing trigger operation, the target storage space consumed by the data files to be balanced under the same data file category in each storage node is calculated, and according to the difference in target storage space between each storage node, the data files to be balanced of the same data file category are migrated between the storage nodes until the difference in storage space occupied by the data files to be balanced of the same data file category in each storage node is less than a preset difference threshold.

[0094] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0095] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0096] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A distributed data balancing method, characterized in that: The method comprises: Counting the data files to be balanced distributed in all preset storage nodes, and determining the file feature vectors corresponding to the data files to be balanced according to the attributes of the data files to be balanced; Classify the data files to be balanced based on the file feature vectors to obtain multiple data file categories; In response to a balancing trigger operation, the target storage space consumed by the data files to be balanced under the same data file category in each of the storage nodes is calculated, and according to the difference in the target storage space between the storage nodes, the data files to be balanced of the same data file category are migrated between the storage nodes until the difference in the storage space occupied by the data files to be balanced of the same data file category in each of the storage nodes is less than a preset difference threshold, including: determining the distribution of the data files to be balanced of the same data file category in each of the storage nodes, and calculating the ideal distribution value of the data files to be balanced based on the distribution; comparing the target storage space consumed by the data files to be balanced in each of the storage nodes with the ideal distribution value, and after detecting that the difference between the target storage space consumed by the data files to be balanced in the storage node and the ideal distribution value is greater than the difference threshold, migrating the target data files to be balanced that are greater than the ideal distribution value to the storage node with the smallest storage space, and repeating the above method until the difference in the storage space occupied by the data files to be balanced of the same data category in each of the storage nodes is less than the difference threshold.

2. The method according to claim 1, characterized in that The classifying process of the to-be-balanced data files based on the file feature vectors to obtain multiple data file categories includes: Obtaining a preset attribute weight corresponding to the attribute; The attributes are weighted based on the attribute weights, and the file feature vector is obtained through normalization calculation. The data files to be balanced are classified based on the file feature vector to obtain the data file category.

3. The method according to claim 2, characterized in that The attributes also include static attributes and dynamic attributes; The data files to be balanced are classified based on the file feature vectors to obtain multiple data file categories, including: Determine the dynamic attribute corresponding to the to-be-balanced data file, and regularly update the dynamic attribute according to a preset update cycle; and adjust the attribute weight of the dynamic attribute according to a preset weight adjustment strategy to obtain an updated dynamic attribute weight; Determining a preset static attribute weight corresponding to the static attribute; Based on the dynamic attribute and the static attribute, in combination with the corresponding dynamic attribute weight and the static attribute weight, the data file to be balanced is reclassified to obtain an updated data file category, wherein the dynamic attribute represents that the corresponding attribute is a dynamically changing attribute, and the static attribute represents that the corresponding attribute is an attribute that does not dynamically change within a preset time period threshold, and the weight adjustment strategy is that the dynamic change frequency of the dynamic attribute is inversely proportional to the dynamic attribute weight.

4. The method according to claim 1, characterized in that: The triggering conditions of the equalization triggering operation include automatic triggering and manual triggering.

5. The method according to claim 1, characterized in that The classifying process of the to-be-balanced data files based on the file feature vectors to obtain multiple data file categories includes: The similarity between each feature value in the file feature vector is calculated, and the file feature vector is classified according to the similarity calculation result to obtain the data file category, wherein the corresponding feature value is obtained according to the attribute.

6. A distributed data balancing device, characterized in that: The device comprises: An acquisition module, used for counting the data files to be balanced distributed in all preset storage nodes, and determining the file feature vector corresponding to the data file to be balanced according to the attribute of each data file to be balanced; A calculation module, used for classifying the data files to be balanced based on the file feature vectors to obtain multiple data file categories; A generation module is used to calculate the target storage space consumed by the data files to be balanced under the same data file category in each of the storage nodes in response to the balancing trigger operation, and migrate the data files to be balanced of the same data file category between the storage nodes according to the difference in the target storage space between the storage nodes until the difference in the storage space occupied by the data files to be balanced of the same data file category in each of the storage nodes is less than a preset difference threshold, including: determining the distribution of the data files to be balanced of the same data file category in each of the storage nodes, and calculating the ideal distribution value of the data files to be balanced based on the distribution; comparing the target storage space consumed by the data files to be balanced in each of the storage nodes with the ideal distribution value, and after detecting that the difference between the target storage space consumed by the data files to be balanced in the storage node and the ideal distribution value is greater than the difference threshold, migrating the target data files to be balanced that are greater than the ideal distribution value to the storage node with the smallest storage space, and repeating the above method until the difference in the storage space occupied by the data files to be balanced of the same data category in each of the storage nodes is less than the difference threshold.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images