A file storage method, apparatus and device for a Hadoop platform

By using a file popularity ranking calculation model and server cluster status analysis, the file storage method of the Hadoop platform is dynamically adjusted, solving the problems of difficult management and high maintenance costs of useless data, and achieving efficient data storage and resource utilization.

CN116226056BActive Publication Date: 2026-03-06XIAMEN YOUWEI TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202211724841.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-30
Publication Date
2026-03-06
Estimated Expiration
2042-12-30

AI Technical Summary

Technical Problem

Existing Hadoop platform file storage methods cannot effectively distinguish between useless data and data that has not been used for a long time, resulting in difficult data management and high maintenance costs. Furthermore, they cannot perform hierarchical storage, leading to a waste of computing resources.

Method used

The system automatically classifies and grades files based on their popularity rating. It dynamically adjusts the file storage distribution and the number of copies based on the server cluster status. The system uses the file popularity rating calculation model to calculate the number of file copies, and combines the remaining storage ratio of the server cluster with the file popularity weight to calculate the number of file copies. The system then groups and stores file copies based on server performance and historical operation records.

Benefits of technology

It enables dynamic adjustment of file storage while ensuring high data availability, reducing data maintenance costs, avoiding server resource waste and read/write efficiency impacts, and improving data management efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116226056B_ABST
    Figure CN116226056B_ABST
Patent Text Reader

Abstract

This paper relates to the field of computer technology, and in particular to a file storage method, apparatus, and device for the Hadoop platform. The method includes: calculating the popularity level of files to be stored using a file popularity level calculation model; calculating the number of file replicas for each popularity level based on the remaining storage ratio of the server cluster, the percentage of newly added data per unit time, and the file popularity weight corresponding to the popularity level of the files to be stored; grouping the servers in the server cluster into multiple server groups based on multiple performance indicators of each server and their historical file operation records; and storing file replicas of the files to be stored in the server groups according to their popularity levels. The embodiments described in this paper achieve automatic classification and grading based on file read / write activity, dynamically adjusting file storage distribution and the number of file replicas according to a strategy, thereby reducing enterprise operating costs while ensuring high data availability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This article relates to the field of computer technology, and in particular to a file storage method, apparatus and device for the Hadoop platform. Background Technology

[0002] Currently, various business processes generate massive amounts of data. Most technical fields use Hadoop as a distributed data storage and computing platform, storing data in file format. For data security and integrity, a file is typically copied multiple times, and each copy is further divided into multiple blocks. These blocks are then stored on one or more servers. In real-world environments, data is constantly being written, including both formal and temporary data, which are stored indiscriminately. Without effective storage management, useless or long-unused data will be lost in the sea of ​​data. Since data is valuable, it requires analysis and mining of massive amounts of data; simply deleting it is insufficient. Continuing in this way will increase data management and maintenance costs, placing a heavier burden on enterprises and necessitating continuous server purchases, further increasing operating costs.

[0003] How to effectively manage and efficiently store data is a problem faced by all enterprises. Currently, the common practice is to classify data manually or semi-automatically, and delete or reduce the number of copies of some unused data. This has the following problems: (1) How to determine whether data can be deleted, whether it is currently unused but may be needed in the future. (2) Degrading some data copies will still result in the data being distributed on high-performance servers, but the data will not be used for a long time, which will waste computing resources. (3) It requires constant manual sorting and cannot be dynamically automated.

[0004] There is an urgent need for a new file storage method for the Hadoop platform to solve the problems that existing file storage methods on the Hadoop platform cannot distinguish between useless or long-unused data, and cannot perform hierarchical storage of data, leading to increasingly difficult data management and higher maintenance costs. Summary of the Invention

[0005] To address the problems existing in the prior art, this embodiment provides a file storage method, apparatus, and device for the Hadoop platform. It realizes automatic classification and hierarchical classification based on file read and write status, dynamically adjusts file storage distribution and file copy number according to strategy, reduces data maintenance costs, and reduces enterprise operating costs while ensuring high data availability.

[0006] To solve the above-mentioned technical problems, the specific technical solution presented in this paper is as follows:

[0007] On the one hand, this embodiment provides a file storage method for the Hadoop platform, the method including,

[0008] The popularity level of the file to be stored is calculated using a file popularity level calculation model. The file popularity level calculation model is obtained by training multiple files and their popularity tags.

[0009] Based on the remaining storage ratio of the server cluster, the percentage of newly added data per unit time, and the file popularity weight corresponding to the popularity level of the file to be stored, the number of file copies of the file to be stored at multiple popularity levels is calculated respectively.

[0010] The servers in the server cluster are grouped according to multiple performance indicators of each server and their stored historical file operation records to obtain multiple server groups.

[0011] The file copies of the files to be stored are stored in the server group according to the stated popularity level.

[0012] Furthermore, based on the remaining storage ratio of the server cluster, the proportion of newly added data per unit time, and the file popularity weight corresponding to the popularity level of the file to be stored, the formula for calculating the number of file replicas for the file to be stored at multiple popularity levels is as follows:

[0013] dup_cnt=MAX(a,MIN([value],b));

[0014] value=PoRS / PoNAD_PT / St×DTW;

[0015] Wherein, dup_cnt represents the number of file replicas, MAX represents the maximum value function, MIN represents the minimum value function, [] represents the integer function, a represents the lower limit of the predetermined replicas, b represents the upper limit of the predetermined replicas, PoRS represents the remaining storage percentage, PoNAD_PT represents the percentage of new data per unit time, St represents the predetermined time, and DTW represents the file popularity weight, wherein the predetermined time is an integer multiple of the unit time corresponding to the percentage of new data per unit time.

[0016] Furthermore, the access frequency of the files to be stored corresponding to the higher popularity levels is higher than that of the files to be stored corresponding to the lower popularity levels.

[0017] Based on the remaining storage percentage of the server cluster, the percentage of newly added data per unit time, and the file popularity weight corresponding to the popularity level of the files to be stored, the number of file replicas for the files to be stored at multiple popularity levels is calculated, including...

[0018] Based on the remaining storage ratio of the server cluster, the percentage of newly added data per unit time, and the file popularity weight corresponding to the popularity level of the file to be stored, the number of file copies of the file to be stored for each of the multiple popularity levels whose popularity ranking exceeds the first preset threshold is calculated.

[0019] Furthermore, the servers in the server cluster are grouped based on multiple performance metrics of each server and their stored historical file operation records, resulting in multiple server groups, which further include:

[0020] The servers in the server cluster are sorted in descending order of performance according to the aforementioned performance metrics.

[0021] The files in the historical file operation records are grouped according to multiple predetermined file operation frequency intervals, and the number of the file operation frequency intervals is the same as the number of the server groups.

[0022] Calculate the ratio of the sum of the sizes of all files in each group to the total size of the files in the historical file operation records;

[0023] The servers are grouped according to the percentage to obtain the server groups.

[0024] Furthermore, the sorted servers are grouped according to the stated proportion, resulting in server groups that further include,

[0025] Calculate the storage space size corresponding to the percentage based on the percentage and the total storage space size of the server cluster;

[0026] Multiple servers are selected sequentially from the sorted server cluster to form the server group, wherein the sum of the storage space of all servers in the server group is close to the storage space size of the corresponding proportion of the server group, and the generation order of the multiple server groups is the same as the order of the average file operation frequency corresponding to the file operation frequency range from large to small.

[0027] Furthermore, sorting the servers in the server cluster according to the performance indicators in descending order of performance further includes,

[0028] Multiple performance metrics corresponding to any one server are used as multiple benchmark metrics.

[0029] For a given server, calculate the ratio of each performance metric to the corresponding benchmark metric, and calculate the server's overall score based on the obtained ratios.

[0030] The servers are sorted in descending order of their overall scores.

[0031] Furthermore, storing file copies of the files to be stored in the server group according to the stated popularity level further includes,

[0032] According to the sorting of the popularity levels, a predetermined number of file copies for each popularity level are stored in multiple servers of the server group corresponding to that popularity level, and the remaining unstored file copies for that popularity level are stored in multiple servers of the server group with the largest proportion of remaining storage space.

[0033] Furthermore, if the storage space of the server group corresponding to the predetermined number of file copies of the popularity level is insufficient, the method further includes,

[0034] Determine the next server group that has insufficient storage space based on the order of the server groups;

[0035] Storing the predetermined number of file copies in multiple servers corresponding to the next server group. Further, storing the predetermined number of file copies in multiple servers corresponding to the next server group further includes...

[0036] The servers corresponding to the next server group are sorted according to their remaining storage space and their overall score.

[0037] The predetermined number of file copies are stored sequentially on a number of servers that are sorted first.

[0038] Furthermore, storing the predetermined number of file copies in multiple servers corresponding to the next server group also includes,

[0039] In the next server group, select multiple servers whose remaining storage is greater than the second preset threshold value;

[0040] Determine whether the number of selected servers is greater than the number of file blocks in the predetermined number of replicas;

[0041] If so, the selected servers are sorted according to the comprehensive score.

[0042] The predetermined number of file copies are stored sequentially on a number of servers that are sorted first.

[0043] Furthermore, after calculating the ratio of the sum of the sizes of all files in each group to the total size of the files in the historical file operation records, the method further includes,

[0044] Using the stated percentage as the first percentage, multiple first percentages are weighted to obtain a second percentage corresponding to the operation frequency of each file;

[0045] Summing the multiple second percentages yields the total percentage;

[0046] The ratio of each second percentage to the total percentage is calculated to obtain a third percentage, so that the sorted servers can be grouped according to the third percentage.

[0047] On the other hand, this embodiment also provides a file storage device for a Hadoop platform, including,

[0048] The file popularity level calculation unit is used to calculate the popularity level of the file to be stored using the file popularity level calculation model. The popularity level calculation model is obtained by training multiple files and the popularity tags of the multiple files.

[0049] The file copy number calculation unit is used to calculate the number of file copies of a file to be stored for multiple popularity levels based on the remaining storage ratio of the server cluster, the proportion of newly added data per unit time, and the file popularity weight corresponding to the popularity level of the file to be stored.

[0050] The server grouping unit is used to group the servers in the server cluster according to multiple performance indicators of each server and their stored historical file operation records, so as to obtain multiple server groups.

[0051] The file copy storage unit is used to store file copies of the file to be stored in the server group according to the popularity level.

[0052] On the other hand, this embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above.

[0053] Finally, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0054] Using the embodiments described in this paper, the popularity level of the file to be stored is calculated using a file popularity level calculation model. Then, based on the current state of the server cluster used to store the file to be stored, the number of file copies for each popularity level is calculated. Compared with the method of setting the number of file copies for the file to be stored based on manual experience, this method determines the optimal number of file copies for each popularity level based on the current state of the server cluster, thereby avoiding excessive file copies causing server operating pressure or insufficient file copies affecting file read and write efficiency. To allocate copies of files with different popularity levels to servers with performance best suited to their respective levels, this application groups servers in the cluster based on multiple performance metrics and their historical file operation records. This results in multiple server groups, and copies of the files to be stored are then stored within these groups according to their popularity levels. This approach ensures that the most suitable server is assigned to each file copy based on its popularity level, avoiding issues such as inefficient file read / write operations and wasted server resources due to improper server allocation. Furthermore, it addresses the problem that existing Hadoop platform file storage methods cannot distinguish between useless and long-unused data, nor can they perform hierarchical data storage, leading to increasingly difficult data management and higher maintenance costs. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments or prior art described herein, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this article. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 The figure shown is a schematic diagram of an implementation system for a file storage method on the Hadoop platform according to an embodiment of this paper;

[0057] Figure 2 The diagram shown is a flowchart of a file storage method for the Hadoop platform according to an embodiment of this paper;

[0058] Figure 3 The following diagram illustrates the steps for partitioning server groups in an embodiment of this paper.

[0059] Figure 4 The figure shows the steps in this embodiment of the process of grouping the sorted servers according to their proportions to obtain server groups;

[0060] Figure 5 The following is a description of the steps in this embodiment to sort the servers in the server cluster in descending order of performance according to performance metrics.

[0061] Figure 6 The diagram illustrates the steps in this embodiment of storing file copies of the files to be stored in a server group according to their popularity level.

[0062] Figure 7 The figure shows the steps for storing files to be stored in an embodiment of this paper when the storage space of the server group corresponding to a predetermined number of file copies of a popularity level is insufficient.

[0063] Figure 8 The illustration shows the steps in this embodiment of the process of storing a predetermined number of file copies on multiple servers corresponding to the next server group.

[0064] Figure 9 The diagram illustrates the steps in this embodiment to store a predetermined number of file copies across multiple servers corresponding to the next server group.

[0065] Figure 10 The figure shows the steps for weighted calculation of the proportion in the embodiment of this article;

[0066] Figure 11 The diagram shown is a structural schematic of a file migration device implemented in this paper;

[0067] Figure 12 The diagram shown is a structural schematic of the computer device in the embodiment of this article.

[0068] [Explanation of Figure Markers]:

[0069] 101. File source end;

[0070] 102. File storage service;

[0071] 103. Server cluster;

[0072] 1101. Calculation unit for document popularity level;

[0073] 1102. Unit for calculating the number of document copies;

[0074] 1103. Server group division unit;

[0075] 1104. File copy storage unit;

[0076] 1202. Computer equipment;

[0077] 1204. Processing equipment;

[0078] 1206. Storage resources;

[0079] 1208. Drive mechanism;

[0080] 1210. Input / output module;

[0081] 1212. Input devices;

[0082] 1214. Output devices;

[0083] 1216. Presentation equipment;

[0084] 1218. Graphical User Interface;

[0085] 1220. Network interface;

[0086] 1222. Communication link;

[0087] 1224. Communication bus. Detailed Implementation

[0088] The technical solutions in the embodiments described below will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments described herein, and not all of the embodiments. Based on the embodiments described herein, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this document.

[0089] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0090] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0091] like Figure 1The diagram shows a schematic of the system structure for implementing a file storage method on a Hadoop platform according to an embodiment of this paper. It may include a file source 101, a file storage service 102, and a server cluster 103. The file storage service 102 may be a standalone server or an application deployed on the file source 101 or the server cluster 103. Preferably, the file storage service 102 described in this embodiment is an application deployed on the server cluster 103 and runs on the servers in the server cluster 103. The file storage service 102 can receive files to be stored sent by the file source 101 and then store them on the servers in the server cluster 103.

[0092] The file source 101 and the server cluster 103 can communicate via a network, which may include a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, and be connected to a website, user equipment (e.g., computing devices), and backend systems.

[0093] The file storage service 102 can periodically obtain the operating status of each server in the server cluster 103, such as the remaining storage ratio of the server cluster 103, the percentage of newly added data per unit time, and multiple performance indicators, and then perform file storage based on the multiple performance indicators of the server cluster 103. Optionally, the server in the file source 101 or the server cluster 103 can be a node of a cloud computing system (not shown in the figure), or each server can be a separate cloud computing system, including multiple computers interconnected by a network and working as a distributed processing system. This embodiment does not impose any limitations.

[0094] To address the problems existing in the prior art, this embodiment provides a file storage method for the Hadoop platform, which realizes automatic classification and hierarchical classification based on file read and write status, dynamically adjusts file storage distribution and the number of file replicas according to strategies, reduces data maintenance costs, and reduces enterprise operating costs while ensuring high data availability. Figure 2 The diagram shows a flowchart of a file storage method for a Hadoop platform according to an embodiment of this paper. This diagram illustrates the process of storing a file to be stored in the servers of a server cluster, but based on conventional or non-creative labor, it may include more or fewer operational steps. The order of steps listed in the embodiment is merely one possible execution order among many, and does not represent the only execution order. In actual system or device products, the methods shown in the embodiment or the accompanying drawings can be executed sequentially or in parallel. Specifically, as shown... Figure 2 As shown, the method can be executed by the file storage service 102 and may include:

[0095] Step 201: Calculate the popularity level of the file to be stored using the file popularity level calculation model. The file popularity level calculation model is obtained by training multiple files and their popularity tags.

[0096] Step 202: Calculate the number of file copies of the files to be stored for each of the following popularity levels based on the remaining storage ratio of the server cluster, the proportion of newly added data per unit time, and the file popularity weight corresponding to the popularity level of the files to be stored.

[0097] Step 203: Group the servers in the server cluster according to multiple performance indicators of each server and their stored historical file operation records to obtain multiple server groups;

[0098] Step 204: Store a copy of the file to be stored in the server group according to the popularity level.

[0099] The method described in this embodiment calculates the popularity level of the file to be stored using a file popularity level calculation model. Then, based on the current state of the server cluster used to store the file to be stored, the number of file copies for each popularity level is calculated. Compared to the method of setting the number of file copies for the file to be stored based on manual experience, this method determines the optimal number of file copies for each popularity level based on the current state of the server cluster. This avoids excessive file copies causing server operating pressure or insufficient file copies affecting file read and write efficiency. To allocate copies of files with different popularity levels to servers with performance best suited to their respective levels, this application groups servers in the cluster based on multiple performance metrics and their historical file operation records. This results in multiple server groups, and copies of the files to be stored are then stored within these groups according to their popularity levels. This approach ensures that the most suitable server is assigned to each file copy based on its popularity level, avoiding issues such as inefficient file read / write operations and wasted server resources due to improper server allocation. Furthermore, it addresses the problem that existing Hadoop platform file storage methods cannot distinguish between useless and long-unused data, nor can they perform hierarchical data storage, leading to increasingly difficult data management and higher maintenance costs.

[0100] In this embodiment, model training is first performed using multiple files and their popularity tags. All file operation records can be recorded through the HDFS gateway. File operations include file reading, file writing, and file creation. These operations are then written to a log file and stored in HDFS. The file record content includes file path, time, operation type, operation method, and operator. After collecting the file operation records, the data needs to be formatted and the data tags expanded. The formatted data is then stored in a relational database (MySQL). Specific information includes the file's parent path, the folder to which the file belongs, the operation time, the operator, the operation type (read, write), and the operation method. Then, feature calculations are performed on the formatted data to obtain as many rich features as possible to improve the accuracy of the trained model. The file path is parsed, and all parent directories of the file are calculated as independent features. The parent department of the operator is queried and used as a feature calculation. The operation time period is calculated based on the data operation time.

[0101] Then, the feature data is sampled, and a portion of the data is used as samples for analysis, file hot / cold classification, and labeling. In this embodiment, the label can be understood as the file popularity level. The frequency of file operations can be calculated based on historical operation records, and then the file popularity is classified according to the access frequency. The XCBOOST model is trained based on the processed sample data, and multiple file popularity levels are obtained as shown in Table 1.

[0102] Table 1

[0103] Popularity level describe LV1 I have been performing read and write operations every day for the past year. LV2 Read and write operations have been performed monthly over the past year, filtering out the parts that overlap with LV1. LV3 Read and write operations within the past year, filtering out the parts that overlap with LV1 and LV2. LV4 Read and write operations have been performed within the past two years; the parts that overlap with LV1, LV2, and LV3 have been filtered out. LV5 Long-term cold data that has not been read or written for more than 2 years

[0104] Then, files are categorized and tagged based on their popularity. There are two ways to tag files: 1. Manually setting file category tags by creating a whitelist; 2. Automatically setting file category tags using the XGBOOST model. Files tagged using the whitelist method will not participate in the subsequent automatic tag calculation by the algorithm model. If the model needs to calculate the category tags, the files must be removed from the whitelist.

[0105] After training the XGBOOST model (i.e., the file popularity level calculation model described in this embodiment), the model is used to calculate the popularity level of the file to be stored.

[0106] Then, based on the remaining storage ratio of the server cluster, the percentage of new data added per unit time, and the file popularity weight corresponding to the popularity level of the file to be stored, the number of file copies of the file to be stored for multiple popularity levels is calculated respectively.

[0107] In this embodiment, the remaining storage percentage of the server cluster is the ratio of the remaining storage space in the server cluster to the total storage space of the server cluster. The percentage of newly added data per unit time is the ratio of the size of newly added files to the total storage space of the server within a unit time. This unit time can be set according to actual needs, such as 1 day. The file popularity weight corresponding to the file popularity level can be set based on experience. The higher the file popularity level, the greater the corresponding file popularity weight. This ensures that when calculating the number of file replicas, the higher the file popularity level, the greater the number of file replicas. This ensures the operational efficiency of high-popularity files (e.g., parallel operation on multiple file replicas). The lower the file popularity level, the smaller the number of file replicas, thus avoiding the waste of storage space (e.g., files with low file operation frequency do not need too many file replicas). For example, the popularity level LV1 corresponds to a popularity level weight of 1, the popularity level LV2 corresponds to a popularity level weight of 0.8, the popularity level LV3 corresponds to a popularity level weight of 0.5, and so on.

[0108] Then, the servers are grouped. Since different servers have different performance levels, files that are frequently accessed should be stored on servers with higher performance to improve operational efficiency, while files that are not frequently accessed should be stored on servers with lower performance to avoid wasting server resources. Therefore, this embodiment of the paper groups the servers and stores the files to be stored according to different popularity levels based on the groups.

[0109] Furthermore, since the storage space size of each server may be different, in order to avoid wasting storage space, this embodiment of the invention also considers the operation frequency of historical files in the server when grouping the servers. For each operation frequency range, the required storage space size is calculated, and then multiple servers are allocated to meet the storage space requirements.

[0110] Finally, based on the popularity level, copies of the files to be stored are stored on servers within the server group.

[0111] According to one embodiment of this paper, the formulas for calculating the number of file copies of a file to be stored for multiple popularity levels are (1)-(2), based on the remaining storage ratio of the server cluster, the proportion of newly added data per unit time, and the file popularity weight corresponding to the popularity level of the file to be stored:

[0112] dup_cnt = MAX(a, MIN([value], b)) (1)

[0113] value = PoRS / PoNAD_PT / St×DTW (2)

[0114] Wherein, dup_cnt represents the number of file replicas, MAX represents the maximum value function, MIN represents the minimum value function, [] represents the integer function, a represents the lower limit of the predetermined replicas, b represents the upper limit of the predetermined replicas, PoRS represents the remaining storage percentage, PoNAD_PT represents the percentage of new data per unit time, St represents the predetermined time, and DTW represents the file popularity weight, wherein the predetermined time is an integer multiple of the unit time corresponding to the percentage of new data per unit time.

[0115] In this embodiment, the rounding function [] can specifically be a rounding function or a rounding function that discards decimals; this specification does not impose any limitations on the embodiments. The predetermined time is an integer multiple of the unit time corresponding to the percentage of newly added data per unit time. For example, if the unit time is in days, then the percentage of newly added data per unit time is the percentage of newly added data per day. If the file storage requirement is to ensure that the allocated daily storage percentage is sufficient for one year, then the predetermined time can be 365 days. In this embodiment, the lower and upper limits of the predetermined replicas can be set based on manual experience. For example, for a file to be stored with a popularity level of LV1, the lower limit of the predetermined replicas can be 3, and the upper limit of the predetermined replicas can be 6, indicating that the number of replicas of the file to be stored with a popularity level of LV1 is in the range [3, 6].

[0116] Then, based on the remaining storage percentage, the percentage of newly added data each day, the scheduled time, and the file popularity weight, the calculated value of the number of file copies is determined. If the calculated value is within the range of [3, 6], then the final number of file copies is that calculated value. Compared to methods that directly set the number of file copies to an empirical value, the file copy calculation method described in this embodiment can be flexibly adjusted according to the actual situation of the server cluster, resulting in the number of file copies that best matches the actual situation of the server cluster, thereby making full use of the server cluster's resources and improving file operation efficiency.

[0117] According to one embodiment of this article, the access frequency of the file to be stored corresponding to the higher popularity level is higher than that of the file to be stored corresponding to the lower popularity level.

[0118] Based on the remaining storage percentage of the server cluster, the percentage of newly added data per unit time, and the file popularity weight corresponding to the popularity level of the files to be stored, the number of file replicas for the files to be stored at multiple popularity levels is calculated, including...

[0119] Based on the remaining storage ratio of the server cluster, the percentage of newly added data per unit time, and the file popularity weight corresponding to the popularity level of the file to be stored, the number of file copies of the file to be stored for each of the multiple popularity levels whose popularity ranking exceeds the first preset threshold is calculated.

[0120] In this embodiment, the specific levels of popularity are set manually based on experience. This is equivalent to using a file popularity level calculation model to classify the files to be stored into specific popularity levels. Therefore, some files with very low popularity may not be processed for a long time. If the number of file copies is still calculated based on the remaining storage ratio of the server cluster, it may lead to a waste of storage resources. Therefore, in this embodiment, the number of file copies can be calculated only for files with popularity levels that exceed the first preset threshold. For example, only the number of file copies of files to be stored from LV1 to LV3 can be calculated. For files to be stored from LV4 to LV5, the number of file copies can be stored directly according to the experience-set number of file copies.

[0121] According to one embodiment of this article, such as Figure 3 As shown, the servers in the server cluster are grouped according to multiple performance indicators of each server and their stored historical file operation records, resulting in multiple server groups, which further include...

[0122] Step 301: Sort the servers in the server cluster in descending order of performance according to the performance indicators;

[0123] Step 302: Group the files in the historical file operation records according to multiple predetermined file operation frequency intervals, wherein the number of file operation frequency intervals is the same as the number of server groups;

[0124] Step 303: Calculate the ratio of the sum of the sizes of all files in each group to the total size of the files in the historical file operation records;

[0125] Step 304: Group the sorted servers according to the percentage to obtain the server groups.

[0126] In this embodiment, the servers in the server cluster are first sorted in descending order of performance based on performance metrics, so that higher-performing servers can be selected to form higher-performing server groups. Then, to ensure that the total storage space of the divided service groups can meet the file storage requirements of the files to be stored for a specified time period within the corresponding file operation frequency range, the files in the historical operation file records are grouped according to multiple predetermined file operation frequency ranges. The number of file operation frequency ranges is the same as the number of server groups; that is, multiple server groups are created according to multiple predetermined file operation frequency ranges, so that the files to be stored within the predetermined file operation frequency ranges can be stored in the corresponding server groups. To ensure that the total storage space of the server group can meet the above storage requirements, this embodiment calculates the ratio of the sum of the sizes of all files in each group to the total size of the files in the historical file operation records. In other words, based on the historical file operation records, the proportion of the total file size corresponding to each file operation frequency range is calculated. Then, the sorted servers are grouped according to this proportion to obtain the server group. The obtained server group ensures that the performance meets the operation requirements of the corresponding file operation frequency range, and also ensures that the total storage space of the server group can meet the storage requirements of the corresponding file operation frequency range within a specified time period.

[0127] Furthermore, according to one embodiment of this document, such as Figure 4 As shown, the servers are grouped according to the stated proportion, and the resulting server group further includes,

[0128] Step 401: Calculate the storage space size corresponding to the percentage based on the percentage and the total storage space size of the server cluster;

[0129] Step 402: Select multiple servers sequentially from the sorted server cluster to form the server group, wherein the sum of the storage space of all servers in the server group is close to the storage space size of the corresponding proportion of the server group, and the generation order of the multiple server groups is the same as the order of the average file operation frequency corresponding to the file operation frequency range from large to small.

[0130] In this embodiment, the storage space size corresponding to the percentage is first calculated based on the percentage and the total storage space size of the server cluster, which is equivalent to calculating the storage space size corresponding to a file operation frequency range. Then, multiple servers are sequentially selected from the sorted server cluster to form server groups. The sum of the total storage space of each server group is close to the storage space size corresponding to the percentage of the server group. In other words, the storage space size of the server group corresponding to the file operation frequency range is close to the storage space size required for that file operation frequency range. Furthermore, the order in which the server groups are generated is the same as the order from largest to smallest average file operation frequency corresponding to the file operation frequency range.

[0131] For example, the frequency ranges for file operations are: daily, monthly, annual, and more than one year. That is, the proportion of the total file size of files that are operated daily, the proportion of the total file size of files that are operated monthly (it should be noted that the files that are operated monthly in this embodiment do not include the files to be operated, that is, files that are operated once every day or more but within 30 days), the proportion of the total file size of files that are operated annually (it should be noted that the files that are operated annually in this embodiment do not include the files to be operated, that is, files that are operated once every month or more but within 12 months), and the proportion of the total file size of files that are operated only once every year or more.

[0132] Then, the proportions of file sizes operated on daily, monthly, yearly, and for more than one year relative to the total file size are calculated (the sum of these four proportions equals 1), denoted as dr, mr, yr, and or, respectively. Based on these proportions and the total storage space of the server cluster, the corresponding storage space size is calculated, resulting in Mdr for dr, Mmr for mr, Myr for yr, and Mor for or.

[0133] Then, following the descending order of operation frequency, multiple servers are sequentially selected from the sorted server cluster to form server groups. For example, servers with a file operation frequency range of "daily" are first selected to form a server group corresponding to a file operation frequency range of "daily". In this process, starting from the first server in the sorting (i.e., the server with the highest performance), servers are selected sequentially, and their storage space is accumulated until the sum of the storage space of the selected servers is close to the storage space size Mdr corresponding to the percentage dr. The selected servers are then formed into the first server group GROUP1. It can be understood that the first server group GROUP1 has the strongest overall performance and corresponds to the most frequently operated files, and its storage space size can meet the storage requirements of the most frequently operated files for a specified time period. Then, servers are selected again to obtain server group GROUP2 corresponding to a file operation frequency range of "monthly", server group GROUP3 corresponding to a file operation frequency range of "yearly", and server group GROUP4 corresponding to a file operation frequency range of "more than one year".

[0134] According to one embodiment of this article, such as Figure 5 As shown, sorting the servers in the server cluster according to the performance indicators from highest to lowest further includes:

[0135] Step 501: Use multiple performance metrics corresponding to any one server as multiple benchmark metrics.

[0136] Step 502: For a server, calculate the ratio of each performance metric to the corresponding benchmark metric, and calculate the server's overall score based on the obtained ratios.

[0137] Step 503: Sort the servers in descending order of their comprehensive scores.

[0138] In this embodiment, to sort servers according to multiple performance indicators, these indicators need to be integrated into a single overall performance indicator before sorting. Current methods typically involve weighted calculations of multiple performance indicators, but this requires manually setting the weights for each indicator. Inappropriate weighting can affect the accuracy of the sorting results. Therefore, this embodiment selects multiple performance indicators corresponding to any given server as benchmark indicators. For each server, the ratio of each performance indicator to its corresponding benchmark indicator is calculated, and a comprehensive score is calculated based on these ratios. Finally, the servers are sorted in descending order of their comprehensive scores. This means that regardless of the ranking of the server selected as the benchmark indicator among all servers, the final sorting result remains unaffected. Furthermore, selecting any server's performance indicator as the benchmark avoids the need to set weights for each indicator, resulting in a more accurate and realistic sorting result.

[0139] Preferably, the formula for calculating the comprehensive score in this embodiment is (3):

[0140]

[0141] Where Scor represents the server's overall score, cpu_core_num represents the number of CPU cores on the server, p_cpu_core represents the CPU core count baseline, mem_val represents the server's memory size, p_mem_val represents the memory size baseline, disk_val represents the server's hard disk size, p_disk_val represents the hard disk size baseline, and disk_type_scor represents the disk type score, which is a predetermined value, such as in formula (4):

[0142] disk_type_scor=(SSD=1,SAS=0.1) (4)

[0143] SSD and SAS represent disk types, and disk type scores can be set based on experience according to the disk performance corresponding to the disk type.

[0144] IO_scor represents the network IO score, which is also a predetermined value, for example, in formula (5):

[0145] IO_scor = (10 Gigabit = 1, 1 Gigabit = 0.5) (5)

[0146] Among these, the network I / O score can be set based on experience and the corresponding performance of network I / O.

[0147] server_time represents the server's online time. Since a long online time may result in lower performance, this embodiment takes the server's online time into account when calculating the server's overall score. Taking the online time unit "month" as an example, the formula for calculating server_time is (6):

[0148] server_time = (current date - launch date) / 30(6)

[0149] After obtaining the overall score for each server, the servers are sorted in descending order of their overall scores.

[0150] According to one embodiment of this article, such as Figure 6 As shown, storing file copies of the files to be stored in the server group according to the popularity level further includes,

[0151] Step 601: According to the sorting of the popularity levels, store a predetermined number of file copies for each popularity level in multiple servers of the server group corresponding to that popularity level.

[0152] Step 602: Store the remaining unstored file copies of the popularity level in multiple servers corresponding to the server group with the largest proportion of remaining storage space.

[0153] In this embodiment, a mapping between popularity levels and server groups can be pre-established based on experience regarding the frequency of file storage and the required server performance. Therefore, files are first sorted according to predetermined levels, and a predetermined number of file copies for each popularity level are stored on multiple servers within the corresponding server group, ensuring the operational efficiency requirements for files at that popularity level. If there are any remaining unstored file copies, the server group with the largest remaining storage space is selected for storage, thereby ensuring file security and recoverability in case of loss.

[0154] According to one embodiment of this article, such as Figure 7 As shown, if the storage space of the server group corresponding to a predetermined number of file copies of the popularity level is insufficient, the method further includes,

[0155] Step 701: Determine the next server group for the server group with insufficient storage space according to the order of the server groups;

[0156] Step 702: Store the predetermined number of file copies in multiple servers corresponding to the next server group.

[0157] In this embodiment, if the storage space of a pre-established server group with corresponding relationships is insufficient, in order to ensure operational efficiency, this embodiment determines the next server group after the server group with insufficient storage space according to the order of the server groups. That is, among the multiple server groups sorted by performance, the next server group after the server group with insufficient storage space can be understood as having performance second only to the server group with insufficient storage space, thus ensuring operational efficiency as much as possible. Then, a predetermined number of file copies of the file to be stored are stored on the multiple servers corresponding to the next server group.

[0158] According to one embodiment of this article, such as Figure 8 As shown, storing the predetermined number of file copies in multiple servers corresponding to the next server group further includes,

[0159] Step 801: Sort the multiple servers corresponding to the next server group according to the remaining storage space and the comprehensive score;

[0160] Step 802: Store the predetermined number of file copies of multiple file blocks sequentially on the sorted servers.

[0161] In this embodiment, to ensure file operation efficiency as much as possible when the storage space of the server group corresponding to the predetermined number of file copies of the popularity level is insufficient and it is necessary to store the predetermined number of file copies in multiple servers of the next server group, this embodiment sorts the multiple servers corresponding to the next server group according to the remaining storage space size and the comprehensive score. That is, the servers with high performance are sorted out, and then the multiple file blocks of the predetermined number of file copies are stored sequentially in the multiple servers that are sorted first, thereby ensuring file operation efficiency as much as possible.

[0162] According to one embodiment of this article, in order to ensure sorting accuracy, such as Figure 9 As shown, storing the predetermined number of file copies in multiple servers corresponding to the next server group also includes,

[0163] Step 901: Select multiple servers in the next server group whose remaining storage is greater than the second preset threshold value;

[0164] Step 902: Determine whether the number of selected servers is greater than the number of file blocks in the predetermined number of replicas;

[0165] Step 903: If so, sort the selected servers according to the comprehensive score;

[0166] Step 904: Store the predetermined number of file copies of multiple file blocks sequentially on the sorted servers.

[0167] In this embodiment, firstly, multiple servers with remaining storage greater than a second preset threshold are selected from the next server group. These servers have the most remaining storage space. Since this embodiment stores data in file blocks (BLOCKs) of file replicas, storing multiple BLOCKs across multiple servers, it is determined whether the number of selected servers is greater than the number of file blocks of the predetermined number of replicas. If so, it indicates that a large number of servers have been selected, and multiple servers with superior performance can be further selected. This involves sorting the selected servers according to their overall score, and finally, storing the file blocks of the predetermined number of file replicas sequentially on the top-ranked servers.

[0168] According to one embodiment of this article, such as Figure 10 As shown, after calculating the ratio of the sum of the sizes of all files in each group to the total size of the files in the historical file operation records, the method further includes,

[0169] Step 1001: Using the stated proportion as the first proportion, weight the multiple first proportions to obtain the second proportion corresponding to the operation frequency of each file;

[0170] Step 1002: Summate the multiple second percentages to obtain the total percentage;

[0171] Step 1003: Calculate the ratio of each second percentage to the total percentage to obtain the third percentage, so as to group the sorted servers according to the third percentage.

[0172] In this embodiment, when grouping servers and calculating their proportions, manual experience can be incorporated to further refine the proportions. For example, files with high operation frequency may have a larger proportion to improve access efficiency, while files with low operation frequency may have a smaller proportion to avoid wasting storage resources. Therefore, this embodiment uses the stated proportion as the first proportion. Multiple first proportions are weighted to obtain a second proportion corresponding to the operation frequency of each file. This weight is set based on experience. Then, to ensure that the sum of multiple weighted proportions is still 1 (indicating a reasonable allocation of all storage space in the server cluster), multiple second proportions are summed to obtain a total proportion. The ratio of each second proportion to the total proportion is then calculated to obtain a third proportion. Thus, the sum of multiple third proportions is still 1. This facilitates grouping the sorted servers according to the third proportion.

[0173] Based on the same inventive concept, this embodiment also provides a file storage device for a Hadoop platform, such as... Figure 11 As shown, including,

[0174] The file popularity level calculation unit 1101 is used to calculate the popularity level of the file to be stored using the file popularity level calculation model. The popularity level calculation model is obtained by training multiple files and the popularity tags of the multiple files.

[0175] The file copy number calculation unit 1102 is used to calculate the number of file copies of a file to be stored for multiple popularity levels based on the remaining storage ratio of the server cluster, the proportion of newly added data per unit time, and the file popularity weight corresponding to the popularity level of the file to be stored.

[0176] Server grouping unit 1103 is used to group the servers in the server cluster according to multiple performance indicators of each server and its stored historical file operation records to obtain multiple server groups.

[0177] File copy storage unit 1104 is used to store file copies of the file to be stored in the server group according to the popularity level.

[0178] The beneficial effects obtained by the above-described device are the same as those obtained by the above-described method, and will not be described in detail in the embodiments of this specification.

[0179] like Figure 12The diagram illustrates the structure of a computer device according to an embodiment of the present invention. The apparatus in this invention can be the computer device described in this embodiment, executing the method of the present invention as described above. The computer device 1202 may include one or more processing devices 1204, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 1202 may also include any storage resource 1206 for storing any kind of information, such as code, settings, data, etc. Non-limitingly, for example, the storage resource 1206 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any storage resource can use any technology to store information. Further, any storage resource can provide volatile or non-volatile retention of information. Further, any storage resource can represent a fixed or removable component of the computer device 1202. In one case, when the processing device 1204 executes associated instructions stored in any storage resource or combination of storage resources, the computer device 1202 can perform any operation of the associated instructions. The computer device 1202 also includes one or more drive mechanisms 1208 for interacting with any storage resource, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0180] Computer device 1202 may also include an input / output module 1210 (I / O) for receiving various inputs (via input device 1212) and providing various outputs (via output device 1214). A specific output mechanism may include a presentation device 1216 and an associated graphical user interface (GUI) 1218. In other embodiments, the input / output module 1210 (I / O), input device 1212, and output device 1214 may be omitted, and the device may function solely as a computer device within a network. Computer device 1202 may also include one or more network interfaces 1220 for exchanging data with other devices via one or more communication links 1222. One or more communication buses 1224 couple the components described above together.

[0181] Communication link 1222 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 1222 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0182] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0183] This embodiment also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the above-described method.

[0184] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0185] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0186] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0187] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0188] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0190] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0191] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] This document uses specific embodiments to illustrate the principles and implementation methods of this document. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this document. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this document. Therefore, the content of this specification should not be construed as a limitation of this document.

Claims

1. A Hadoop platform file storage method, characterized by, The application comprises the following steps: The file hotness level calculation model is obtained by training a plurality of files and hotness labels of the plurality of files; The file copy number of the plurality of hotness level files is calculated according to the residual storage proportion of the server cluster, the unit time new data proportion and the file hotness weight corresponding to the hotness level of the to-be-stored file; The servers in the server cluster are grouped according to a plurality of performance indicators of each server in the server cluster and historical file operation records stored by the server, and a plurality of server groups are obtained; The file copy of the to-be-stored file is stored in the server group according to the hotness level; The access frequency of the to-be-stored file corresponding to the hotness level in front of the sequence is higher than the access frequency of the to-be-stored file corresponding to the hotness level behind the sequence; The file copy number of the plurality of hotness level files is calculated according to the residual storage proportion of the server cluster, the unit time new data proportion and the file hotness weight corresponding to the hotness level of the to-be-stored file, and the formula is as follows: dup_cnt = MAX(a, MIN([value], b)); value = PoRS / PoNAD PT / St×DTW; ​ ​ ​ ​ 2. The method of claim 1, wherein, ​ ​ ​ Wherein, dup_cnt represents the file copy quantity, MAX represents the maximum function, MIN represents the minimum function, [ ] represents the integral function, a represents the predetermined copy lower limit value, b represents the predetermined copy upper limit value, PoRS represents the remaining storage proportion, PoNAD PT represents the unit time new data proportion, St represents the predetermined time, DTW represents the file hotness weight, and the predetermined time is an integer multiple of the unit time corresponding to the unit time new data proportion.

3. The method of claim 1, wherein, Grouping the sorted servers according to the proportions to obtain the server groups further includes, Calculating the storage space size corresponding to the proportion according to the proportion and the total storage space size of the server cluster; Selecting multiple servers from the sorted server cluster in sequence as the server groups, wherein the sum of the storage spaces of all servers in the server groups is close to the storage space size of the proportion corresponding to the server groups, and the generation order of the multiple server groups is the same as the order of the average file operation frequency from large to small corresponding to the file operation frequency interval.

4. The method of claim 1, wherein, Sorting the servers in the server cluster according to the performance indicators in the order from high to low further includes, Taking multiple performance indicators corresponding to any one server as multiple benchmark indicators; Calculating the ratio of each performance indicator to the corresponding benchmark indicator for one server, and calculating the comprehensive score of the server according to the multiple ratios obtained; Sorting the servers in the order from large to small according to the comprehensive scores.

5. The method of claim 4, wherein, Storing the file copies of the to-be-stored file in the server groups according to the hotness levels further includes, According to the sorting of the hotness levels, storing a predetermined number of file copies of each hotness level in multiple servers of the server group corresponding to the hotness level, and storing the remaining file copies of the hotness level in multiple servers of the server group corresponding to the maximum remaining storage space proportion.

6. The method of claim 5, wherein, If the storage space of the server group corresponding to the predetermined number of file copies of the hotness level is insufficient, the method further includes, Determining the next server group of the server group with insufficient storage space according to the order of the server groups; Storing the predetermined number of file copies in multiple servers corresponding to the next server group.

7. The method of claim 6, wherein, Storing the predetermined number of file copies in multiple servers corresponding to the next server group further includes, Sorting the multiple servers corresponding to the next server group according to the remaining storage space size and the comprehensive score; Storing multiple file blocks of the predetermined number of file copies in multiple servers in sequence.

8. The method of claim 6, wherein, Storing the predetermined number of file copies in multiple servers corresponding to the next server group further includes, Selecting multiple servers with a remaining storage greater than a second preset threshold value in the next server group; Determining whether the number of selected servers is greater than the number of multiple file blocks of the predetermined number of copies; If yes, the selected servers are ranked according to the comprehensive scores; The file blocks of the predetermined number of file copies are sequentially stored in the servers ranked in the front.

9. The method of claim 1, wherein, After calculating the proportion of the sum of the sizes of all files in each group to the total size of files in the historical file operation record, the method further comprises, The proportion is taken as a first proportion, and a plurality of first proportions are weighted to obtain a second proportion corresponding to each file operation frequency; The second proportions are summed to obtain a total proportion; The ratio of each second proportion to the total proportion is calculated to obtain a third proportion, so as to group the ranked servers according to the third proportion.

10. A Hadoop platform file storage device, characterized by, Comprise, The file hotness level calculation unit is configured to calculate the to-be-stored file by using a file hotness level calculation model to obtain the hotness level of the to-be-stored file, wherein the file hotness level calculation model is obtained by training a plurality of files and hotness labels of the plurality of files; The file copy number calculation unit is configured to calculate the file copy number of the to-be-stored file of each hotness level according to the remaining storage proportion of the server cluster, the data addition proportion per unit time, and the file hotness weight corresponding to the hotness level of the to-be-stored file; The server group division unit is configured to group the servers in the server cluster according to a plurality of performance indicators of each server in the server cluster and historical file operation records stored by the servers to obtain a plurality of server groups; The file copy storage unit is configured to store the file copies of the to-be-stored file in the server groups according to the hotness levels; The access frequency of the to-be-stored file corresponding to the hotness level ranked in the front is higher than the access frequency of the to-be-stored file corresponding to the hotness level ranked in the back; The file copy number of the to-be-stored file of each hotness level is calculated according to the remaining storage proportion of the server cluster, the data addition proportion per unit time, and the file hotness weight corresponding to the hotness level of the to-be-stored file further comprises, The file copy number of the to-be-stored file of each hotness level ranked beyond a first preset threshold value is calculated according to the remaining storage proportion of the server cluster, the data addition proportion per unit time, and the file hotness weight corresponding to the hotness level of the to-be-stored file; The server group division unit is configured to group the servers in the server cluster according to a plurality of performance indicators of each server in the server cluster and historical file operation records stored by the servers to obtain a plurality of server groups further comprises, The servers in the server cluster are ranked in the order from high to low according to the performance indicators; The files in the historical file operation record are grouped according to a plurality of predetermined file operation frequency intervals, and the number of file operation frequency intervals is the same as the number of server groups; The proportion of the sum of the sizes of all files in each group to the total size of files in the historical file operation record is calculated; The ranked servers are grouped according to the proportions to obtain the server groups.

11. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method in any one of claims 1-9 when executing the computer program.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-9.

Citation Information

Patent Citations

  • Dynamic replica management method based on file heat

    CN103150347A

  • Heterogeneous Hadoop storage method based on data temperatures and joint performance

    CN106156283A