Data processing method and device, computer readable storage medium and electronic equipment

Abstract

The invention relates to the technical field of computers, and provides a data processing method, a data processing device, a computer readable storage medium and electronic device.The data processing method is applied to a data hierarchical management node in a distributed storage system, and the distributed storage system further comprises a plurality of data processing nodes. Each data processing node manages a data set, each data processing node is used for regularly screening out local hot spot data from the managed data set, and the method comprises the following steps: receiving the local hot spot data regularly reported by each data processing node and heat statistical information of the local hot spot data; screening out first global hotspot data which is not stored in a high-performance storage medium from the local hotspot data according to the popularity statistical information reported by the plurality of data processing nodes; and migrating the first global hotspot data into a high-performance storage medium. According to the invention, reasonable hierarchical storage of the hotspot data in the distributed storage system can be realized.

Description

Background Art

[0002] With the rapid development and advancement of the Internet and computer technology, related data storage technologies are also booming.

[0003] In the current distributed storage system, in order to reduce the storage cost of data and maximize the performance of the entire storage system, the data in the system can be stored in layers, for example, hot data and cold data can be stored in layers. However, the existing storage solutions cannot accurately and comprehensively determine the hot data, resulting in unreasonable data stratification.

[0004] In view of this, there is an urgent need in the art to develop a new data processing method and device.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure. Summary of the invention

[0006] The purpose of the present disclosure is to provide a data processing method, a data processing device, a computer-readable storage medium and an electronic device, thereby overcoming the technical problem of unreasonable data stratification caused by the limitations of related technologies at least to a certain extent.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a data processing method is provided, which is applied to a data hierarchical management node in a distributed storage system, wherein the distributed storage system further comprises a plurality of data processing nodes, each of which manages a data set, and each of which is used to regularly filter out local hotspot data from the data set it manages, wherein the method comprises:

[0009] Receiving the local hotspot data and heat statistics information of the local hotspot data regularly reported by each of the data processing nodes;

[0010] Filtering out first global hotspot data that is not stored in a high-performance storage medium from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes;

[0011] Migrate the first global hotspot data to the high-performance storage medium.

[0012] In an exemplary embodiment of the present disclosure, the filtering out first global hotspot data that is not stored in a high-performance storage medium from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes includes:

[0013] Filtering global hotspot data from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes;

[0014] The second global hotspot data stored in the high-performance storage medium is removed from the global hotspot data to obtain the first global hotspot data not stored in the high-performance storage medium.

[0015] In an exemplary embodiment of the present disclosure, after filtering out first global hotspot data that is not stored in a high-performance storage medium from the local hotspot data according to the heat statistics information reported by the plurality of data processing nodes, the data hierarchical management node temporarily stores the first global hotspot data in a temporary storage area;

[0016] The migrating the first global hotspot data to the high-performance storage medium includes:

[0017] Each of the data processing nodes pulls the designated global hotspot data within its management scope from the temporary storage area, and migrates the designated global hotspot data to a high-performance storage medium.

[0018] In an exemplary embodiment of the present disclosure, after migrating the first global hotspot data to the high-performance storage medium, the method further includes:

[0019] In response to receiving the migration completion message, the first global hotspot data is written into a hotspot storage file; a plurality of the hotspot storage files are arranged in the order of generation time to form a time-series file linked list;

[0020] And, writing the first global hot spot data and its associated heat statistics information into the heat statistics database.

[0021] In an exemplary embodiment of the present disclosure, the method further includes:

[0022] Monitoring the data occupied capacity of the high performance storage medium;

[0023] In response to the data occupying a capacity greater than a preset capacity threshold, acquiring cold data in the high-performance storage medium;

[0024] The cold data is migrated from the high-performance storage medium to a low-performance storage medium.

[0025] In an exemplary embodiment of the present disclosure, the acquiring of cold data in the high-performance storage medium includes:

[0026] Acquire a historical hotspot storage file whose generation time is earlier than a preset time point from the time-series file linked list;

[0027] For each of the historical global hot spot data contained in the historical hot spot storage file, query the heat statistics information corresponding to each of the historical global hot spot data from the heat statistics database;

[0028] If the heat statistics information is found, determining whether each of the historical global hot data is the cold data according to the heat statistics information;

[0029] If the heat statistics information is not found, each of the historical global hot data is determined as the cold data.

[0030] In an exemplary embodiment of the present disclosure, after acquiring the cold data in the high-performance storage medium, the data tiering management node temporarily stores the cold data in a temporary storage area;

[0031] Migrating the cold data from the high-performance storage medium to the low-performance storage medium includes:

[0032] Each of the data processing nodes pulls designated cold data within its management scope from the temporary storage area, and migrates the designated cold data from the high-performance storage medium to the low-performance storage medium.

[0033] In an exemplary embodiment of the present disclosure, after migrating the cold data from the high-performance storage medium to the low-performance storage medium, the method further includes:

[0034] In response to receiving a migration completion message, deleting the historical hotspot storage file;

[0035] And, deleting the cold data and its heat statistics information from the heat statistics database.

[0036] In an exemplary embodiment of the present disclosure, the method further includes:

[0037] Monitoring the amount of data in the heat statistics database;

[0038] In response to the data volume in the heat statistics database being greater than a preset data volume threshold, the heat statistics database is set to a read-only state, and a new heat statistics database is created.

[0039] In an exemplary embodiment of the present disclosure, the method further includes:

[0040] Monitoring the number of the heat statistics databases that have been created;

[0041] In response to the number of the created heat statistics databases being greater than a preset number threshold, determining an invalid database from the created heat statistics databases according to a creation time; the creation time of the invalid database being earlier than a target time;

[0042] The invalid database is deleted.

[0043] According to a second aspect of the present disclosure, there is provided a data processing device, which is arranged in a data hierarchical management node in a distributed storage system, wherein the distributed storage system further comprises a plurality of data processing nodes, each of which manages a data set, and each of which is used to regularly filter out local hotspot data from the data set it manages, wherein the device comprises:

[0044] An information receiving module, used for receiving the local hotspot data and heat statistics information of the local hotspot data regularly reported by each of the data processing nodes;

[0045] A hotspot data screening module, configured to screen out first global hotspot data that is not stored in a high-performance storage medium from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes;

[0046] A data migration module is used to migrate the first global hotspot data to the high-performance storage medium.

[0047] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data processing method described in the first aspect is implemented.

[0048] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the data processing method described in the first aspect above by executing the executable instructions.

[0049] It can be seen from the above technical solutions that the data processing method, data processing device, computer-readable storage medium and electronic device in the exemplary embodiments of the present disclosure have at least the following advantages and positive effects:

[0050] In the technical solutions provided in some embodiments of the present disclosure, a new data tiering management node is added to the distributed storage system, and each data processing node periodically filters out local hotspot data from the data set it manages. After that, the above local hotspot data and the heat statistics of the local hotspot data are periodically reported to the data tiering management node. The data tiering management node filters out the first global hotspot data that is not stored in the high-performance storage medium from the local hotspot data according to the heat statistics reported by multiple data processing nodes, and finally migrates the first global hotspot data to the high-performance storage medium. On the one hand, it can solve the problem that the global hotspot data cannot be accurately determined in the relevant distributed storage system, resulting in the high-performance storage medium being excessively occupied by non-global hotspot data, thereby improving the performance of the entire storage system; further, migrating the global hotspot data to the high-performance storage medium can significantly reduce data access delays, increase data access speeds, and thus improve the responsiveness of the overall system; on the other hand, it can store hotspot data separately from cold data, optimize the use of storage resources, use limited high-performance storage media to store the most frequently accessed data, and store other data on lower-cost media, thereby reducing system costs.

[0051] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0053] Figure 1 A schematic diagram showing a flow chart of a data processing method in an embodiment of the present disclosure;

[0054] Figure 2 A schematic diagram showing a flow chart of how to migrate global hotspot data to a high-performance storage medium in an embodiment of the present disclosure;

[0055] Figure 3 A schematic diagram showing a flow chart of another data processing method in an embodiment of the present disclosure;

[0056] Figure 4 A schematic diagram showing a process of obtaining cold data in a high-performance storage medium in an embodiment of the present disclosure;

[0057] Figure 5 A schematic diagram showing a flow chart of how to migrate global hotspot data to a high-performance storage medium in an embodiment of the present disclosure;

[0058] Figure 6 The overall architecture diagram of the distributed storage system in the embodiment of the present disclosure is shown;

[0059] Figure 7 A schematic diagram showing the structure of a data processing device in an exemplary embodiment of the present disclosure is shown;

[0060] Figure 8 A schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0061] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0062] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0063] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0064] In the current storage implementation, in order to reduce the storage cost of data while maximizing the performance of the entire storage system, the data in the system can be stored in layers. For example, data with low access frequency can be stored in a large-capacity storage medium with low price and poor performance (such as a hard disk), while data with high access frequency can be stored in a storage medium with high price but better performance (such as a solid-state drive). Therefore, in a KV storage system where Key and Value are stored separately (such as wiscKey, etc.), Value can usually be stored in storage media of different layers according to the access frequency.

[0065] In the existing tiered storage systems, most of them perform hot and cold statistics on the keys on a storage node and store the hot data in a storage medium with good performance. However, for a distributed storage cluster system, the high-performance storage medium space is usually shared by the entire cluster. Although each storage node can count its own hottest keys and migrate them to high-performance media, it cannot guarantee that the hot keys on a single node are also the hottest keys from the perspective of the entire cluster. Therefore, this solution may result in the data after migration being unable to maximize the performance of the entire system. At the same time, it may also cause relatively non-hot keys to occupy too much high-performance medium space, and the real hot keys cannot be migrated later.

[0066] In addition, the existing tiered storage system also has another problem. In order not to maintain too many hot and cold statistical data, the statistics for hot and cold data are calculated according to a certain data range. For example, for block storage, a certain LBA range (Logical Block Addressing) may be used as a statistical point. For KV-based storage systems, a group of keys with similar characteristics may be used as a statistical point. When this statistical point is determined to be a hot spot, all data within this range will be moved to the high-performance layer. Although this will reduce the hot and cold statistical data that need to be maintained, the range migration will cause some keys that were originally cold data to be moved to the hot layer medium, which will also reduce the efficiency of the entire storage tiering.

[0067] Based on the above problems, the present invention proposes a data tiered storage solution based on a distributed KV storage system to realize the statistics of hot and cold data based on the hot and cold degree of data access in the entire cluster. At the same time, it can also migrate hot and cold data with Key as the smallest unit without maintaining too large statistical data.

[0068] In the embodiments of the present disclosure, a data processing method is first provided, which at least to some extent overcomes the defect of unreasonable data stratification in the current distributed storage system.

[0069] Figure 1 A flow chart of a data processing method in an embodiment of the present disclosure is shown. The execution subject of the data processing method may be a data hierarchical management node in a distributed storage system.

[0070] refer to Figure 1 According to an embodiment of the present disclosure, a data processing method comprises the following steps:

[0071] Step S110, receiving local hotspot data and heat statistics information of the local hotspot data regularly reported by each data processing node;

[0072] Step S120, filtering out first global hotspot data that is not stored in a high-performance storage medium from local hotspot data according to heat statistics information reported by multiple data processing nodes;

[0073] Step S130: Migrate the first global hotspot data to a high-performance storage medium.

[0074] exist Figure 1 In the technical solution provided by the illustrated embodiment, a new data tiering management node is added to the distributed storage system, and each data processing node periodically filters out local hotspot data from the data set it manages. Thereafter, the above-mentioned local hotspot data and the heat statistics information of the local hotspot data are periodically reported to the data tiering management node. The data tiering management node filters out the first global hotspot data that is not stored in the high-performance storage medium from the local hotspot data according to the heat statistics information reported by multiple data processing nodes, and finally migrates the first global hotspot data to the high-performance storage medium. On the one hand, it can solve the problem that the global hotspot data cannot be accurately determined in the relevant distributed storage system, resulting in the high-performance storage medium being excessively occupied by non-global hotspot data, thereby improving the performance of the entire storage system; further, migrating the global hotspot data to the high-performance storage medium can significantly reduce data access delays, increase data access speeds, and thus improve the responsiveness of the overall system; on the other hand, it can store hotspot data separately from cold data, optimize the use of storage resources, use limited high-performance storage media to store the most frequently accessed data, and store other data on lower-cost media, thereby reducing system costs.

[0075] The following Figure 1 The specific implementation process of each step is described in detail:

[0076] The distributed storage system in the present disclosure refers to a distributed key-value storage system. A distributed key-value (KV) storage system is a distributed storage system for storing and retrieving key-value pairs, which allows users to store and query data values ​​through unique keys. The data in the present disclosure may refer to the Key in the Key-Value.

[0077] The present disclosure may add a data tiering management node (which may be called a Tier-controller) in the distributed storage system, and execute the data tiering control process (ie, steps S110 to S130 in the present disclosure) in the entire distributed storage system through the data tiering management node.

[0078] The above-mentioned distributed storage system also includes multiple data processing nodes (which can be called KV-servers). In the distributed storage system, the keys stored in the entire system can be divided into slices by range (for example: according to the hash value range of the key or according to the dictionary order of the key). Therefore, each data processing node is responsible for managing a data set (each data set contains multiple keys) to achieve the function of load balancing.

[0079] Thus, each data processing node can regularly collect heat statistics information of each Key in the data set it manages (for example, the heat statistics information can be determined based on the access frequency and the most recent access time), and filter out local hotspot data from the data set it manages based on the heat statistics information. For example, the Key with a higher access frequency can be determined as the local hotspot data, or the Key with the most recent access time closest to the current time can be determined as the local hotspot data, or the Key with a higher access frequency and the most recent access time closest to the current time can be determined as the local hotspot data. All of these can be set according to actual conditions, and the present disclosure does not make any special limitations on this.

[0080] Afterwards, each data processing node may report the filtered local hotspot data and heat statistics of the local hotspot data to the data hierarchical management node.

[0081] Next reference Figure 1 In step S110, local hotspot data and heat statistics information of the local hotspot data regularly reported by each data processing node are received.

[0082] In this step, the data hierarchical management node may receive local hotspot data and heat statistics information of the local hotspot data regularly reported by each data processing node.

[0083] In step S120, first global hotspot data that is not stored in a high-performance storage medium is screened out from local hotspot data according to heat statistics information reported by multiple data processing nodes.

[0084] In this step, the data hierarchical management node may filter out the first global hotspot data that is not stored in the high-performance storage medium from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes.

[0085] Exemplarily, the data hierarchical management node can sort the heat statistical information corresponding to the multiple local hotspot data received from high to low in terms of heat, and first filter out the global hotspot data based on the sorting result. Then, the second global hotspot data stored in the high-performance storage medium can be filtered out from the above global hotspot data. Then, the above second global hotspot data can be eliminated to obtain the first global hotspot data not stored in the high-performance storage medium.

[0086] It should be noted that, after determining the second global hot spot data stored in the high-performance storage medium, the data hierarchy management node may also update the heat statistics information corresponding to the second global hot spot data into the heat statistics database.

[0087] In addition, the data hierarchical management node may temporarily store the first global hotspot data in a temporary storage area. Exemplarily, the temporary storage area may be a cache, thereby speeding up the response speed when the data is subsequently read.

[0088] In step S130, the first global hotspot data is migrated to a high-performance storage medium.

[0089] In this step, since the first global hotspot data is temporarily stored in the cache, each data processing node can pull the designated global hotspot data within its management scope from the cache and migrate the designated global hotspot data to a high-performance storage medium.

[0090] The above-mentioned high-performance storage media refer to storage devices or technologies that can provide high-speed data reading and writing capabilities. These storage devices are generally used in application scenarios that require high I / O performance, such as enterprise data centers, high-performance computing, cloud computing platforms, real-time data analysis, and other services that require fast data access. Exemplarily, the high-performance storage media in the present disclosure can be a solid-state drive (SSD), a non-volatile memory express solid-state drive (NVMe SSD), a random access memory (RAM), a phase change memory (PCM), etc., which can be set according to actual conditions, and the present disclosure does not make special restrictions on this.

[0091] After each data processing node migrates the designated global hotspot data within its management scope to the high-performance storage medium, each data processing node can send a migration completion message to the data tiering management node. After that, the data tiering management node can clear the designated global hotspot data corresponding to each data processing node temporarily stored in the temporary storage area until the above-mentioned first global hotspot data in the cache is cleared.

[0092] After migrating the first global hotspot data to the high-performance storage medium, the data tiering management node can write the above first global hotspot data into the hotspot storage file (which can be called the Tier-data file). Multiple hotspot storage files can be arranged in the order of generation time to form a time-series file linked list. Since the earliest file is always arranged at the head of the file, it is convenient to quickly determine the cold Key according to the time-series file linked list, thereby facilitating efficient data management.

[0093] In addition, the data tiering management node can also write the first global hot spot data and its associated heat statistics information into the heat statistics database (which can be called the Tier-db database), so as to facilitate the subsequent rapid search for the heat statistics information of each first global hot spot data, and determine whether the data has become cold data based on the heat statistics information.

[0094] refer to Figure 2 , Figure 2 A flow chart showing how to migrate global hotspot data to a high-performance storage medium in an embodiment of the present disclosure includes steps S201 to S209:

[0095] In step S201, each data processing node collects statistics of the heat statistics of the keys within its management scope and determines candidate hot keys;

[0096] In step S202, each data processing node reports the candidate hotspot Key and its heat statistics information to the data hierarchical management node;

[0097] In step S203, the data hierarchy management node calculates the first global hotspot Key that needs to be relocated and records it in the cache, and selects the second global hotspot Key that has been relocated;

[0098] In step S204, the data layer management node updates the heat statistics information corresponding to the second global hot spot Key;

[0099] In step S205, each data processing node pulls the key that needs to be relocated from the temporary storage area;

[0100] In step S206, the data hierarchical management node returns the Key that needs to be relocated to each data processing node;

[0101] In step S207, each data processing node returns a migration success message to the data layer management node;

[0102] In step S208, the data hierarchy management node clears the first global hotspot Key that needs to be relocated from the cache;

[0103] In step S209, the data tier management node records the relocated first global hotspot Key to the Tier-data file and the Tier-db database.

[0104] Based on the above technical solutions, the present disclosure can solve the problem that the global hotspot data cannot be accurately determined in the related distributed storage system, resulting in the high-performance storage medium being excessively occupied by non-global hotspot data, thereby improving the performance of the entire storage system.

[0105] In order to avoid the problem that the storage space in the high-performance storage medium is too high and thus affects the system response speed, the present disclosure also provides another data processing method, referring to Figure 3 , Figure 3 A flow chart showing another data processing method in an embodiment of the present disclosure is shown, including steps S301 to S303:

[0106] In step S301, the data occupied capacity of the high-performance storage medium is monitored.

[0107] In this step, the data tiering management node can monitor the data occupancy capacity of the high-performance storage medium (ie, the size of the space occupied by the data on the high-performance storage medium) in real time to avoid data overload in the high-performance storage medium.

[0108] In step S302, in response to the data occupied capacity being greater than a preset capacity threshold, cold data in a high-performance storage medium is acquired.

[0109] In this step, if it is detected that the capacity occupied by the above data is greater than the preset capacity threshold, cold data in the high-performance storage medium can be obtained. Cold data can be data that is not frequently accessed or rarely used.

[0110] Specifically, refer to Figure 4 , Figure 4 A schematic diagram showing a process of obtaining cold data in a high-performance storage medium in an embodiment of the present disclosure includes steps S401 to S404:

[0111] In step S401, a historical hotspot storage file whose generation time is earlier than a preset time point is obtained from a time-series file linked list.

[0112] In this step, when it is detected that the capacity occupied by the above data is greater than the preset capacity threshold, the historical hotspot storage file with a generation time earlier than the preset time point can be obtained from the above time sequence file linked list. Exemplarily, the historical hotspot storage file can be one or more hotspot storage files in the time sequence file linked list with a generation time earlier than the preset time point (which can be set according to actual conditions, and the present disclosure does not make special restrictions on this).

[0113] In step S402, for each piece of historical global hotspot data contained in the historical hotspot storage file, the heat statistics information corresponding to each piece of historical global hotspot data is queried from the heat statistics database.

[0114] In this step, since each historical hotspot storage file contains a batch of historical global hotspot data, for each historical global hotspot data, the heat statistics database can be queried for the heat statistics information corresponding to the historical global hotspot data.

[0115] In step S403, if the heat statistics information is found, it is determined whether each historical global hot data is cold data according to the heat statistics information.

[0116] In this step, if the heat statistics information corresponding to each historical global hot data is queried, it can be determined whether each historical global hot data is cold data based on the heat statistics information. Specifically, when the heat statistics information indicates that the access heat of the historical global hot data is high, it can be determined that the historical global hot data is not cold data, and when the heat statistics information indicates that the access heat of the historical global hot data is low, it can be determined that the historical global hot data is cold data.

[0117] In step S404, if no heat statistics information is found, each historical global hot data is determined as cold data.

[0118] In this step, if the heat statistics information corresponding to a certain historical global hot data is not found, it can be explained that the heat statistics information of the historical global hot data has been cleared, and thus the historical global hot data is cold data.

[0119] After determining the cold data, the data tiering management node may temporarily store the cold data in a cache so that each subsequent data processing node can quickly pull the corresponding cold data.

[0120] Next reference Figure 3 In step S303, the cold data is migrated from the high-performance storage medium to the low-performance storage medium.

[0121] In this step, each data processing node can pull out designated cold data within its management scope from the cache, and migrate the designated cold data from the high-performance storage medium to the low-performance storage medium.

[0122] Among them, low-performance storage media generally refers to storage devices or technologies that perform poorly in terms of data read and write speed, latency, throughput, etc. Such storage media are mainly used to store infrequently accessed data (i.e., cold data), or to make a trade-off between cost and performance. Exemplarily, the low-performance storage media in the present disclosure may be tape storage, hard disk drive (HDD), optical storage, etc., which can be set according to actual conditions, and the present disclosure does not make any special restrictions on this.

[0123] After migrating the cold data from the high-performance storage medium to the low-performance storage medium, each data processing node can send a migration completion message to the data tiering management node, after which the data tiering management node can clear the specified cold data temporarily stored in the temporary storage area until all cold data in the cache is cleared.

[0124] After migrating the cold data to a low-performance storage medium, the data tiering management node can also delete the above-mentioned historical hotspot storage file in response to the migration completion message, and rewrite the historical global hotspot data in the historical hotspot storage file that has not become cold data into a new hotspot storage file. At the same time, the above-mentioned cold data and its heat statistics information can also be deleted from the above-mentioned heat statistics database.

[0125] refer to Figure 5 , Figure 5 A flow chart showing how to migrate global hotspot data to a high-performance storage medium in an embodiment of the present disclosure includes steps S501 to S509:

[0126] In step S501, the data tiering management node determines the data occupied capacity of the high-performance storage medium;

[0127] In step S502, the data layer management node obtains the earliest historical hot spot storage file from the time-series file linked list;

[0128] In step S503, the hot spot statistics database is queried to determine whether each historical global hot spot data has become cold data;

[0129] In step S504, cold data is put into cache;

[0130] In step S505, each data processing node pulls cold data from the cache;

[0131] In step S506, the data tiering management node returns cold data to each data processing node;

[0132] In step S507, each data processing node returns a migration success message to the data layer management node;

[0133] In step S508, the data hierarchical management node deletes cold data from the hot storage file and the heat statistics database;

[0134] In step S509, the data tiering management node rewrites the historical global hotspot data that has not become cold data into a new hotspot storage file.

[0135] It should be noted that the present disclosure can also monitor the amount of data in the heat statistics database. When the amount of data in the heat statistics database is greater than the preset data amount threshold, the heat statistics database can be set to read-only state, and a new heat statistics database can be created to continue to store new heat data and its heat statistics data. For example, assuming that the upper limit of the key in a heat statistics database is 1000W, then when the key in heat statistics database 1 reaches 1000W, we can seal this heat statistics database, and then create a heat statistics database 2, and write all new data to heat statistics database 2.

[0136] The present disclosure can also monitor the number of heat statistics databases that have been created. When the number of heat statistics databases that have been created is greater than a preset number threshold, invalid databases can be determined from the heat statistics databases that have been created according to the creation time (the invalid database can be the heat statistics database with the earliest creation time or earlier than the specified time point). For example, a maximum of 5 heat statistics databases can be created. Then, when the number of creations has reached 5, the oldest heat statistics database 1 can be deleted. Because at this time the information in the heat statistics database 1 has been out for a long time, it can be considered that the data here has no reference value, and thus the invalid database can be deleted.

[0137] The present disclosure records the relevant information of the hot spot keys that have been migrated to the high-performance storage medium in two places (the heat storage file Tier-data is used to store the hot spot keys, and the heat statistics database Tier-db is used to store the hot spot keys and their detailed heat statistics information). Therefore, since the heat storage file can be used directly to obtain the header file in the time series file linked list each time, and does not involve any search and calculation operations, the hot spot keys recorded therein can be quickly accessed, greatly improving the data acquisition efficiency. At the same time, for the heat statistics database Tier-db, the present disclosure monitors its data volume in real time, and creates a new heat statistics database when the data volume exceeds the upper limit, thereby avoiding the problem of low data search efficiency caused by excessive occupation of the capacity of a single heat statistics database.

[0138] Therefore, compared with the solution in the related art that only stores the relevant information of the hotspot key in the database, the present disclosure can optimize the data storage method, thereby reducing the data storage pressure of the database and improving the data access speed.

[0139] refer to Figure 6 , Figure 6 The overall architecture diagram of the distributed storage system in the embodiment of the present disclosure is shown as follows: Figure 6 As shown:

[0140] The distributed storage system includes a data tier management node (Tier-controller) and multiple data processing nodes (such as Figure 6 KV server1, KV server2, KV server3) shown in ;

[0141] KV-server is responsible for taking over the actual IO writes. In a distributed KV system, all keys in the entire system are sliced ​​by range. Each KV-server is only responsible for processing keys within a range, thereby achieving load balancing.

[0142] When processing IO reads and writes, the KV-server can obtain the local hotspot Key within its scope of responsibility based on the Key access frequency and the most recent access time. The KV-server will send the local hotspot Key and its heat statistics to the Tier-controller.

[0143] Tier-controller is responsible for collecting data reported by all KV-servers in the cluster and calculating the global hotspot keys that need to be relocated;

[0144] Tier-controller returns the global hotspot key that needs to be relocated to the corresponding KV-server, and the KV-server performs specific data relocation;

[0145] Tier-controller saves the hot and cold statistics of the relocated keys. When the hot layer space occupancy reaches a certain threshold, it calculates the keys that have become cold and sends a command to the KV-server to relocate the cold keys to the cold layer storage space.

[0146] Based on the above technical solutions, the present disclosure can at least achieve the following technical effects:

[0147] First, by adding a central control node Tier-controller to the distributed KV storage system, the hot and cold keys are counted from the perspective of the entire cluster for migration.

[0148] Second, we use the time-series file Tier-data and multi-instance Tier-db together, and discard the oldest statistical information at the same time, which reduces the statistical information that needs to be maintained and enables the migration of hot and cold data based on the key granularity.

[0149] The present disclosure also provides a data processing device, which is arranged in a data hierarchical management node in a distributed storage system, wherein the distributed storage system further comprises a plurality of data processing nodes, each of which manages a data set, and each of which is used to regularly filter out local hotspot data from the data set it manages. Figure 7 A schematic diagram showing the structure of a data processing device in an exemplary embodiment of the present disclosure is shown; Figure 7 As shown, the data processing device 700 may include an information receiving module 710, a hot data screening module 720 and a data migration module 730. Among them:

[0150] The information receiving module 710 is used to receive the local hotspot data and the heat statistics information of the local hotspot data regularly reported by each of the data processing nodes;

[0151] A hotspot data screening module 720, configured to screen out first global hotspot data that is not stored in a high-performance storage medium from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes;

[0152] The data migration module 730 is used to migrate the first global hotspot data to the high-performance storage medium.

[0153] In an exemplary embodiment of the present disclosure, the hotspot data screening module 720 screens out first global hotspot data that is not stored in a high-performance storage medium from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes, including:

[0154] Filtering global hotspot data from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes;

[0155] The second global hotspot data stored in the high-performance storage medium is removed from the global hotspot data to obtain the first global hotspot data not stored in the high-performance storage medium.

[0156] In an exemplary embodiment of the present disclosure, after filtering out first global hotspot data that is not stored in a high-performance storage medium from the local hotspot data according to the heat statistics information reported by the plurality of data processing nodes, the data hierarchical management node temporarily stores the first global hotspot data in a temporary storage area;

[0157] The data migration module 730 migrates the first global hotspot data to the high-performance storage medium, including:

[0158] Each of the data processing nodes pulls the designated global hotspot data within its management scope from the temporary storage area, and migrates the designated global hotspot data to a high-performance storage medium.

[0159] In an exemplary embodiment of the present disclosure, after migrating the first global hotspot data to the high-performance storage medium, the data migration module 730 is further configured to:

[0160] In response to receiving the migration completion message, the first global hotspot data is written into a hotspot storage file; a plurality of the hotspot storage files are arranged in the order of generation time to form a time-series file linked list;

[0161] And, writing the first global hot spot data and its associated heat statistics information into the heat statistics database.

[0162] In an exemplary embodiment of the present disclosure, the data migration module 730 is further configured to:

[0163] Monitoring the data occupied capacity of the high performance storage medium;

[0164] In response to the data occupying a capacity greater than a preset capacity threshold, acquiring cold data in the high-performance storage medium;

[0165] The cold data is migrated from the high-performance storage medium to a low-performance storage medium.

[0166] In an exemplary embodiment of the present disclosure, the data migration module 730 acquires cold data in the high-performance storage medium, including:

[0167] Acquire a historical hotspot storage file whose generation time is earlier than a preset time point from the time-series file linked list;

[0168] For each of the historical global hot spot data contained in the historical hot spot storage file, query the heat statistics information corresponding to each of the historical global hot spot data from the heat statistics database;

[0169] If the heat statistics information is found, determining whether each of the historical global hot data is the cold data according to the heat statistics information;

[0170] If the heat statistics information is not found, each of the historical global hot data is determined as the cold data.

[0171] In an exemplary embodiment of the present disclosure, after acquiring the cold data in the high-performance storage medium, the data tiering management node temporarily stores the cold data in a temporary storage area;

[0172] The data migration module 730 migrates the cold data from the high-performance storage medium to the low-performance storage medium, including:

[0173] Each of the data processing nodes pulls designated cold data within its management scope from the temporary storage area, and migrates the designated cold data from the high-performance storage medium to the low-performance storage medium.

[0174] In an exemplary embodiment of the present disclosure, after migrating the cold data from the high-performance storage medium to the low-performance storage medium, the data migration module 730 is further configured to:

[0175] In response to receiving a migration completion message, deleting the historical hotspot storage file;

[0176] And, deleting the cold data and its heat statistics information from the heat statistics database.

[0177] In an exemplary embodiment of the present disclosure, the data migration module 730 is further configured to:

[0178] Monitoring the amount of data in the heat statistics database;

[0179] In response to the data volume in the heat statistics database being greater than a preset data volume threshold, the heat statistics database is set to a read-only state, and a new heat statistics database is created.

[0180] In an exemplary embodiment of the present disclosure, the data migration module 730 is further configured to:

[0181] Monitoring the number of the heat statistics databases that have been created;

[0182] In response to the number of the created heat statistics databases being greater than a preset number threshold, determining an invalid database from the created heat statistics databases according to a creation time; the creation time of the invalid database being earlier than a target time;

[0183] The invalid database is deleted.

[0184] The specific details of each module in the above data processing device have been described in detail in the corresponding data processing method, so they will not be repeated here.

[0185] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0186] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0187] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0188] The present disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or may exist independently without being assembled into the electronic device.

[0189] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0190] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0191] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiments.

[0192] In addition, an electronic device capable of implementing the above method is also provided in an embodiment of the present disclosure.

[0193] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0194] Refer to the following Figure 8 The electronic device 800 according to this embodiment of the present disclosure is described. Figure 8 The electronic device 800 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0195] like Figure 8 As shown, the electronic device 800 is in the form of a general computing device. The components of the electronic device 800 may include but are not limited to: at least one processor 810, at least one memory 820, a bus 830 connecting different system components (including the memory 820 and the processor 810), and a display 840.

[0196] The memory stores program codes, which can be executed by the processor 810, so that the processor 810 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processor 810 can perform the following steps: Figure 1 As shown in: Step S110, receiving the local hotspot data and the heat statistics information of the local hotspot data periodically reported by each of the data processing nodes; Step S120, filtering out the first global hotspot data that is not stored in the high-performance storage medium from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes; Step S130, migrating the first global hotspot data to the high-performance storage medium.

[0197] The memory 820 may include a readable medium in the form of a volatile storage, such as a random access memory (RAM) 8201 and / or a cache memory 8202 , and may further include a read-only memory (ROM) 8203 .

[0198] The memory 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0199] Bus 830 may be a bus representing one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0200] The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 800, and / or communicate with any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 850. Furthermore, the electronic device 800 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0201] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A data processing method, characterized in that: A data hierarchical management node applied to a distributed storage system, wherein the distributed storage system further comprises a plurality of data processing nodes, each of which manages a data set, and each of which is used to regularly filter out local hotspot data from the data set it manages, wherein the method comprises: Receiving the local hotspot data and heat statistics information of the local hotspot data regularly reported by each of the data processing nodes; Filtering out first global hotspot data that is not stored in a high-performance storage medium from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes; Migrate the first global hotspot data to the high-performance storage medium.

2. The method according to claim 1, characterized in that The step of filtering out first global hotspot data that is not stored in a high-performance storage medium from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes includes: Filtering global hotspot data from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes; The second global hotspot data stored in the high-performance storage medium is removed from the global hotspot data to obtain the first global hotspot data not stored in the high-performance storage medium.

3. The method according to claim 1 or 2, characterized in that: After filtering out first global hotspot data that is not stored in a high-performance storage medium from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes, the data hierarchical management node temporarily stores the first global hotspot data in a temporary storage area; The migrating the first global hotspot data to the high-performance storage medium includes: Each of the data processing nodes pulls the designated global hotspot data within its management scope from the temporary storage area, and migrates the designated global hotspot data to a high-performance storage medium.

4. The method according to claim 3, characterized in that After migrating the first global hotspot data to the high-performance storage medium, the method further includes: In response to receiving the migration completion message, the first global hotspot data is written into a hotspot storage file; a plurality of the hotspot storage files are arranged in the order of generation time to form a time-series file linked list; And, writing the first global hot spot data and its associated heat statistics information into the heat statistics database.

5. The method according to claim 4, characterized in that The method further comprises: Monitoring the data occupied capacity of the high performance storage medium; In response to the data occupying a capacity greater than a preset capacity threshold, acquiring cold data in the high-performance storage medium; The cold data is migrated from the high-performance storage medium to a low-performance storage medium.

6. The method according to claim 5, characterized in that The obtaining of cold data in the high-performance storage medium includes: Acquire a historical hotspot storage file whose generation time is earlier than a preset time point from the time-series file linked list; For each of the historical global hot spot data contained in the historical hot spot storage file, query the heat statistics information corresponding to each of the historical global hot spot data from the heat statistics database; If the heat statistics information is found, determining whether each of the historical global hot data is the cold data according to the heat statistics information; If the heat statistics information is not found, each of the historical global hot data is determined as the cold data.

7. The method according to claim 6, characterized in that After acquiring the cold data in the high-performance storage medium, the data tiering management node temporarily stores the cold data in a temporary storage area; Migrating the cold data from the high-performance storage medium to the low-performance storage medium includes: Each of the data processing nodes pulls designated cold data within its management scope from the temporary storage area, and migrates the designated cold data from the high-performance storage medium to the low-performance storage medium.

8. The method according to claim 6, characterized in that After migrating the cold data from the high-performance storage medium to the low-performance storage medium, the method further includes: In response to receiving a migration completion message, deleting the historical hotspot storage file; And, deleting the cold data and its heat statistics information from the heat statistics database.

9. The method according to claim 2, characterized in that: The method further comprises: Monitoring the amount of data in the heat statistics database; In response to the data volume in the heat statistics database being greater than a preset data volume threshold, the heat statistics database is set to a read-only state, and a new heat statistics database is created.

10. The method according to claim 9, characterized in that The method further comprises: Monitoring the number of the heat statistics databases that have been created; In response to the number of the created heat statistics databases being greater than a preset number threshold, determining an invalid database from the created heat statistics databases according to a creation time; the creation time of the invalid database being earlier than a target time; The invalid database is deleted.

11. A data processing device, characterized in that: A data hierarchical management node is provided in a distributed storage system, wherein the distributed storage system further comprises a plurality of data processing nodes, each of which manages a data set, and each of which is used to regularly filter out local hotspot data from the data set it manages, wherein the device comprises: An information receiving module, used for receiving the local hotspot data and heat statistics information of the local hotspot data regularly reported by each of the data processing nodes; A hotspot data screening module, configured to screen out first global hotspot data that is not stored in a high-performance storage medium from the local hotspot data according to the heat statistics information reported by the multiple data processing nodes; A data migration module is used to migrate the first global hotspot data to the high-performance storage medium.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 10 is implemented.

13. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the data processing method according to any one of claims 1 to 10 by executing the executable instructions.

Images