Data processing

Abstract

Provided in one or more embodiments of the present specification are a data processing method and apparatus, and an electronic device and a storage medium. The method comprises: during data access of each node in a distributed data system, correspondingly determining popularity values of a plurality of pieces of data in each node, wherein the popularity value is used for representing an access probability of the data; and loading, into a cache space corresponding to the node, some data that ranks higher in a data popularity sequence of each node in the distributed data system, wherein the data popularity sequence is a sequence formed by arranging the plurality of pieces of data in the node in a descending order of the popularity values. By means of the method, data having a high access probability can be loaded into a cache space during data access, such that the hit rate of data in each node is increased when same is accessed, that is, the hit rate of data in a distributed data system is increased when same is accessed, thereby improving the quality of service that the distributed data system provides for the outside.

Description

Data processing Technical Field

[0001] One or more embodiments of this specification relate to the field of database technology, and in particular, to data processing methods and devices, electronic devices, and storage media. Background Art

[0002] With the rapid development of the internet and information technology, data generation is exploding, placing increasing demands on databases and their management. Database data must be loaded into cache (commonly known as memory) before it can be accessed. However, the database cache is often smaller, or even significantly smaller, than the database's persistent storage. This means that only a portion of the database's data can be loaded into the cache at once. If data is found within the cache at the time of access, it can be accessed directly without loading. This is called a data hit.

[0003] In related technologies, distributed data systems frequently undergo expansion, contraction, and data balancing, which results in data transfers between different nodes in the distributed data system. This means that the data within each node frequently changes. Consequently, the hit rate when accessing data within the distributed data system is low, resulting in poor service quality provided by the distributed data system. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a data processing method and apparatus, an electronic device, and a storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions.

[0006] According to a first aspect of one or more embodiments of the present specification, a data processing method is proposed, the method comprising: in a data access process of each node in a distributed data system, determining the heat value of multiple data in each node, wherein the heat value is used to characterize the access probability of the data; loading a front portion of the data in the data heat sequence of each node in the distributed data system into a cache space corresponding to the node, wherein the data heat sequence is a sequence formed by the multiple data in the node according to the heat value from high to low.

[0007] In one embodiment of the present specification, during the data access process of each node in the distributed data system, the heat values ​​of multiple data in each node are correspondingly determined, including: determining the heat value of the data based on the most recent access time and / or access frequency of the data in the node.

[0008] In one embodiment of the present specification, determining the heat value of the data based on the most recent access time and access frequency of the data in the node includes: performing weighted summation of the most recent access time and access frequency of the data in the node to obtain the heat value of the data.

[0009] In one embodiment of the present specification, the method further includes: in response to the space occupied by the data being less than a space threshold, determining the weight of the data's most recent access time as a first weight and the weight of the data's access frequency as a second weight, wherein the first weight is greater than the second weight; in response to the space occupied by the data being not less than a space threshold, determining the weight of the data's most recent access time as a third weight and the weight of the data's access frequency as a fourth weight, wherein the third weight is less than the fourth weight.

[0010] In one embodiment of the present specification, during the data access process of each node in the distributed data system, the heat values ​​of multiple data in each node are correspondingly determined, including: during the data access process of each node in the distributed data system, the heat values ​​of N data in each node are correspondingly determined, wherein the space occupied by the N data is greater than the cache space corresponding to the node, and N is an integer greater than or equal to 1.

[0011] In one embodiment of the present specification, during the data access process of each node in the distributed data system, the heat values ​​of multiple data in each node are correspondingly determined, including: in response to the data of any node in the distributed data system being accessed, determining the heat values ​​of multiple data in the node.

[0012] In one embodiment of the present specification, loading the front portion of data in the data heat sequence of each node in the distributed data system into the cache space corresponding to the node includes: loading the front M data in the data heat sequence of the node into the cache space corresponding to the node, wherein the space occupied by the M data is less than or equal to the effective space of the cache space, and M is an integer greater than or equal to 1.

[0013] In one embodiment of the present specification, the method further includes at least one of the following: in response to target data of any node in the distributed data system being transferred or deleted, updating the data heat sequence of the node according to the heat value of the target data; in response to target data being written to any node in the distributed data system, updating the data heat sequence of the node according to the heat value of the target data.

[0014] In one embodiment of the present specification, the method further includes: in response to changes in the top M data in the data heat sequence of the node during update, adjusting the data in the cache space corresponding to the node according to the updated data heat sequence.

[0015] In one embodiment of the present specification, the method further includes: saving the data heat sequence of each node in the distributed data system to the disk space corresponding to the node.

[0016] According to a second aspect of one or more embodiments of the present specification, a data processing device is proposed, comprising: a heat determination module for determining the heat value of multiple data in each node during data access at each node in a distributed data system, wherein the heat value is used to characterize the access probability of the data; and a data cache module for loading a front portion of data in a data heat sequence of each node in the distributed data system into a cache space corresponding to the node, wherein the data heat sequence is a sequence of multiple data in the node formed from high to low according to heat values.

[0017] In one embodiment of the present specification, the heat determination module is used to determine the heat value of the data according to the most recent access time and / or access frequency of the data in the node.

[0018] In one embodiment of the present specification, the heat determination module is used to determine the heat value of the data based on the most recent access time and access frequency of the data in the node, and is used to: perform weighted summation of the most recent access time and access frequency of the data in the node to obtain the heat value of the data.

[0019] In one embodiment of the present specification, the device also includes a weight module for: in response to the space occupied by the data being less than a space threshold, determining the weight of the data's most recent access time as a first weight and the weight of the data's access frequency as a second weight, wherein the first weight is greater than the second weight; in response to the space occupied by the data being not less than a space threshold, determining the weight of the data's most recent access time as a third weight and the weight of the data's access frequency as a fourth weight, wherein the third weight is less than the fourth weight.

[0020] In one embodiment of the present specification, the heat determination module is used to: during the data access process of each node in a distributed data system, determine the heat value of N data in each node, wherein the space occupied by the N data is greater than the cache space corresponding to the node, and N is an integer greater than or equal to 1.

[0021] In one embodiment of the present specification, the heat determination module is used to: in response to data of any node in the distributed data system being accessed, determine the heat values ​​of multiple data in the node.

[0022] In one embodiment of the present specification, the data cache module is used to: load the top M data in the data heat sequence of the node into the cache space corresponding to the node, wherein the space occupied by the M data is less than or equal to the effective space of the cache space, and M is an integer greater than or equal to 1.

[0023] In one embodiment of the present specification, the device also includes an update module, which is used for at least one of the following: in response to the target data of any node in the distributed data system being transferred or deleted, updating the data heat sequence of the node according to the heat value of the target data; in response to the target data being written to any node in the distributed data system, updating the data heat sequence of the node according to the heat value of the target data.

[0024] In one embodiment of the present specification, the device further includes an adjustment module for: in response to changes in the top M data in the data heat sequence of the node during an update, adjusting the data in the cache space corresponding to the node according to the updated data heat sequence.

[0025] In one embodiment of the present specification, the apparatus further includes a persistence module configured to save the data heat sequence of each node in the distributed data system to a disk space corresponding to the node.

[0026] According to a fourth aspect of one or more embodiments of this specification, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor executes the executable instructions to implement the method described in the first aspect.

[0027] According to a fifth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0028] The technical solutions provided by the embodiments of this specification may include the following beneficial effects.

[0029] The data processing method provided in the embodiments of this specification can determine the heat value of multiple data in each node during the data access process of each node in a distributed data system, and load the front part of the data heat sequence of each node in the distributed data system into the cache space corresponding to the node. Since the heat value of the data represents the access probability of the data, this method can load data with a higher access probability into the cache space during the data access process, thereby improving the hit rate of data in each node when it is accessed, that is, improving the hit rate of data in the distributed data system when it is accessed, so as to improve the quality of service provided by the distributed data system to the outside world. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] FIG1 is a flowchart of a data processing method provided by an exemplary embodiment.

[0031] FIG2 is a schematic diagram of a data heat sequence update process provided by an exemplary embodiment.

[0032] FIG3 is a schematic diagram of a capacity expansion process of a distributed data system provided by an exemplary embodiment.

[0033] FIG4 is a schematic diagram of a scaling-down process of a distributed data system provided by an exemplary embodiment.

[0034] FIG5 is a schematic structural diagram of a device provided by an exemplary embodiment.

[0035] FIG6 is a block diagram of a data processing device provided by yet another exemplary embodiment. DETAILED DESCRIPTION

[0036] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0037] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0038] With the rapid development of the internet and information technology, data generation is exploding, placing increasing demands on databases and their management. Database data must be loaded into cache (commonly known as memory) before it can be accessed. However, the database cache is often smaller, or even significantly smaller, than the database's persistent storage. This means that only a portion of the database's data can be loaded into the cache at once. If data is found within the cache at the time of access, it can be accessed directly without loading. This is called a data hit.

[0039] In related technologies, distributed data systems frequently undergo expansion, contraction, and data balancing, which results in data transfers between different nodes in the distributed data system. This means that the data within each node frequently changes. Consequently, the hit rate when accessing data within the distributed data system is low, resulting in poor service quality provided by the distributed data system.

[0040] Based on this, on the first aspect, at least one embodiment of this specification provides a data processing method, which can select in real time the data with a higher access probability in each node in the distributed data system and load it into the cache space, so as to improve the hit rate of the data when it is accessed in the distributed data system, especially when the distributed data system undergoes data transfer such as expansion, reduction, and data balancing (that is, data is transferred between different nodes), it can still ensure the hit rate of the data when it is accessed.

[0041] Exemplarily, the method can be applied to a database management system, that is, a database management system of a distributed data system.

[0042] First, a brief introduction to distributed data systems is provided. A distributed data system includes multiple nodes, each of which can be an independent storage unit, such as a physical server or a virtual server. Each node in a distributed data system is configured with a corresponding cache space. The cache space can be part of the node's own storage space, or part of the storage space of a specific storage unit within the distributed data system (i.e., this specific storage unit provides independent cache space for each node). If the cache space is part of the node's own storage space, then the storage space in the node other than the cache space is persistent space. If the cache space is part of the storage space of a specific storage unit within the distributed data system, then all storage space in the node is persistent space. The difference between persistent space and cache space is that data stored in persistent space will not disappear due to factors such as downtime or power outages, while data stored in cache space will disappear due to factors such as downtime or power outages. If a specific storage unit within the distributed data system provides cache space for each node, then the database management system can also be installed on this specific storage unit.

[0043] Distributed data systems often require operations such as scaling up and down, load balancing, and replacing failed nodes. During scaling down, the number of nodes in a distributed data system decreases, and the data from the scaled-down nodes is transferred to one or more target nodes. During scaling up, data from one or more source nodes is migrated to one or more target nodes. Data migration between nodes is also required when performing load balancing or adding new data replicas.

[0044] Please refer to FIG. 1 , which exemplarily shows the flow of the data processing method, including steps S101 to S102 .

[0045] In step S101 , during a data access process of each node in a distributed data system, the heat values ​​of multiple data in each node are determined, wherein the heat values ​​are used to characterize the access probability of the data.

[0046] Among them, data access to nodes within a distributed data system may include data reading, data writing, data transfer, data deletion, etc. For example, when a database management system responds to an instruction from an upper-layer application and reads data from a node within a distributed data system, it is considered data reading; for another example, when a database management system responds to an instruction from an upper-layer application and writes data to a node within a distributed data system, it is considered data writing; for another example, when a database management system responds to an instruction from an upper-layer application and transfers data from a node within a distributed data system to another node, it is considered data transfer; for another example, when a database management system responds to an instruction from an upper-layer application and deletes data from a node within a distributed data system, it is considered data deletion. It should be understood that data access also includes other forms, and the above forms are only examples and do not constitute a form of limitation on data access.

[0047] The heat value of the data may be represented by the most recent access time of the data, the frequency of data access, etc. That is, this step may determine the heat value of the data according to the most recent access time and / or frequency of access of the data in the node.

[0048] For example, if the heat value of data is determined based on the most recent access time of the data in the node, the later the most recent access time (i.e., the smaller the time interval from the current time), the higher the heat value, and the earlier the most recent access time (i.e., the larger the time interval from the current time), the lower the heat value.

[0049] For example, if the popularity of data is determined based on the access frequency of the data in the node, the higher the access frequency, the higher the popularity, and the lower the access frequency, the lower the popularity. It should be understood that the access frequency of data in the node can be determined by the number of accesses within a preset time period before the current moment.

[0050] As another example, if the popularity of the data is determined based on the most recent access time and access frequency of the data in the node, the most recent access time and access frequency of the data in the node can be weighted and summed to obtain the popularity of the data. Before performing the weighted sum of the most recent access time and access frequency, the most recent access time and access frequency can be unified into a value representing the access popularity of the data, and then a weighted sum is performed according to their respective weights, where the sum of the weights can be 1.

[0051] For example, the inverse of the ratio of the interval between the most recent access time and the current time to the preset duration (the ratio can also be rounded, etc.) is used as the value representing the data access heat; the ratio of the access frequency to the preset number of times is used as the value representing the data access heat.

[0052] For example, the weight of the most recent access time and the weight of the access frequency can be preset. Alternatively, the weight of the most recent access time and the weight of the access frequency corresponding to each data can be determined separately based on information such as the space occupied by each data (commonly known as data memory, which can be expressed in the number of bits of data). For example: in response to the space occupied by the data being less than the space threshold, the weight of the most recent access time of the data can be determined to be the first weight and the weight of the access frequency of the data can be determined to be the second weight, wherein the first weight is greater than the second weight; in response to the space occupied by the data being not less than the space threshold, the weight of the most recent access time of the data can be determined to be the third weight and the weight of the access frequency of the data can be determined to be the fourth weight, wherein the third weight is less than the fourth weight. That is, when the space occupied by the data is small, the weight of the most recent access time is greater than the weight of the access frequency; when the space occupied by the data is large, the weight of the most recent access time is less than the weight of the access frequency.

[0053] The relationship between the heat values ​​of different data within a node can represent the probability of different data being accessed. This relative concept is instructive for selecting data to load into the cache. Optionally, after determining the heat values ​​of the data within a node, a data heat sequence (or heat map) for the node can be determined. This is a sequence of multiple data within the node, ranked from high to low in terms of heat value.

[0054] The data heat sequence may include a sequence of all data in a node in descending order of heat value. In this case, this step may determine the heat value of each data in each node during the data access process of each node in the distributed data system.

[0055] Alternatively, the data heat sequence may include a sequence of N data (i.e., partial data) in a node, formed in descending order of heat values. In this case, this step may determine the heat values ​​of the N data in each node during the data access process of each node in the distributed data system. It should be understood that the space occupied by the N data may be larger than the cache space corresponding to the node, and N is an integer greater than or equal to 1; that is, the amount of data in the data heat sequence determined and maintained for each node by this method is greater than, or even much greater than, the amount of data that can be cached in the cache space corresponding to the node. This enables, when part or all of the data in the cache space is deleted or transferred, to quickly select part of the data with a higher access probability from the remaining data in the node and load it into the cache space, thereby improving the hit rate when the node data is accessed.

[0056] It should be understood that as time goes by, the most recent access time and access frequency of the data in the node will change, but if the data in the node has not been accessed, the relative relationship of the heat values ​​between the data will not change. As mentioned above, the relative relationship of the heat values ​​between the data in the node is instructive for selecting part of the data to be loaded into the cache space. Therefore, this step can determine the heat values ​​of multiple data in the node in response to the data of any node in the distributed data system being accessed. In other words, when the data in a node is accessed, the relative relationship of the heat values ​​between the data in the node changes, so the heat value of the data in the node is re-determined to update the relative relationship of the heat values ​​between the data, such as the data heat sequence.

[0057] It should be understood that the heat value of the data in each node in the distributed data system and the data heat sequence of the node can be saved in the cache space of the node for easy reading. Of course, the data heat sequence of each node in the distributed data system can also be saved (i.e., persisted) in the disk space corresponding to the node (i.e., persistent space) to avoid the heat value and data heat sequence in the cache space from disappearing due to downtime, power outage, etc., so that after the node returns to normal, data can be directly loaded into the cache space corresponding to each node according to the data heat sequence in the disk space to improve the data access hit rate of each node after downtime or power outage. For example, the data heat value and data heat sequence can be updated in real time in the cache space; and updated in the disk space according to preset intervals.

[0058] Please refer to Figure 2, which illustrates a schematic diagram of adjusting the data heat sequence of a node according to step S101. As can be seen from the figure, data within the master replica node can be processed based on service access traffic, for example, by moving data in the cache space to local storage or loading data from local storage into the cache space. Furthermore, the data heat sequence can be updated based on service access traffic. Finally, the updated data heat sequence within the master replica node is synchronized to each slave replica node.

[0059] In step S102, a portion of data at the front of the data heat sequence of each node in the distributed data system is loaded into the cache space corresponding to the node, wherein the data heat sequence is a sequence formed by multiple data in the node in descending order of heat value.

[0060] For example, for each node in the distributed data system, the top M data in the data heat sequence of the node are loaded into the cache space corresponding to the node, wherein the occupied space of the M data is less than or equal to the effective space of the cache space, and M is an integer greater than or equal to 1. It should be understood that the effective space of the cache space can be all or part of the cache space, for example, a preset occupiable proportion of the cache space.

[0061] Since this step is to load the front part of the data heat sequence of the node into the cache space corresponding to the node, the data loaded into the cache space corresponding to the node can be updated synchronously when the data heat sequence of the node is updated (for example, due to step S101, or due to other reasons). In other words, when the data in a certain node is accessed, the relative relationship between the heat values ​​of the data in the node changes, so the heat value of the data in the node is re-determined to update the relative relationship between the heat values ​​of the data, such as the data heat sequence; and then the data loaded into the cache space corresponding to the node is updated according to the updated data heat sequence, that is, the front part of the data in the updated data heat sequence of the node is loaded into the cache space corresponding to the node.

[0062] For example, data a of node A is relatively early in its data heat sequence, so data a is loaded into the cache space corresponding to node A; data a of node A is deleted by the database management system in response to instructions from the upper-level application, so the data with the highest heat value among the data that node A has not loaded into the cache space can be added to the cache space.

[0063] The data processing method provided in the embodiments of this specification can determine the heat value of multiple data in each node during the data access process of each node in a distributed data system, and load the front part of the data heat sequence of each node in the distributed data system into the cache space corresponding to the node. Since the heat value of the data represents the access probability of the data, this method can load data with a higher access probability into the cache space during the data access process, thereby improving the hit rate of data in each node when it is accessed, that is, improving the hit rate of data in the distributed data system when it is accessed, so as to improve the quality of service provided by the distributed data system to the outside world.

[0064] As previously mentioned, the embodiments of this specification can update the heat value of the data in the node, the data heat sequence of the node, and the data in the cache space corresponding to the node during the data access process of the node in the distributed data system. When the distributed data system undergoes operations such as expansion, contraction, data balancing, and adding data copies, data access to at least one node is involved. Therefore, when the distributed data system undergoes operations such as expansion, contraction, data balancing, and adding data copies, the data in the cache space corresponding to the node involved can be adjusted to ensure that the data in the node involved has a high hit rate when being accessed.

[0065] Operations like expansion, contraction, and data balancing in distributed data systems primarily involve data transfer, deletion, and writing. The data heat value determined by this method will be used as a data attribute as the data is transferred, deleted, or written. Therefore, during these operations, you can maintain the node data heat sequence according to the following optional example.

[0066] Optional Example 1: In response to target data of any node in the distributed data system being transferred or deleted, the data heat sequence of the node is updated according to the heat value of the target data. Optionally, the target data is deleted from the data heat sequence of the node.

[0067] For example, when a distributed data system expands, new nodes are added to the existing nodes, and some or all of the existing nodes will transfer some of their data to the new nodes, so some of the data in the existing nodes will be transferred. For another example, when a distributed data system shrinks, some of the existing nodes will be retained while data in other nodes will be transferred to the retained nodes, so some of the data in the nodes that are not retained will be transferred.

[0068] Furthermore, in response to changes in the top M data in the node's data heat sequence during an update, the data in the cache space corresponding to the node may be adjusted according to the updated data heat sequence, where M is the upper limit of the amount of data loaded into the cache space corresponding to the node. For example, if the top data in the node's data heat sequence is deleted during an update, the data with the highest heat value among the data not loaded into the cache space of the node may be added to the cache space.

[0069] Optional Example 2: In response to target data being written to any node in the distributed data system, the data heat sequence of the node is updated according to the heat value of the target data. Optionally, the target data is added to the data heat sequence of the node according to the heat value of the target node.

[0070] For example, when a distributed data system expands, new nodes are added to the existing nodes, and some or all of the existing nodes will transfer some of their data to the new nodes, so data will be written to the new nodes. For another example, when a distributed data system shrinks, some of the existing nodes will be retained, while data from other nodes will be transferred to the remaining nodes, so data will be written to the remaining nodes.

[0071] Furthermore, in response to changes in the top M data in the data heat sequence of the node during an update, the data in the cache space corresponding to the node can be adjusted according to the updated data heat sequence, where M is the upper limit of the amount of data loaded into the cache space corresponding to the node. For example, if the target data is added to the Mth position in the back row of the data heat sequence of the node, the data with the lowest heat value in the cache space corresponding to the node can be deleted, and the target data can be loaded into the cache space.

[0072] Next, the effect of executing this method when the distributed data system is expanded will be described in detail with reference to FIG3 .

[0073] Figure 3 shows the heat map of each node before expansion (i.e., data heat sequence) and the heat map of each node after expansion (i.e., data heat sequence). The heat map of the node before expansion includes cached data and uncached data, and the heat map of the node after expansion includes cached data, new cached data, and uncached data.

[0074] The expansion process in Figure 3 is to add a new node C on the basis of the original two nodes A and Node B. Therefore, both nodes A and B transfer part of the data to node C. Among them, the data retained by node A is called dataset 1, and the data transferred to node C is called dataset 2. The data retained by node B is called dataset 3, and the data transferred to node C is called dataset 4.

[0075] As can be seen from Figure 3, since dataset 2 in node A is transferred to node C, the cached data in dataset 2 is also transferred, and the space occupied by the cached data in dataset 2 in node A is released. Since this method determines the heat value of each data in dataset 1 in real time, the part of the data with higher heat values ​​in the uncached data of dataset 1 before expansion can be loaded into the cache space of node A, so that after dataset 2 is transferred, the part of the data with higher heat values ​​in the remaining data can be loaded into the cache space immediately, thereby improving the utilization of the cache space and the hit rate of data access.

[0076] As can be seen from Figure 3, since data set 4 in node B is transferred to node C, the cached data in data set 4 is also transferred, and the space occupied by the cached data in data set 2 in node B is released. Since this method determines the heat value of each data in data set 3 in real time, the part of the data with higher heat values ​​in the uncached data of data set 3 before expansion can be loaded into the cache space of node B, so that after data set 4 is transferred, the part of the data with higher heat values ​​in the remaining data can be loaded into the cache space immediately, thereby improving the utilization of the cache space and the hit rate of data access.

[0077] As can be seen from FIG3 , when dataset 2 in node A and dataset 4 in node B are transferred to node C, the heat map of dataset 2 (i.e., the sequence of heat values ​​of each data in dataset 2 from high to low) and the heat map of dataset 4 (i.e., the sequence of heat values ​​of each data in dataset 4 from high to low) are also transferred to node C. Therefore, part of the data in dataset 2 that was originally loaded into the cache space of dataset A and part of the data in dataset 4 that was originally loaded into the cache space of dataset B can be directly loaded into the cache space of node C; if there is still free space in the cache space, part of the data with higher heat values ​​can be further loaded into the cache space of node C according to the heat maps of datasets 2 and 4 to make full use of the cache space. It can be seen that during the transfer of datasets 2 and 4, their heat maps are transferred synchronously, so that data with higher heat values ​​can be loaded immediately after the transfer of datasets 2 and 4, thereby improving the utilization of the cache space and the hit rate of data access.

[0078] Next, the effect of executing the method when the distributed data system is scaled down will be described in detail with reference to FIG4 .

[0079] Figure 4 shows the heat map of each node before scaling down (i.e., data heat sequence) and the heat map of each node after scaling down (i.e., data heat sequence). The heat map of the node before scaling down includes old cached data and uncached data, and the heat map of the node after scaling down includes old cached data, newly eliminated cached data (i.e., data in the old cached data transferred from the cache space to the disk) and uncached data.

[0080] The scaling process in Figure 4 reduces the original nodes D and E to just node D. Therefore, node E transfers all its data to node D. The data in node D before scaling is called dataset 5, and the data in node E before scaling is called dataset 6. It should be understood that datasets 5 and 6 occupied the cache space of nodes D and E before scaling, but after scaling, datasets 5 and 6 only occupy the cache space of node D. That is, the cache space is also reduced during the scaling process.

[0081] As can be seen from Figure 4, since this method determines the heat value of each data in data sets 5 and 6 in real time, and the heat value of each data in data set 6 is also transferred to node D at any time during the process of transferring data set 6 to node D, all data in data sets 5 and 6 can be re-sorted according to the heat value, and some data with higher heat values ​​in the sorting results are loaded into the cache space of node D. Then, some old cache data in data set 5 are eliminated to form new eliminated cache data, and some old cache data in data set 6 are eliminated to form new eliminated cache data. Therefore, after nodes D and E are reduced in size, some data with higher heat values ​​in all data (i.e., data sets 5 and 6) can be immediately retained in the cache space, thereby improving the utilization of the cache space and the hit rate of data access.

[0082] As can be seen, during the transfer of dataset 6, its heat map was also transferred simultaneously. This allowed the high-heat data in datasets 5 and 6 to be loaded immediately after they were merged to node D, thereby improving cache space utilization and data access hit rate. Imagine if, after dataset 6 was merged to node D, the data in the cache space was not adjusted according to this method, the hit rate of the data in dataset 6 would be 0 when accessed.

[0083] FIG5 is a schematic structural diagram of a device provided by an exemplary embodiment. Referring to FIG5 , at the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and may also include hardware required for other tasks. One or more embodiments of this specification may be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or logic devices.

[0084] Please refer to Figure 6. The data processing device can be applied to the device shown in Figure 5 to implement the technical solution of this specification. The device includes: a heat determination module 601, which is used to determine the heat value of multiple data in each node during the data access process of each node in the distributed data system, wherein the heat value is used to characterize the access probability of the data; a data cache module 602, which is used to load the front part of the data heat sequence of each node in the distributed data system into the cache space corresponding to the node, wherein the data heat sequence is a sequence formed by the multiple data in the node according to the heat value from high to low.

[0085] In one embodiment of the present specification, the heat determination module is used to determine the heat value of the data according to the most recent access time and / or access frequency of the data in the node.

[0086] In one embodiment of the present specification, the heat determination module is used to determine the heat value of the data based on the most recent access time and access frequency of the data in the node, and is used to: perform weighted summation of the most recent access time and access frequency of the data in the node to obtain the heat value of the data.

[0087] In one embodiment of the present specification, the device also includes a weight module for: in response to the space occupied by the data being less than a space threshold, determining the weight of the data's most recent access time as a first weight and the weight of the data's access frequency as a second weight, wherein the first weight is greater than the second weight; in response to the space occupied by the data being not less than a space threshold, determining the weight of the data's most recent access time as a third weight and the weight of the data's access frequency as a fourth weight, wherein the third weight is less than the fourth weight.

[0088] In one embodiment of the present specification, the heat determination module is used to: during the data access process of each node in a distributed data system, determine the heat value of N data in each node, wherein the space occupied by the N data is greater than the cache space corresponding to the node, and N is an integer greater than or equal to 1.

[0089] In one embodiment of the present specification, the heat determination module is used to: in response to data of any node in the distributed data system being accessed, determine the heat values ​​of multiple data in the node.

[0090] In one embodiment of the present specification, the data cache module is used to: load the top M data in the data heat sequence of the node into the cache space corresponding to the node, wherein the space occupied by the M data is less than or equal to the effective space of the cache space, and M is an integer greater than or equal to 1.

[0091] In one embodiment of the present specification, the device also includes an update module, which is used for at least one of the following: in response to the target data of any node in the distributed data system being transferred or deleted, updating the data heat sequence of the node according to the heat value of the target data; in response to the target data being written to any node in the distributed data system, updating the data heat sequence of the node according to the heat value of the target data.

[0092] In one embodiment of the present specification, the device further includes an adjustment module for: in response to changes in the top M data in the data heat sequence of the node during an update, adjusting the data in the cache space corresponding to the node according to the updated data heat sequence.

[0093] In one embodiment of the present specification, the apparatus further includes a persistence module configured to save the data heat sequence of each node in the distributed data system to a disk space corresponding to the node.

[0094] One or more embodiments of this specification also provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0095] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0096] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0097] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0098] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0099] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0100] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0101] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0102] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0103] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0104] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0105] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A data processing method, comprising: During a data access process at each node in the distributed data system, a heat value of a plurality of data in each node is correspondingly determined, wherein the heat value is used to characterize an access probability of the data; The front part of the data heat sequence of each node in the distributed data system is loaded into the cache space corresponding to the node, wherein the data heat sequence is a sequence formed by multiple data in the node according to the heat value from high to low.

2. The data processing method according to claim 1, wherein during the data access process of each node in the distributed data system, determining the heat values ​​of the plurality of data in each node comprises: The heat value of the data is determined according to the most recent access time and / or access frequency of the data in the node.

3. The data processing method according to claim 2, wherein determining the heat value of the data based on the most recent access time and access frequency of the data in the node comprises: The most recent access time and access frequency of the data in the node are weightedly summed to obtain the heat value of the data.

4. The data processing method according to claim 3, further comprising: In response to the space occupied by the data being less than a space threshold, determining a weight of a most recent access time of the data as a first weight and a weight of an access frequency of the data as a second weight, wherein the first weight is greater than the second weight; In response to the space occupied by the data being no less than a space threshold, the weight of the most recent access time of the data is determined to be a third weight and the weight of the access frequency of the data is determined to be a fourth weight, wherein the third weight is less than the fourth weight.

5. The data processing method according to any one of claims 1 to 4, wherein during the data access process of each node in the distributed data system, determining the heat values ​​of the plurality of data in each node comprises: During the data access process of each node in the distributed data system, the heat values ​​of N data in each node are correspondingly determined, wherein the space occupied by the N data is greater than the cache space corresponding to the node, and N is an integer greater than or equal to 1.

6. The data processing method according to any one of claims 1 to 4, wherein during the data access process of each node in the distributed data system, determining the heat values ​​of the plurality of data in each node comprises: In response to data of any node in the distributed data system being accessed, heat values ​​of multiple data in the node are determined.

7. The data processing method according to claim 1, wherein loading the first portion of the data heat sequence of each node in the distributed data system into the cache space corresponding to the node comprises: The top M data in the data heat sequence of the node are loaded into the cache space corresponding to the node, wherein the space occupied by the M data is less than or equal to the effective space of the cache space, and M is an integer greater than or equal to 1.

8. The data processing method according to claim 7, further comprising at least one of the following: In response to target data of any node in the distributed data system being transferred or deleted, updating the data heat sequence of the node according to the heat value of the target data; In response to target data being written to any node in the distributed data system, the data heat sequence of the node is updated according to the heat value of the target data.

9. The data processing method according to claim 8, further comprising: In response to changes in the top M data in the data heat sequence of the node during update, the data in the cache space corresponding to the node is adjusted according to the updated data heat sequence.

10. The data processing method according to claim 1, further comprising: The data heat sequence of each node in the distributed data system is saved to the disk space corresponding to the node.

11. A data processing device, comprising: a heat determination module, configured to determine heat values ​​of a plurality of data in each node during a data access process of each node in a distributed data system, wherein the heat values ​​are used to characterize access probabilities of the data; The data cache module is used to load the front part of the data heat sequence of each node in the distributed data system into the cache space corresponding to the node, wherein the data heat sequence is a sequence formed by multiple data in the node according to the heat value from high to low.

12. An electronic device comprising: processor; as well as a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 10 by running the executable instructions.

13. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images