Data processing method and device, electronic equipment and readable medium

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment and a readable medium, and relates to the technical field of computers, in the method, in response to a data access operation, a key name accessed by the data access operation is determined as a key name to be counted; and based on a hash function corresponding to a first fragment in a preset fragment sequence, determining a statistical bit corresponding to the to-be-counted key name in the first fragment. And setting a statistical bit corresponding to the to-be-counted key name in the first fragment as a preset value. Under the condition that the number of key names recorded in the preset fragment sequence reaches a preset updating condition, a second fragment in the preset fragment sequence is removed, a new fragment is added to the preset fragment sequence, and the second fragment is located behind the first fragment. And identifying the hot data based on a preset fragment sequence. Therefore, based on the key names of the key-value pairs and the maintained preset fragment sequence, whether a certain key-value pair is hot data can be conveniently distinguished.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, device, electronic device and readable medium. Background Art

[0002] At present, with the continuous development of computer technology, there are more and more data-intensive applications such as search engine applications, distributed data system applications, OLTP / OLAP databases, etc. Correspondingly, the amount of data required to be stored is growing, and the application of key-value storage systems is becoming more and more extensive. Among them, the key-value storage system, also known as the key-value database, is one of the basic components of computer systems and software, and is the most basic underlying storage engine of the data center. As a data storage and retrieval system, the performance and storage efficiency of the key-value storage system directly affect the operating efficiency of the system and user experience. The key-value storage system can store key-value objects.<key.value> ,The key-value storage system stores and retrieves associated key values through uniquely identified key names, making data access and management more efficient.

[0003] In order to improve the performance of key-value storage systems, how to count data to facilitate the identification of hot data has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] Embodiments of the present invention provide a data processing method, device, electronic device, and readable medium, which can solve the problem of how to perform statistics on data to facilitate the distinction of hot data.

[0005] In order to solve the above problems, an embodiment of the present invention discloses a data processing method, which includes:

[0006] In response to a data access operation, determining the key name accessed by the data access operation as the key name to be counted;

[0007] Determining, based on a hash function corresponding to a first shard in a preset shard sequence, a statistical bit corresponding to the key name to be counted in the first shard; the preset shard sequence includes M shards, one shard corresponds to multiple statistical bits, and the first shard is the first K consecutive shards in the preset shard sequence, where K is less than M;

[0008] Setting the statistical bit corresponding to the key name to be counted in the first shard to a preset value; the preset value is different from the default value of the statistical bit;

[0009] When the number of key names recorded in the preset shard sequence reaches a preset update condition, remove the second shard in the preset shard sequence and add a new shard to the preset shard sequence; the second shard is located after the first shard;

[0010] Hot data is identified based on the preset sharding sequence.

[0011] In another aspect, an embodiment of the present invention discloses a data processing device, comprising:

[0012] A first determining module is configured to, in response to a data access operation, determine a key name accessed by the data access operation as a key name to be counted;

[0013] a second determining module, configured to determine, based on a hash function corresponding to a first shard in a preset shard sequence, a statistical bit corresponding to the key name to be counted in the first shard; the preset shard sequence includes M shards, one shard corresponds to multiple statistical bits, the first shard is the first K consecutive shards in the preset shard sequence, and K is less than M;

[0014] A setting module, configured to set the statistical bit corresponding to the key name to be counted in the first shard to a preset value; the preset value is different from the default value of the statistical bit;

[0015] A first processing module is configured to remove a second shard from the preset shard sequence and add a new shard to the preset shard sequence when the number of key names recorded in the preset shard sequence reaches a preset update condition; the second shard is located after the first shard;

[0016] An identification module is used to identify hot data based on the preset fragmentation sequence.

[0017] On the other hand, an embodiment of the present invention discloses an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the aforementioned method.

[0018] An embodiment of the present invention further discloses a machine-readable medium having instructions stored thereon. When executed by one or more processors, the processors are enabled to execute the method described above.

[0019] The embodiments of the present invention include the following advantages: The data processing method provided by the embodiments of the present invention, in response to the data access operation, determines the key name accessed by the data access operation as the key name to be counted. Based on the hash function corresponding to the first shard in the preset shard sequence, determine the statistical bit corresponding to the key name to be counted in the first shard; the preset shard sequence includes M shards, one shard corresponds to multiple statistical bits, the first shard is the first K consecutive shards in the preset shard sequence, and K is less than M. Set the statistical bit corresponding to the key name to be counted in the first shard to a preset value, which is different from the default value of the statistical bit. When the number of key names recorded in the preset shard sequence reaches the preset update condition, remove the second shard in the preset shard sequence, and add a new shard to the preset shard sequence, and the second shard is located after the first shard. Identify hot data based on the preset shard sequence. In this way, access statistics are collected for accessed key names using a preset sharding sequence. By automatically updating the shards in the preset sharding sequence, the preset sharding sequence records recently accessed key names. This allows for convenient identification of whether a key-value pair is hot data based on the key name and the maintained preset sharding sequence. Furthermore, calculations based on the hash function corresponding to the shards can conveniently determine the corresponding statistical position in the shard for the key name to be counted. This improves overall processing efficiency to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 This is a flowchart of a data processing method provided by an embodiment of the present invention;

[0022] Figure 2 This is a sharding schematic diagram provided by an embodiment of the present invention;

[0023] Figure 3 1 is a schematic structural diagram of a hot and cold data distinguisher provided by an embodiment of the present invention;

[0024] Figure 4 This is a schematic diagram of an application of a hot and cold data distinguisher provided by an embodiment of the present invention;

[0025] Figure 5 is a block diagram of a data processing device provided by an embodiment of the present invention;

[0026] Figure 6 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0028] In large-scale data storage scenarios, the access frequency distribution of data is usually significantly unbalanced. A small amount of data is frequently accessed, while the majority of data is accessed less frequently. How to efficiently identify and optimize the management of hot data and cold data is one of the key technologies in the optimization of key-value storage systems. In an embodiment of the present invention, hot data refers to key-value pairs with a high access frequency. Cold data refers to key-value pairs with a low access frequency. The data processing method provided in an embodiment of the present invention can record the accessed key-value pairs based on a preset sharding sequence and the accessed key name, thereby facilitating the distinction of hot data based on the maintained preset sharding sequence.

[0029] The data processing method is described in detail below.

[0030] Figure 1 This is a flow chart of the steps of a data processing method provided by an embodiment of the present invention. Figure 1 As shown, the data processing method may include the following steps:

[0031] Step 101: In response to a data access operation, determine the key name accessed by the data access operation as the key name to be counted.

[0032] Step 102: Based on the hash function corresponding to the first shard in the preset shard sequence, determine the statistical bit corresponding to the key name to be counted in the first shard; the preset shard sequence includes M shards, one shard corresponds to multiple statistical bits, and the first shard is the first K consecutive shards in the preset shard sequence, and K is less than M.

[0033] Step 103: Set the statistical bit corresponding to the key name to be counted in the first shard to a preset value; the preset value is different from the default value of the statistical bit.

[0034] Step 104: When the number of key names recorded in the preset shard sequence reaches a preset update condition, remove the second shard in the preset shard sequence and add a new shard to the preset shard sequence; the second shard is located after the first shard.

[0035] Step 104: Identify hot data based on the preset sharding sequence.

[0036] Among them, the data access operation includes the key-value pair write operation and the key-value pair read operation of the key-value storage system. Accordingly, the key name in the key-value pair written by the key-value pair write operation and the key name in the key-value pair read by the key-value pair read operation are the accessed key names. Accordingly, these key names can be used as key names to be counted. The preset shard sequence is a shard sequence created in advance for counting the access popularity of key-value pairs. The preset shard sequence consists of M shards, each of which includes multiple statistical bits. Specifically, one shard is an array including N bits. Among them, the specific values of M and N can be set according to actual needs, and the embodiment of the present invention does not limit this. For example, assuming that M and N are both 5, the preset shard sequence includes 5 arrays, each of which is 5 bits.

[0037] Among them, the first K consecutive shards in the preset shard sequence are the first shards, and the other shards in the preset shard sequence except the first shard are the second shards. The hash function corresponding to the first shard includes the hash function corresponding to each of the K consecutive shards. The preset value and the default value can be pre-set. For example, the preset value can be 1 and the default value can be 0. In an embodiment of the present invention, each time an access occurs, the corresponding statistical bit in the K consecutive shards is set to a preset value for the key name accessed to record the access. Among them, the specific value of K can be set according to actual needs, and K is a positive integer less than M. In an embodiment of the present invention, data access statistics can be achieved by determining the statistical bit corresponding to the key name to be counted and setting the corresponding statistical bit to a preset value. Therefore, the efficiency of data statistics is relatively high.

[0038] At the same time, in an embodiment of the present invention, the number of key names recorded in the preset shard sequence is monitored. When the number of recorded key names reaches the preset update condition, the original at least one second shard is automatically removed, and a new shard with the same data is added to the preset shard sequence, thereby updating the preset shard sequence and eliminating some of the previous records. In this way, it is ensured that the preset shard sequence only records the key names that have been accessed recently, and then based on the numerical value of the statistical bit of the existing shard in the preset shard sequence, it is possible to distinguish whether a key-value pair is hot data. Among them, the preset update condition can be set in advance, and the embodiment of the present invention does not limit this. The number of shards M included in the updated preset shard sequence remains unchanged.

[0039] In summary, the data processing method provided by the embodiment of the present invention, in response to the data access operation, determines the key name accessed by the data access operation as the key name to be counted. Based on the hash function corresponding to the first shard in the preset shard sequence, determine the statistical bit corresponding to the key name to be counted in the first shard; the preset shard sequence includes M shards, one shard corresponds to multiple statistical bits, the first shard is the first K consecutive shards in the preset shard sequence, and K is less than M. Set the statistical bit corresponding to the key name to be counted in the first shard to a preset value, which is different from the default value of the statistical bit. When the number of key names recorded in the preset shard sequence reaches the preset update condition, remove the second shard in the preset shard sequence, and add a new shard to the preset shard sequence, and the second shard is located after the first shard. Identify hot data based on the preset shard sequence. In this way, access statistics are collected for accessed key names using a preset sharding sequence. By automatically updating the shards in the preset sharding sequence, the preset sharding sequence records recently accessed key names. This allows for convenient identification of whether a key-value pair is hot data based on the key name and the maintained preset sharding sequence. Furthermore, calculations based on the hash function corresponding to the shards can conveniently determine the corresponding statistical position in the shard for the key name to be counted. This improves overall processing efficiency to a certain extent.

[0040] Optionally, in an embodiment of the present invention, the second shard is the last M-K shards in the preset shard sequence. Accordingly, the step of determining the statistical position corresponding to the key name to be counted in the first shard based on the hash function corresponding to the first shard in the preset shard sequence may specifically include:

[0041] Step 1021: Use the key name to be counted as input to the hash function corresponding to each shard in the first shard.

[0042] Step 1021: Obtain the output of the hash function corresponding to each shard in the first shard, and obtain the statistical position corresponding to the key name to be counted in each first shard.

[0043] Accordingly, the step of setting the statistical bit corresponding to the key name to be counted in the first shard to a preset value may specifically include: step 1031, setting the statistical bit corresponding to the key name to be counted in each of the first shards to 1.

[0044] Specifically, the key name to be counted can be used as the input of the hash function corresponding to each of the first K shards, and the output of the hash function corresponding to each of the first K shards can be obtained as the statistical bit corresponding to the key name to be counted in the first K shards. That is, the key name to be counted is mapped to the corresponding bit based on the hash function corresponding to the shard. Then, the statistical bits corresponding to the key name to be counted in the first K shards are set to 1 respectively. Among them, setting the statistical bit corresponding to the key name to be counted in the shard to a preset value is regarded as inserting the key name to be counted in the shard, and recording the key name to be counted in the shard. That is, for a key name to be counted, the statistical overhead of the key name to be counted is K bits, so the storage overhead is low. In an embodiment of the present invention, by setting the statistical bit corresponding to the key name to be counted in each first shard to 1, this is equivalent to inserting the key name to be counted in the preset shard sequence, thereby realizing the recording of the key name to be counted, wherein the statistical bit corresponding to the key name to be counted in these K shards represents the existence information of the key name to be counted, and the K shards into which the key name to be counted is inserted can be regarded as including the existence information of the key name to be counted.

[0045] Assume K=3, the key to be counted is <key1-value1>, then key1 can be used as the input to the hash functions corresponding to the shards numbered 1, 2, and 3 in the preset shard sequence. The output of the hash function corresponding to the first shard (i.e., shard numbered 1) is used as the statistical bit corresponding to key1 in the first shard. The output of the hash function corresponding to the second shard (i.e., shard numbered 2) is used as the statistical bit corresponding to key1 in the second shard. The output of the hash function corresponding to the third shard (i.e., shard numbered 3) is used as the statistical bit corresponding to key1 in the third shard. Furthermore, assuming that the statistical bits corresponding to key1 in the first shard, the statistical bits corresponding to the second shard, and the statistical bits corresponding to the third shard are 3, 4, and 1, respectively. Then the first three shards after this setting are: S1 = [0 0 1 0 0], S2 = [0 0 0 1 0], and S3 = [1 0 0 0 0].

[0046] In an embodiment of the present invention, the key name to be counted is used as the input of the hash function corresponding to each shard in the first shard. The output of the hash function corresponding to each shard in the first shard is obtained to obtain the statistical bit corresponding to the key name to be counted in each first shard. The statistical bit corresponding to the key name to be counted in each first shard is set to 1, thereby inserting the key name to be counted into the preset shard sequence, realizing statistics for the key name to be counted, and ensuring data statistical efficiency to a certain extent.

[0047] Optionally, after the step of setting the statistical bit corresponding to the key name to be counted in the first shard to a preset value, the embodiment of the present invention may further include the following steps:

[0048] Step S21: add 1 to the value of the second counter; the second counter is used to record the number of key names recorded in the preset fragment sequence.

[0049] Step S22: When the value of the second counter is an integer multiple of the preset number of key names, determine whether the number of recorded key names reaches the preset update condition.

[0050] In an embodiment of the present invention, the second counter may be pre-created to count the number of key names recorded since the creation of the preset sharding sequence (i.e., the number of key names recorded in the preset sharding sequence). Specifically, each time a key name to be counted is inserted into the preset sharding sequence, the value of the second counter may be increased by 1, so that the value of the second counter can accurately represent the number of key names recorded in the preset sharding sequence. One second counter is configured for each preset sharding sequence, i.e., one second counter is configured for each M shard.

[0051] Furthermore, it is possible to detect whether the value of the second counter is an integer multiple of the preset number of key names. The preset number of key names can be pre-set by the user, and the larger M is, the larger the preset number of key names is. The preset number of key names is represented by g. The value of the second counter can be divided modulo g. If the result of the modulo division is 0, it means that the value of the second counter is divisible by g, that is, the value of the second counter is an integer multiple of g. Conversely, if the result of the modulo division is not 0, it means that the value of the second counter is not divisible by g, that is, the value of the second counter is not an integer multiple of g.

[0052] The preset update condition can be that the value of the second counter is an integer multiple of the preset number of key names. Accordingly, when the modulo result is 0, it can be determined that the number of recorded key names has reached the preset update condition, thereby triggering the shard update operation. Conversely, when the modulo result is not 0, it is determined that the number of recorded key names has not reached the preset update condition. In this way, it is equivalent to treating the insertion of g key names as a generation, and inserting g key names in a generation. A shard update is performed every time a generation is completed, that is, a shard update is performed every time g key names are inserted. For example, assuming that g is 10, then a shard update is performed when 10 key names have been inserted, another shard update is performed when 20 key names have been inserted, and another shard update is performed when 30 key names have been inserted. That is, a shard update is performed each time the number of recorded key names reaches an integer multiple of 10.

[0053] In this embodiment of the present invention, the number of keys recorded in a preset shard sequence is recorded by maintaining the value of a second counter. By detecting whether the value of the second counter is an integer multiple of the preset number of keys, it is possible to determine whether the number of recorded keys has met the preset update condition, thereby improving the efficiency of triggering shard updates to a certain extent.

[0054] Optionally, the step of removing the second fragment in the preset fragment sequence and adding a new fragment to the preset fragment sequence may specifically include:

[0055] Step 1041: Clear the last fragment in the second fragment in the preset fragment sequence.

[0056] Step 1042: Move the remaining fragments backward and add a new fragment at the beginning of the preset fragment sequence.

[0057] In an embodiment of the present invention, each time a shard update operation is performed, the last shard in the preset shard sequence may be discarded, and a new shard may be added at the first position of the preset shard sequence to record the latest key name. The newly added shard is located at the first position of the preset shard sequence, and therefore, is equivalent to adding a new first shard. The other shards (i.e., the remaining shards) except the last shard may be moved backward in sequence, i.e., the original xth shard becomes the x+1th shard after movement. For example, the shard sequence number of the original first shard is updated to 2, becoming the second shard. The shard sequence number of the original second shard is updated to 3, becoming the third shard.

[0058] By shifting shards, the existence information of recorded key names can be gradually discarded, gradually eliminating expired information. Assuming K is 1, the key name to be counted is only recorded in one shard at a time. Assuming the key name to be counted is A, then A is only recorded in this one shard. Each time a shard is shifted, one shard is discarded. After the shift, if A is not accessed again, the existence information of A will be completely discarded, and A will not be queried. The larger K is, the longer it takes for the existence information of A to be completely discarded. For example, if K is 5, then after these five shards are discarded, the existence information of the key names recorded in these five shards will be completely discarded.

[0059] It should be noted that, in the embodiments of the present invention, a fragment essentially consists of n bits. Discarding a fragment can be accomplished by clearing the n bits of the fragment, for example, restoring them to their initial value of 0. Accordingly, the fragment sequence number of the fragment represented by these n cleared bits can be set to the minimum sequence number, i.e., to 1, thereby placing these n cleared bits as the first newly added fragment in the preset fragment sequence. Furthermore, the fragment is moved backward, specifically by incrementing the fragment sequence number by 1, thereby shifting the fragment's position in the preset fragment sequence. However, the physical storage position of the n bits corresponding to the fragment remains unchanged, i.e., the physical position of each fragment remains unchanged. Furthermore, the hash function corresponding to each fragment in the preset fragment sequence corresponds to the physical position of each fragment. Since moving a fragment does not change the physical position of each fragment, the corresponding hash function does not change. For example, if the physical position of the first fragment is aaa, the hash function corresponding to the first fragment is the hash function corresponding to physical position aaa. After the shard is moved, the original first shard becomes the second shard, but the physical location is still aaa. Accordingly, the hash function corresponding to the second shard is the hash function corresponding to the physical location aaa.

[0060] Optionally, in the embodiment of the present invention, step 105 includes the following steps:

[0061] Step 1051: In response to a data query operation, determine the key name queried by the data query operation as the key name to be queried.

[0062] Step 1052: Search the preset shard sequence for the number of consecutive shards whose target statistical bit is the preset value as the target number; the target statistical bit is the statistical bit corresponding to the key name to be queried.

[0063] Step 1053: When the target number is not less than K, the key-value pair to which the key name to be queried belongs is determined as the hot data.

[0064] Accordingly, after the above step 1052, the embodiment of the present invention may further include the following steps:

[0065] Step S31: When the target number is less than K, the key-value pair to which the key name to be queried belongs is determined as cold data.

[0066] In an embodiment of the present invention, hot data and cold data can be identified based on a preset shard sequence. Specifically, a data query instruction can be generated to trigger a data query operation. The data query instruction carries the key name of the key-value pair that needs to distinguish between hot and cold data. Accordingly, the key name carried in the data query instruction can be used as the key name to be queried. Furthermore, for a recorded key name, there will be K consecutive shards in the preset shard sequence where the statistical bit corresponding to the key name is a preset value. Therefore, the number of consecutive shards whose target statistical bit is the preset value can be found as the target number. Based on whether the target number reaches K, it is determined whether the key name to be queried exists in the preset shard sequence. In this way, data query can be conveniently implemented. Specifically, when the key name is inserted for the first time, the statistical bit corresponding to the key name in the first K shards is the preset value. As the shards move, the K shards whose statistical bits corresponding to the key name are the preset value are positioned further and further back in the preset shard sequence.

[0067] If the target number is not less than K, that is, there are currently K consecutive shards in the preset shard sequence with the corresponding statistical bit of the key to be queried being the preset value. Therefore, it can be considered that the key to be queried has been queried. Accordingly, the key-value pair to which the key to be queried belongs can be determined as hot data. Conversely, if the target number is less than K, that is, due to shard updates, there are currently K consecutive shards in the preset shard sequence with the corresponding statistical bit of the key to be queried being the preset value. Therefore, it can be considered that the key to be queried has not been queried. Accordingly, the key-value pair to which the key to be queried belongs can be determined as cold data.

[0068] L represents the number of second shards, and L can be understood as the total length available for shard movement. L can represent the length of the time window. The larger L is, the longer the time window is, and the longer the existence information of the recorded key name is retained in the preset shard sequence. The smaller L is, the smaller the time window is, and the shorter the existence information of the recorded key name is retained in the preset shard sequence. In the embodiment of the present invention, by maintaining the preset shard sequence, the preset shard sequence can represent the heat information of the data. Specifically, for a key name to be queried, if the consecutive K shards that record the key name to be queried have not been accessed again after moving L+1 times, that is, the key name to be queried has not been accessed again within the time window, then the key name to be queried cannot be queried from the preset shard sequence. And even if there are other key names that are mapped to the same statistical bit as the key name to be queried in a shard, thereby causing a false positive situation, as the number of moves of the consecutive K shards recording the key name increases, the probability of misjudgment due to false positives in finding the consecutive K shards that hit the key name to be queried from the preset shard sequence becomes lower and lower. Among them, the shard where the key name to be queried hits refers to the shard where the statistical bit corresponding to the key name to be queried in the shard is a preset value, and the statistical bit corresponding to the key name to be queried in the shard is determined based on the hash function corresponding to the shard. In this way, by performing shard movement updates on the preset shard sequence, key names that have not been accessed recently can be gradually eliminated. In an embodiment of the present invention, the role of the first shard is to record the inserted key name. For example, Figure 2 This is a sharding diagram provided by an embodiment of the present invention. Figure 2 As shown in the figure, when A is first inserted, it is inserted into the first three shards. Specifically, you can first determine the statistical bit corresponding to the key name of A in the first three shards. Set the statistical bit in the array corresponding to each of the first three shards to 1.

[0069] After one shift, the three consecutive shards of A are recorded and moved back one position. After another shift, the three consecutive shards of A become shards 3 to 5. Finally, after three shifts, the last shard of the three consecutive shards of A is discarded, and only the fourth and fifth shards in the preset shard sequence contain A. After K+L shifts, the shards that recorded the key name are completely cleared.

[0070] Conversely, if the key name is accessed again within the time window, the key name can still be retrieved from the preset sharding sequence. Therefore, the key name retrieved from the preset sharding sequence is equivalent to the key name that has been accessed multiple times within the time window, indicating that the key name is accessed at the access frequency of the time interval represented by the time window. That is, the preset sharding sequence maintained by the embodiment of the present invention can effectively represent the access frequency and access time interval of the key-value pair. In this way, even for data with complex access patterns, it is possible to accurately identify whether the data is hot data, reduce the misjudgment of hot and cold data, and avoid affecting the storage optimization effect for hot and cold data.

[0071] Accordingly, if the key name is queried, it can be considered that the key name is frequently accessed and the access heat of the key name is high enough. Accordingly, the key-value pair to which the key name belongs can be determined as hot data. Accordingly, the key name that is not queried from the preset shard sequence is equivalent to the key name that has not been accessed again within the time window. Therefore, it can be considered that the access frequency of the key name is low and the access heat of the key name is low. Accordingly, the key-value pair to which the key name belongs can be determined as cold data. For example, assuming that a key-value pair is accessed once every 10 seconds, it takes 3 seconds to move L+1 times through the K consecutive shards that record the key name of the key-value pair, then the key name of the key-value pair cannot be queried therefrom, and the key-value pair can be determined as cold data. In this way, by querying whether there are L consecutive shards in the preset shard sequence whose corresponding statistical bit of the key name is the preset value, it is possible to determine whether the key name has been accessed within a certain time range (i.e., time window), and then conveniently identify whether the key-value pair is hot data, so that hot and cold data can be identified conveniently and more accurately.

[0072] In the embodiment of the present invention, the preset sharding sequence is equivalent to a hot and cold data distinguisher. Figure 3 Schematic diagram of the structure of a hot and cold data distinguisher provided by an embodiment of the present invention. Figure 3 As shown, the hot and cold data distinguisher adopts a segmented design, including M shards, that is, one shard is regarded as one segment. Each shard corresponds to an independent hash function, for example, shard 1 corresponds to Hash 1, and shard K corresponds to Hash K. When updating a shard, the last shard (shard M) can be discarded first. The hot and cold data distinguisher supports two basic operations: insert and query. In special cases, for example, when the statistical bits corresponding to other key names are the same as those of the key name to be queried, the key name to be queried will be mistakenly considered to have been queried, resulting in a misjudgment problem. In an embodiment of the present invention, by setting different hash functions for different shards, efficient operation is supported, so that the hash function corresponding to each shard can be used to determine the statistical bits corresponding to the key name in the shard, thereby improving processing efficiency. At the same time, the situation where different key names correspond to the same statistical bits in the same shard can be reduced, thereby reducing the misjudgment rate.

[0073] The total number of distinct independent elements (i.e., the upper limit on the number of keys that can be recorded) Q that can be accommodated by the entire hot and cold data distinguisher is related to the number of bits N in the shard. Specifically, N = Q / ln². The accuracy and false positive rate of the hot and cold data distinguisher are related to the specific values of K and L. Furthermore, the specific values of K and L can be pre-adjusted to achieve different target false positive rates. Specifically, the larger K and N are, the lower the false positive rate. In one implementation, K is a positive integer not less than 2. This avoids the problem of setting K to 1 and recording keys in only one shard, which results in an excessively high false positive rate. For example, if K is 1, then only one statistical bit in one shard is set to 1 for key A. Subsequently, any query that finds that statistical bit set to 1 in a shard indicates that key A has been queried. Therefore, if the statistical bits corresponding to other keys match the statistical bit corresponding to key A, the query will be considered to have found key A, leading to an excessively high false positive rate.

[0074] Furthermore, the larger K and N are, the higher the memory consumption. For example, if K is 6 and L is 14, the total memory usage is (6+14)*N, and the false positive rate is 1 in 10. For example, if K is 10 and L is 25, the total memory usage is (10+25)*N, and the false positive rate is 1 in 100.

[0075] Optionally, the step of searching, from the preset fragment sequence, the number of consecutive fragments having a target statistical bit equal to the preset value as the target number may specifically include:

[0076] Step S32a: Use the first fragment of the last K fragments in the preset fragment sequence as the fragment to be queried.

[0077] Step S32b: Based on the hash function corresponding to the shard to be queried, determine the statistical bit corresponding to the key name to be queried in the shard to be queried as the target statistical bit.

[0078] Step S32c: If the target statistical bit is the preset value, add 1 to the value of the first counter; the first counter is used to record the target number.

[0079] Step S32d, taking the next shard of the shard to be queried as a new shard to be queried, and returning the hash function corresponding to the shard to be queried, determining the statistical bit corresponding to the key name to be queried in the shard to be queried, as the target statistical bit step, until the preset stop condition is reached.

[0080] In an embodiment of the present invention, the first counter may be a pre-created counter for counting the target number. The earlier the consecutive K shards that the key to be queried is hit, the later the key to be queried was inserted. When performing a query, K shards are used as query units, and the query is first started from the end of the preset shard sequence, that is, the query is given priority from the previously inserted shards. Specifically, the last K shards are first taken as a shard group, and when querying the current shard group, the first shard therein can be queried first. For example, assuming that M is 10 and K is 3, then for the 8th to 10th shards, the 8th shard can be used as the shard to be queried first. Then, based on the hash function corresponding to the 8th shard, the statistical bit corresponding to the key to be queried in the shard to be queried is determined, and the target statistical bit corresponding to the key to be queried in the 8th shard is obtained. Specifically, the key name to be queried can be used as the input of the current shard to be queried (i.e., the 8th shard), and then the output of the current shard to be queried is obtained to obtain the target statistical bit corresponding to the key name to be queried in the 8th shard. Assuming that the target statistical bit is 3, it can be detected whether the value of the 3rd bit in the 8th shard is 1. If so, it is considered that the key name to be queried hits the current shard to be queried, and the current shard to be queried includes the existence information of the key name to be queried. Accordingly, the value of the first counter can be increased by 1, so that the value of the first counter can accurately represent the number of consecutive shards in the preset shard sequence whose target statistical bit is a preset value. In an embodiment of the present invention, the first counter can be a counter of a pre-created target number.

[0081] Furthermore, if the key to be queried hits the current shard to be queried, the query can continue to the next shard of the current shard to be queried. For example, the 9th shard can be used as the new shard to be queried, and then return to the above step S32b to continue the query.

[0082] In embodiments of the present invention, the preset stop condition may be pre-set. Optionally, in embodiments of the present invention, the preset stop condition may be determined to have been met when the value of the first counter reaches K. That is, when the query key name is found to hit K consecutive shards, the preset stop condition is considered to have been met, and the shard query is terminated. This avoids unnecessary query operations on the shards.

[0083] In an embodiment of the present invention, K shards are sequentially used as shards to be queried, and each shard to be queried is tested separately to determine whether the queried key name hits the shard to be queried, and the first counter is used to count the number of shards to be queried that hit, so that the target number can be accurately counted.

[0084] Optionally, the embodiment of the present invention may further include the following steps:

[0085] Step S32e: If the target statistical bit is not the preset value, the designated shard is used as a new shard to be queried, and the hash function corresponding to the shard to be queried is returned to determine the statistical bit corresponding to the key name to be queried in the shard to be queried, as the step of the target statistical bit; wherein the designated shard is the first shard among the first K shards of the current shard to be queried.

[0086] In an embodiment of the present invention, if the target statistical bit is not a preset value, it is considered that the key name to be queried does not hit the current shard to be queried, and the current shard to be queried does not include the existence information of the key name to be queried. Accordingly, the value of the first counter is not increased by 1, and at the same time, the first shard of the first K shards of the current shard to be queried is used as the new shard to be queried, and the query is continued by returning to the above step S32b. Assuming that the current shard to be queried is the 8th shard, the first K shards (5th to 7th shards) can be used as a new shard group, and the first shard in the shard group: shard 5, is used as the new shard to be queried. Assuming that the current shard to be queried is the 9th shard, the first K shards (6th to 8th shards) can be used as a new shard group, and the first shard in the shard group: shard 6, is used as the new shard to be queried.

[0087] Let p represent the value of the second counter. For the first K shards of the current shard to be queried, if the key name to be queried hits the first Kp shards among these K shards, then when traversing to the Kp-th shard among these K shards, the value of the second counter reaches K, that is, the key name to be queried hits K consecutive shards, and then the query process will be terminated, and it is determined that the key name to be queried is queried. In an embodiment of the present invention, by reusing the value previously counted by the second counter when the current shard to be queried is not hit, and re-querying backwards from the first K shards of the current shard to be queried, it is possible to accurately identify the consecutive K shards in the preset shard sequence while avoiding repeated queries on shards that have already been queried.

[0088] Optionally, in an embodiment of the present invention, when there is no new shard to be queried in the preset shard sequence, it is determined that the preset stop condition is reached. Specifically, if the target statistical bit of the current shard to be queried is not the preset value, that is, the key name to be queried does not hit the current shard to be queried, the first shard among the first K shards of the current shard to be queried will be returned to start the query. As the return continues, if the first shard of the preset shard sequence is finally reached, that is, the first shard of the preset shard sequence is included in the first K shards of the current shard to be queried, the new shard group includes the first shard. Then, when the key name to be queried does not hit the first shard or there is no unqueried shard in the new shard group, it is determined that there is no new shard to be queried in the preset shard sequence. It should be noted that if the target statistical bit of the current shard to be queried is not the preset value, and the number of shards before the current shard to be queried is less than K, all shards before the current shard to be queried can be used as a new shard group.

[0089] In an embodiment of the present invention, when there is no new shard to be queried in the preset shard sequence, it is determined that the preset stop condition is reached. In this way, it is possible to avoid omitting the shard to be queried in the query process, thereby avoiding the problem of deviation in the query results.

[0090] The data processing method provided in the embodiments of the present invention can be applied to a key-value store system based on a Log Structured Merge Tree (LSM-Tree). LSM-Tree key-value stores are widely used in large-scale data management scenarios. LSM-Tree is a hierarchical, ordered, disk-oriented data structure. Its core concept is to fully utilize the sequential batch write capability of the disk to ensure data write performance. Figure 4 This is a schematic diagram of an application of a hot and cold data distinguisher provided by an embodiment of the present invention. Figure 4 As shown, the hot and cold data distinguisher can be integrated with the LSM-Tree key-value storage system. In response to a write or read operation on a key-value pair, the written key name / read key name can be used as a key name to be counted, and the key name to be counted is recorded in the hot and cold data distinguisher. The specific data stored in the key-value pair can be set according to actual needs. For example, objects such as images, text, videos, applications, and user accounts registered in the platform can be used as key values, and the object identifier can be used as the key name, and stored in the form of a key-value pair.

[0091] The LSM-Tree key-value storage system is divided into multiple levels, for example, Figure 4 L0 to L6 in the memory. The data to be stored in the memory is stored in the form of a sorted string table (SST) in these multiple levels on the disk. Among them, a level can include multiple SSTs, and an SST can include multiple key-value pairs. Specifically, the LSM structure used by the key-value database is mainly composed of three parts: memory table (Memory Table, Memtable), sorted string table (Sorted String Table, SSTable) and Log. Specifically, Log can be a write-ahead log (WAL). Memtable is an ordered associative array in memory (Memory). Memtable can store the most recently written records as a cache. When the Memtable is full, it will be frozen as an immutable memory table (Immutable Memtable) and wait to be serialized into an SSTable to be written to the storage device (i.e. Figure 4 SSTable can include data blocks and index blocks. Data blocks store key-value pair objects sorted by key.<key.value> ,The index block stores the key range of all data blocks.,WAL,stores the write records of data to ensure crash consistency.

[0092] The LSM-tree structure fully utilizes sequential disk writes, maintaining a memtable in memory. Once the memtable is full, data is appended to the disk in batches, converting random data writes into sequential writes. This ensures the orderliness of data at each layer, providing good query performance. However, this structure searches each layer from top to bottom when querying a key. This leads to a significant read amplification problem, which is particularly severe for range queries and can create performance bottlenecks for key-value storage systems. Read amplification refers to the situation where the amount of data required to be read during a read is less than the actual amount of data read.

[0093] Accordingly, in an embodiment of the present invention, when a key-value pair is identified as hot data, storage optimization can be performed to allocate more resources to the hot data to speed up access to the hot data, optimize data access performance, reduce latency and improve throughput. Specifically, the hot data can be stored in a storage medium with a higher access speed. For example, the hot data is saved in the memory, for example, in a designated area of the memory, to avoid reading data from each layer of disk, thereby avoiding the problem of read amplification, and thus improving the performance of the key-value storage system and improving the adaptability of the key-value storage system to workloads. Alternatively, the hot data can be moved to an upper layer to reduce the number of layers required to query when accessing the hot data subsequently, thereby reducing read amplification to a certain extent. In an embodiment of the present invention, the key-value pairs identified as hot data are stored in the memory or moved to an upper layer, thereby improving the utilization rate of these storage resources.

[0094] For key-value pairs identified as cold data, the key-value pairs may be stored in a storage medium with a slower access speed.

[0095] In one existing technology, based on the Least Recently Used (LRU) algorithm, the most recently accessed data blocks are directly stored in the memory as hot data. In this way, the data blocks are saved as units, the storage cost is high, and there is a problem that the data blocks written to the memory include unnecessary other key-value pairs and key values, which leads to write amplification. In the embodiment of the present invention, the recognition accuracy is higher based on the key name as the unit. By identifying whether the key-value pair to which the key name belongs is hot data, if the key-value pair is identified as hot data, only the key-value pair is saved in the memory. Compared with the method based on the data block as the unit, the storage cost is lower, the waste of storage resources is reduced, and the write amplification problem is avoided.

[0096] In another prior art, hot data is managed by simulating clock hands, thereby achieving heat statistics and management. In this method, a heat bit array needs to be associated with the key name of each key-value pair. If the data is accessed, the heat bit is increased by 1. In this method, since a heat bit array is associated with the key name of each key-value pair, the more key names, the more heat bits, the higher the storage overhead, and the additional burden on system resources. In the embodiment of the present invention, heat statistics can be achieved by maintaining only a fixed number of bits. Therefore, the storage overhead is low and no additional burden is placed on system resources. In addition, when selecting cold data in the prior art, the heat bit arrays of all key names need to be scanned one by one, and the algorithm efficiency is low. In this solution, only a hash function is used to perform a hash query based on the key name of the key-value pair, that is, to determine whether the corresponding statistical bit of the key name to be queried is 1, it can be determined whether the key name to be queried is hot data. The algorithm has higher query speed and efficiency, can better meet the needs of data-intensive applications, and is suitable for efficient storage management scenarios such as large-scale distributed storage and cloud storage platforms.

[0097] In the embodiment of the present invention, a heat aging mechanism is introduced. By automatically triggering the movement of the shard array when the preset update conditions are met, the heat information of the data is dynamically updated, so that the heat information of the data can be represented more accurately and in real time based on the preset shard sequence, and the hot and cold data can be dynamically distinguished based on the preset shard sequence. By continuously moving the shards, the embodiment of the present invention is equivalent to clearing the previously inserted key names, thereby realizing automatic cleaning of expired data, automatic expiration management, and high data management efficiency. Furthermore, the preset shard sequence in the embodiment of the present invention can insert a key name, and can also delete the key name that has been inserted before. And deletion can be achieved by only triggering the movement of the shard, so the cost of the deletion operation is low. And when the shard is deleted, a new shard will be added, so the total number of shard arrays remains unchanged, and thus to a certain extent, the problem of increased misjudgment rate due to deletion can be avoided.

[0098] Reference Figure 5 , shows a block diagram of a data processing device provided by an embodiment of the present invention, such as Figure 5 As shown, the data processing device may specifically include:

[0099] A first determining module 201 is configured to, in response to a data access operation, determine a key name accessed by the data access operation as a key name to be counted;

[0100] A second determining module 202 is configured to determine, based on a hash function corresponding to a first shard in a preset shard sequence, a statistical bit corresponding to the key name to be counted in the first shard; the preset shard sequence includes M shards, each shard corresponds to multiple statistical bits, and the first shard is the first K consecutive shards in the preset shard sequence, where K is less than M;

[0101] The setting module 203 is configured to set the statistical bit corresponding to the key name to be counted in the first shard to a preset value; the preset value is different from the default value of the statistical bit;

[0102] The first processing module 204 is configured to remove the second shard in the preset shard sequence and add a new shard to the preset shard sequence if the number of key names recorded in the preset shard sequence reaches a preset update condition; the second shard is located after the first shard;

[0103] The identification module 205 is configured to identify hot data based on the preset sharding sequence.

[0104] Optionally, the second fragment is the last M-K fragments in the preset fragment sequence; the first processing module 204 is specifically configured to:

[0105] Clear the last fragment of the second fragment in the preset fragment sequence;

[0106] The remaining fragments are moved backward, and a new fragment is added to the first position of the preset fragment sequence.

[0107] Optionally, the identification module 205 is specifically configured to:

[0108] In response to a data query operation, determining the key name queried by the data query operation as the key name to be queried;

[0109] Searching the preset shard sequence for the number of consecutive shards whose target statistical bit is the preset value as the target number; the target statistical bit is the statistical bit corresponding to the key name to be queried;

[0110] When the target number is not less than K, the key-value pair to which the key name to be queried belongs is determined as the hot data;

[0111] When the target number is smaller than K, the key-value pair to which the key name to be queried belongs is determined as cold data.

[0112] Optionally, the identification module 205 is further configured to:

[0113] The first fragment among the last K fragments in the preset fragment sequence is used as the fragment to be queried;

[0114] Determine, based on the hash function corresponding to the shard to be queried, the statistical bit corresponding to the key name to be queried in the shard to be queried as the target statistical bit;

[0115] If the target statistical bit is the preset value, the value of the first counter is increased by 1; the first counter is used to record the target number;

[0116] The next shard of the shard to be queried is used as a new shard to be queried, and the hash function corresponding to the shard to be queried is returned to determine the statistical bit corresponding to the key name to be queried in the shard to be queried as the target statistical bit step, until a preset stop condition is reached.

[0117] Optionally, the identification module 205 is further configured to:

[0118] If the target statistical bit is not the preset value, the designated shard is used as a new shard to be queried, and the hash function corresponding to the shard to be queried is returned to determine the statistical bit corresponding to the queried key name in the shard to be queried as the target statistical bit;

[0119] The designated shard is the first shard among the first K shards of the shard to be queried.

[0120] Optionally, the identification module 205 is further configured to:

[0121] When the value of the first counter reaches K, or when there is no new fragment to be queried in the preset fragment sequence, it is determined that the preset stop condition is met.

[0122] Optionally, the second determining module 202 is specifically configured to:

[0123] Using the key name to be counted as input to the hash function corresponding to each shard in the first shard;

[0124] Obtaining the output of the hash function corresponding to each shard in the first shard, and obtaining the statistical position corresponding to the key name to be counted in each of the first shards;

[0125] The setting the statistical bit corresponding to the key name to be counted in the first shard to a preset value includes: setting the statistical bit corresponding to the key name to be counted in each of the first shards to 1.

[0126] Optionally, the device further comprises:

[0127] A second processing module is configured to add 1 to the value of a second counter after the setting module 203 sets the statistical bit corresponding to the key name to be counted in the first shard to a preset value; the second counter is configured to record the number of key names recorded in the preset shard sequence;

[0128] The third determining module is configured to determine that the number of recorded key names reaches the preset update condition when the value of the second counter is an integer multiple of the preset number of key names.

[0129] In summary, in the data processing device provided by the embodiment of the present invention, in response to the data access operation, the key name accessed by the data access operation is determined as the key name to be counted. Based on the hash function corresponding to the first shard in the preset shard sequence, the statistical bit corresponding to the key name to be counted in the first shard is determined; the preset shard sequence includes M shards, one shard corresponds to multiple statistical bits, and the first shard is the first K consecutive shards in the preset shard sequence, and K is less than M. The statistical bit corresponding to the key name to be counted in the first shard is set to a preset value, and the preset value is different from the default value of the statistical bit. When the number of key names recorded in the preset shard sequence reaches the preset update condition, the second shard in the preset shard sequence is removed, and a new shard is added to the preset shard sequence, and the second shard is located after the first shard. Identify hot data based on the preset shard sequence. In this way, access statistics are collected for accessed key names using a preset sharding sequence. By automatically updating the shards in the preset sharding sequence, the preset sharding sequence records recently accessed key names. This allows for convenient identification of whether a key-value pair is hot data based on the key name and the maintained preset sharding sequence. Furthermore, calculations based on the hash function corresponding to the shards can conveniently determine the corresponding statistical position in the shard for the key name to be counted. This improves overall processing efficiency to a certain extent.

[0130] Reference Figure 6 Schematic diagram of the structure of the electronic device provided by the embodiment of the present invention. Figure 6 As shown, the electronic device includes: a processor, a memory, a communication interface and a communication bus.

[0131] The processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, which enables the processor to execute the data processing method of the above embodiment. The executable instructions can form a program.

[0132] An embodiment of the present invention provides a machine-readable medium having instructions stored thereon, which, when executed by one or more processors, enables the processors to execute the data processing method of the aforementioned embodiment.

[0133] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0134] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located and with the authorization given by the owner of the corresponding device.

[0136] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing terminal device to operate in a predictable manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0139] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0140] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0141] Moreover, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal device that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0142] The above describes in detail a data processing method, a data processing device, an electronic device, and a readable medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: In response to a data access operation, determining the key name accessed by the data access operation as the key name to be counted; Determining, based on a hash function corresponding to a first shard in a preset shard sequence, a statistical bit corresponding to the key name to be counted in the first shard; the preset shard sequence includes M shards, one shard corresponds to multiple statistical bits, and the first shard is the first K consecutive shards in the preset shard sequence, where K is less than M; Setting the statistical bit corresponding to the key name to be counted in the first shard to a preset value; the preset value is different from the default value of the statistical bit; When the number of key names recorded in the preset shard sequence reaches a preset update condition, remove the second shard in the preset shard sequence and add a new shard to the preset shard sequence; the second shard is located after the first shard; Hot data is identified based on the preset sharding sequence.

2. The method according to claim 1, characterized in that The second fragment is the last M-K fragments in the preset fragment sequence; The removing the second fragment in the preset fragment sequence and adding a new fragment to the preset fragment sequence includes: Clear the last fragment of the second fragment in the preset fragment sequence; The remaining fragments are moved backward, and a new fragment is added to the first position of the preset fragment sequence.

3. The method according to claim 1, characterized in that The identifying hot data based on the preset sharding sequence includes: In response to a data query operation, determining the key name queried by the data query operation as the key name to be queried; Searching the preset shard sequence for the number of consecutive shards whose target statistical bit is the preset value as the target number; the target statistical bit is the statistical bit corresponding to the key name to be queried; When the target number is not less than K, the key-value pair to which the key name to be queried belongs is determined as the hot data; After searching the preset fragment sequence for a number of consecutive fragments whose target statistical bit is the preset value and setting the number as the target number, the method further includes: When the target number is smaller than K, the key-value pair to which the key name to be queried belongs is determined as cold data.

4. The method according to claim 3, characterized in that The step of searching, from the preset fragment sequence, for a number of consecutive fragments whose target statistical bit is the preset value as the target number includes: The first fragment among the last K fragments in the preset fragment sequence is used as the fragment to be queried; Determine, based on the hash function corresponding to the shard to be queried, the statistical bit corresponding to the key name to be queried in the shard to be queried as the target statistical bit; If the target statistical bit is the preset value, the value of the first counter is increased by 1; the first counter is used to record the target number; The next shard of the shard to be queried is used as a new shard to be queried, and the hash function corresponding to the shard to be queried is returned to determine the statistical bit corresponding to the key name to be queried in the shard to be queried as the target statistical bit step, until a preset stop condition is reached.

5. The method according to claim 4, characterized in that The method further comprises: If the target statistical bit is not the preset value, the designated shard is used as a new shard to be queried, and the hash function corresponding to the shard to be queried is returned to determine the statistical bit corresponding to the queried key name in the shard to be queried as the target statistical bit; The designated shard is the first shard among the first K shards of the shard to be queried.

6. The method according to claim 4, characterized in that The method further comprises: When the value of the first counter reaches K, or when there is no new fragment to be queried in the preset fragment sequence, it is determined that the preset stop condition is met.

7. The method according to claim 1, characterized in that The determining, based on the hash function corresponding to the first shard in the preset shard sequence, the statistical bit corresponding to the key name to be counted in the first shard, includes: Using the key name to be counted as input to the hash function corresponding to each shard in the first shard; Obtaining the output of the hash function corresponding to each shard in the first shard, and obtaining the statistical position corresponding to the key name to be counted in each of the first shards; The setting the statistical bit corresponding to the key name to be counted in the first shard to a preset value includes: setting the statistical bit corresponding to the key name to be counted in each of the first shards to 1.

8. The method according to claim 1, characterized in that After setting the statistical bit corresponding to the key name to be counted in the first shard to a preset value, the method further includes: Add 1 to the value of the second counter; the second counter is used to record the number of key names recorded in the preset sharding sequence; When the value of the second counter is an integer multiple of the preset number of key names, it is determined that the number of recorded key names reaches the preset update condition.

9. A data processing device, characterized in that: The device comprises: A first determining module is configured to, in response to a data access operation, determine a key name accessed by the data access operation as a key name to be counted; a second determining module, configured to determine, based on a hash function corresponding to a first shard in a preset shard sequence, a statistical bit corresponding to the key name to be counted in the first shard; the preset shard sequence includes M shards, one shard corresponds to multiple statistical bits, the first shard is the first K consecutive shards in the preset shard sequence, and K is less than M; A setting module, configured to set the statistical bit corresponding to the key name to be counted in the first shard to a preset value; the preset value is different from the default value of the statistical bit; A first processing module is configured to remove a second shard from the preset shard sequence and add a new shard to the preset shard sequence when the number of key names recorded in the preset shard sequence reaches a preset update condition; the second shard is located after the first shard; An identification module is used to identify hot data based on the preset fragmentation sequence.

10. An electronic device, characterized in that: include: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store executable instructions, and the executable instructions enable the processor to execute the method according to any one of claims 1 to 8.

11. A machine-readable medium, characterized in that Instructions are stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 8.

Images