Redis storage optimization method and device, equipment and storage medium

Abstract

The invention discloses a Redis storage optimization method and device, equipment and a storage medium, and relates to the technical field of data storage, the method comprises the following steps: obtaining an original key of storage data in a Redis database, and determining a hash value corresponding to the original key; generating a hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key; taking the hash identification number corresponding to the original key as a to-be-stored key, and storing the to-be-stored key in a plurality of storage buckets in a scattered manner; and based on the keys to be stored in the multiple storage buckets, storage coding is performed again to optimize the memory space of the Redis database. By means of the mode, the Key cached in the Redis is regularly dispersed into the multiple storage buckets, and the Key of each storage bucket is the character string with the same length, so that a bottom layer coding mode capable of saving more space can be adopted, memory alignment and space saving are achieved, the storage space can be effectively utilized, and memory occupation and memory fragments are reduced.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a Redis storage optimization method, device, equipment and storage medium. Background Art

[0002] With the expansion of in-bank business, the amount of data has exploded, which has put forward higher requirements for data processing and storage technology. As a high-performance key-value storage system, Redis (Remote Dictionary Server) is widely used in scenarios such as caching and real-time data analysis due to its excellent read and write performance and rich data structure. However, the traditional Redis storage mode has the problems of high memory waste and poor storage economy, especially in the scenario of massive data storage. The data itself will occupy a large amount of space, and the data will also generate a large number of keys. These keys are numerous, of different lengths, and have different underlying encodings, resulting in the inability to effectively align the memory. The free space left after the expired data is deleted cannot be effectively utilized, which reduces the amount of data that can be saved, occupies a lot of memory, and causes a waste of memory space.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a Redis storage optimization method, device, equipment and storage medium, aiming to solve the technical problem in the prior art that the different key lengths of Redis storage lead to a lot of memory waste.

[0005] To achieve the above purpose, the present application provides a Redis storage optimization method, the method comprising:

[0006] Obtain the original key of the data stored in the Redis database, and determine the hash value corresponding to the original key;

[0007] Generate a hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key;

[0008] Using the hash identification number corresponding to the original key as the key to be stored, and distributing and storing the key to be stored in multiple storage buckets;

[0009] Based on the keys to be stored in the multiple storage buckets, storage encoding is re-performed to optimize the memory space of the Redis database.

[0010] In one embodiment, the step of generating a hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key includes:

[0011] Calculate the optimal identification number length;

[0012] Determine a valid hash value corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key;

[0013] Based on the ASCII code value range, the effective hash value corresponding to the original key is adjusted to obtain an initial hash identification number corresponding to the original key;

[0014] The initial hash identification number corresponding to the original key is converted into a corresponding ASCII code to obtain the hash identification number corresponding to the original key.

[0015] In one embodiment, the step of calculating the optimal identification number length includes:

[0016] Using the number of bits of the bucket space of the storage bucket as the number of bits of the identification number;

[0017] Based on the number of effective bits of the byte, the number of bits of the identification number is adjusted to obtain the optimal identification number length.

[0018] In one embodiment, the step of adjusting the number of bits of the identification number based on the number of effective bits of the byte to obtain the optimal identification number length includes:

[0019] Based on the number of digits of the identification number, divide the identification number bytes and determine the length of the identification number bytes;

[0020] Based on the number of effective digits of the byte, determining whether the number of digits of the identification number is divisible by the number of effective digits of the byte;

[0021] When the number of digits of the identification number cannot be divided evenly by the number of valid digits of the byte, calculate the remainder corresponding to the number of digits of the identification number;

[0022] Obtaining the length of the last byte in the identification number bytes, and adjusting the length of the last byte based on the remainder;

[0023] The optimal identification number length is determined based on the adjusted length of the last byte.

[0024] In one embodiment, before the step of determining the number of digits of the identification number based on the number of digits of the bucket space of the storage bucket, the step further includes:

[0025] Get the correspondence between the number of original keys, the estimated average number of buckets, and the number of bucket spaces;

[0026] The number of bits of the bucket space is determined based on the number of the original keys, the estimated average number of buckets, and the corresponding relationship.

[0027] In one embodiment, before the step of determining the number of digits of the identification number based on the number of digits of the bucket space of the storage bucket, the step further includes:

[0028] Acquire the data scale of the original key, and generate test data based on the data scale;

[0029] Testing the bucket consumption data of the storage bucket based on the preset test space bit number, and determining the bucket consumption data corresponding to different preset test space bit numbers when the storage bucket stores the test data;

[0030] The bucket consumption data corresponding to different preset test space bit numbers are analyzed to determine the optimal space bit number, and the optimal space bit number is used as the bucket space bit number.

[0031] In one embodiment, the step of re-encoding the storage based on the keys to be stored in the multiple storage buckets to optimize the memory space of the Redis database further includes:

[0032] Obtaining access timestamp information of the key to be stored;

[0033] Based on a preset byte position, inserting the access timestamp information of the key to be stored into the storage data corresponding to the key to be stored;

[0034] After the step of re-encoding the storage based on the keys to be stored in the multiple storage buckets to optimize the memory space of the Redis database, the following steps are further included:

[0035] When receiving a write instruction, extracting a preset number of sample storage buckets from the storage buckets;

[0036] When the capacity of the sample storage bucket is greater than a preset capacity threshold, obtaining access timestamp information of the data stored in the sample storage bucket;

[0037] Based on the access timestamp information of the data stored in the sample storage bucket, determining the non-access duration of the key to be stored corresponding to the data stored in the sample storage bucket;

[0038] Based on the non-access duration of the to-be-stored key corresponding to the data stored in the sample storage bucket, determining, among the to-be-stored keys in the sample storage bucket, expired keys whose non-access duration is greater than a preset time period;

[0039] The expired key is deleted to write new data.

[0040] In addition, to achieve the above purpose, the present application also proposes a Redis storage optimization device, which includes:

[0041] A hash calculation module is used to obtain the original key of the data stored in the Redis database and determine the hash value corresponding to the original key;

[0042] A data dispersing module, used for generating a hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key;

[0043] The data dispersing module is further used to use the hash identification number corresponding to the original key as the key to be stored, and to disperse and store the key to be stored in multiple storage buckets;

[0044] The storage optimization module is used to re-encode the storage based on the keys to be stored in the multiple storage buckets to optimize the memory space of the Redis database.

[0045] In addition, to achieve the above objectives, the present application also proposes a Redis storage optimization device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the Redis storage optimization method as described above.

[0046] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the Redis storage optimization method as described above are implemented.

[0047] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the Redis storage optimization method as described above are implemented.

[0048] The present application provides a Redis storage optimization method, which obtains the original key of the data stored in the Redis database, determines the hash value corresponding to the original key; generates the hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key; uses the hash identification number corresponding to the original key as the key to be stored, and stores the key to be stored in multiple buckets in a dispersed manner; re-encodes the storage based on the keys to be stored in multiple buckets to optimize the memory space of the Redis database. The present application distributes the Key cached in Redis into multiple buckets in an orderly manner, and the Key of each bucket is a string of the same length, so that a more space-saving underlying encoding method can be used to achieve memory alignment and space saving, which can effectively utilize storage space, reduce memory occupancy and memory fragmentation, and reduce the waste of memory space. It can also be applied to tens of billions of data storage scenarios to ensure efficient use of memory, and solve the technical problem of more memory waste caused by different lengths of Key stored in Redis. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0051] Figure 1 This is a flow chart of the first embodiment of the Redis storage optimization method of the present application;

[0052] Figure 2 A storage optimization diagram of the Redis storage optimization method provided in Example 1 of the present application;

[0053] Figure 3 A query optimization diagram of the Redis storage optimization method provided in Example 1 of the present application;

[0054] Figure 4 This is a flow chart of the second embodiment of the Redis storage optimization method of the present application;

[0055] Figure 5 A schematic diagram of a hash identification number generation process for the Redis storage optimization method provided in Example 2 of the present application;

[0056] Figure 6 This is a flow chart of the third embodiment of the Redis storage optimization method of the present application;

[0057] Figure 7 A schematic diagram of the expiration strategy flow of the Redis storage optimization method provided in Example 3 of the present application;

[0058] Figure 8 A brief flowchart of the Redis storage optimization method provided in Example 3 of the present application;

[0059] Fig. 9 This is a schematic diagram of the module structure of the Redis storage optimization device according to an embodiment of the present application;

[0060] Fig.10 This is a schematic diagram of the device structure of the hardware operating environment involved in the Redis storage optimization method in the embodiment of the present application.

[0061] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0062] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0063] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0064] The main solution of the embodiment of the present application is: obtain the original key of the data stored in the Redis database, and determine the hash value corresponding to the original key; generate a hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key; use the hash identification number corresponding to the original key as the key to be stored, and store the key to be stored in multiple storage buckets in a dispersed manner; re-encode the storage based on the keys to be stored in multiple storage buckets to optimize the memory space of the Redis database.

[0065] At present, the traditional Redis storage model has the problems of high memory waste and poor storage economy, especially in the scenario of massive data storage. The value of the data itself will occupy a large amount of space, and the data will also generate a large number of keys. These keys are numerous, of different lengths, and have different underlying encodings, resulting in the inability to effectively align the memory. The free space left after the expired data is deleted cannot be effectively utilized, which reduces the amount of data that can be saved, occupies a lot of memory, and causes waste of memory space.

[0066] The present application provides a solution to distribute the keys cached in Redis into multiple storage buckets in an orderly manner. The keys of each storage bucket are strings of the same length, so that a more space-saving underlying encoding method can be used to achieve memory alignment and space saving, and the storage space can be effectively utilized, memory usage and memory fragmentation can be reduced, and the waste of memory space can be reduced. The application can also be applied to tens of billions of data storage scenarios to ensure efficient use of memory, and solve the technical problem that the different lengths of keys stored in Redis lead to a lot of memory waste.

[0067] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a Redis storage optimization device, etc., and this embodiment does not specifically limit this. The following takes the Redis storage optimization device as an example to illustrate this embodiment and the following embodiments.

[0068] The present application embodiment provides a Redis storage optimization method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the Redis storage optimization method of the present application.

[0069] In this embodiment, the Redis storage optimization method includes steps S10 to S40:

[0070] Step S10, obtaining the original key of the data stored in the Redis database, and determining the hash value corresponding to the original key;

[0071] It should be noted that the Redis database is a key-value cache database, and all stored data is stored in the memory in the form of key-value pairs. Each key is a string type object that can contain any characters such as letters, numbers, symbols, etc., which enables the key to uniquely identify the stored data in a concise and descriptive way, and the value is the specific data stored in Redis. This design allows Redis operations to be performed in units of keys, including operations such as acquisition, modification, and deletion, so as to efficiently manage the stored data.

[0072] In this embodiment, the stored data, i.e., the data stored in the Redis database, can be considered as Value. The original key, i.e., the key corresponding to the stored data, has not been processed, and usually has different lengths and different underlying encodings, which results in the inability to effectively align the memory. The free space left after the expired data is deleted cannot be effectively utilized, which reduces the amount of data that can be saved, occupies more memory, and causes a waste of memory space. Therefore, this embodiment regularizes these original keys to achieve memory alignment.

[0073] In addition, it should be noted that the hash value refers to the conversion of the input storage data into an output value of fixed length through a hash function (hash function), which has the characteristics of irreversibility, uniqueness and anti-collision. The hash function can be a SHA (Secure Hash Algorithm) hash function, an MD5 (Message-Digest Algorithm 5) hash function, etc. Since the number of keys determines the collision rate, in the specific implementation, the appropriate hash function can be selected according to the number of original keys.

[0074] Exemplarily, this embodiment uses the MD5 hash function to calculate the hash value corresponding to the original key. First, the string of the original key is converted into a byte array, which can be converted using UTF-8 encoding. Each Chinese character occupies 2 bytes, and other characters occupy 1 byte; then determine the number of bits of the converted array, fill the array, fill a 1 after the array, and then fill it with 0 until the number of bits after the fill is equal to 448 modulo 512, at least 1 bit is filled, and at most 512 bits can be filled; then add a 64-bit value to the end of the array after the fill is completed. The content of the value is the original byte array. If the length of the original byte array is less than 64 bits, it is still filled to 64 bits. If the length of the original byte array is greater than 64 bits, the content of the last 64 bits is selected. If the length of the original byte array is equal to 64 bits, the content of the original byte array is directly used, thereby obtaining a 32 bits; MD5 operation requires a 128-bit MD5 buffer to store intermediate variables and final results. The buffer can be seen as four 32-bit registers A, B, C, and D; every 512 bits (64 bytes) is a segment, the data is divided into n segments (n is greater than or equal to 1), and four non-linear functions F, G, H, and I are defined. Four rounds of logical processing are performed on each data segment, and four different functions F, G, H, and I are used in the four rounds. Each round takes ABCD and the current 512-bit block as input, and the processed data is sent to ABCD (128 bits). The data is processed and output in the form of [A, B, C, D], that is, starting from the sequential bytes of A with the low bit in front and ending at the sequential bytes of D with the high bit in front, and finally the hash value of each original key is obtained.

[0075] Step S20, generating a hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key;

[0076] It should be noted that since the MD5 used is a 32-bit hexString (hexadecimal characters), its space is 128 bits, which is relatively large. Assuming that the stored data is in the tens of billions, it is about 33 bits. Therefore, it is necessary to calculate a hash with a suitable number of bits as the new key. The optimal identification number length refers to the most appropriate length of the generated new key. The hash identification number is the generated fixed-length BucketId, which is used to store the new key. Since the new key is generated according to the optimal identification number length, the new key at this time has the same length.

[0077] It is understandable that this embodiment uses the MD5 hash function to generate a fixed-length BucketId for storing new keys. In specific implementations, the front part of the hash value is usually used as the hash identification number, that is, according to the optimal identification number length, the values ​​corresponding to the corresponding digits in the hash value are selected from the front to the back to obtain the final BucketId.

[0078] Furthermore, in order to reduce the number of keys in the Redis database, the keys can be allowed to collide in the BucketId space. In this case, it can be considered that multiple keys are hung under one BucketId. For example, on average, 10 keys are hung under each BucketId. At this time, the number of keys will be reduced by more than 90%.

[0079] Step S30, using the hash identification number corresponding to the original key as the key to be stored, and distributing and storing the key to be stored in multiple storage buckets;

[0080] It should be noted that the key to be stored refers to the key stored in the storage bucket, that is, the generated new key. In this embodiment, the hash identification number corresponding to the original key is used as the new key, and these new keys are stored in multiple storage buckets in a dispersed manner.

[0081] It is understandable that the number of buckets is usually determined by the scale of the stored data. For example, if you plan to store tens of billions of data and plan to have each bucket share 10 key-value pairs, you will need 2 buckets. 30 = 1073741824 storage buckets, which is the number of final keys. The magnitude of data storage should be planned in advance, and the floating range is about ten to fifteen times the number of buckets. For example, to store tens of billions of data, 30 bits to 31 bits are usually selected as the number of buckets. At this time, the business growth is in a reasonable range (10 to 15 times growth). If the business grows too much, it will increase the query time and even cause a sharp increase in memory.

[0082] It should be understood that in order to shorten the length of the Key and save memory, the hash identifier is usually converted into an ASCII code, so that the length of the Key can be reduced by half.

[0083] Step S40, re-encoding the storage based on the keys to be stored in the multiple storage buckets to optimize the memory space of the Redis database.

[0084] It should be noted that the cached keys are regularly distributed into multiple storage buckets. After processing, the keys of each storage bucket are strings of the same length, and the underlying storage will automatically adopt the space-saving Embstr encoding method to achieve memory alignment and space saving, thereby optimizing the memory space of the Redis database.

[0085] Understandably, reference Figure 2After optimization, the key-value pairs in the Redis database can be stored in a HashMap (an implementation of the Map interface based on a hash table, used to store key-value pair mappings) structure, so that when querying, the hash identifier corresponding to the Key can be calculated according to the above process, so as to query the target Key corresponding to the hash identifier, and then query the target Value corresponding to the target Key. For example, refer to Figure 3 When querying, the traditional key acquisition, i.e., get(key), is converted to first calculating the hash identifier hashKey, i.e., hashKey=hash(key), and then obtaining the key corresponding to the hash identifier hashKey, i.e., hget(hashKey,key), so as to obtain the value corresponding to the key.

[0086] This embodiment provides a Redis storage optimization method, which obtains the original key of the data stored in the Redis database, determines the hash value corresponding to the original key; generates a hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key; uses the hash identification number corresponding to the original key as the key to be stored, and stores the key to be stored in multiple buckets in a dispersed manner; re-encodes the storage based on the keys to be stored in multiple buckets to optimize the memory space of the Redis database. The keys cached in Redis are regularly dispersed into multiple buckets, and the keys of each bucket are strings of the same length, so that a more space-saving underlying encoding method can be used to achieve memory alignment and space saving, which can effectively utilize storage space, reduce memory occupancy and memory fragmentation, and reduce waste of memory space. It can also be applied to tens of billions of data storage scenarios to ensure efficient use of memory.

[0087] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 4 , step S20 may include steps S201 to S204:

[0088] Step S201, calculating the optimal identification number length;

[0089] In a feasible implementation, step S201 may include steps S2011 to S2012:

[0090] S2011, using the number of bits of the bucket space of the storage bucket as the number of bits of the identification number;

[0091] It should be noted that the number of bits in the bucket space is the bucket space size (bucket-bit), that is, the bucket size. The number of bits in the identification number is the initial number of bits of the hash identification number, that is, the initial length of the hash identification number. In this embodiment, the number of bits in the bucket space of the storage bucket is used as the number of bits of the identification number, that is, the bucket space size is used as the initial length of the hash identification number.

[0092] In a feasible implementation, the step of determining the number of bucket space bits of the storage bucket may include steps A11 to A12:

[0093] A11, obtains the correspondence between the number of original keys, the estimated average number of buckets, and the number of bits in the bucket space;

[0094] It should be noted that the number of original keys is the number of original keys, and the estimated average number of buckets is the estimated number of keys stored in each bucket. The corresponding relationship between the number of original keys, the estimated average number of buckets, and the number of bucket space bits is the calculation relationship of the number of bucket space bits, as shown below:

[0095] bucket-bit=log2(N / M)

[0096] Among them, bucket-bit is the number of bits in the bucket space, M is the estimated average number of buckets, and N is the number of original keys.

[0097] A12: Determine the number of bits of the bucket space based on the number of original keys, the estimated average number of buckets, and the corresponding relationship.

[0098] It can be understood that by substituting the number of original keys and the estimated average number of buckets into the above calculation formula, the corresponding number of bucket space bits can be calculated.

[0099] In another feasible implementation, the step of determining the number of bucket space bits of the storage bucket may include steps B11 to B13:

[0100] B11, obtaining the data scale of the original key, and generating test data based on the data scale;

[0101] It is understandable that Key regularization can greatly save storage space, but it needs to be based on a reasonable bucket space setting. If there are too many buckets, each bucket will store fewer elements on average. Although Redis uses compressed lists internally at this time, it is easy to cause waste of empty buckets; if there are too few buckets, each bucket will have too many elements on average, which can easily cause Redis to be converted into a hash table internally, and the storage space of the Value in Redis will increase exponentially. This embodiment controls the consumption distribution of buckets to a certain extent by controlling the bucket-bit parameter.

[0102] It should be noted that the data scale refers to the data magnitude of the Key. According to the data magnitude of the Key, a series of data is set for testing, namely, test data.

[0103] B12, testing the bucket consumption data of the storage bucket based on the preset test space bit number, and determining the bucket consumption data corresponding to different preset test space bit numbers when the storage bucket stores the test data;

[0104] It should be noted that the bucket consumption data refers to the consumption distribution of the storage bucket. The preset test space bit number is the bucket space bit number used for testing set in this embodiment, and usually multiple are set. The specific value and specific number can be determined according to the actual situation, and no specific limitation is made to this.

[0105] It can be understood that, in this embodiment, under different preset test space bit numbers, the bucket consumption data of the storage bucket when storing the test data is tested.

[0106] B13, analyzing the bucket consumption data corresponding to different preset test space bit numbers, determining the optimal space bit number, and using the optimal space bit number as the bucket space bit number.

[0107] It should be noted that the optimal number of spaces is the most appropriate number of bucket spaces obtained according to the test results. According to the bucket consumption data corresponding to different preset test space digits, the most appropriate number of bucket spaces is found in the preset test space digits as the optimal number of spaces.

[0108] In the specific implementation, through statistical analysis, when storing 2 billion key-value data, it is determined that the optimal space bit number is 29 bits and the average bucket capacity is 4.18, making the entire space controllable and highly utilized.

[0109] In this implementation, bucket space is planned based on statistical rules, which effectively reduces memory expansion rate and improves memory usage efficiency.

[0110] The above are only two feasible implementations of step S201 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S201.

[0111] S2012, adjusting the number of bits of the identification number based on the number of effective bits of the byte to obtain the optimal length of the identification number.

[0112] It should be noted that the effective digits of a byte refer to the number of effective digits in a byte. Since the hash identification number finally obtained is in the form of ASCII code, and ASCII code (0-127), only 7 bits are available in any byte, that is, the effective digits of a byte are 7 bits.

[0113] In a feasible implementation, step S2012 may include steps C11 to C15:

[0114] C11, dividing the identification number bytes based on the number of digits of the identification number, and determining the length of the identification number bytes;

[0115] It should be noted that the identification number byte is the byte of the hash identification number, and the bytes of the hash identification number are divided according to the number of digits of the identification number.

[0116] It is understandable that since a byte is usually 8 bits, it is first divided into 8 bits per byte. At this time, the first identification number bytes are all 8 bits, and the last identification number byte can be 1 to 8 bits, so that the length of each identification number byte can be determined.

[0117] C12, based on the number of effective digits of the byte, determining whether the number of digits of the identification number is divisible by the number of effective digits of the byte;

[0118] It should be noted whether the number of digits in the identification number is divisible by the effective number of the byte, that is, whether the number of digits in the identification number is divisible by 7. According to the effective number of the byte (7 digits), calculate the number of digits in the identification number and divide it by 7. If there is no remainder, it means that the number of digits in the identification number is divisible by the effective number of the byte. If there is a remainder, it means that the number of digits in the identification number is not divisible by the effective number of the byte.

[0119] It can be understood that if the number of bits of the identification number is evenly divisible by the number of valid bits of the byte, the number of bits of the identification number can be used as the optimal identification number length; if the number of bits of the identification number is not evenly divisible by the number of valid bits of the byte, the number of bits of the identification number needs to be further adjusted to obtain the optimal identification number length.

[0120] C13, when the number of digits of the identification number cannot be divided evenly by the number of valid digits of the byte, calculate the remainder corresponding to the number of digits of the identification number;

[0121] It should be noted that if the number of digits of the identification number cannot be divided evenly by the number of valid digits of the byte, the remainder of dividing the number of digits of the identification number by 7 is obtained.

[0122] C14, obtaining the length of the last byte in the identification number byte, and adjusting the length of the last byte based on the remainder;

[0123] It should be noted that the length of the last byte is the length of the last byte. In this embodiment, the length of the last byte in the identification number byte is adjusted according to the calculated remainder so that the number of digits of the identification number can be divided by 7.

[0124] It can be understood that if the length of the last byte is greater than the remainder, the length of the last byte can be reduced, and the reduced value is the remainder. For example, if the length of the last byte is 4 bits and the remainder is 3 bits, the length of the last byte is reduced by 3 bits, and the adjusted length of the last byte is 1 bit. If the length of the last byte is less than the remainder, the length of the last byte can be increased, and the increased value is the remainder. For example, if the length of the last byte is 2 bits and the remainder is 3 bits, the length of the last byte is increased by 2 bits, and the adjusted length of the last byte is 5 bits. If the length of the last byte is equal to the remainder and the remainder is less than or equal to 4 bits, the length of the last byte can be increased or reduced, and the increased / reduced value is the remainder. For example, if the length of the last byte is 4 bits and the remainder is 4 bits, the length of the last byte can be reduced by 4 bits, and the adjusted length of the last byte is 0 bits, or the length of the last byte is increased by 4 bits, and the adjusted length of the last byte is 8 bits. If the length of the last byte is equal to the remainder and the remainder is greater than 4 bits, the length of the last byte can be reduced, and the reduced value is the remainder. For example, if the length of the last byte is 5 bits and the remainder is 5 bits, the length of the last byte is reduced by 5 bits. At this time, the adjusted length of the last byte is 0 bits.

[0125] C15, determining the optimal identification number length based on the adjusted length of the last byte.

[0126] It should be noted that the adjusted last byte length is obtained and used as the optimal identification number length.

[0127] In this implementation, the number of bits of the identification number is adjusted based on the number of effective bits of the byte to obtain an accurate optimal identification number length, so as to facilitate subsequent ASCII code conversion and reduce the Key length.

[0128] Step S202, determining a valid hash value corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key;

[0129] It should be noted that the valid hash value refers to the part selected as the hash identification number. Usually, the front part of the hash value is selected as the hash identification number. That is to say, according to the optimal identification number length, the values ​​corresponding to the corresponding number of bits in the hash value are selected from front to back to obtain the final hash identification number.

[0130] Step S203, adjusting the valid hash value corresponding to the original key based on the ASCII code value range to obtain an initial hash identification number corresponding to the original key;

[0131] It should be noted that the ASCII code value range refers to the range of values ​​that can be represented by the ASCII code, that is, 0 to 127. The initial hash identification number is the adjusted valid hash value. If the valid hash value of the original key exceeds the ASCII code value range, it cannot be successfully converted into the corresponding ASCII code. Therefore, this embodiment traverses all valid hash values, finds out the valid hash values ​​that exceed the ASCII code value range, and makes corresponding adjustments to obtain the initial hash identification number, so that all initial hash identification numbers can be within the ASCII code value range.

[0132] Step S204: convert the initial hash identification number corresponding to the original key into a corresponding ASCII code to obtain the hash identification number corresponding to the original key.

[0133] It can be understood that the initial hash identification numbers need to be converted into corresponding ASCII codes, and finally the hash identification numbers corresponding to each original key, that is, each BucketId, are obtained. At this time, the hash identification numbers are filled with ASCII codes, and the length can be reduced by half, thereby reducing the length of the new Key by half.

[0134] In the specific implementation, refer to Figure 5 , create a MessageDigest instance of the MD5 algorithm, prepare for hash calculation, and update the MD5 instance with the Key to be stored; calculate the MD5 hash result and obtain the original hash byte array; calculate the length of BucketId according to the specified number of bits, ensuring that the number of bits is correct; adjust the last byte according to bit%7 to further ensure the accuracy of the BucketId bit; copy the front part of the MD5 hash result to the BucketId array to form a preliminary BucketId; traverse the BucketId array and adjust the byte values ​​to ensure that all bytes are within the ASCII code range; return the processed BucketId array to complete the BucketId generation.

[0135] This embodiment provides a Redis storage optimization method, which uses a custom Hash algorithm to regularly distribute the keys cached in Redis into multiple storage buckets. The keys of each storage bucket are strings of the same length, so that a more space-saving underlying encoding method can be used to achieve memory alignment and space saving, and can effectively utilize storage space, reduce memory usage and memory fragmentation, save 80% of the original memory usage, reduce memory space waste, and can be applicable to tens of billions of data storage scenarios to ensure efficient use of memory. At the same time, bucket space planning based on statistical laws can effectively reduce memory expansion rate and improve memory usage efficiency.

[0136] Based on the above embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 6 , before step S40, steps S31 to S32 may be included:

[0137] Step S31, obtaining access timestamp information of the key to be stored;

[0138] It should be noted that the access timestamp information refers to the time when the storage key was last accessed. A TTL (Time To Live) timestamp may be used. Based on the access timestamp information, the length of time between the last access time of the key to be stored and the current time may be determined.

[0139] Step S32: based on a preset byte position, insert the access timestamp information of the key to be stored into the storage data corresponding to the key to be stored.

[0140] It should be noted that the preset byte position is a preset position for inserting the access timestamp information, for example, the first 32 bits of the data, which can be set according to actual needs and is not specifically limited.

[0141] It can be understood that, according to the preset byte position, the access timestamp information of the key to be stored is inserted into the corresponding storage data.

[0142] Step S40 may include steps S401 to S405:

[0143] Step S401, when a write instruction is received, extracting a preset number of sample storage buckets from the storage buckets;

[0144] It should be noted that the Redis data stored in bucket form cannot directly use the cache expiration policy natively supported by Redis. Therefore, this embodiment implements it through write-time deletion.

[0145] In addition, it should be noted that a write instruction is an instruction to write new data, and it is generally considered that a write operation is about to be performed. The preset number is a preset number of extractions, and the specific value can be flexibly adjusted according to actual needs, and there is no specific limitation on this. The sample storage bucket is the storage bucket extracted as a sample.

[0146] It is understandable that before the write operation is performed, samples are extracted from the storage buckets, and a preset number of storage buckets are extracted as sample storage buckets.

[0147] Step S402, when the capacity of the sample storage bucket is greater than a preset capacity threshold, obtaining access timestamp information of the data stored in the sample storage bucket;

[0148] It should be noted that the preset capacity threshold refers to the threshold of the capacity of the pre-set storage bucket. The specific value can be flexibly adjusted according to actual needs and is not specifically limited. If the capacity of the sample storage bucket is greater than the preset capacity threshold, it means that there is not much available capacity left in the sample storage bucket. At this time, it is usually necessary to process the stored Keys and delete the expired Keys. In the specific implementation, the number of elements (entries) in the HashMap can be used as the capacity, and the maximum value of the set number of elements can be used as the preset capacity threshold. For example, assuming that the maximum value of the number of elements is 15, when the entry in the HashMap is greater than 15, it is considered that the capacity has exceeded the limit and the expired Key needs to be deleted in order to write new data.

[0149] It is understandable that in order to determine whether the Key is expired, it is necessary to obtain the last access time of the Key. Since this embodiment stores the access timestamp information of the Key in the first 32 bits of the corresponding data, the access timestamp information of the stored data can be obtained to determine the last access time of the Key.

[0150] Step S403, based on the access timestamp information of the data stored in the sample storage bucket, determining the non-access duration of the key to be stored corresponding to the data stored in the sample storage bucket;

[0151] It should be noted that the non-access duration refers to the length of time from the last access time to the current time, which can be used to characterize the hotness or coldness of data.

[0152] Step S404, based on the non-access duration of the to-be-stored key corresponding to the data stored in the sample storage bucket, determine, among the to-be-stored keys in the sample storage bucket, expired keys whose non-access duration is greater than a preset time period;

[0153] It should be noted that the preset time period is a set appropriate time threshold, for example, 35 days, and the specific value can be flexibly adjusted according to actual needs, and there is no specific limitation on this. The expired key is the expired key, that is, the cold data.

[0154] Generally speaking, the longer the duration of non-access of the key to be stored is, the more likely it is to be cold data. Therefore, this embodiment sets a preset time period. If the duration of non-access of the storage key is greater than the preset time period, the storage key is considered to be cold data and is treated as an expired key. If the duration of non-access of the storage key is less than the preset time period, the storage key is considered to be hot data.

[0155] Step S405, deleting the expired key to write new data.

[0156] It is understandable that by deleting expired keys, cold data can be eliminated in a timely manner, and hot data can be retained, so that new data can be written and query efficiency can be improved.

[0157] In the specific implementation, refer to Figure 7 , store the TTL timestamp in the first 32 bits of the value, sample according to a certain ratio during write operations, and use the HLEN command to determine whether the bucket capacity exceeds a certain threshold. If it exceeds, delete the expired Key to implement a custom data expiration strategy, eliminate cold data in time, retain hot data, and improve query efficiency.

[0158] This embodiment provides a Redis storage optimization method, which distributes the keys cached in Redis into multiple storage buckets in an orderly manner. The keys of each storage bucket are strings of the same length, so that a more space-saving underlying encoding method can be used to achieve memory alignment and space saving, and can effectively utilize storage space, reduce memory usage and memory fragmentation, and reduce waste of memory space. It can also be applicable to tens of billions of data storage scenarios to ensure efficient use of memory. At the same time, a custom expiration strategy is used to eliminate cold data in a timely manner, retain hot data, improve query efficiency, and reduce the impact of invalid data on system performance.

[0159] For example, to help understand the implementation process of the Redis storage optimization method obtained by combining this embodiment with the above-mentioned embodiment 3, please refer to Figure 8 , Figure 8 The following is an overall schematic diagram of a Redis storage optimization method. Specifically:

[0160] Using custom Hash algorithms to organize Key storage, greatly reducing memory usage and memory fragmentation, saving 80% of the original memory usage; planning bucket space based on statistical rules, effectively reducing memory expansion rate, and improving memory usage efficiency; custom expiration policies to eliminate cold data in a timely manner, retain hot data, improve query efficiency, and reduce the impact of invalid data on system performance. By optimizing Key storage (Key regularization and dispersion), planning bucket space (storage planning) and custom expiration policies, efficient memory utilization is achieved and query efficiency is improved, which is especially suitable for tens of billions of data cache scenarios.

[0161] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the Redis storage optimization method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0162] This application also provides a Redis storage optimization device, please refer to Fig. 9 , the Redis storage optimization device includes:

[0163] The hash calculation module 10 is used to obtain the original key of the data stored in the Redis database and determine the hash value corresponding to the original key;

[0164] A data dispersing module 20, configured to generate a hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key;

[0165] The data dispersing module 20 is further configured to use the hash identification number corresponding to the original key as the key to be stored, and to disperse and store the key to be stored in multiple storage buckets;

[0166] The storage optimization module 30 is used to re-encode the storage based on the keys to be stored in the multiple storage buckets to optimize the memory space of the Redis database.

[0167] In a feasible implementation manner, the data dispersing module 20 is further used to calculate the optimal identification number length;

[0168] Determine a valid hash value corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key;

[0169] Based on the ASCII code value range, the effective hash value corresponding to the original key is adjusted to obtain an initial hash identification number corresponding to the original key;

[0170] The initial hash identification number corresponding to the original key is converted into a corresponding ASCII code to obtain the hash identification number corresponding to the original key.

[0171] In a feasible implementation manner, the data dispersing module 20 is further configured to use the number of bits of the bucket space of the storage bucket as the number of bits of the identification number;

[0172] Based on the number of effective bits of the byte, the number of bits of the identification number is adjusted to obtain the optimal identification number length.

[0173] In a feasible implementation manner, the data dispersing module 20 is further used to divide the identification number bytes based on the number of bits of the identification number, and determine the length of the identification number bytes;

[0174] Based on the number of effective digits of the byte, determining whether the number of digits of the identification number is divisible by the number of effective digits of the byte;

[0175] When the number of digits of the identification number cannot be divided evenly by the number of valid digits of the byte, calculate the remainder corresponding to the number of digits of the identification number;

[0176] Obtaining the length of the last byte in the identification number bytes, and adjusting the length of the last byte based on the remainder;

[0177] The optimal identification number length is determined based on the adjusted length of the last byte.

[0178] In a feasible implementation manner, the data dispersing module 20 is further used to obtain the corresponding relationship between the number of original keys, the estimated average number of buckets, and the number of bucket space bits;

[0179] The number of bits of the bucket space is determined based on the number of the original keys, the estimated average number of buckets, and the corresponding relationship.

[0180] In a feasible implementation manner, the data dispersing module 20 is further used to obtain the data scale of the original key, and generate test data based on the data scale;

[0181] Testing the bucket consumption data of the storage bucket based on the preset test space bit number, and determining the bucket consumption data corresponding to different preset test space bit numbers when the storage bucket stores the test data;

[0182] The bucket consumption data corresponding to different preset test space bit numbers are analyzed to determine the optimal space bit number, and the optimal space bit number is used as the bucket space bit number.

[0183] In a feasible implementation manner, the storage optimization module 30 is further used to obtain access timestamp information of the key to be stored;

[0184] Based on a preset byte position, the access timestamp information of the key to be stored is inserted into the storage data corresponding to the key to be stored.

[0185] In a feasible implementation manner, the storage optimization module 30 is further configured to extract a preset number of sample storage buckets from the storage bucket when receiving a write instruction;

[0186] When the capacity of the sample storage bucket is greater than a preset capacity threshold, obtaining access timestamp information of the data stored in the sample storage bucket;

[0187] Based on the access timestamp information of the data stored in the sample storage bucket, determining the non-access duration of the key to be stored corresponding to the data stored in the sample storage bucket;

[0188] Based on the non-access duration of the to-be-stored key corresponding to the data stored in the sample storage bucket, determining, among the to-be-stored keys in the sample storage bucket, expired keys whose non-access duration is greater than a preset time period;

[0189] The expired key is deleted to write new data.

[0190] The Redis storage optimization device provided by the present application adopts the Redis storage optimization method in the above embodiment, which can solve the technical problem that the different lengths of the Keys stored in Redis lead to a lot of memory waste. Compared with the prior art, the beneficial effects of the Redis storage optimization device provided by the present application are the same as the beneficial effects of the Redis storage optimization method provided by the above embodiment, and other technical features in the Redis storage optimization device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0191] The present application provides a Redis storage optimization device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Redis storage optimization method in the above-mentioned embodiment 1.

[0192] Reference below Fig.10 , which shows a schematic diagram of the structure of a Redis storage optimization device suitable for implementing the embodiment of the present application. The Redis storage optimization device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The Redis storage optimization device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0193] like Fig.10As shown, the Redis storage optimization device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the Redis storage optimization device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the Redis storage optimization device to communicate with other devices wirelessly or by wire to exchange data. Although the Redis storage optimization device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided alternatively.

[0194] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0195] The Redis storage optimization device provided by the present application adopts the Redis storage optimization method in the above embodiment, which can solve the technical problem that the different lengths of the Keys stored in Redis lead to a lot of memory waste. Compared with the prior art, the beneficial effects of the Redis storage optimization device provided by the present application are the same as the beneficial effects of the Redis storage optimization method provided by the above embodiment, and the other technical features in the Redis storage optimization device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0196] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0197] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0198] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the Redis storage optimization method in the above-mentioned embodiment.

[0199] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0200] The computer-readable storage medium may be included in the Redis storage optimization device; or may exist independently without being assembled into the Redis storage optimization device.

[0201] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the Redis storage optimization device, the Redis storage optimization device: obtains the original key for storing data in the Redis database, and determines the hash value corresponding to the original key; generates a hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key; uses the hash identification number corresponding to the original key as the key to be stored, and stores the key to be stored in multiple storage buckets in a dispersed manner; re-encodes the storage based on the keys to be stored in multiple storage buckets to optimize the memory space of the Redis database.

[0202] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0203] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0204] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0205] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned Redis storage optimization method, and can solve the technical problem that the different lengths of the Keys stored in Redis lead to more memory waste. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the Redis storage optimization method provided by the above-mentioned embodiment, and will not be repeated here.

[0206] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned Redis storage optimization method when executed by a processor.

[0207] The computer program product provided by this application can solve the technical problem that the different lengths of the keys stored in Redis lead to a lot of memory waste. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the Redis storage optimization method provided in the above embodiment, which will not be repeated here.

[0208] The above are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A Redis storage optimization method, characterized in that: The method comprises: Obtain the original key of the data stored in the Redis database, and determine the hash value corresponding to the original key; Generate a hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key; Using the hash identification number corresponding to the original key as the key to be stored, and distributing and storing the key to be stored in multiple storage buckets; Based on the keys to be stored in the multiple storage buckets, storage encoding is re-performed to optimize the memory space of the Redis database.

2. The method according to claim 1, characterized in that The step of generating a hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key comprises: Calculate the optimal identification number length; Determine a valid hash value corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key; Based on the ASCII code value range, the effective hash value corresponding to the original key is adjusted to obtain an initial hash identification number corresponding to the original key; The initial hash identification number corresponding to the original key is converted into a corresponding ASCII code to obtain the hash identification number corresponding to the original key.

3. The method according to claim 2, characterized in that The step of calculating the optimal identification number length comprises: Using the number of bits of the bucket space of the storage bucket as the number of bits of the identification number; Based on the number of effective bits of the byte, the number of bits of the identification number is adjusted to obtain the optimal identification number length.

4. The method according to claim 3, characterized in that The step of adjusting the number of bits of the identification number based on the number of effective bits of the byte to obtain the optimal identification number length comprises: Based on the number of digits of the identification number, divide the identification number bytes and determine the length of the identification number bytes; Based on the number of effective digits of the byte, determining whether the number of digits of the identification number is divisible by the number of effective digits of the byte; When the number of digits of the identification number cannot be divided evenly by the number of valid digits of the byte, calculate the remainder corresponding to the number of digits of the identification number; Obtaining the length of the last byte in the identification number bytes, and adjusting the length of the last byte based on the remainder; The optimal identification number length is determined based on the adjusted length of the last byte.

5. The method according to claim 3, characterized in that Before the step of determining the number of digits of the identification number based on the number of digits of the bucket space of the storage bucket, the following step is further included: Get the correspondence between the number of original keys, the estimated average number of buckets, and the number of bucket spaces; The number of bits of the bucket space is determined based on the number of the original keys, the estimated average number of buckets, and the corresponding relationship.

6. The method according to claim 3, characterized in that Before the step of determining the number of digits of the identification number based on the number of digits of the bucket space of the storage bucket, the following step is further included: Acquire the data scale of the original key, and generate test data based on the data scale; Testing the bucket consumption data of the storage bucket based on the preset test space bit number, and determining the bucket consumption data corresponding to different preset test space bit numbers when the storage bucket stores the test data; The bucket consumption data corresponding to different preset test space bit numbers are analyzed to determine the optimal space bit number, and the optimal space bit number is used as the bucket space bit number.

7. The method according to any one of claims 1 to 6, characterized in that The step of re-encoding the storage based on the keys to be stored in the multiple storage buckets to optimize the memory space of the Redis database also includes: Obtaining access timestamp information of the key to be stored; Based on a preset byte position, inserting the access timestamp information of the key to be stored into the storage data corresponding to the key to be stored; After the step of re-encoding the storage based on the keys to be stored in the multiple storage buckets to optimize the memory space of the Redis database, the following steps are further included: When receiving a write instruction, extracting a preset number of sample storage buckets from the storage buckets; When the capacity of the sample storage bucket is greater than a preset capacity threshold, obtaining access timestamp information of the data stored in the sample storage bucket; Based on the access timestamp information of the data stored in the sample storage bucket, determining the non-access duration of the key to be stored corresponding to the data stored in the sample storage bucket; Based on the non-access duration of the to-be-stored key corresponding to the data stored in the sample storage bucket, determining, among the to-be-stored keys in the sample storage bucket, expired keys whose non-access duration is greater than a preset time period; The expired key is deleted to write new data.

8. A Redis storage optimization device, characterized in that: The device comprises: A hash calculation module is used to obtain the original key of the data stored in the Redis database and determine the hash value corresponding to the original key; A data dispersing module, used for generating a hash identification number corresponding to the original key based on the optimal identification number length and the hash value corresponding to the original key; The data dispersing module is further used to use the hash identification number corresponding to the original key as the key to be stored, and to disperse and store the key to be stored in multiple storage buckets; The storage optimization module is used to re-encode the storage based on the keys to be stored in the multiple storage buckets to optimize the memory space of the Redis database.

9. A Redis storage optimization device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the Redis storage optimization method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the Redis storage optimization method according to any one of claims 1 to 7 are implemented.

Images