Method and system for detecting and sharding value of redis data cache

Abstract

The application provides a detection and fragmentation method of redis data cache value, which comprises the following steps: 1, setting the threshold of redis slow query, recording through the log, calculating the reading time, and writing the slow query into the log if the threshold is exceeded; 2, monitoring the slow query log, reading the redis key of the slow query, and processing the redis value between 10-100 KB; 3, fragmenting according to 10 KB, adding the key suffix of the next fragment to the end of each group of value, the value of the first group of fragments is ended with _1, representing that the key of the next group of fragments is key_1, the value of the second group of fragments is ended with _2, and so on, _3 to _9, until the last fragment is not added with the suffix; 4, providing a general redis reading interface, reading the value of the redis key, if the value is ended with _1, indicating that the redis is a fragmented large value, continuing to read the data of the next data fragment key_1, and so on until the last fragment, splicing the characters of all fragments and returning to the client; effectively reducing the network blocking problem caused by the client timeout blocking.

Description

Technical Field

[0001] This invention relates to the field of data caching technology, and in particular to a method and system for detecting and sharding values ​​in Redis data caching. Background Technology

[0002] Redis is a commonly chosen middleware for data caching, a key-value storage system. Redis is suitable for storing small amounts of frequently accessed data. While the keys for Redis data caching can be strictly defined during system operation, the values ​​are difficult for users to determine, and their size is sometimes hard to control. If a Redis value is too large, it may cause client timeouts and network congestion leading to read timeouts. If the size of Redis values ​​is calculated and monitored synchronously during Redis operations, a few large values ​​might impact overall storage efficiency, as most values ​​are small. However, these large values ​​cannot be ignored, as they could cause client timeouts and network congestion. Summary of the Invention

[0003] To overcome the above problems, the purpose of this invention is to provide a method for detecting and sharding values ​​in a Redis data cache. By sharding, a giant Redis data cache value is split into multiple smaller values, which effectively alleviates problems such as network congestion caused by client timeout blocking.

[0004] This invention employs the following scheme: a method for detecting and sharding values ​​in a Redis data cache, the method comprising the following steps:

[0005] Step S1: Set the threshold for slow queries in the Redis data cache. Obtain the Redis read time by the time difference between the log records before and after reading the Redis data cache. When the Redis read time exceeds the threshold, write the key of the slow query in the Redis data cache to the log asynchronously.

[0006] Step S2: Monitor the slow query log, read the key of the Redis data cache for slow queries, and if the value corresponding to the key in the Redis data cache is less than 10KB, no operation is performed and the process ends. If it is greater than 100KB, the key in the Redis data cache is deleted. If the value is between 10-100KB, proceed to step S3.

[0007] Step S3: Divide the Redis data cache values ​​into groups according to preset values. The content of each group of values ​​is value_1, value_2, value_3, and so on until the last shard is not appended. The resulting set of shards is {(key_value, value_1), (key_value_1, value_2)...(key_value_N-1, value_N),(key_value_N, value)}.

[0008] Step S4: Provide a Redis read interface to read the value of the Redis data cache key. If the value ends with _1, it means that this Redis data cache is a sharded value. Continue to read the data of the next data shard key value _1, and so on until the last shard. Concatenate the characters of all shards and return them to the client.

[0009] Furthermore, the implementation of step S1 is further specified as follows: set a Redis slow query threshold of 100ms. Queries exceeding 100ms are slow queries caused by values ​​that are too large. Obtain the Redis read time by the time difference between reading the log records before and after Redis. If the threshold of 100ms is exceeded, write the Redis data cache key of the slow query asynchronously to the log. The synchronous writing method is not used in order not to reduce the system throughput.

[0010] Furthermore, step S3 is implemented as follows: the values ​​in the Redis data cache are sharded into groups according to preset values ​​to avoid single queries timeouts due to large individual values; the grouping method is: key: value_1; key_1: value_2; key_2: value_3; ... key_N-1: value_N; key_N: value, where N is an integer greater than 1; then the shard set after grouping is {(key value, value_1), (key value_1, value_2) ... (key value_N-1, value_N), (key value_N, value)}.

[0011] Furthermore, step S4 is implemented as follows: A Redis read interface is provided. The value of a Redis data cache key is read through this interface. If the value ends with _1, it indicates that this Redis data cache is a sharded value. The next shard key value _1 is then read. If the value corresponding to shard key value _1 ends with _2, the next shard key value _2 is read. That is, the suffix of the value is used as an index to find the same suffix for the key corresponding to the next shard. If the last value of a shard has no suffix, it is considered that all shard data has been read. Finally, the complete data is returned to the client by concatenating value_1 + value_2 + ... + value_N + value.

[0012] The present invention also provides a system for detecting and sharding values ​​in a Redis data cache, the system comprising a Redis read time acquisition module, a value size detection module, a sharding module, and a concatenation module;

[0013] The Redis read time acquisition module sets a threshold for slow queries in the Redis data cache, obtains the Redis read time by the time difference between log records before and after reading the Redis data cache, and writes the key of the slow query Redis data cache to the log asynchronously when the Redis read time exceeds the threshold.

[0014] The value size detection module monitors the slow query log, reads the key of the Redis data cache for slow queries, and if the value corresponding to the key in the Redis data cache is less than 10KB, no operation is performed and the process ends. If it is greater than 100KB, the key in the Redis data cache is deleted. If the value is between 10-100KB, the process proceeds to the sharding module.

[0015] The sharding module shards the Redis data cache value into groups according to preset values. The content of each group of values ​​is value_1, value_2, value_3, and so on until the last shard is not appended. The shard set after grouping is {(key value, value_1), (key value_1, value_2)...(key value_N-1, value_N),(key value_N, value)}.

[0016] The concatenation module provides a Redis read interface to read the value of the Redis data cache key. If the value ends with _1, it indicates that this Redis data cache is a sharded value. Continue reading the data of the next data shard key value _1, and so on until the last shard. Concatenate the characters of all shards and return them to the client.

[0017] Furthermore, the implementation of the Redis read time acquisition module is further specified as follows: a Redis slow query threshold of 100ms is set, and queries exceeding 100ms are slow queries caused by values ​​that are too large; the Redis read time is obtained by the time difference between the log records before and after the Redis read; if the threshold of 100ms is exceeded, the Redis data cache key of the slow query is asynchronously written to the log. The synchronous writing method is not used in order not to reduce the system throughput.

[0018] Furthermore, the implementation of the sharding module is further specified as follows: the values ​​of the Redis data cache are sharded according to preset values ​​to avoid single query timeouts caused by a single value being too large; the grouping method is: key: value_1; key_1: value_2; key_2: value_3; ... key_N-1: value_N; key_N: value, where N is an integer greater than 1; then the shard set after grouping is {(key value, value_1), (key value_1, value_2) ... (key value_N-1, value_N), (key value_N, value)}.

[0019] Furthermore, the implementation of the concatenation module is further specified as follows: A Redis read interface is provided. The value of a Redis data cache key is read through this interface. If the value ends with _1, it indicates that this Redis data cache is a sharded value. The system continues reading the data of the next shard key value _1. If the value corresponding to shard key value _1 ends with _2, the system continues reading the data of the next shard key value _2. That is, the suffix of the value is used as an index to find the same suffix for the key corresponding to the next shard. If the last value of a shard has no suffix, it is considered that all shard data has been read. Then, the complete data is returned to the client by concatenating value_1 + value_2 + ... + value_N + value.

[0020] The beneficial effects of this invention are as follows: This invention can asynchronously record slow query logs of Redis data cache, detect whether the key of the slow query in the Redis data cache is a large Redis value, and split a giant Redis data cache value into multiple smaller values ​​through sharding. This avoids the client timeout caused by a single slow query of a large value, and changes the single timeout read to multiple small value reads, effectively mitigating the problems of network congestion caused by client timeout blocking. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the workflow of the method of the present invention.

[0022] Figure 2 This is a system principle block diagram of the present invention. Detailed Implementation

[0023] The invention will now be further described with reference to the accompanying drawings.

[0024] Please see Figure 1 As shown, the present invention provides a method for detecting and sharding values ​​in a Redis data cache, the method comprising the following steps:

[0025] Step S1: Set a threshold for slow queries in the Redis data cache. Obtain the Redis read time by the time difference between reading the log records before and after reading the Redis data cache. When the Redis read time exceeds the threshold, the key of the slow query in the Redis data cache is asynchronously written to the log. The implementation of step S1 is further specified as follows: Set a threshold of 100ms for slow queries in Redis. Queries exceeding 100ms are slow queries caused by values ​​that are too large. Obtain the Redis read time by the time difference between reading the log records before and after reading the Redis data cache. If it exceeds the threshold of 100ms, the key of the slow query in the Redis data cache is asynchronously written to the log. The synchronous writing method is not used to avoid reducing the system throughput.

[0026] Step S2: Monitor the slow query log, read the key of the Redis data cache for slow queries, and if the value corresponding to the key in the Redis data cache is less than 10KB, no operation is performed and the process ends. If it is greater than 100KB, the key in the Redis data cache is deleted. If the value is between 10-100KB, proceed to step S3.

[0027] Step S3: The Redis cached values ​​are sharded into groups based on preset values. Each group's value is named value_1, value_2, value_3, and so on until the last shard is not appended. The resulting shard set is {(key_value, value_1), (key_value_1, value_2)...(key_value_N-1, value_N), (key_value_N, value)}. Step S3 is further implemented as follows: The Redis cached values ​​are sharded into groups... The data is sharded according to preset values ​​to avoid query timeouts caused by a single value being too large. The grouping method is as follows: key: value_1; key_1: value_2; key_2: value_3; ... key_N-1: value_N; key_N: value, where N is an integer greater than 1. The shard set after grouping is {(key value, value_1), (key value_1, value_2) ... (key value_N-1, value_N), (key value_N, value)}.

[0028] Step S4: Provide a Redis read interface to read the value of the Redis data cache key. If the value ends with _1, it means that this Redis data cache is a sharded value. Continue to read the data of the next data shard key value _1, and so on until the last shard. Concatenate the characters of all shards and return them to the client.

[0029] The implementation of step S4 is further specified as follows: A Redis read interface is provided. The value of a Redis data cache key is read through this interface. If the value ends with _1, it indicates that this Redis data cache is a sharded value. The next shard key value _1 is then read. If the value corresponding to shard key value _1 ends with _2, the next shard key value _2 is read. That is, the suffix of the value is used as an index to find the same suffix for the key corresponding to the next shard. If the last value of a shard has no suffix, it is considered that all shard data has been read. Finally, the complete data is returned to the client by concatenating value_1 + value_2 + ... + value_N + value.

[0030] The present invention will be further described below with reference to a specific embodiment:

[0031] A method for detecting and sharding values ​​in a Redis data cache, comprising the following steps:

[0032] Step 1: Set the Redis slow query threshold to 100ms. Calculate the Redis read time by the time difference between the log points before and after the Redis read. If the threshold of 100ms is exceeded, write the slow query Redis key to the log asynchronously.

[0033] A Redis slow query threshold of 100ms is set. Queries exceeding 100ms may be caused by excessively large values. The Redis read time is calculated by tracking the time difference before and after the read. If the time exceeds the 100ms threshold, the slow query Redis key is asynchronously written to the log. Synchronous logging is avoided to prevent reducing system throughput. For example, if the queried Redis key is 'user:xxx', the log entry will be 'user:xxx'.

[0034] Step 2: Monitor the slow query log, read the Redis data cache key for the slow query, and read the Redis data cache value. If it is less than 10KB, skip it; if it is greater than 100KB, delete it.

[0035] Monitoring the slow query logs revealed that queries for the Redis key "user:xxx" exceeded 100ms. Reading the Redis value for "user:xxx" showed the value as 'aaabbbccc........zzz'. Since UTF-8 encoding limits each character to a maximum of 3 bytes, 10KB would contain 3 * 10 * 1024 = 30720 characters. If the value is less than 30720 characters, it is skipped. If the value is greater than 30720 characters, the Redis key is deleted because excessively large keys would severely impact the system and are therefore not stored.

[0036] Step 3: If the Redis value is between 10-100KB, then shard it in groups of 10KB. Add the key suffix of the next shard to the value of each group as the end. The value of the first group of shards ends with _1, which means that the key of the next group of shards is key_1. The value of the second group of shards ends with _2, and so on from _3 to _9, until the last shard does not have a suffix.

[0037] If a Redis value is between 10-100KB (30720-307200 characters), it is sharded for storage to prevent single queries from timing out due to excessively large values. Using 10KB groups, assuming the value 'aaabbbccc........zzz' has 61443 characters, the first 30720 characters form a group. The first shard key is 'user:xxx', and the value is 'aaabbbccc...._1', with the suffix '_1' added to the key of the next shard. The next group is from 30721 to 61440 characters, with the key 'user:xxx_1' and the value '..._2'. The third group is from 61441 to 92160 characters. Since the entire value only has 61443 characters, it can be divided into a maximum of three groups, with the value 'zzz'. As this is the last group, no suffix is ​​added.

[0038] The rules are summarized as follows

[0039] key: value_1

[0040] key_1: value_2

[0041] key_2: value_3

[0042] key_3: value_4

[0043] key_4: value_5

[0044] key_5: value_6

[0045] key_6: value_7

[0046] key_7: value_8

[0047] key_8: value_9

[0048] key_9: value_10

[0049] key_N-1: value_N;

[0050] key_N: value.

[0051] Step 4: Provide a general Redis read interface. If the value of a Redis key ends with _1, it means that this Redis key is a large value that has been sharded. Continue reading the data of the next data shard key_1, and so on until the last shard. Concatenate the characters of all shards and return them to the client.

[0052] To identify and read large sharded Redis values, a general Redis read interface is provided as the entry point. Reading the Redis key 'user:xxx' yields value1 = 'aaabbbccc...._1'. The presence of '_1' at the end indicates this is a large sharded value. The next shard key, 'user:xxx_1', is then read, yielding value2 which ends with '_2'. The next shard, 'user:xxx_2', yields value3 = 'zzz'. The absence of a suffix indicates that all shards have been read. Value1 + value2 + value3 are then concatenated to obtain the complete data 'aaabbbccc........zzz', which is returned to the client as organized data.

[0053] Please see Figure 2 As shown, the present invention also provides a system for detecting and sharding the value of a Redis data cache, the system including a Redis read time acquisition module, a value size detection module, a sharding module, and a splicing and combining module;

[0054] The Redis read time acquisition module sets a threshold for slow queries in the Redis data cache, obtains the Redis read time by the time difference between log records before and after reading the Redis data cache, and writes the key of the slow query Redis data cache to the log asynchronously when the Redis read time exceeds the threshold.

[0055] The value size detection module monitors the slow query log, reads the key of the Redis data cache for slow queries, and if the value corresponding to the key in the Redis data cache is less than 10KB, no operation is performed and the process ends. If it is greater than 100KB, the key in the Redis data cache is deleted. If the value is between 10-100KB, the process proceeds to the sharding module.

[0056] The sharding module shards the Redis data cache value into groups according to preset values. The content of each group of values ​​is value_1, value_2, value_3, and so on until the last shard is not appended. The shard set after grouping is {(key value, value_1), (key value_1, value_2)...(key value_N-1, value_N),(key value_N, value)}.

[0057] The concatenation module provides a Redis read interface to read the value of the Redis data cache key. If the value ends with _1, it indicates that this Redis data cache is a sharded value. Continue reading the data of the next data shard key value _1, and so on until the last shard. Concatenate the characters of all shards and return them to the client.

[0058] The implementation of the Redis read time acquisition module is further specified as follows: a Redis slow query threshold of 100ms is set, and queries exceeding 100ms are slow queries caused by values ​​that are too large; the Redis read time is obtained by the time difference between the log records before and after the Redis read; if the threshold of 100ms is exceeded, the Redis data cache key of the slow query is asynchronously written to the log. The synchronous writing method is not used in order not to reduce the system throughput.

[0059] The implementation of the sharding module is further specified as follows: the values ​​of the Redis data cache are sharded according to preset values ​​to avoid single query timeouts caused by a single value being too large; the grouping method is: key: value_1; key_1: value_2; key_2: value_3; ... key_N-1: value_N; key_N: value, where N is an integer greater than 1; then the shard set after grouping is {(key value, value_1), (key value_1, value_2) ... (key value_N-1, value_N), (key value_N, value)}.

[0060] The implementation of the concatenation module is further specified as follows: A Redis read interface is provided. The value of a Redis data cache key is read through this interface. If the value ends with _1, it indicates that this Redis data cache is a sharded value. The system continues reading the data of the next shard key value _1. If the value corresponding to shard key value _1 ends with _2, the system continues reading the data of the next shard key value _2. That is, the suffix of the value is used as an index to find the same suffix for the key corresponding to the next shard. If the last value of a shard has no suffix, it is considered that all shard data has been read. Then, the complete data is returned to the client by concatenating value_1 + value_2 + ... + value_N + value.

[0061] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. A method for detecting and sharding values of a redis data cache, the method comprising: The method comprises the following steps: ​ Step S1, setting a threshold value of redis data cache slow query, obtaining the redis reading time through the time difference of reading the front and rear log records of the redis data cache, and when the redis reading time exceeds the threshold value, the key of the slow query redis data cache is written into the log asynchronously; Step S2, monitoring the slow query log, reading the key of the slow query redis data cache, reading the value corresponding to the key of the redis data cache, if the value is less than 10KB, no operation is performed, and the process is ended, if the value is greater than 100KB, the key of the redis data cache is deleted, and if the value is between 10-100KB, step S3 is entered; Step S3, the value of the redis data cache is divided into groups according to a preset value, the content of each group value is value_1, value_2, value_3, and so on until the last piece is not followed by a suffix; the grouped piece set is {(key value, value value_1), (key value_1, value value_2)...(key value_N-1, value value_N), (key value_N, value value)}; Step S4, providing a redis reading interface, reading the value of the redis data cache key, if the value is followed by _1, it indicates that the redis data cache is a divided value, the data of the next data piece key value_1 is read, and so on until the last piece is read, and all the pieces of characters are spliced and returned to the client; The implementation mode of step S4 is further specific: providing a redis reading interface, reading the value of the redis data cache key through the redis reading interface, if the value is followed by _1, it indicates that the redis data cache is a divided value, the data of the next data piece key value_1 is read, and so on until the last piece is read, and all the pieces of characters are spliced and returned to the client.

2. The method for detecting and sharding a redis data cache value according to claim 1, characterized in that: The implementation mode of step S1 is further specific: setting a threshold value of 100ms for the redis slow query, and the query exceeding 100ms is a slow query caused by a large value; the reading time of the redis is obtained through the time difference of reading the front and rear log records of the redis, if the reading time exceeds the threshold value of 100ms, the key of the slow query redis data cache is written into the log asynchronously, and the synchronous writing mode is not adopted in order to not reduce the throughput of the system.

3. The method for detecting and sharding a redis data cache value according to claim 1, characterized in that: The implementation manner of step S3 is further specific: the value of the redis data cache is divided into groups according to a preset value, so as to avoid that a single value is too large to cause a single query timeout; the grouping manner is: key:value_1; key_1:value_2; key_2:value_3;... key_N-1:value_N; key_N:value, wherein N is an integer greater than 1; and the grouped shard set is {(key value, value value_1), (key value_1, value value_2)... (key value_N-1, value value_N), (key value_N, value value)}.

4. A system for detection and sharding of values of a redis data cache, characterized in that: The system comprises a redis reading time acquisition module, a value size detection module, a sharding module and a splicing combination module. The redis reading time acquisition module sets a threshold of a slow query of the redis data cache, acquires a redis reading time through a time difference between reading logs before and after the redis data cache, and writes the key of the slow query of the redis data cache into a log asynchronously when the redis reading time exceeds the threshold. The value size detection module monitors a slow query log, reads the key of the slow query of the redis data cache, reads the value corresponding to the key of the redis data cache, does not perform an operation if the value is less than 10 KB, ends the process, deletes the key of the redis data cache if the value is greater than 100 KB, and enters the sharding module if the value is between 10-100 KB. The sharding module divides the value of the redis data cache into groups according to a preset value, and the content of each group of values is value_1, value_2, value_3 and so on until the last shard without a suffix at the end; and the grouped shard set is {(key value, value value_1), (key value_1, value value_2)... (key value_N-1, value value_N), (key value_N, value value)}. The splicing combination module provides a redis reading interface, reads the value of the key of the redis data cache, reads the data of the next data shard key value_1 if the value ends with _1, indicating that the redis data cache is a divided value, and reads the last shard and splices the characters of all shards to return to the client. The implementation manner of the splicing combination module is further specifically: a redis reading interface is provided, redis data cache key value is read through the redis reading interface, if the value is ended with _1, it indicates that the redis data cache is a value of a split piece, the data of the next data split piece key value _1 is read; and the value corresponding to the data split piece key value _1 is ended with _2, the data of the next data split piece key value _2 is read, that is, the tail suffix of the value is taken as an index to find the same tail suffix of the key corresponding to the next data split piece, the last value of the data split piece is not taken as a suffix, and all split piece data is read; the complete data is obtained by splicing value value _1+value value _2+...+value value N+value value, and the complete data is returned to the client.

5. The system for detection and sharding of a redis data cache's value according to claim 4, characterized in that: The implementation manner of the redis reading time acquisition module is further specifically: a threshold of a redis slow query is set as 100 ms, and a query exceeding 100 ms is a slow query caused by a too large value; the reading time of redis is obtained through the time difference of the log records before and after the redis reading, if the reading time of redis exceeds the threshold 100 ms, the slow query redis data cache key is written into a log asynchronously, and the synchronous writing into the log is not adopted, so as to not reduce the throughput of the system.

6. The system for detection and sharding of a redis data cache's value according to claim 4, wherein: The implementation manner of the split piece module is further specifically: the value of the redis data cache is split according to a preset value, so that the query single time is prevented from being timed out due to a too large single value; the grouping manner is: key: value _1; key _1: value _2; key _2: value _3;... key N-1: value N; key N: value, wherein N is an integer greater than 1; and the split piece set after the grouping is {(key value, value value _1), (key value _1, value value _2)... (key value N-1, value value N), (key value N, value value)}. The implementation manner of the split piece module is further specifically: the value of the redis data cache is split according to a preset value, so that the query single time is prevented from being timed out due to a too large single value; the grouping manner is: key: value _1; key _1: value _2; key _2: value _3;... key N-1: value N; key N: value, wherein N is an integer greater than 1; and the split piece set after the grouping is {(key value, value value _1), (key value _1, value value _2)... (key value N-1, value value N), (key value N, value value)}.

Images