Key detection method, system, device and storage medium based on remote dictionary

Abstract

The present invention provides a key detection method, system, device and storage medium based on a remote dictionary, the method comprising: obtaining a memory snapshot of a Redis server in a Redis cluster whose retrieval memory occupancy is greater than a first preset threshold and whose load is lower than a second preset threshold; inspecting the memory snapshot according to preset attributes, and storing the inspection results in a performance data warehouse; tracking all keys in the performance data warehouse, screening keys whose occupied capacity is greater than a preset key occupied capacity threshold, and generating an alarm email. The present invention can automatically match Redis cluster shards through a logical algorithm, can perform priority checks in chronological order, store data in a data warehouse to ensure data security, regularly push emails to relevant owners or users, understand the bigkey status of Redis in real time, and make corresponding adjustments, thereby improving the stability of Redis and improving work efficiency.

Description

Background Art

[0002] Redis (Remote Dictionary Server), also known as remote dictionary service, is an open source log-type, Key-Value database written in ANSI C language, supporting the network, memory-based or persistent, and providing APIs in multiple languages.

[0003] Redis is a key-value storage system. Similar to Memcached, it supports storing more value types, including string, list, set, zset (sorted set), and hash. These data types support push / pop, add / remove, intersection, union, difference, and richer operations, and these operations are atomic. On this basis, Redis supports various different sorting methods. Like memcached, in order to ensure efficiency, data is cached in memory. The difference is that Redis will periodically write updated data to disk or write modification operations to additional record files, and on this basis, master-slave synchronization is implemented.

[0004] Redis is a high-performance key-value database. The emergence of Redis has largely compensated for the shortcomings of key / value storage such as memcached, and can be a good supplement to relational databases in some situations. It provides clients such as Java, C / C++, C#, PHP, JavaScript, Perl, Object-C, Python, Ruby, Erlang, etc., which is very convenient to use.

[0005] Redis supports master-slave synchronization. Data can be synchronized from a master to any number of slaves, which can be masters that are linked to other slaves. This allows Redis to perform single-layer tree replication. Data can be written to the disk intentionally or unintentionally. Since the publish / subscribe mechanism is fully implemented, slaves can subscribe to a channel and receive the complete message publishing record of the master server when synchronizing the tree anywhere. Synchronization is very helpful for scalability of read operations and data redundancy.

[0006] In Redis, bigkey refers to a large value corresponding to a key in Redis. Operating bigkey is usually time-consuming, which means that the possibility of blocking Redis is greater. Therefore, it is necessary to check the bigkey in advance to avoid affecting online production. The Redis client has a built-in bigkey query function, which will traverse all keys in the server online. During the traversal, it consumes a lot of server resources and affects the performance of the production environment. In addition, the bigkey search that comes with the client will only check the largest key of each type each time, and cannot view more bigkeys, which cannot meet the bigkey inspection requirements of the production environment.

[0007] Therefore, the present invention provides a key detection method, system, device and storage medium based on a remote dictionary. Summary of the invention

[0008] In view of the problems in the prior art, the purpose of the present invention is to provide a key detection method, system, device and storage medium based on a remote dictionary, which overcomes the difficulties of the prior art, can automatically match Redis cluster shards through logical algorithms, can perform priority checks in chronological order, store data in a data warehouse to ensure data security, push emails to relevant owners or users at regular intervals, understand the bigkey status of Redis in real time, and make corresponding adjustments, thereby improving the stability of Redis and improving work efficiency.

[0009] An embodiment of the present invention provides a key detection method based on a remote dictionary, comprising the following steps:

[0010] S110, obtaining a memory snapshot of a Redis server in a Redis cluster whose retrieval memory occupancy is greater than a first preset threshold and whose load is lower than a second preset threshold;

[0011] S120, inspecting the memory snapshot according to preset attributes, and storing the inspection result in a performance data warehouse;

[0012] S130: Track all the keys in the performance data warehouse, filter the keys whose key thresholds are greater than the capacity occupied by preset keys, and generate an alarm email.

[0013] Preferably, in step S120, the keys contained in the memory snapshot are inspected by using a preset key prefix, and the keys satisfying the preset key prefix are stored in the performance data warehouse.

[0014] Preferably, in step S120, the keys contained in the memory snapshot are inspected by using a preset key suffix, and the keys satisfying the preset key suffix are stored in the performance data warehouse.

[0015] Preferably, in step S120, the keys included in the memory snapshot are inspected according to the capacity occupied by the preset keys, and the keys that are larger than the capacity occupied by the preset keys are stored in the performance data warehouse.

[0016] Preferably, in step S120, the prefix and suffix of the preset key and the key that is larger than the capacity occupied by the preset key are stored in the performance data warehouse.

[0017] Preferably, in step S130, the performance data warehouse is updated in real time.

[0018] Preferably, the capacity occupied by the preset key is the number of bytes of the preset key, and step S130 further includes step S140, in which the key is split into multiple sub-keys according to the number of bytes of the preset key.

[0019] An embodiment of the present invention further provides a remote dictionary-based key detection system, which is used to implement the above-mentioned remote dictionary-based key detection method. The remote dictionary-based key detection system includes:

[0020] A memory snapshot module is used to obtain a memory snapshot of a Redis server in a Redis cluster whose retrieval memory occupancy is greater than a first preset threshold and whose load is lower than a second preset threshold;

[0021] The performance data warehouse module inspects the memory snapshot according to preset attributes and stores the inspection results in the performance data warehouse;

[0022] The monitoring and alarm module tracks all the keys in the performance data warehouse, filters the keys whose thresholds are greater than the capacity occupied by the preset keys, and generates an alarm email.

[0023] Preferably, it also includes a key update module, which splits the key into multiple sub-keys according to the number of bytes of the preset key.

[0024] An embodiment of the present invention further provides a key detection device based on a remote dictionary, comprising:

[0025] processor;

[0026] a memory storing executable instructions of the processor;

[0027] The processor is configured to perform the steps of the above-mentioned remote dictionary-based key detection method by executing the executable instructions.

[0028] An embodiment of the present invention further provides a computer-readable storage medium for storing a program, which implements the steps of the above-mentioned key detection method based on a remote dictionary when executed.

[0029] The purpose of the present invention is to provide a key detection method, system, device and storage medium based on a remote dictionary, which can automatically match Redis cluster shards through a logical algorithm, perform priority checks in chronological order, store data in a data warehouse to ensure data security, push emails to relevant owners or users at regular intervals, understand the bigkey status of Redis in real time, and make corresponding adjustments, thereby improving the stability of Redis and improving work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Other features, objectives and advantages of the present invention will become more apparent from a reading of the detailed description of non-limiting embodiments made with reference to the following accompanying drawings.

[0031] Figure 1 It is a flow chart of the key detection method based on remote dictionary of the present invention.

[0032] Figure 2 It is a schematic diagram of the implementation process of the key detection method based on the remote dictionary of the present invention.

[0033] Figure 3 It is a module schematic diagram of the key detection system based on the remote dictionary of the present invention.

[0034] Figure 4 It is a structural schematic diagram of a key detection device based on a remote dictionary of the present invention.

[0035] Figure 5 It is a schematic diagram of the structure of a computer-readable storage medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention will be comprehensive and complete and fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their repeated description will be omitted.

[0037] Figure 1 Flowchart of the key detection method based on remote dictionary of the present invention. Figure 1 As shown, an embodiment of the present invention provides a key detection method based on a remote dictionary, comprising the following steps:

[0038] S110, obtaining a memory snapshot of a Redis server in a Redis cluster whose retrieval memory occupancy is greater than a first preset threshold and whose load is lower than a second preset threshold;

[0039] S120, inspect the memory snapshot according to preset attributes, and store the inspection result in the performance data warehouse;

[0040] S130. Track all keys in the performance data warehouse, filter out keys whose occupied capacity is greater than a preset key occupied capacity threshold, and generate an alarm email. The preset key occupied capacity is the number of bytes of the preset key.

[0041] S140, split the key into multiple sub-keys according to the number of bytes of the preset key.

[0042] The purpose of the present invention is to propose a bigkey inspection method that has no impact on online business and to capture the bigkey of Redis in full at one time. The present invention formulates an inspection process for Redis clusters that require bigkey inspection. For Redis instances with high memory occupancy and low load in the cluster, a memory snapshot is pulled. Pulling a snapshot will derive child process operations, reducing the impact on server performance. While controlling the server load, the memory snapshot is retrieved and analyzed, and the key prefix and suffix and key size can be specified for on-demand inspection, and the inspection results are stored in the performance data warehouse. According to the specified threshold, an alarm email is sent regularly to promote the rectification of bigkey.

[0043] In a preferred embodiment, in step S120, the keys included in the memory snapshot are inspected by using a preset key prefix, and the keys satisfying the preset key prefix are stored in the performance data warehouse, but not limited thereto.

[0044] In a preferred embodiment, in step S120, the keys included in the memory snapshot are inspected by using a preset key suffix, and the keys satisfying the preset key suffix are stored in the performance data warehouse, but not limited thereto.

[0045] In a preferred embodiment, in step S120, the keys included in the memory snapshot are inspected by the capacity occupied by the preset keys, and the keys larger than the capacity occupied by the preset keys are stored in the performance data warehouse, but not limited thereto.

[0046] In a preferred embodiment, in step S120, the prefix and suffix of the preset key and the key that is larger than the capacity occupied by the preset key are stored in the performance data warehouse, but not limited to this.

[0047] In a preferred embodiment, in step S130, the performance data warehouse is updated in real time, but the present invention is not limited thereto.

[0048] The remote dictionary-based key detection method of the present invention can automatically match Redis cluster shards through a logical algorithm, can perform priority checks in chronological order, store data in a data warehouse to ensure data security, regularly push emails to relevant owners or users, understand the bigkey status of Redis in real time, and make corresponding adjustments, thereby improving the stability of Redis and improving work efficiency.

[0049] See also Figure 1 and 2 The implementation process of the present invention is as follows: obtaining the full amount of production data from the production cluster and updating the new cluster time information. Obtaining the corresponding instance information according to the production cluster.

[0050] Get the instance RDB information and obtain the big key through logical algorithm.

[0051] The resulting memory profile is stored in the data warehouse. Alarms are sent through the alarm platform using the specified alarm method.

[0052] The alarm conditions include:

[0053] 1. Select the largest group in each cluster for analysis

[0054] 2. The value corresponding to the key is larger than 50M

[0055] Take a key of a defendant as an example.

[0056] Redis large keys have many hidden dangers (program timeout / network congestion / QPS drop / Redis switch, etc.). In order to eliminate hidden dangers and reduce accidents, the following is the inspection result of cluster large keys. Please split and optimize as soon as possible. Cluster name: demo_Redis

[0057] db key type size encoding Number of elements Expiration time Recent Visits 0 SUB2021040505 hash 52.94M hashtable 382764 2021-04-04 11 days ago

[0058] As shown in the table above, the size of key [SUB2021040505] (52.94M) exceeds the preset threshold of 50M, which is likely to block Redis. Therefore, key [SUB2021040505] is sent an alarm through the alarm platform. Subsequently, key [SUB2021040505] is split to reduce its occupied capacity and ensure the security and smooth operation of the server.

[0059] Figure 3 Schematic diagram of the module of the key detection system based on the remote dictionary of the present invention. Figure 3 As shown, the remote dictionary-based key detection system 5 of the present invention includes:

[0060] The memory snapshot module obtains a memory snapshot of a Redis server in a Redis cluster whose retrieval memory occupancy is greater than a first preset threshold and whose load is lower than a second preset threshold.

[0061] The performance data warehouse module inspects memory snapshots according to preset attributes and stores the inspection results in the performance data warehouse.

[0062] The monitoring and alarm module tracks all keys in the performance data warehouse, filters out keys whose capacity is greater than the preset key capacity threshold, and generates alarm emails.

[0063] The key update module splits the key into multiple sub-keys according to the number of bytes of the preset key.

[0064] The remote dictionary-based key detection system of the present invention can automatically match Redis cluster shards through logical algorithms, perform priority checks in chronological order, store data in a data warehouse to ensure data security, regularly push emails to relevant owners or users, understand the bigkey status of Redis in real time, and make corresponding adjustments, thereby improving the stability of Redis and improving work efficiency.

[0065] The embodiment of the present invention further provides a remote dictionary-based key detection device, comprising a processor and a memory, wherein executable instructions of the processor are stored, wherein the processor is configured to execute the steps of the remote dictionary-based key detection method by executing the executable instructions.

[0066] As shown above, the remote dictionary-based key detection system of this embodiment of the present invention can automatically match Redis cluster shards through logical algorithms, can perform priority checks in chronological order, store data in a data warehouse to ensure data security, regularly push emails to relevant owners or users, understand the bigkey status of Redis in real time, and make corresponding adjustments, thereby improving the stability of Redis and improving work efficiency.

[0067] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "platform".

[0068] Figure 4 Schematic diagram of the structure of the key detection device based on the remote dictionary of the present invention. Figure 4 The electronic device 600 according to this embodiment of the present invention is described. Figure 4The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0069] like Figure 4 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0070] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the electronic prescription circulation processing method section of this specification. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0071] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0072] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0073] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0074] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0075] The embodiment of the present invention also provides a computer-readable storage medium for storing a program, and the steps of the key detection method based on a remote dictionary are implemented when the program is executed. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned electronic prescription circulation processing method section of this specification.

[0076] As shown above, the remote dictionary-based key detection system of this embodiment of the present invention can automatically match Redis cluster shards through logical algorithms, can perform priority checks in chronological order, store data in a data warehouse to ensure data security, regularly push emails to relevant owners or users, understand the bigkey status of Redis in real time, and make corresponding adjustments, thereby improving the stability of Redis and improving work efficiency.

[0077] Figure 5 Schematic diagram of the structure of the computer-readable storage medium of the present invention. Figure 5 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0078] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0079] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0080] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0081] In summary, the purpose of the present invention is to provide a key detection method, system, device and storage medium based on a remote dictionary, which can automatically match Redis cluster shards through a logical algorithm, perform priority checks in chronological order, store data in a data warehouse to ensure data security, push emails to relevant owners or users at regular intervals, understand the bigkey status of Redis in real time, and make corresponding adjustments, thereby improving the stability of Redis and improving work efficiency.

[0082] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A key detection method based on a remote dictionary, characterized in that: The following steps are involved: S110, obtaining a memory snapshot of a Redis server in a Redis cluster whose retrieval memory occupancy is greater than a first preset threshold and whose load is lower than a second preset threshold; S120, deriving a child process operation, inspecting the memory snapshot according to preset attributes, storing the inspection result in the performance data warehouse, inspecting the key contained in the memory snapshot according to the preset key prefix and the preset key suffix, and storing the key that satisfies the preset key prefix or the preset key suffix in the performance data warehouse; S130, track all keys in the performance data warehouse, filter the keys whose thresholds are greater than the capacity occupied by the preset keys, generate an alarm email, and update the performance data warehouse in real time. The capacity occupied by the preset keys is the number of bytes of the preset keys; S140. Split the key into multiple sub-keys according to the number of bytes of the preset key.

2. A remote dictionary-based key detection system, used to implement the remote dictionary-based key detection method according to claim 1, characterized in that: include: A memory snapshot module is used to obtain a memory snapshot of a Redis server in a Redis cluster whose retrieval memory occupancy is greater than a first preset threshold and whose load is lower than a second preset threshold; The performance data warehouse module derives a subprocess operation, inspects the memory snapshot according to preset attributes, stores the inspection results in the performance data warehouse, inspects the keys contained in the memory snapshot by the preset key prefix and the preset key suffix, and stores the keys that meet the preset key prefix or the preset key suffix in the performance data warehouse; A monitoring and alarm module tracks all keys in the performance data warehouse, filters the keys whose thresholds are greater than the capacity occupied by a preset key, and generates an alarm email, where the capacity occupied by the preset key is the number of bytes of the preset key; The key update module splits the key into multiple sub-keys according to the number of bytes of the preset key.

3. A key detection device based on a remote dictionary, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to perform the steps of the remote dictionary-based key detection method of claim 1 by executing the executable instructions.

4. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the remote dictionary-based key detection method according to claim 1 are implemented.

Images