A hot data real-time detection system and method under high concurrency

Abstract

The present application relates to hotspot data real-time detection method technical field, especially relates to a kind of high concurrency under hotspot data real-time detection system and method.The present application relates to hotspot data real-time detection method technical field, especially relates to a kind of high concurrency under hotspot data real-time detection system and method.It includes zookeeper cluster, client end sdk package, worker end cluster and dashboard console;Zookeeper cluster is used to store rule configuration, the ip address of each worker and the hot key detected, and the hot key manually added;Client end sdk package is used to judge whether a key is hot key in a convenient way, and completes key reporting, listens to the change of rule in zookeeper, the change of worker information, the change of hot key, and carries out local caffeine caching to hot key.

Description

Technical Field

[0001] This invention relates to the technical field of real-time hotspot data detection methods, and in particular to a real-time hotspot data detection system and method under high concurrency. Background Technology

[0002] In systems with a large number of concurrent users, hot keys have always been an unavoidable problem. This could be due to a sudden surge in popularity of certain products, a massive influx of users into a store, a sudden surge of requests during a flash sale, or a sudden influx of web crawlers during a coupon giveaway. These sudden, unpredictable hot keys pose significant potential risks to the system.

[0003] The impact of hot keys on the data layer is obvious. For example, if data is stored in Redis or MySQL, taking Redis as an example, the unknown hot data will be stored on a specific Redis shard according to the hash rules, and data is usually retrieved from that shard. Since Redis has decent performance, and in cluster mode, let's assume it can support 50,000 reads per second, which is sufficient for most daily use. However, when the platform suddenly receives hundreds of thousands or even millions of requests per second, although most of the traffic will disappear within a few seconds, this short period of hot key usage can instantly paralyze the Redis shard cluster it resides in. The reason is simple: Redis is a single-threaded structure, and all incoming requests are queued. When the request volume far exceeds its processing capacity, subsequent requests will be stuck waiting or timeout. Because the Redis shard is completely filled with requests for this key, all other data operations on that shard cannot continue to provide services. In other words, the hot key not only affects itself but also the data it shares. Obviously, within this extremely short time window, we cannot quickly scale up Redis by more than 10 times to support this hotspot. Although Redis is already excellent, it often becomes the biggest bottleneck in this kind of scenario. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a real-time hot data detection system and method under high concurrency. It can accurately detect any sudden and unpredictable hot requests, including but not limited to hot data (such as a sudden large number of requests for the same product), hot users (such as web crawlers and spammers), and hot interfaces (a sudden massive number of requests for the same interface). Then, it pushes these hot data and hot users to the JVM memory of all machines deployed with the application to greatly reduce the impact on the backend data storage layer. The client can decide how to use these hot keys, and these hot keys maintain consistency throughout the entire application cluster.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is: a real-time hot data detection system under high concurrency, including a zookeeper cluster, a client-side SDK package, a worker-side cluster, and a dashboard console;

[0006] The ZooKeeper cluster is used to store rule configurations, the IP addresses of each worker, as well as detected hot keys and manually added hot keys;

[0007] The client SDK package connects to the ZooKeeper cluster and is used to conveniently determine whether a key is a hot key. It also performs key reporting, listens for rule changes, worker information changes, and hot key changes in ZooKeeper, and caches hot keys locally in Caffeine.

[0008] The worker cluster is connected to the ZooKeeper cluster and is used to periodically report its own IP information so that the client can obtain the address and establish a long connection. Then, the test keys sent by each client are accumulated and calculated. When the rule threshold set in ZooKeeper is reached, the hot key is pushed to each client.

[0009] The dashboard console is connected to the ZooKeeper cluster and is used to set key rules for each application. It also allows manual addition and deletion of hot keys for each client to listen to.

[0010] Preferably, the ZooKeeper cluster is a high-performance configuration center that can provide efficient listening and subscription services with minimal resource consumption.

[0011] Preferably, the client-side SDK package is a reference SDK added to the service.

[0012] Preferably, the dashboard console is a Java program with a visual interface.

[0013] A method for real-time detection of hotspot data under high concurrency, characterized by the following steps:

[0014] Step 1: Set up a ZooKeeper cluster

[0015] ZooKeeper, as a globally shared configuration center, will allow all clients to read completely consistent worker and rule information;

[0016] Step 2: Launch the dashboard console visual interface

[0017] Add the test rules for each app to the interface;

[0018] Step 3: Start the worker cluster

[0019] The worker cluster can be configured with application-level isolation, or it can be unisolated. When isolation is configured, the application can only use these few workers to avoid competition for performance resources from other applications. After the worker starts, it will read the previously configured rules from ZooKeeper and continuously listen for changes in the rules.

[0020] Then, the worker will periodically report its IP information to ZooKeeper. If it does not report for a period of time, ZooKeeper will delete the worker information. The IP reported by the worker is used by the client for long-term connection. Each client establishes a long-term connection based on the worker information that the app can use in ZooKeeper. The key to be tested will be hashed and evenly distributed to each worker according to the number of workers.

[0021] After that, the worker starts receiving and calculating the keys sent by each client. When a key reaches the threshold set in the rules, it is pushed to all client SDKs of the APP, and then pushed to ZooKeeper for the dashboard to listen and record.

[0022] Step 4: Client-side SDK package

[0023] After the client starts, it connects to ZooKeeper to obtain the rules and the IP information of the dedicated worker, and then continuously listens for this information; after obtaining the IP information, it establishes a long connection with the worker through Netty.

[0024] The client will start a scheduled task to send the key to be tested to the corresponding worker machine in batches every 500ms. The sending rule is to take the remainder of the key's hashcode with the number of workers, so a fixed key will definitely be sent to the same worker. During these 500ms, the client will collect and accumulate the key to be tested and its quantity locally, and then send it out in batches when the time expires.

[0025] Once the worker detects a hot key, it will be pushed over. The framework uses Caffeine for local caching and will set the local expiration time according to the expiration time set in the rule. Each hot key maintains consistency throughout the entire client cluster.

[0026] Preferably, in step four, a key that has already been "hot" will not be sent again unless the local cache for that key has expired.

[0027] Preferably, in step four, if a hot key is manually added or deleted in the console, the client will also listen for it and add or delete it in the local Caffeine.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] 1. This invention can accurately detect any sudden, unpredictable hotspot requests at the millisecond level.

[0030] 2. In this invention, all hot keys can maintain consistency throughout the entire application cluster.

[0031] 3. The computing cluster of this invention supports distributed cluster deployment.

[0032] 4. The hot key rules of this invention support online real-time modification and real-time effect. Attached Figure Description

[0033] Figure 1 This is a framework diagram of the present invention. Detailed Implementation

[0034] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that these descriptions are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0035] Reference Figure 1 The present invention provides a real-time hotspot data detection system under high concurrency, comprising a ZooKeeper cluster, a client SDK package, a worker cluster, and a dashboard console.

[0036] The ZooKeeper cluster is used to store rule configurations, the IP addresses of each worker, as well as detected hot keys and manually added hot keys;

[0037] The client SDK package connects to the ZooKeeper cluster and is used to conveniently determine whether a key is a hot key. It also performs key reporting, listens for rule changes, worker information changes, and hot key changes in ZooKeeper, and caches hot keys locally in Caffeine.

[0038] The worker cluster is connected to the ZooKeeper cluster and is used to periodically report its own IP information so that the client can obtain the address and establish a long connection. Then, the test keys sent by each client are accumulated and calculated. When the rule threshold set in ZooKeeper is reached, the hot key is pushed to each client.

[0039] The dashboard console is connected to the ZooKeeper cluster and is used to set key rules for each application. It also allows manual addition and deletion of hot keys for each client to listen to.

[0040] The present invention provides a method for real-time detection of hotspot data under high concurrency, comprising the following steps:

[0041] Step 1

[0042] First, set up a ZooKeeper cluster.

[0043] ZooKeeper, as a globally shared configuration center, ensures that all clients can read completely consistent worker and rule information.

[0044] Step Two

[0045] Launch the dashboard visual interface

[0046] Add test rules for each app to the interface. For example, dcb-mall-shopping-cart contains a rule that uses keys starting with "commodity-detail-", such as "commodity-detail-123". A key that appears 10 times every 30 seconds is considered a "hot key". Only keys that match the rules will be sent to the worker for calculation.

[0047] Step 3

[0048] Start the worker cluster

[0049] Worker clusters can be configured with application-level isolation, or not. With isolation enabled, the application can only use these specific workers to avoid competition for performance resources from other applications. After a worker starts, it reads the previously configured rules from ZooKeeper and continuously monitors for rule changes.

[0050] Then, the worker periodically reports its IP information to ZooKeeper. If it fails to report for a period of time, ZooKeeper will delete the worker's information. The IP reported by the worker is used by the client for long-term connections. Each client establishes a long-term connection based on the worker information available to the application in ZooKeeper, and the key to be tested is hashed and evenly distributed among the workers according to the number of workers.

[0051] After that, the worker starts receiving and calculating the keys sent by each client. When a key reaches the threshold set in the rules, it is pushed to all client SDKs of the APP, and then a copy is pushed to ZooKeeper for the dashboard to listen for and record.

[0052] Step Four

[0053] client side

[0054] After the client starts, it connects to ZooKeeper to obtain rules and unique worker IP information, and then continuously listens for this information. After obtaining the IP information, it establishes a long-lived connection with the worker through Netty.

[0055] The client starts a scheduled task that sends a batch of test keys to the corresponding worker machines every 500ms (configurable). The sending rule is the modulo of the key's hashcode divided by the number of workers, so a fixed number of keys will definitely be sent to the same worker. During these 500ms, the client collects and accumulates the test keys and their quantities locally, and then sends them out in batches when the deadline arrives. Note that keys that have already been "hot" (i.e., are no longer actively used) will not be sent again unless the local cache for that key has expired.

[0056] Once a worker detects a hot key, it pushes it over. The framework uses Caffeine for local caching, and the local expiration time is set according to the expiration time configured in the original rule. Of course, if a hot key is manually added or deleted in the console, the client will also listen for it and add or delete it in the local Caffeine accordingly. In this way, all hot keys remain consistent across the entire client cluster.

[0057] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A real-time hotspot data detection system under high concurrency, characterized in that: This includes a ZooKeeper cluster, client SDK package, worker cluster, and dashboard console; The ZooKeeper cluster is used to store rule configurations, the IP addresses of each worker, as well as detected hot keys and manually added hot keys; The client SDK package connects to the ZooKeeper cluster and is used to conveniently determine whether a key is a hot key. It also performs key reporting, listens for rule changes, worker information changes, and hot key changes in ZooKeeper, and caches hot keys locally in Caffeine. The worker cluster is connected to the ZooKeeper cluster and is used to periodically report its own IP information so that the client can obtain the address and establish a long connection. Then, the test keys sent by each client are accumulated and calculated. When the rule threshold set in ZooKeeper is reached, the hot key is pushed to each client. The dashboard console is connected to the ZooKeeper cluster and is used to set key rules for each application. It also allows manual addition and deletion of hot keys for each client to listen to.

2. The real-time hotspot data detection system under high concurrency as described in claim 1, characterized in that: The ZooKeeper cluster is a high-performance configuration center that can provide efficient listening and subscription services with minimal resource consumption.

3. The real-time hotspot data detection system under high concurrency as described in claim 1, characterized in that, Its features are: The client-side SDK package is a reference SDK added to the service.

4. The real-time hotspot data detection system under high concurrency as described in claim 1, characterized in that, Its features are: The dashboard console is a Java program with a visual interface.

5. A method for real-time detection of hotspot data under high concurrency, characterized in that, Includes the following steps: Step 1: Set up a ZooKeeper cluster ZooKeeper, as a globally shared configuration center, will allow all clients to read completely consistent worker and rule information; Step 2: Launch the dashboard console visual interface Add the test rules for each app to the interface; Step 3: Start the worker cluster The worker cluster can be configured with application-level isolation, or it can be unisolated. When isolation is configured, the application can only use these few workers to avoid competition for performance resources from other applications. After the worker starts, it will read the previously configured rules from ZooKeeper and continuously listen for changes in the rules. Then, the worker will periodically report its IP information to ZooKeeper. If it does not report for a period of time, ZooKeeper will delete the worker information. The IP reported by the worker is used by the client for long-term connection. Each client establishes a long-term connection based on the worker information that the app can use in ZooKeeper. The key to be tested will be hashed and evenly distributed to each worker according to the number of workers. After that, the worker starts receiving and calculating the keys sent by each client. When a key reaches the threshold set in the rules, it is pushed to all client SDKs of the APP, and then pushed to ZooKeeper for the dashboard to listen and record. Step 4: After the client starts, it will connect to ZooKeeper to obtain the rules and the IP information of the dedicated worker, and then continuously listen for this information. After obtaining the IP information, a long connection will be established with the worker via Netty; The client will start a scheduled task to send the key to be tested to the corresponding worker machine in batches every 500ms. The sending rule is to take the remainder of the key's hashcode with the number of workers, so a fixed key will definitely be sent to the same worker. During these 500ms, the client will collect and accumulate the key to be tested and its quantity locally, and then send it out in batches when the time expires. Once the worker detects a hot key, it will be pushed over. The framework uses Caffeine for local caching and will set the local expiration time according to the expiration time set in the rule. Each hot key maintains consistency throughout the entire client cluster.

6. The method for real-time detection of hotspot data under high concurrency according to claim 5, characterized in that: In step four, a key that has already been "hot" will not be sent again unless the local cache for that key has expired.

7. The method for real-time detection of hotspot data under high concurrency as described in claim 5, characterized in that: In step four, if a hot key is manually added or deleted in the console, the client will also listen for it and add or delete it in the local Caffeine.

Images