A method, device and system for managing hotspot data

By monitoring and migrating hotspot data in the Redis cluster, the overload problem caused by hotspot keys is solved, load balancing and stable performance of the main cluster are achieved, and the reliability of the system is ensured.

CN110569233BActive Publication Date: 2024-11-22HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201910878310.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-06-30
Filing Date
2019-09-17
Publication Date
2024-11-22
Estimated Expiration
2039-09-17

AI Technical Summary

Technical Problem

When facing frequent access requests from a large number of single data, Redis clusters may lead to hot-spot key phenomena, resulting in excessive load on the server, which may cause crashes, and then break down the cache mechanism and database, causing business crashes.

Method used

By monitoring the process to identify hotspot data and migrate it to the secondary cluster, it can reduce the business pressure of the primary cluster, achieve load balancing, and ensure the performance and reliability of the primary cluster.

Benefits of technology

Effectively distribute the load of hotspot data, avoid crashes on a single server, ensure the stable operation of the cache mechanism and database, and improve the reliability and performance of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110569233B_ABST
    Figure CN110569233B_ABST
Patent Text Reader

Abstract

The present application provides a hotspot data management method, device and system. The method includes: the first cluster and the second cluster include at least one single-threaded main control unit, the main control unit processes one access request for data at a time, and the data accessed by the access request is jointly managed by the first cluster and the second cluster; the database server determines the hotspot data managed by the first cluster and determines whether the migration conditions for migrating the hotspot data to the second cluster are met, and the second cluster is used to manage the hotspot data that meets the migration conditions; the database server migrates the hotspot data that meets the migration conditions to the second cluster and triggers the update of the access path to the hotspot data. The above method can reduce the business pressure of the first cluster when hotspot data appears, and ensure the performance and reliability of the first cluster.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a hotspot data management method, device and system. Background Art

[0002] In order to ensure that the database can provide services stably and efficiently, a cache mechanism is usually set up between the database (DB) and the client. The use of the cache mechanism can not only relieve the pressure on the database server, but also improve the data reading and writing speed. In-memory databases are usually used as a cache layer between clients and databases. All data access controls of in-memory databases are performed in memory. Since the memory itself has a fast reading and writing speed, the in-memory database has better performance.

[0003] Redis is an in-memory database that can be used as a cache layer for databases and clients. Redis is a key-value lock-free database written in C language, compliant with the Berkeley Software Distribution (BSD) protocol, supporting networks, and can be based on memory or persistence. Redis cluster is a distributed solution for Redis, which effectively solves the distributed requirements of Redis. The cluster architecture can achieve the purpose of load balancing.

[0004] In actual usage scenarios, Redis Cluster may receive a large number of frequent access requests for a single data. Clients may request a large number of data corresponding to the same keyword (key) (which may correspond to a promotional product, hot news, hot comments, etc. of an application). According to the hash value of the key, all access requests will be concentrated on the same physical device, and the load of the physical device will be seriously aggravated, causing the main control unit deployed on the physical device and the physical device itself to face the risk of crashing and paralysis. This phenomenon is called the "hot key" problem.

[0005] The hot key phenomenon will cause a large amount of traffic to concentrate, reaching the upper limit of the server. At the same time, too many requests are concentrated on one server, which will cause the server to crash, and then cause the entire cache mechanism to be penetrated by the hot key. At this time, business requests directly access the backend database, further penetrating the DB and causing a series of business crashes. Summary of the invention

[0006] The embodiment of the present invention discloses a hotspot data management method, device and system, which can migrate the hotspot data to the second cluster when the hotspot data appears in the first cluster, thereby reducing the business pressure of the first cluster and ensuring the performance and reliability of the first cluster.

[0007] In a first aspect, the present application provides a hotspot data management method, which is executed by a monitoring process in a database server, comprising: a monitoring module in the monitoring process determines the hotspot data managed by a first cluster, the first cluster includes at least one single-threaded main control unit, the single-threaded main control unit processes one access request at a time, and the access request includes an access request for data; the monitoring module determines whether a migration condition for migrating the hotspot data to a second cluster is met, the second cluster includes at least one single-threaded main control unit, the data accessed by the access request is jointly managed by the first cluster and the second cluster, and the second cluster is used to manage the hotspot data that meets the migration condition; when the migration condition is met, the migration module in the monitoring process migrates the hotspot data to the second cluster and triggers an update of the access path to the hotspot data.

[0008] In an embodiment of the present application, the first cluster may be a redis cluster, the main control unit of the first cluster may be deployed on multiple physical devices, the second cluster is a sub-cluster of the first cluster, and the main control unit of the second cluster may be deployed on the same physical device as the main control unit of the first cluster. In the case where there is hotspot data in the first cluster, by migrating the hotspot data to the second cluster, the business pressure of the first cluster can be reduced, load balancing between the first cluster and the second cluster can be achieved, and the high performance and reliability of the first cluster can be ensured. In combination with the first aspect, in a possible implementation of the first aspect, the first main control unit in the first cluster and the first main control unit in the second cluster are deployed on different physical devices.

[0009] In combination with the first aspect, in a possible implementation of the first aspect, the main control unit manages at least one hash slot, the identifier of the hash slot is an integer obtained by mapping the data through a preset algorithm, and the hash slot includes at least one data; the hash slot managed by the first main control unit in the first cluster and the hash slot managed by the first main control unit in the second cluster have the same identifier.

[0010] Optionally, the preset algorithm includes a hash algorithm.

[0011] In an embodiment of the present application, by deploying the main control units that manage hash slots with the same identifier in two clusters on different physical devices, the business pressure of a single physical device can be dispersed, ensuring that the cluster can provide services to the outside world normally.

[0012] In combination with the first aspect, in a possible implementation of the first aspect, the monitoring module counts the number of accesses to the data managed by the first cluster within a preset time, and determines data whose accesses exceed a first threshold as hot data.

[0013] In combination with the first aspect, in a possible implementation of the first aspect, the migration module reads the hot data and the keywords corresponding to the hot data from the first cluster; determines a first hash slot in the second cluster based on the keywords corresponding to the hot data; and migrates the hot data and the keywords corresponding to the hot data to the first hash slot.

[0014] In the solution provided in the present application, the migration module completes the migration of hot data by reading the hot data and the keyword corresponding to the hot data in the first cluster, and then writing the keyword into the first hash slot after determining the first hash slot in the second cluster.

[0015] In combination with the first aspect, in a possible implementation of the first aspect, the migration module sends migration information to a client communicating with the first cluster and the second cluster, the migration information carries keywords corresponding to the hotspot data, and the migration information is used to instruct the client to record the keywords corresponding to the hotspot data into a keyword set, and the keyword set is used by the client to determine the target cluster corresponding to the access request, and the target cluster includes the first cluster and the second cluster.

[0016] In the solution provided in the present application, the client receives migration information carrying keywords corresponding to hot spot data sent by the migration module, and records the keywords corresponding to the hot spot data into a keyword set. When there is an access request for the hot spot data, the client can accurately find the target cluster that manages the hot spot data based on the keyword set.

[0017] In combination with the first aspect, in a possible implementation of the first aspect, before the migration module sends migration information to the client communicating with the first cluster and the second cluster, the migration module reads the hotspot data managed by the first cluster after the migration is completed, and verifies whether the hotspot data is consistent with the hotspot data managed by the first cluster before the migration; when the hotspot data managed by the first cluster after the migration is completed is consistent with the hotspot data managed by the first cluster before the migration, the migration information is sent to the client communicating with the first cluster and the second cluster.

[0018] In the solution provided by the present application, after completing the migration of hotspot data, the migration module performs data consistency judgment, that is, judging whether the hotspot data managed by the first cluster before and after the migration are consistent, so as to determine whether the hotspot data has changed during the migration process. And if the data is consistent, the migration information including the keywords corresponding to the hotspot data is sent to the client, so that the client can accurately access the hotspot data.

[0019] In combination with the first aspect, in a possible implementation of the first aspect, the migration module sends migration information to a client communicating with the first cluster and the second cluster, the migration information carries the hotspot data and keywords corresponding to the hotspot data, and the migration information is used to instruct the client to record the hotspot data and the keywords corresponding to the hotspot data to a first set, and the first set is used by the client to perform data consistency judgment.

[0020] In the solution provided in the present application, the client maintains a first set, which records the hotspot data and the keywords corresponding to the hotspot data. The client can determine whether the hotspot data has changed during the migration process based on the first set, thereby ensuring data consistency.

[0021] In combination with the first aspect, in a possible implementation of the first aspect, the monitoring module calculates the resource usage of each main control unit in the first cluster, and monitors the main control units whose resource usage exceeds a first threshold; counts the number of accesses to each data managed by the main control unit whose resource usage exceeds the first threshold within a preset time, and determines the data whose access times exceed a second threshold as hot data.

[0022] In the solution provided in the present application, the monitoring module calculates the resource usage of each main control unit, monitors the main control units whose resource usage exceeds a first threshold, and further counts each data access managed by the main control whose resource usage exceeds the first threshold, thereby determining the hotspot data. In this way, it is possible to avoid monitoring all the main control units to affect the performance of the first cluster, thereby improving the monitoring efficiency.

[0023] In combination with the first aspect, in a possible implementation of the first aspect, the monitoring module counts the total number of accesses to the data managed by the main control unit whose resource usage exceeds the first threshold, and when the total number of accesses exceeds a third threshold, it is determined that the hotspot data managed by the main control unit whose resource usage exceeds the first threshold meets the migration condition.

[0024] In the solution provided by the present application, after determining the hotspot data, the monitoring module needs to further determine whether the migration conditions of the hotspot data are met, that is, the monitoring module counts the total number of accesses to all data managed by the main control unit that manages the hotspot data, and determines that the migration conditions are met when the total number of accesses exceeds a preset threshold. This can avoid frequent migration of hotspot data and ensure the performance of the first cluster.

[0025] In combination with the first aspect, in a possible implementation of the first aspect, the monitoring module monitors the second cluster, and the second cluster manages the hotspot data migrated from the first cluster to the second cluster; the number of accesses to the hotspot data within a preset time is counted, and when the number of accesses to the hotspot data is less than a fourth threshold, the hotspot data is marked as non-hotspot data, the migration module migrates the non-hotspot data to the first cluster, and triggers an update of the access path to the non-hotspot data.

[0026] In the solution provided in the present application, the monitoring module determines the non-hotspot data by monitoring the second cluster and counting the number of accesses to the data managed by the second cluster, and after determination, migrates the non-hotspot data to the first cluster through the migration module. This can achieve load balancing between the first cluster and the second cluster, and avoid constantly migrating hotspot data to the second cluster, which may cause the second cluster to crash due to excessive business pressure.

[0027] In combination with the first aspect, in a possible implementation of the first aspect, the first cluster or the second cluster includes at least one main control unit, and the at least one main control unit manages the same number of hash slots; or manages different numbers of hash slots according to preset rules; wherein the total number of hash slots is 16384.

[0028] In combination with the first aspect, in a possible implementation manner of the first aspect, the resource usage includes CPU resource usage, memory resource usage, or network capability resource usage.

[0029] In combination with the first aspect, in a possible implementation of the first aspect, the main control unit includes at least one slave control unit, and the slave control unit is used to replace the main control unit and become a new main control unit when the main control unit fails.

[0030] In combination with the first aspect, in a possible implementation of the first aspect, each main control unit corresponds to a unique IP address and port number, and the monitoring process manages each main control unit according to the IP address and port number.

[0031] In combination with the first aspect, in a possible implementation of the first aspect, before migrating the hotspot data to the second cluster, the monitoring module classifies the hotspot data by priority and preferentially migrates the hotspot data with the highest priority to the second cluster.

[0032] In combination with the first aspect, in a possible implementation of the first aspect, the monitoring process communicates with the client through a transmission control protocol (TCP) connection or a dedicated channel connection, and the migration module sends migration information to the client through the TCP connection or the dedicated channel connection.

[0033] In combination with the first aspect, in a possible implementation manner of the first aspect, a reply confirmation mechanism is adopted between the monitoring process and the client to ensure that all clients communicating with the migration module successfully receive the migration information.

[0034] In a second aspect, the present application provides a method for accessing hot data, including: a client queries whether a keyword set includes the keyword based on a keyword in a business request, the keyword set includes the keyword corresponding to the hot data migrated from the first cluster to the second cluster, the first cluster and the second cluster include at least one single-threaded main control unit, the single-threaded main control unit processes one access request at a time, the access request includes an access request for data, and the data accessed by the access request is jointly managed by the first cluster and the second cluster; the client determines a target cluster based on the query result, the target cluster includes the first cluster or the second cluster; the client sends an access request to the target cluster, the access request includes the keyword in the business request.

[0035] In the solution provided in the present application, the client maintains a keyword set, which includes keywords corresponding to all hot data migrated from the first cluster to the second cluster. When the client generates a business request, it can select the correct cluster for access by determining whether the keyword in the business request exists in the keyword set.

[0036] In combination with the second aspect, in a possible implementation manner of the second aspect, the client receives migration information sent by the database server, the migration information includes keywords corresponding to the hot data, and records the keywords corresponding to the hot data into the keyword set.

[0037] In combination with the second aspect, in a possible implementation of the second aspect, the client receives first migration information sent by the database server, the first migration information includes the hotspot data and the keywords corresponding to the hotspot data, and records the hotspot data and the keywords corresponding to the hotspot data to a first set; receives second migration information sent by the database server, the second migration information includes non-hotspot data and the keywords corresponding to the non-hotspot data, and records the non-hotspot data and the keywords corresponding to the non-hotspot data to a second set; wherein the first set and the second set are respectively used to cache data for which data consistency judgment has not been performed.

[0038] In the solution provided in the present application, the client maintains a first set and a second set. The first set records hotspot data and keywords corresponding to the hotspot data. The client can judge whether the hotspot data has changed during the migration process based on the first set, thereby ensuring data consistency; the second set records non-hotspot data and keywords corresponding to the non-hotspot data. The client can judge whether the non-hotspot data has changed during the migration process based on the second set, thereby ensuring data consistency.

[0039] In combination with the second aspect, in a possible implementation of the second aspect, if the keyword set and the second set include the keyword in the business request, and the data corresponding to the keyword in the second set is consistent with the data corresponding to the keyword managed by the second cluster, the first cluster is determined to be the target cluster; when they are inconsistent, the second cluster is determined to be the target cluster; if the keyword set does not include the keyword in the business request, the first set includes the keyword in the business request, and the data corresponding to the keyword in the first set is consistent with the data corresponding to the keyword managed by the first cluster, the second cluster is determined to be the target cluster; when they are inconsistent, the first cluster is determined to be the target cluster.

[0040] In a third aspect, the present application provides a network device, comprising: a processing module, used to determine hotspot data managed by a first cluster, the first cluster comprising at least one single-threaded main control unit, the single-threaded main control unit processing one access request at a time, the access request comprising an access request for data; a judgment module, used to judge whether a migration condition for migrating the hotspot data to a second cluster is met, the second cluster comprising at least one single-threaded main control unit, the data accessed by the access request is jointly managed by the first cluster and the second cluster, and the second cluster is used to manage hotspot data that meets the migration condition; a migration module, used to migrate the hotspot data to the second cluster when the migration condition is met, and trigger an update of the access path to the hotspot data.

[0041] In combination with the third aspect, in a possible implementation manner of the third aspect, the first main control unit in the first cluster and the first main control unit in the second cluster are deployed on different physical devices.

[0042] In combination with the third aspect, in a possible implementation of the third aspect, the main control unit manages at least one hash slot, the identifier of the hash slot is an integer obtained by mapping the data through a preset algorithm, and the hash slot includes at least one data; the hash slot managed by the first main control unit in the first cluster and the hash slot managed by the first main control unit in the second cluster have the same identifier.

[0043] In combination with the third aspect, in a possible implementation of the third aspect, the processing module is also used to read the hot data and the keywords corresponding to the hot data from the first cluster, and determine the first hash slot in the second cluster based on the keywords corresponding to the hot data; the migration module is also used to migrate the hot data and the keywords corresponding to the hot data to the first hash slot.

[0044] In combination with the third aspect, in a possible implementation of the third aspect, the network device also includes a sending module, and the sending module is used to send migration information to a client communicating with the first cluster and the second cluster, and the migration information includes keywords corresponding to the hotspot data. The migration information is used to instruct the client to record the keywords corresponding to the hotspot data into a keyword set, and the keyword set is used by the client to determine the target cluster corresponding to the access request, and the target cluster includes the first cluster and the second cluster.

[0045] In combination with the third aspect, in a possible implementation of the third aspect, the processing module is also used to read the hotspot data managed by the first cluster after the migration is completed, and verify whether the hotspot data is consistent with the hotspot data managed by the first cluster before the migration; the sending module is also used to send the migration information to the client communicating with the first cluster and the second cluster when the hotspot data managed by the first cluster after the migration is completed is consistent with the hotspot data managed by the first cluster before the migration.

[0046] In combination with the third aspect, in a possible implementation of the third aspect, the network device also includes a sending module, and the sending module is used to send migration information to a client communicating with the first cluster and the second cluster, and the migration information carries the hotspot data and keywords corresponding to the hotspot data. The migration information is used to instruct the client to record the hotspot data and the keywords corresponding to the hotspot data to a first set, and the first set is used by the client to perform data consistency judgment.

[0047] In combination with the third aspect, in a possible implementation of the third aspect, the processing module is also used to calculate the resource usage of each main control unit in the first cluster, and monitor the main control units whose resource usage exceeds the first threshold; count the number of accesses to each data managed by the main control unit whose resource usage exceeds the first threshold within a preset time, and determine the data whose access times exceed the second threshold as hot data.

[0048] In combination with the third aspect, in a possible implementation of the third aspect, the judgment module is also used to count the total number of accesses to the data managed by the main control unit whose resource usage exceeds the first threshold. When the total number of accesses exceeds the third threshold, it is determined that the hotspot data managed by the main control unit whose resource usage exceeds the first threshold meets the migration condition.

[0049] In combination with the third aspect, in a possible implementation of the third aspect, the processing module is also used to monitor the second cluster, and the second cluster manages the hotspot data migrated from the first cluster to the second cluster; counts the number of accesses to the hotspot data within a preset time, and when the number of accesses to the hotspot data is less than a fourth threshold, marks the hotspot data as non-hotspot data; the migration module is also used to migrate the non-hotspot data to the first cluster and trigger an update of the access path to the non-hotspot data.

[0050] In a fourth aspect, the present application provides a client device, comprising: a processing module, used to query whether a keyword set includes the keyword based on the keyword in the business request, the keyword set includes the keyword corresponding to the hot data migrated from the first cluster to the second cluster, the first cluster and the second cluster include at least one single-threaded main control unit, the single-threaded main control unit processes one access request at a time, the access request includes an access request for data, and the data accessed by the access request is jointly managed by the first cluster and the second cluster; determine the target cluster based on the query result, the target cluster includes the first cluster or the second cluster; a sending module, used to send an access request to the target cluster, the access request includes the keyword in the business request.

[0051] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the client device also includes a receiving module, which is used to receive migration information sent by a database server, and the migration information includes keywords corresponding to the hotspot data; the processing module is also used to record the keywords corresponding to the hotspot data into the keyword set.

[0052] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the client device also includes a receiving module, the receiving module is used to receive first migration information sent by the database server, the first migration information includes the hotspot data and the keywords corresponding to the hotspot data; the processing module is also used to record the hotspot data and the keywords corresponding to the hotspot data to a first set; the receiving module is also used to receive second migration information sent by the database server, the second migration information includes non-hotspot data and the keywords corresponding to the non-hotspot data; the processing module is also used to record the non-hotspot data and the keywords corresponding to the non-hotspot data to a second set; wherein, the first set and the second set are respectively used to cache data for which data consistency judgment has not been performed.

[0053] In combination with the fourth aspect, in a possible implementation of the fourth aspect, the processing module is also used to determine that the first cluster is the target cluster when the keyword set and the second set include the keyword in the business request, and the data corresponding to the keyword in the second set is consistent with the data corresponding to the keyword managed by the second cluster, and when they are inconsistent, determine the second cluster as the target cluster; the processing module is also used to determine that the second cluster is the target cluster when the keyword set does not include the keyword in the business request, the first set includes the keyword in the business request, and the data corresponding to the keyword in the first set is consistent with the data corresponding to the keyword managed by the first cluster, and when they are inconsistent, determine the first cluster as the target cluster.

[0054] In a fifth aspect, the present application provides a network device, comprising a processor and a memory, wherein the memory is used to store program code, and the processor is used to call the program code in the memory to execute the above-mentioned first aspect and a method combined with any one of the implementation methods of the above-mentioned first aspect.

[0055] In a sixth aspect, the present application provides a client device, comprising a processor and a memory, wherein the memory is used to store program code, and the processor is used to call the program code in the memory to execute the above-mentioned second aspect and a method combined with any one of the implementation methods of the above-mentioned second aspect.

[0056] In the seventh aspect, the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the process of the method provided in the above-mentioned first aspect and any one of the implementation methods in combination with the above-mentioned first aspect.

[0057] In an eighth aspect, the present application provides a computer storage medium storing a computer program, which is implemented when the computer program is executed by a processor and can implement the process of the method provided in the above-mentioned second aspect and in combination with any one of the implementation methods in the above-mentioned second aspect.

[0058] In a ninth aspect, the present application provides a computer program, which includes instructions. When the computer program is executed by a computer, the computer can execute the process of the method provided in the above-mentioned first aspect and in combination with any one of the implementation methods in the above-mentioned first aspect.

[0059] In a tenth aspect, the present application provides a computer program, which includes instructions. When the computer program is executed by a computer, the computer can execute the process of the method provided in the above-mentioned second aspect and in combination with any one of the implementation methods of the above-mentioned second aspect.

[0060] In an eleventh aspect, the present application provides a hotspot data migration system, comprising a network device and a client device, wherein the network device and the client device can communicate with each other;

[0061] The network device is used to execute the method as described in any one of the first aspects;

[0062] The client device is used to execute the method as described in any one of the second aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0064] Figure 1 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0065] Figure 2A is a schematic diagram of the internal structure of a network node device provided in an embodiment of the present application;

[0066] Figure 2B is a schematic diagram of the internal structure of a cluster provided in an embodiment of the present application;

[0067] Figure 2C is a schematic diagram of the internal structure of a user equipment provided in an embodiment of the present application;

[0068] Figure 3 It is a flowchart of a hotspot data management method provided in an embodiment of the present application;

[0069] Figure 4 It is a schematic diagram of a cluster deployment provided in an embodiment of the present application;

[0070] Figure 5 It is a flowchart of another hotspot data management method provided in an embodiment of the present application;

[0071] Figure 6 It is a flowchart of a data consistency verification method provided by an embodiment of the present application;

[0072] Fig. 7A It is a structural diagram of a network device provided in an embodiment of the present application;

[0073] Figure 7B It is a structural diagram of another network device provided in an embodiment of the present application;

[0074] Fig. 8A It is a structural diagram of a client device provided in an embodiment of the present application;

[0075] Figure 8B is a structural diagram of another client device provided in an embodiment of the present application;

[0076] Fig. 9 It is a structural diagram of a network device provided in an embodiment of the present application;

[0077] Fig.10 It is a structural diagram of a client device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present application are clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0079] First, some of the terms and related technologies involved in this application are explained in conjunction with the accompanying drawings to facilitate understanding by those skilled in the art.

[0080] Redis cluster is a distributed solution for redis. Redis cluster uses virtual slot partitioning and uses a well-dispersed hash algorithm to map all data to hash slots with different identifiers. Each slot contains one or more data. The identifier of each slot is an integer. The value range of the integer is 0-16383, and the identifiers corresponding to different slots are different. Rediscluster contains a total of 16384 slots. Slots are the basic unit for data management and migration within the cluster. Each node in the cluster (which can be called a master control unit or process) manages a certain number of slots. Data is stored in the redis cluster in the form of key-value pairs. When a data needs to be stored in the redis cluster, the keyword (key) corresponding to the data will first be hashed to obtain a data result, and then the data result will be modulo 16384. The remainder is the identifier of a slot. According to the identifier, a unique slot can be determined, and then the specific value of the data and the corresponding key can be determined. Which master control unit manages it. Once a master control unit fails, the slot managed by the master control unit cannot be accessed, causing the entire redis cluster to fail to work properly. When deploying a redis cluster, all master control units are distributed on different physical devices to achieve load balancing.

[0081] Application programming interface (API) is a set of predefined functions, the purpose of which is to provide applications with the ability to access a set of routines based on certain software or hardware without having to access the source code or understand the details of the internal working mechanism. Routine is a collection of functional interfaces provided by the system to the outside, which is similar to a function. For example, services, standard functions, library functions, etc. are routines. The API calls provided to the outside make the content correlation between devices stronger. In order to provide a unified API interface to the outside, it is necessary to provide an open and unified API interface environment for the device that calls the API to help users access the functions and resources of the device. For example, in an embodiment of the present application, the client can schedule through the API interface provided by the server to achieve data access to the server, and the client itself does not need to understand its complex scheduling mechanism and hot data migration mechanism.

[0082] Figure 1A possible application scenario of an embodiment of the present application is shown. In this application scenario, device 110, device 120 and device 130 are network node devices, such as database servers, etc., which can deploy a main control unit belonging to the same cluster, such as deploying a main control unit belonging to cluster 140. Device 150, device 160 and device 170 are user devices, such as mobile phones, tablet computers, personal computers or servers, etc. Interface 1 is a logical interface provided by cluster 140 to the outside, and interface 1 can be an API interface. Device 150, device 160 and device 170 can access data of cluster 140 through interface 1, that is, device 150, device 160 and device 170 can access any one of device 110, device 120 and device 130 through interface 1. Device 110, device 120 and device 130 all include one or more master control units, each of which can be configured with one or more slave control units. All master control units in device 110, device 120 and device 130 provide services to the outside as a whole (i.e., in the form of cluster 140), and the cluster must ensure its integrity to provide services normally. It must ensure that the slot managed by the cluster is 16384. In other words, if a master control unit fails, the slot managed by the master control unit will not be accessible, and the cluster to which the master control unit belongs cannot work normally, that is, the entire cluster crashes. When a master control unit fails, a slave control unit can be selected from the slave control units configured by the master control unit to replace the function of the original master control unit as a new master control unit. Any of device 150, device 160 and device 170 includes a client process, and the client process includes a client interface. The client process calls interface 1 through the client interface to access the data managed by any master control unit in device 110, device 120 or device 130.

[0083] When a main control unit in a network node device receives a large number of access requests for a single data, a hot key phenomenon may occur. For example, if a main control unit in device 110 receives a large number of access requests for a single data, the load on the main control unit increases, which may cause the main control unit to fail to work properly. If the main control unit fails, after the slave control unit takes over the main control unit as the new main control unit, there are still a large number of access requests for a single data, that is, the slave control unit still needs to process these access requests, which causes the slave control unit to be overloaded and crash. After the slave control unit also fails to work properly, the integrity of cluster 140 is destroyed, that is, the slot managed by cluster 140 is missing and is no longer 16384, which ultimately results in cluster 140 being unable to provide services.

[0084] In order to solve the above problems, the present application provides a hotspot data migration method, device and system. When hotspot data exists in the main cluster, the hotspot data is migrated to the secondary cluster to reduce the business pressure of the main cluster and ensure the performance and reliability of the main cluster.

[0085] See also Figure 2A , Figure 2A 2 shows a schematic diagram of the internal structure of the network node device 210. It should be understood that Figure 2A Only some hardware structures and some software modules in the device 210 are shown. In specific implementation, the device 210 may also include more hardware structures, such as communication interfaces, and more software modules, such as various application programs. Figure 2A As shown, the device 210 includes a central processing unit (CPU) 2110, a memory 2120, and a monitoring process 2130. The CPU 2110 calls and executes the program stored in the memory 2120 to generate a main control unit 2141, a main control unit 2142, and a main control unit 2143, as well as a slave control unit 2144 of the main control unit 2141, a slave control unit 2145 of the main control unit 2142, and a slave control unit 2146 of the main control unit 2143. Only three main control units are shown here by way of example, and more main control units can certainly be generated. In addition, multiple slave control units can be configured for each main control unit. Figure 2A It is only shown in one example. Figure 2A In the figure, below the dotted line are the hardware structure CPU 2110 and memory 2120 of the device 210, and above the dotted line are the monitoring process 2130, the main control unit 2141, the main control unit 2142, the main control unit 2143, and the slave control unit 2144 of the main control unit 2141, the slave control unit 2145 of the main control unit 2142, and the slave control unit 2146 of the main control unit 2143, which are run through hardware in the device 210. For example, the CPU 2110 calls the program in the memory 2120 to implement the monitoring management function of the monitoring process 2130. Figure 2AIn the memory 2120, data managed by the main control unit 2141, the main control unit 2142 and the main control unit 2143 are stored. Each main control unit or slave control unit is a process in the device 210, and each main control unit is also called a redis instance, which is responsible for managing a certain number of slots. Each main control unit is single-threaded, that is, for the same main control unit, only one access request can be processed at a time; that is, only one access request can be processed at any certain time, and multiple access requests cannot be processed to ensure high performance requirements. It is worth noting that each main control unit will occupy a certain storage resource when running, and the storage resource stores data or data sets. It should also be understood that the main control unit 2141, the main control unit 2142 and the main control unit 2143 can be main control units belonging to the same cluster, or they can be main control units belonging to different clusters. It is worth noting that each main control unit corresponds to a unique identifier, which consists of an Internet protocol (IP) and a port number (port). The device 210 can identify each main control unit through the identifier. In addition, the device 210 can also select part of the main control units as the main control units of the main cluster through the identifier, and select part of the main control units as the main control units of the secondary cluster through the identifier. For example, the main control unit 2141 and the main control unit 2142 are selected as the main control units in the main cluster, and the main control unit 2143 is selected as the main control unit in the secondary cluster. The main control unit in the main cluster and the main control unit in the secondary cluster use the memory resources of the device 210 together. The data managed by the main cluster (such as hot data) can be migrated to the secondary cluster and managed by the secondary cluster to disperse the business pressure of the main cluster; the data managed by the secondary cluster (such as non-hot data) can also be migrated back to the main cluster and managed by the main cluster to avoid the risk of collapse of the secondary cluster due to excessive business pressure. It should be noted that in the subsequent description of this application, the main control unit of the secondary cluster is deployed on the physical device where the main control unit of the main cluster is deployed. For example, of course, the main control unit of the main cluster and the main control unit of the secondary cluster can also be deployed on different physical devices respectively, and this application does not limit this. The monitoring process 2130 is connected to the main control unit 2141, the main control unit 2142 and the main control unit 2143 and monitors and manages them. The monitoring process 2130 includes a monitoring module 2131 and a migration module 2132. The monitoring module 2131 is responsible for monitoring the main control unit 2141, the main control unit 2142 and the main control unit 2143 and identifying the hotspot data therein. After identifying the hotspot data, the monitoring module 2131 notifies the migration module 2132. The migration module 2132 migrates the hotspot data identified by the monitoring module 2131 and sends migration information to the user device connected to the device 210.

[0086] See also Figure 2B , Figure 2B The diagram shows the internal structure of the cluster. The main control unit managed by cluster 240 is deployed on device 210, device 220 and device 230 at the same time. The internal structure of device 220 and device 230 is similar to that of device 210, except that device 220 and device 230 do not include monitoring process 2130 in device 210. It should be understood that for a cluster, only one monitoring process needs to be deployed to manage and monitor the cluster. Optionally, the monitoring process can be deployed on any physical device belonging to the cluster to implement monitoring and management of the main control units deployed on all physical devices in the cluster. Figure 2B In the example, the monitoring process 2130 is deployed on the device 210. Of course, the monitoring process 2130 can also be deployed on the device 220 or the device 230. The devices 210, 220 and 230 are connected to each other. The monitoring process 2130 in the device 210 can access all the main control units belonging to the cluster 240 in the devices 220 and 230, and is responsible for monitoring and managing all the main control units in the cluster 240. The total number of slots managed by all the main control units in the cluster 240 is 16384. The number of slots reflects the integrity of the cluster 240. Only when the total number of slots remains at 16384, the cluster 240 can provide services normally. If a main control unit fails, since each main control unit is responsible for managing some slots, it will cause some slots to be missing, then the total number of slots will no longer be 16384, that is, the integrity of the cluster 240 is destroyed, and the cluster 240 will not work normally. The monitoring module 2131 is responsible for monitoring all the main control units in the cluster 240 and identifying the hot data therein. The migration module 2132 migrates the hotspot data identified by the monitoring module 2131, and sends migration information to the user equipment connected to the cluster 240. It should be understood that the cluster 240 can also include more network node devices, that is, the main control unit in the cluster 240 can also be deployed on more physical devices. The above cluster 240 is described by taking the device 210, the device 220 and the device 230 as an example, which should not constitute a specific limitation.

[0087] It should also be understood that all the main control units in the above-mentioned devices 210, 220 and 230 may all belong to the main control unit in the cluster 240; or, some of the main control units in the devices 210, 220 and 230 belong to the main control unit in the cluster 240, and the other main control units belong to the main control unit of the secondary cluster or other clusters. The formation of the secondary cluster is consistent with the main cluster (i.e., cluster 240), and the CPU in the physical device calls the program stored in the memory to generate the main control unit, and then different physical devices select which main control units to use to form a complete secondary cluster together. The selection process can be determined according to the IP address and port. It should be understood that after generating each main control unit, the physical device can actively select to determine which cluster the main control unit belongs to. Exemplarily, after the CPU 2310 in the device 230 calls the program stored in the memory 2320 to generate the main control unit, the main control unit 2331 and the main control unit 2332 are selected as the main control units of the main cluster, and the main control unit 2333 is selected as the main control unit of the secondary cluster.

[0088] See also Figure 2C , Figure 2C It is shown in the figure a schematic diagram of the internal structure of the user equipment. It should be understood that Figure 2C Only some of the hardware structures and some of the software modules in the device 250 are shown. In a specific implementation, the device 250 may also include more hardware structures, such as a power supply, and more software modules, such as various application programs. The device 250 includes a CPU 2510, a memory 2520, and a client process 2530. The client process 2530 includes a client interface 2531, and the client interface 2531 includes a scheduling module 2532. It should be noted that in Figure 2C In the figure, below the dotted line are the hardware structure CPU 2510 and memory 2520 of device 250, and above the dotted line is the client process 2530 implemented by hardware in device 250. CPU 2510 calls the program in memory 2520 to implement the function of scheduling module 2532 in client process 2530. Memory 2520 is also used to store data received by scheduling module 2532. Scheduling module 2532 in client interface 2531 determines the cluster where the keyword in the current service request is located, and selects the correct cluster for access. Device 250 can use Figure 2BThe services provided by the cluster 240 shown in the figure, the device 250 is connected to the cluster 240 through the network, and the scheduling module 2532 can access the data managed by any one of the main control units in the device 210, the device 220 and the device 230. The scheduling module 2532 can also receive the migration information sent by the migration module 2132. Exemplarily, the device 210 and the device 250 are connected through the network. Since the device 210, the device 220 and the device 230 are interconnected, the device 250 can access any one of the main control units in the device 210, the device 220 and the device 230. In addition, it is assumed that the main control unit 2141 and the main control unit 2142 in the device 210, the main control unit 2231 and the main control unit 2232 in the device 220, and the main control unit 2331 and the main control unit 2332 in the device 230 belong to the main cluster, and the remaining main control units belong to the secondary cluster. The monitoring module 2131 monitors the main cluster and the secondary cluster. When the monitoring module 2131 detects that there is hotspot data in the main control unit 2331 in the main cluster, the migration module 2132 is notified, and the migration module 2132 migrates the hotspot data in the main control unit 2331 to the main control unit 2233 in the secondary cluster. After the migration is completed, the migration module 2132 sends migration information to the scheduling module 2532 in the device 250 to notify the device 250 that if the hotspot data is to be accessed, the main control unit 2233 in the device 220 needs to be accessed. After receiving the migration information, the scheduling module 2532 stores the keywords corresponding to the hotspot data carried in the migration information in the keyword set, and uses the keyword set to judge the current service request and select the correct cluster for access.

[0089] This application deploys the main control units in the main cluster and the secondary cluster on multiple identical physical devices, and selects one physical device to deploy the monitoring process. Figure 2BAs shown, the main control units of the main cluster and the secondary cluster can be deployed simultaneously in the device 210, the device 220 and the device 230, and the monitoring process 2130 in the device 210 is responsible for monitoring and managing the main control units in the two clusters. The monitoring process is used to monitor and manage all the main control units and slave control units in the main cluster, as well as monitor and manage all the main control units and slave control units in the secondary cluster. In other words, the monitoring process monitors and manages the main cluster and the secondary cluster at the same time. Specifically, since each main control unit has a unique IP address and port number, the monitoring process can distinguish each main control unit in the main cluster and the secondary cluster by different IP addresses and port numbers and establish a connection with them. In addition, after the connection is established, the monitoring process can use the monitor (monitoring management) function in redis to monitor and manage each main control unit connected. The monitoring process includes a monitoring module and a migration module. After the monitoring module monitors the hot data in the main cluster, it notifies the migration module, and the migration module migrates the hot data to the secondary cluster to disperse the pressure of the main cluster and achieve load balancing between clusters.

[0090] Combination Figure 2A , Figure 2B and Figure 2C For the application scenario shown, see Figure 3 , Figure 3 FIG. 1 is a flow chart of a hotspot data management method provided in an embodiment of the present application. Figure 3 As shown, the method includes but is not limited to the following steps:

[0091] S301: The database server determines hotspot data managed by the first cluster.

[0092] Specifically, the first cluster can be a redis cluster, which can also be called a main cluster. The database server here can be any physical device that has a monitoring process deployed on it, such as the above Figure 2B Device 210 in the main cluster. The monitoring process monitors and identifies the data managed by the main cluster, and determines the hot data managed by the main cluster. More specifically, the hot data managed by the main cluster is determined by the monitoring module in the monitoring process. It should be understood that in the subsequent description of this application, the database server and the monitoring process refer to the same execution subject, that is, the actions performed by the database server can be understood as the actions performed by the monitoring process deployed on the database server, and no special explanation or distinction will be made later.

[0093] It should be noted that the main cluster uses a hash algorithm to perform a hash operation on the key corresponding to the data to be stored to obtain a data result, and the data result is modulo 16384. The remainder obtained is the identifier of a certain slot, and the data to be stored is stored in the slot with the identifier. In this way, all data can be mapped to slots with different identifiers, and the identifier of each slot is an integer in the value range of 0-16383. The slots represented by different integers are different. For example, slot1 and slot2 are two different slots. The main cluster contains a total of 16384 slots, and each main control unit is responsible for managing a certain number of slots. Each slot is equivalent to a data set and can contain one or more data. For example, slot1 contains all data corresponding to the data results modulo 16384 with a remainder of 1. Each main control unit can be responsible for managing the same number of slots, or it can manage different numbers of slots according to weight ratios, etc. This application does not limit this. In particular, when the main cluster is deployed, the main control units are evenly deployed on different physical devices to achieve load balancing of the main cluster and disperse the business pressure of a single physical device.

[0094] In a possible implementation, the monitoring module in the monitoring process counts the number of accesses to each data managed by the main control unit within a preset time, and determines the data whose accesses exceed a threshold as hotspot data.

[0095] Specifically, the monitoring module can monitor all main control units in the main cluster in real time or periodically, count the number of accesses to each data managed by each main control unit, and determine the data whose access times exceed the threshold as hot data. The preset time can be a unit time, such as 1 second. The period and threshold can be set as needed, and this application does not limit this.

[0096] In one possible implementation, the monitoring module in the monitoring process calculates the resource usage of each main control unit in the main cluster, monitors the main control units whose resource usage exceeds a first threshold, and counts the number of accesses to each data managed by the main control unit whose resource usage exceeds the first threshold within a preset time, and determines the data whose access times exceed a second threshold as hot data.

[0097] Specifically, the monitoring module in the monitoring process can estimate the size of the first threshold based on the hardware conditions of the physical device, such as the number of CPUs, CPU capabilities, physical memory size, and network capabilities of the physical device, and cluster configuration information, such as the number of master control units deployed on each physical device, etc. The current resource usage of each master control unit, such as CPU resource usage or physical memory usage, is compared with the estimated first threshold, and the master control unit whose resource usage exceeds the first threshold is monitored.

[0098] Further, after determining that the resource usage of the main control unit exceeds the first threshold, the monitoring module counts the number of accesses to each data in each slot managed by the main control unit within a preset time, and if the number of accesses to a certain data exceeds a second threshold, the data is determined as hot data. The preset time may be a unit time, such as 1 second, and the second threshold may be set as needed, which is not limited in this application.

[0099] For example, see above Figure 2B The monitoring module 2131 determines that the resource usage of the main control unit 2141 exceeds the first threshold, and monitors the main control unit 2141. The main control unit 2141 manages 1000 slots, which are slot1-slot1000. Each slot contains 10 data, and the second threshold is 50000. The monitoring module counts the number of accesses to each data in each slot within 1 second. If the second data in slot1 is accessed 80000 times within 1 second, which exceeds the second threshold of 50000, it can be determined that the second data in slot1 is hot data.

[0100] It is understandable that monitoring the cluster by the monitoring module will affect and cause loss to the performance of the cluster. Therefore, when determining hot spot data, the monitoring module is not used to monitor all the main control units in the main cluster, but to screen them first. The monitoring module is only responsible for monitoring the main control units whose resource usage exceeds the first threshold. This can reduce the loss of cluster performance and ensure that the main cluster can provide better services.

[0101] S302: The database server determines whether a condition for migrating the hotspot data to the second cluster is met.

[0102] Specifically, the second cluster can be a redis cluster, which can also be called a secondary cluster. The secondary cluster can include multiple physical devices, which are interconnected by a network. The specific structure of each physical device can refer to the above Figure 2AIt is worth noting that the physical device used to build the second cluster can be the physical device used to build the first cluster. In other words, the same physical device can be used to build the main cluster and the secondary cluster. Since the physical devices used to build the main cluster and the secondary cluster are the same, for the same physical device, the main control unit in the main cluster and the main control unit in the secondary cluster can be deployed at the same time. For example, in the above Figure 2B In the embodiment, the main control units 2141 and 2142 in the device 210, the main control units 2231 and 2232 in the device 220, and the main control units 2331 and 2332 in the device 230 may be main control units belonging to the main cluster, and the main control unit 2143 in the device 210, the main control unit 2233 in the device 220, and the main control unit 2333 in the device 230 may be main control units belonging to the secondary cluster. In addition, the number of main control units in the secondary cluster may be the same as the number of main control units in the main cluster, and the identifiers of the slots that each main control unit is responsible for may also be the same. Of course, the number of main control units and the identifiers of the slots that each main control unit is responsible for may also be different, and this application does not limit this.

[0103] Optionally, each master control unit in the secondary cluster may also be configured with one or more slave control units. The secondary cluster and the master cluster each include 16384 slots, and each slot may include at least one data.

[0104] In a possible implementation, the first main control unit in the main cluster and the first main control unit in the secondary cluster are deployed on different physical devices, and the slot managed by the first main control unit in the main cluster and the slot managed by the first main control unit in the secondary cluster have the same identifier.

[0105] Specifically, the main cluster and the secondary cluster cooperate with each other, and the secondary cluster is used to receive the hot data migrated from the main cluster to disperse the business pressure of the main cluster. In addition, during the migration of hot data, the corresponding keywords remain unchanged, so the slot identifiers obtained by hashing the keywords are also the same. In other words, if the hot data in slot 1 in the main cluster is migrated to the secondary cluster, it must be migrated to slot 1 of the secondary cluster, and will not be migrated to slots with other identifiers, such as slot 2 or slot 3.

[0106] Therefore, when performing cluster deployment, if the main control units responsible for slots with the same identifier in the two clusters are deployed on the same physical device, the migration of hot data will be completed inside the physical device, that is, for the physical device, the business pressure is still concentrated on the physical device and will not be reduced. Therefore, it is necessary to deploy the main control units responsible for managing slots with the same identifier in the two clusters on different physical devices. Specifically, each physical device (such as a database server) can select the main control units that constitute the main cluster and the secondary cluster based on the IP address and port number. Since the physical device can distinguish different main control units of different clusters by IP address and port number, when deploying slots, you can choose to deploy the main control units that manage slots with the same identifier on different physical devices. For example, there are 10 master control units on a physical device, 8 of which are selected as master control units in the main cluster, and the other two are selected as master control units in the secondary cluster. When deploying slots (the main cluster and the secondary cluster each manage 16384 slots), the physical device can select the 8 master control units in the main cluster to manage slots with identifications of 1-5000, and select the two master control units in the secondary cluster to manage slots with identifications of 5001-10000, ensuring that the identifications of the slots managed by the master control units in different clusters are different. It can be understood that this can make the cluster load more balanced, reduce the impact of hot data on a single physical device, and avoid business pressure concentrated on the same physical device.

[0107] For example, see Figure 4 , Figure 4 Schematic diagram of a cluster deployment provided by an embodiment of the present application. Figure 4As shown, in a specific implementation, the main cluster includes 90 main control units, of which 30 main control units are responsible for slot 0 to slot 5000 and are deployed on the physical device 210; another 30 main control units responsible for slot 5001 to slot 10000 are deployed on the physical device 220; and the last 30 main control units responsible for slot 10001 to slot 16383 are deployed on the physical device 230. The secondary cluster includes 30 main control units, of which 10 main control units are responsible for slot 0 to slot 5000 and are deployed on the physical device 230; another 10 main control units responsible for slot 5001 to slot 10000 are deployed on the physical device 210; and the last 10 main control units responsible for slot 10001 to slot 16383 are deployed on the physical device 220. It can be seen that physical device 210 simultaneously deploys 30 master control units responsible for slot0 to slot5000 in the main cluster and 10 master control units responsible for slot5001 to slot10000 in the secondary cluster. When the data in slot100 in the main cluster is determined as hot data, it needs to be migrated to slot100 in the secondary cluster. Since slot100 in the main cluster is deployed on physical device 210, and slot100 in the secondary cluster is deployed on physical device 230, in fact, the data is migrated from physical device 210 to physical device 230. It is easy to understand that through the above deployment method, the cluster load can be guaranteed to be more balanced and business pressure can be avoided on the same physical device. In addition, the 30 main control units in the secondary cluster are used to assist the main cluster and store the hot spot data determined in the main cluster. Compared with the 90 main control units in the main cluster, the secondary cluster has only 30 main control units, which will not occupy too many resources and can reduce the impact on physical device 210, physical device 220 and physical device 230, while ensuring the high performance of the main cluster.

[0108] It can be understood that by migrating the hotspot data from the main cluster to the secondary cluster, the business pressure of the main cluster can be dispersed, the load of the main cluster can be more balanced, and the high performance and reliability of the main cluster can be guaranteed. Specifically, the monitoring process in the database server can identify each main control unit by IP address and port number and establish a connection with it. After the connection is established, the monitoring process can use the monitor function to monitor it, and when monitoring hotspot data, it can read the hotspot data and the keywords corresponding to the hotspot data, and migrate the read hotspot data and the keywords corresponding to the hotspot data to the secondary cluster. It is worth noting that the migration here is similar to the backup operation, that is, after the migration is completed, the hotspot data and the keywords corresponding to the hotspot data still exist in the main cluster.

[0109] In one possible implementation, statistics are performed on the main control unit whose resource usage (for example, CPU resource usage) exceeds a first threshold and where hot spot data exists, and the total number of accesses to all data managed by the main control unit is counted. If the total number of accesses exceeds a third threshold, it is determined that the main control unit meets the migration conditions, and the hot spot data managed by the main control unit is migrated to the secondary cluster.

[0110] Specifically, the total number of accesses to all data managed by the main control unit (including hotspot data and non-hotspot data) is counted, and the current state of the main control unit and whether the hotspot data needs to be migrated can be determined based on the total number of accesses. When the total number of accesses exceeds the third threshold, it means that there are too many current access requests, which has posed a threat to the security and reliability of the main control unit, that is, the main control unit is at risk of being broken down. Furthermore, since there is hotspot data in the data managed by the main control unit, that is, there is data with an access count exceeding the second threshold, it is necessary to migrate the hotspot data managed by the main control unit to the secondary cluster to disperse the business pressure of the main control unit and avoid the main control unit from being broken down. Optionally, the third threshold can be set as needed, and this application does not limit this.

[0111] In particular, when the monitoring module in the monitoring process monitors that there are multiple hotspot data in the main control unit that meets the migration conditions, for example, in the above Figure 2B In the example, the monitoring module 2131 monitors that the main control unit 2141 meets the migration conditions, and there are multiple hotspot data in the data managed by the main control unit 2141. At this time, if all the hotspot data are migrated to the secondary cluster, the secondary cluster will receive a large number of access requests sent by the client in a short period of time, and the main control unit in the secondary cluster will face huge business pressure, which may cause the main control unit in the secondary cluster to be broken down and unable to work normally, and then cause the physical device of the main control unit deployed in the secondary cluster to crash, so that the entire secondary cluster cannot provide services.

[0112] Optionally, the monitoring module prioritizes the hotspot data. When the monitoring module detects that there are multiple hotspot data in the main control unit in the main cluster that meets the migration conditions, the hotspot data is migrated to the secondary cluster according to the priority. The hotspot data with the highest priority can be migrated to the secondary cluster first, and then the status of the secondary cluster is monitored by the monitoring module. According to the status of the secondary cluster, it is determined whether to migrate the hotspot data with lower priority to the secondary cluster. This can avoid migrating too much hotspot data to the secondary cluster at one time, causing the secondary cluster to crash due to the impact of a large number of access requests in a short period of time, and thus unable to provide services, thereby achieving load balancing between the main cluster and the secondary cluster.

[0113] Exemplarily, the monitoring module statistics show that within 1 second, data 1 managed by the main control unit that meets the migration conditions in the main cluster is accessed 150,000 times, data 2 is accessed 130,000 times, and data 3 is accessed 100,000 times. Data with more than 50,000 access times is determined as hot data (i.e., the second threshold is 50,000). The access times of data 1, data 2, and data 3 all exceed 50,000 times, so they are all hot data. Since data 1 is accessed the most times, its priority level is the highest, so the monitoring module marks data 1 as high-priority hot data. Similarly, data 2 is marked as medium-priority or general-priority hot data, and data 3 is marked as low-priority hot data. The migration module can migrate data 1, data 2, and data 3 to the secondary cluster according to the priority level. For example, the current business pressure of the current secondary cluster is relatively large, that is, there are too many hot data in the current secondary cluster, and the number of hot data exceeds the preset value. For example, the preset value is 1000, which means that the secondary cluster can only manage 1000 hot data under normal circumstances, and there are 1050 hot data in the current secondary cluster. At this time, the business pressure of the secondary cluster is relatively large. At this time, only data 1 can be migrated to the secondary cluster, and data 2 and data 3 are temporarily not migrated to the secondary cluster. If the current business pressure of the secondary cluster is small, data 1, data 2 and data 3 can all be migrated to the secondary cluster.

[0114] It should be understood that the priority here is only divided based on the number of times the hot data is accessed. The priority can also be divided according to other rules or conditions, such as selecting data to be migrated based on the business importance level, etc. This application does not limit this.

[0115] S303: The database server migrates the hotspot data to the second cluster, and triggers an update of the hotspot data access path.

[0116] Specifically, each master control unit corresponds to a unique IP address and port number. All master control units in the same physical device (including the master control unit in the main cluster and the master control unit in the secondary cluster) have the same IP address, but their port numbers are different. The IP addresses of the master control units in different physical devices are different, but their port numbers may be the same. Combining the IP address and port number, a unique master control unit can be determined, that is, the monitoring process is monitored using the IP address and port number, and each master control unit in the main cluster and secondary cluster is managed according to the IP address and port number.

[0117] Further, after the monitoring module deployed in the database server determines the hot data that needs to be migrated in the main cluster, it notifies the migration module, and the migration module reads the hot data and the keywords corresponding to the hot data from the main cluster. After reading the hot data and the keywords corresponding to the hot data, the migration module uses a hash algorithm to perform a hash operation on the read keywords to obtain an identifier corresponding to a slot. According to the identifier, the migration module can determine the main control unit that manages the slot corresponding to the identifier in the secondary cluster, and then obtain the IP address and port number corresponding to the main control unit. It is worth noting that when the main cluster and the secondary cluster are built, the monitoring process has stored the IP address and port number corresponding to each main control unit. After obtaining the IP address and port number, the migration module writes the hot data and the keywords corresponding to the hot data into the slot corresponding to the calculated slot identifier, which is a slot in the slot managed by the main control unit corresponding to the IP address and port number in the secondary cluster. It is worth noting that the migration process and implementation logic of the hot data and the keywords corresponding to the hot data are consistent with the data migration process in the prior art, and will not be repeated here.

[0118] In one possible implementation, after completing the migration of hotspot data, the migration module sends migration information to the client communicating with the primary cluster and the secondary cluster. The migration information includes keywords corresponding to the hotspot data. The migration information instructs the client to record the keywords corresponding to the hotspot data into a keyword set, which includes keywords corresponding to all hotspot data migrated from the primary cluster to the secondary cluster.

[0119] Specifically, after migrating the hotspot data in the main cluster to the secondary cluster, the migration module will send migration information to the client communicating with the main cluster and the secondary cluster to notify the client, and the migration information can be sent in the form of a network packet. Optionally, the client and the migration module can negotiate and define some relevant protocols to ensure that the client can correctly interpret the migration information after receiving it, obtain the keywords corresponding to the hotspot data therein, and add the keywords to the keyword set. Alternatively, a dedicated connection (such as a TCP connection) is established between the migration module and the client to specifically send the migration information to ensure that the client can correctly interpret it, obtain the keywords corresponding to the hotspot data therein, and add the keywords to the keyword set.

[0120] It should be noted that both the main cluster and the secondary cluster use key-value to store data. Each key corresponds to a unique value. In other words, the main control unit needs to manage not only the value of the data itself, but also the key corresponding to the data. The key can also be called a keyword, and no distinction will be made in the following description.

[0121] Furthermore, after the migration module completes the migration of the hotspot data, the storage location of the hotspot data has changed, that is, the storage address and other information of the hotspot data have changed. If the client still sends an access request to the main cluster in the previous manner to request access to the hotspot data, it will not be able to successfully access the hotspot data. Therefore, the migration module needs to send a notification message (i.e., migration information) to all clients communicating with the main cluster and the secondary cluster to notify all clients that the hotspot data has been migrated to the secondary cluster. The client maintains a keyword set. After receiving the migration information, the client adds the keywords corresponding to the hotspot data in the migration information to the keyword set. If the client generates a business request, the client determines whether the keywords in the business request are keywords in the keyword set. If so, an access request is sent to the secondary cluster. If not, an access request is sent to the main cluster.

[0122] Furthermore, the migration module needs to ensure that all clients communicating with the main cluster and the secondary cluster receive the migration information to avoid some clients not receiving the migration information and thus being unable to correctly access the hotspot data. Optionally, the migration module and the client can use a reply confirmation mechanism to ensure that all clients receive the migration information, that is, after the migration module sends the migration information to all clients communicating with the main cluster and the secondary cluster, it receives the confirmation messages replied by all clients. If the migration module receives the confirmation messages replied by all clients within a preset time, it can be determined that all clients have successfully received the migration information. If the migration module does not receive the confirmation message replied by the client within the preset time or receives the confirmation message replied by some clients, it means that not every client has successfully received the migration information. The migration module needs to repeat the above process and send the migration information to all clients again until the confirmation messages replied by all clients are received within the preset time. It can be understood that the migration module can also use other confirmation methods or confirmation mechanisms to ensure that all clients receive the migration information, and this application does not limit this.

[0123] It can be seen that by executing the above method embodiment, the hot data in the main cluster can be migrated to the secondary cluster, dispersing the business pressure of the main cluster, avoiding excessive business pressure in the main cluster, causing the main control unit in the main cluster to be broken down, and even causing the physical device where the main control unit is deployed to be paralyzed, achieving load balancing between the main cluster and the secondary cluster, and ensuring the high performance and reliability of the main cluster. In addition, it can also ensure that the client communicating with the main cluster and the secondary cluster can select the correct cluster for access, ensuring the high reliability of the business.

[0124] It should be understood that steps S301 to S303 involved in the above method embodiment are only schematic descriptions and should not constitute specific limitations. The steps involved can be increased, reduced or combined as needed.

[0125] Combination Figure 2A , Figure 2B and Figure 2C The application scenario shown is shown in Figure 5 , Figure 5 FIG. 1 is a flow chart of another hotspot data management method provided in an embodiment of the present application. Figure 5 As shown, the method includes but is not limited to the following steps:

[0126] S501: The database server determines hotspot data managed by the main cluster.

[0127] Specifically, the identification process and determination process of the hotspot data may refer to the relevant description in the above S301, and for the sake of brevity, it will not be repeated here.

[0128] S502: The database server determines whether a migration condition for migrating the hotspot data to the secondary cluster is met.

[0129] Specifically, the judgment process may refer to the relevant description in the above S302, and for the sake of brevity, it will not be repeated here.

[0130] S503: The database server migrates the hotspot data to the secondary cluster, and triggers the update of the access path of the hotspot data.

[0131] In one possible implementation, after completing the hotspot data migration, the migration module sends first migration information to the client communicating with the main cluster and the secondary cluster. The first migration information includes the hotspot data and the keywords corresponding to the hotspot data. The first migration information instructs the client to record the hotspot data and the keywords corresponding to the hotspot data to the first set.

[0132] Specifically, the migration module reads the hotspot data and the keywords corresponding to the hotspot data from the main cluster, and then writes the hotspot data and the keywords corresponding to the hotspot data into the slot managed by the secondary cluster. The specific migration process can refer to the relevant description in S303 above, which will not be repeated here. However, it should be noted that during the migration process, the migration module may have some clients write operations on the hotspot data, and the write operation will cause the hotspot data to change, that is, the hotspot data read by the migration module from the main cluster is inconsistent with the hotspot data in the main cluster at the current moment, that is, the hotspot data written by the migration module to the secondary cluster is inconsistent with the hotspot data in the main cluster. In addition, if the hotspot data changes during the migration process, the migration module cannot perceive that the hotspot data in the main cluster has changed. After writing the hotspot data and the keywords corresponding to the hotspot data to the secondary cluster, the migration module sends the first migration information carrying the hotspot data and the keywords corresponding to the hotspot data to all clients communicating with the main cluster and the secondary cluster. The hotspot data in the first migration information is inconsistent with the hotspot data in the main cluster at the current moment.

[0133] Furthermore, the client maintains a first set, which is a data buffer set, including all hotspot data and keywords corresponding to the hotspot data that have been migrated from the main cluster to the secondary cluster and have not yet undergone data consistency verification, that is, the first set includes the hotspot data carried in the first migration information and the keywords corresponding to the hotspot data. The first set is used to cooperate with the monitoring process to ensure data consistency. After the client receives the first migration information and writes the hotspot data carried in the migration information and the keywords corresponding to the hotspot data into the first set, or when an access request for the hotspot data is generated in the client, the scheduling module in the client accesses the hotspot data managed by the main cluster, thereby comparing whether the hotspot data in the first set is consistent with the hotspot data managed by the main cluster, so as to determine whether the hotspot data has changed during the migration process. If they are consistent, it means that the migration is successful, that is, the hotspot data has not changed during the migration process, and the hotspot data migrated to the secondary cluster is consistent with the hotspot data managed by the original primary cluster. All access requests for the hotspot data will directly access the secondary cluster; if they are inconsistent, it means that the migration has failed, that is, the hotspot data has changed during the migration process, and the hotspot data migrated to the secondary cluster is inconsistent with the hotspot data managed by the original primary cluster. All access requests for the hotspot data will still access the primary cluster. As for whether the hotspot data needs to be re-migrated, it can be decided by the monitoring process deployed in the database server.

[0134] It should be understood that some specific implementation details of S503 can refer to the relevant description in the above step S303, and for the sake of brevity, they are not repeated here.

[0135] S504: The database server determines the non-hotspot data managed by the secondary cluster.

[0136] Specifically, the monitoring module monitors all main control units managed by the secondary cluster, counts the number of accesses to the data managed by each main control unit within a preset time, and when the number of accesses is less than a fourth threshold, determines the data as non-hotspot data. The preset time may be a unit time, such as 1 second, and the fourth threshold may be set as needed, and may be set to the same value as the above-mentioned second threshold, or may be set to a value less than the second threshold, which is not limited in this application.

[0137] Optionally, the monitoring module can monitor all the main control units managed by the secondary cluster in real time or perform periodic monitoring according to a certain period. The period can be set as needed, and this application does not limit this. It is worth noting that the data managed by the secondary cluster is relatively small compared to the main cluster, so the business pressure of the secondary cluster is relatively small. Therefore, real-time monitoring or periodic monitoring of all the main control units managed by the secondary cluster will not have much impact on the performance of the secondary cluster. The secondary cluster can still provide services normally, which is different from the main cluster.

[0138] It should be noted that there is no logical sequence between S503 and S504, that is, S503 may be executed first and then S504; or S504 may be executed first and then S503; or S503 and S504 may be executed simultaneously, and this application does not limit this.

[0139] S505: The database server migrates the non-hotspot data to the main cluster, and triggers the update of the access path of the non-hotspot data.

[0140] Specifically, after the monitoring module determines the non-hotspot data in the secondary cluster, it needs to migrate the non-hotspot data to the main cluster to reduce the business pressure of the secondary cluster. The migration module reads the non-hotspot data and the keywords corresponding to the non-hotspot data from the secondary cluster, and then the migration module uses a hash algorithm to perform hash calculations on the keywords corresponding to the read non-hotspot data to determine the identifier corresponding to the slot. According to the identifier, the main control unit that manages the slot corresponding to the identifier in the main cluster can be determined, and then the IP address and port number corresponding to the main control unit can be obtained. Based on the obtained IP address and port number, the monitoring module can find the main control unit corresponding to the IP address and port number in the main cluster, and then write the non-hotspot data and the keywords corresponding to the non-hotspot data into the slot corresponding to the calculated slot identifier managed by the main control unit.

[0141] It should be understood that the migration module migrating hot data and keywords corresponding to the hot data to the secondary cluster and the migration module migrating non-hot data and keywords corresponding to the hot data to the primary cluster are opposite processes. For specific migration details, please refer to the relevant description in S303 above.

[0142] In one possible implementation, after the migration module migrates the non-hotspot data from the secondary cluster to the primary cluster, it sends second migration information to the client communicating with the primary cluster and the secondary cluster, where the second migration information includes the non-hotspot data and keywords corresponding to the non-hotspot data, and the second migration information instructs the client to record the non-hotspot data and the keywords corresponding to the non-hotspot data into the second set.

[0143] Specifically, similar to the above-mentioned migration module migrating the hotspot data and the keywords corresponding to the hotspot data to the secondary cluster, in the process of migrating non-hotspot data, the migration module may also have some clients performing write operations on the non-hotspot data, and the write operations will cause the non-hotspot data to change, that is, the non-hotspot data read by the migration module from the secondary cluster is inconsistent with the non-hotspot data in the secondary cluster at the current moment, and the non-hotspot data written by the migration module to the main cluster is inconsistent with the non-hotspot data in the secondary cluster. In addition, if the non-hotspot data changes during the migration process, the migration module cannot perceive that the non-hotspot data in the secondary cluster has changed. After writing the non-hotspot data and the keywords corresponding to the non-hotspot data to the main cluster, the migration module sends a second migration message carrying the non-hotspot data and the keywords corresponding to the non-hotspot data to all clients communicating with the main cluster and the secondary cluster. The non-hotspot data in the second migration message is inconsistent with the non-hotspot data in the secondary cluster at the current moment.

[0144] Furthermore, the client maintains a second set, which is also a data buffer set, including all non-hotspot data migrated from the secondary cluster to the primary cluster and the keywords corresponding to the non-hotspot data that have not yet undergone data consistency verification, that is, the second set includes the non-hotspot data carried in the second migration information and the keywords corresponding to the non-hotspot data. The second set is also used to cooperate with the monitoring process to ensure data consistency. After the client receives the second migration information and writes the non-hotspot data carried in the migration information and the keywords corresponding to the non-hotspot data into the second set, or when an access request for the non-hotspot data is generated in the client, the scheduling module in the client accesses the non-hotspot data managed by the secondary cluster, thereby comparing the non-hotspot data in the second set with the non-hotspot data managed by the secondary cluster at the current moment to see if they are consistent, so as to determine whether the non-hotspot data has changed during the migration process. If they are consistent, it means that the migration is successful, that is, the non-hotspot data has not changed during the migration process, the non-hotspot data migrated to the main cluster is consistent with the non-hotspot data managed by the secondary cluster, and all access requests for the non-hotspot data will directly access the main cluster; if they are inconsistent, it means that the migration has failed, that is, the non-hotspot data has changed during the migration process, the non-hotspot data migrated to the main cluster is inconsistent with the non-hotspot data managed by the secondary cluster, and all access requests for the non-hotspot data in the client will still access the secondary cluster. As for whether the non-hotspot data needs to be re-migrated, it can be decided by the monitoring process deployed in the database server.

[0145] It should be noted that the above-mentioned data consistency judgment process is executed and implemented by the client side, and the data consistency judgment can also be completed by the migration module. Specifically, after completing the data migration, the migration module directly judges whether the data before the migration is consistent with the data in the original cluster after the migration is completed. For example, for hotspot data, after migrating the hotspot data to the secondary cluster, the migration module judges whether the hotspot data in the main cluster before the migration is consistent with the hotspot data in the main cluster after the migration is completed. If they are consistent, it means that the data has not changed. The migration module sends a migration message to all clients communicating with the first cluster and the second cluster. The migration message includes the keywords corresponding to the hotspot data. After receiving the migration message, the client updates the keywords corresponding to the hotspot data to the keyword set; if they are inconsistent, it means that the data has changed and the migration has failed.

[0146] It should be understood that the steps S501 to S505 involved in the above method embodiment are only schematic descriptions and should not constitute specific limitations. The steps involved can be increased, reduced or combined as needed. Figure 5 The specific details of the method embodiment described above can be referred to the relevant description in S301-S303 above, and for the sake of brevity, they will not be repeated here.

[0147] Combination Figure 2A , Figure 2B and Figure 2C The application scenario shown is shown in Figure 6 , Figure 6 Schematic diagram of a data consistency verification method provided by an embodiment of the present application. Figure 6 As shown, the method includes but is not limited to the following steps:

[0148] S601: The scheduling module receives the migration information sent by the migration module, and updates the first set or the second set.

[0149] Specifically, the migration information includes the hot data migrated from the primary cluster to the secondary cluster and the keywords corresponding to the hot data, or the migration information includes the non-hot data migrated from the secondary cluster to the primary cluster and the keywords corresponding to the non-hot data. The first set stores the hot data migrated from the primary cluster to the secondary cluster and the keywords corresponding to the hot data that have not been verified for data consistency, and the second set stores the non-hot data migrated from the secondary cluster to the primary cluster and the keywords corresponding to the non-hot data that have not been verified for data consistency.

[0150] Further, the specific implementation process may refer to the relevant description in the above S503, which will not be repeated here for the sake of brevity.

[0151] S602: The scheduling module determines whether the keyword set includes the keyword in the current service request. If yes, execute step S603; if no, execute step S604.

[0152] Specifically, the scheduling module maintains a keyword set, which includes keywords corresponding to all hot data that have been migrated from the primary cluster to the secondary cluster, that is, the keyword set includes keywords corresponding to all data currently stored and managed by the secondary cluster and that have undergone data consistency verification.

[0153] S603: The scheduling module determines whether the second set includes the keyword in the current service request. If so, execute step S605; if not, execute step S606.

[0154] Specifically, the scheduling module can determine whether the data corresponding to the keyword is non-hotspot data migrated from the secondary cluster to the primary cluster by judging whether the second set includes the keyword in the current business request. If the second set includes the keyword in the current business request, it means that the data corresponding to the keyword is non-hotspot data migrated from the secondary cluster to the primary cluster. Otherwise, it means that the data corresponding to the keyword is not non-hotspot data migrated from the secondary cluster to the primary cluster. The specific judgment process can refer to the relevant description in S505 above.

[0155] S604: The scheduling module determines whether the first set includes the keyword in the current service request. If so, execute step S607; if not, execute step S608.

[0156] S605: The scheduling module determines whether the data corresponding to the keyword in the second set is consistent with the data corresponding to the keyword managed by the secondary cluster at the current moment. If they are consistent, step S609 is executed; if not, step S610 is executed.

[0157] Specifically, the scheduling module determines whether the non-hotspot data has changed during the migration process by determining whether the data corresponding to the keyword in the second set is consistent with the data corresponding to the keyword managed by the secondary cluster at the current moment. If they are consistent, it means that the data corresponding to the keyword has not changed during the migration from the secondary cluster to the primary cluster, and the data has been successfully migrated to the primary cluster. Otherwise, it means that the data has changed during the migration from the secondary cluster to the primary cluster, and the migration has failed. The specific judgment process can refer to the relevant description in S504-S505 above.

[0158] S606: The scheduling module accesses the secondary cluster.

[0159] S607: The scheduling module determines whether the data corresponding to the keyword in the first set is consistent with the data corresponding to the keyword managed by the main cluster at the current moment. If they are consistent, step S611 is executed; if they are inconsistent, step S612 is executed.

[0160] Specifically, the scheduling module determines whether the data has changed during the migration process by determining whether the data corresponding to the keyword in the first set is consistent with the data corresponding to the keyword managed by the primary cluster at the current moment. If they are consistent, it means that the data corresponding to the keyword has not changed during the migration from the primary cluster to the secondary cluster, and the data has been successfully migrated to the secondary cluster. Otherwise, it means that the data has changed during the migration from the primary cluster to the secondary cluster, and the migration has failed. The specific judgment process can refer to the relevant description in S504-S505 above.

[0161] S608: The scheduling module accesses the main cluster.

[0162] S609: The scheduling module deletes the keyword in the keyword set, and deletes the keyword and the data corresponding to the keyword in the second set.

[0163] S610: The scheduling module deletes the keyword and the data corresponding to the keyword in the second set.

[0164] S611: The scheduling module adds the keyword to the keyword set, and deletes the keyword and the data corresponding to the keyword in the first set.

[0165] S612: The scheduling module deletes the keyword and the data corresponding to the keyword in the first set.

[0166] It should be noted that Figure 6 For details of the method embodiments described, please refer to the above Figure 3 Steps S301-S303 and Figure 5 For the sake of brevity, the relevant descriptions of steps S501-S505 in the illustrated method embodiment are not repeated here.

[0167] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, correspondingly, related equipment for cooperating in implementing the above scheme is also provided below.

[0168] See also Fig. 7A , Fig. 7A is a schematic diagram of the structure of a network device provided in an embodiment of the present application. The network device may be the above Figure 3 The database server in the method embodiment described above can execute Figure 3 In the hotspot data management method embodiment, the database server is used as the execution subject of the method and steps. Fig. 7A As shown, the network device 710 includes a processing module 711, a judgment module 712 and a migration module 713.

[0169] A processing module 711 is used to determine hotspot data managed by a first cluster, wherein the first cluster includes at least one single-threaded main control unit, and the single-threaded main control unit processes one access request at a time, wherein the access request includes an access request for data;

[0170] A judgment module 712 is used to judge whether a migration condition for migrating the hotspot data to a second cluster is met, the second cluster includes at least one single-threaded main control unit, the data accessed by the access request is jointly managed by the first cluster and the second cluster, and the second cluster is used to manage the hotspot data that meets the migration condition;

[0171] The migration module 713 is used to migrate the hotspot data to the second cluster and trigger the update of the hotspot data access path when the migration condition is met.

[0172] See also Figure 7B , Figure 7B Schematic diagram of another network device provided in the embodiment of the present application. Figure 7B As shown, the network device 720 includes a processing module 721 , a determination module 722 , a migration module 723 and a sending module 724 .

[0173] As an embodiment, the first main control unit in the first cluster and the first main control unit in the second cluster are deployed on different physical devices.

[0174] As an embodiment, the main control unit manages at least one hash slot, the identifier of the hash slot is an integer obtained by mapping data through a preset algorithm, and the hash slot includes at least one data; the hash slot managed by the first main control unit in the first cluster and the hash slot managed by the first main control unit in the second cluster have the same identifier.

[0175] As an embodiment, the processing module 721 is used to read the hot data and the keywords corresponding to the hot data from the first cluster, and determine the first hash slot in the second cluster based on the keywords corresponding to the hot data; the migration module 723 is also used to migrate the hot data and the keywords corresponding to the hot data to the first hash slot.

[0176] As an embodiment, the sending module 724 is used to send migration information to a client communicating with the first cluster and the second cluster, the migration information including keywords corresponding to the hotspot data, the migration information being used to instruct the client to record the keywords corresponding to the hotspot data into a keyword set, the keyword set being used by the client to determine a target cluster corresponding to the access request, the target cluster including the first cluster and the second cluster.

[0177] As an embodiment, the processing module 721 is also used to read the hotspot data managed by the first cluster after the migration is completed, and to verify whether the hotspot data is consistent with the hotspot data managed by the first cluster before the migration; the sending module 724 is also used to send the migration information to the client communicating with the first cluster and the second cluster when the hotspot data managed by the first cluster after the migration is completed is consistent with the hotspot data managed by the first cluster before the migration.

[0178] As an embodiment, the sending module 724 is used to send migration information to a client communicating with the first cluster and the second cluster, the migration information carries the hotspot data and keywords corresponding to the hotspot data, the migration information is used to instruct the client to record the hotspot data and the keywords corresponding to the hotspot data to a first set, and the first set is used by the client to make data consistency judgments.

[0179] As an embodiment, the processing module 721 is also used to calculate the resource usage of each main control unit in the first cluster, and monitor the main control units whose resource usage exceeds a first threshold; count the number of accesses to each data managed by the main control unit whose resource usage exceeds the first threshold within a preset time, and determine the data whose access times exceed a second threshold as hot data.

[0180] As an embodiment, the judgment module 722 is also used to count the total number of accesses to the data managed by the main control unit whose resource usage exceeds the first threshold. When the total number of accesses exceeds a third threshold, it is determined that the hotspot data managed by the main control unit whose resource usage exceeds the first threshold meets the migration condition.

[0181] As an embodiment, the processing module 721 is also used to monitor the second cluster, and the second cluster manages the hotspot data migrated from the first cluster to the second cluster; counts the number of accesses to the hotspot data within a preset time, and when the number of accesses to the hotspot data is less than a fourth threshold, marks the hotspot data as non-hotspot data; the migration module 723 is also used to migrate the non-hotspot data to the first cluster and trigger an update of the access path to the non-hotspot data.

[0182] It should be noted that the structure of the above network device and the process of using the network device to achieve hotspot data migration are only used as an example and should not constitute a specific limitation. The various units in the network device can be increased, reduced or merged as needed. In addition, the operations and / or functions of the various modules in the network device are to achieve the above Figure 3 For the sake of brevity, the corresponding process of the described method will not be repeated here.

[0183] See also Fig. 8A , Fig. 8A is a schematic diagram of the structure of a client device provided in an embodiment of the present application. The client device may be the above Figure 5 and Figure 6 The device in the method embodiment can execute Figure 5 and Figure 6 The hotspot data management method embodiment described above uses the client as the execution subject of the method and steps. Fig. 8A As shown, the client device 810 includes a processing module 811 and a sending module 812.

[0184] The processing module 811 is used to query whether a keyword set includes the keyword according to the keyword in the business request, the keyword set includes the keyword corresponding to the hot data migrated from the first cluster to the second cluster, the first cluster and the second cluster include at least one single-threaded main control unit, the single-threaded main control unit processes one access request at a time, the access request includes an access request for data, and the data accessed by the access request is jointly managed by the first cluster and the second cluster; determine the target cluster according to the query result, the target cluster includes the first cluster or the second cluster;

[0185] The sending module 812 is configured to send an access request to the target cluster, where the access request includes a keyword in the service request.

[0186] See also Figure 8B , Figure 8B Schematic diagram of another network device provided in the embodiment of the present application. Figure 8B As shown, the client device 820 includes a processing module 821 , a sending module 822 and a receiving module 823 .

[0187] As an embodiment, the receiving module 823 is used to receive migration information sent by a database server, and the migration information includes keywords corresponding to the hot data; the processing module 821 is also used to record the keywords corresponding to the hot data into a keyword set.

[0188] As an embodiment, the receiving module 823 is used to receive first migration information sent by the database server, the first migration information including the hotspot data and the keywords corresponding to the hotspot data; the processing module 821 is also used to record the hotspot data and the keywords corresponding to the hotspot data to a first set; the receiving module 823 is also used to receive second migration information sent by the database server, the second migration information including non-hotspot data and the keywords corresponding to the non-hotspot data; the processing module 821 is also used to record the non-hotspot data and the keywords corresponding to the non-hotspot data to a second set; wherein, the first set and the second set are used to cache data for which data consistency judgment has not been performed.

[0189] As an embodiment, the processing module 821 shown is also used to determine that the second cluster is the target cluster when the keyword set includes the keywords in the business request; the processing module 821 shown is also used to determine that the first cluster is the target cluster when the keyword set does not include the keywords in the business request.

[0190] It should be noted that the structure of the client device and the process of using the client device to achieve hotspot data migration are only used as an example and should not constitute a specific limitation. The various units in the client device can be increased, reduced or merged as needed. In addition, the operations and / or functions of the various modules in the client device are to achieve the above Figure 5 and Figure 6 For the sake of brevity, the corresponding process of the described method will not be repeated here.

[0191] See also Fig. 9 , Fig. 9 Schematic diagram of the structure of a network device provided in an embodiment of the present application. Fig. 9 As shown, the network device 900 includes: a processor 910, a communication interface 920 and a memory 930, and the processor 910, the communication interface 920 and the memory 930 are interconnected via an internal bus 940. It should be understood that the network device may be a database server.

[0192] The processor 910 may be composed of one or more general-purpose processors, such as a central processing unit (CPU), or a combination of a CPU and a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0193] The bus 940 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus 940 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0194] The memory 930 may include a volatile memory, such as a random access memory (RAM); the memory 930 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 930 may also include a combination of the above types. The memory 930 may be used to store programs and data so that the processor 910 can call the program code and data stored in the memory 930 to implement the functions of the above processing modules 711 and 721. The program code may be used to implement Fig. 7A and Figure 7B The functional modules shown in the figure may be used to implement Figure 3 The method embodiment shown uses a database server as the execution subject for the method steps.

[0195] See also Fig.10 , Fig.10 Schematic diagram of the structure of a client device provided in an embodiment of the present application. Fig.10 As shown, the client device 1000 includes: a processor 1110, a communication interface 1120 and a memory 1130, wherein the processor 1110, the communication interface 1120 and the memory 1130 are interconnected via an internal bus 1140. It should be understood that the client device may be a computer or a server.

[0196] The processor 1110 may be composed of one or more general-purpose processors, such as a CPU, or a combination of a CPU and a hardware chip. The hardware chip may be an ASIC, a PLD, or a combination thereof. The PLD may be a CPLD, an FPGA, a GAL, or any combination thereof.

[0197] The bus 1140 may be a PCI bus or an EISA bus, etc. The bus 1140 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0198] The memory 1130 may include a volatile memory, such as a RAM; the memory 1130 may also include a non-volatile memory, such as a ROM, a flash memory, a HDD, or a SSD; the memory 1130 may also include a combination of the above types. The memory 1130 may be used to store programs and data so that the processor 1110 can call the program code and data stored in the memory 1130 to implement the functions of the above processing modules 811 and 821. The program code may be used to implement Fig. 8A and Figure 8B The functional modules shown in the figure may be used to implement Figure 5 and Figure 6 The method steps in the method embodiment shown are performed by using a client device as the execution subject.

[0199] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the program can implement part or all of the steps of any one of the above method embodiments, and implement the above method. Fig. 7A and Figure 7B The function of any functional module described.

[0200] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the program can implement part or all of the steps of any one of the above method embodiments, and implement the above method. Fig. 8A and Figure 8B The function of any functional module described.

[0201] The embodiment of the present application also provides a computer program product, which, when executed on a computer or processor, enables the computer or processor to perform one or more steps in any of the above methods. If the components of the above-mentioned devices are implemented in the form of software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0202] In the above embodiments, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0203] It should be understood that the first, second, third, fourth and various numerical numbers involved in this document are only distinguished for the convenience of description and are not used to limit the scope of the present application.

[0204] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0205] It should also be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0206] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0207] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0208] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0209] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0210] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0211] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0212] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0213] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0214] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for managing hotspot data. It is characterized in that The method comprises: Determining hotspot data managed by a first cluster, the first cluster comprising at least one single-threaded main control unit, the single-threaded main control unit processing one access request at a time, the access request comprising an access request for data; Determine whether a migration condition for migrating the hot data to a second cluster is met, the second cluster includes at least one single-threaded main control unit, the data accessed by the access request is jointly managed by the first cluster and the second cluster, and the second cluster is used to manage the hot data that meets the migration condition; the main control unit is responsible for managing one or more hash slots, each of the hash slots includes one or more data, the identifier of the hash slot is determined based on a keyword of the stored data, and the identifiers of the hash slots managed by the first cluster and the second cluster are the same; When the migration condition is met, the hot data is migrated to the second cluster, and an update of the access path to the hot data is triggered; The migrating of the hot data to the second cluster includes: reading the hot data and the keywords corresponding to the hot data from the first cluster; determining a first hash slot in the second cluster according to the keywords corresponding to the hot data; and migrating the hot data and the keywords corresponding to the hot data to the first hash slot.

2. The method according to claim 1, It is characterized in that The first main control unit in the first cluster and the first main control unit in the second cluster are deployed on different physical devices.

3. The method according to claim 1 or 2, It is characterized in that The main control unit manages at least one hash slot, the identifier of the hash slot is an integer obtained by mapping data through a preset algorithm, and the hash slot includes at least one data; The hash slot managed by the first main control unit in the first cluster and the hash slot managed by the first main control unit in the second cluster have the same identifier.

4. The method according to claim 1 or 2, It is characterized in that Triggering the update of the hotspot data access path includes: Migration information is sent to a client communicating with the first cluster and the second cluster, the migration information carrying keywords corresponding to the hotspot data, the migration information being used to instruct the client to record the keywords corresponding to the hotspot data into a keyword set, the keyword set being used by the client to determine a target cluster corresponding to the access request, the target cluster including the first cluster and the second cluster.

5. The method according to claim 1, It is characterized in that Before sending the migration information to the client communicating with the first cluster and the second cluster, the method further includes: Read the hotspot data managed by the first cluster after the migration is completed; Verify whether the hotspot data is consistent with the hotspot data managed by the first cluster before migration.

6. The method according to any one of claims 1, 2 and 5, It is characterized in that Determine that the hotspot data managed by the first cluster includes: Calculating resource usage of each main control unit in the first cluster, and monitoring the main control unit whose resource usage exceeds a first threshold; The number of accesses to each data managed by the main control unit within a preset time period in which the resource usage exceeds the first threshold is counted, and the data whose accesses exceed the second threshold is determined as hot data.

7. The method according to claim 6, It is characterized in that Determining whether a migration condition for migrating the hotspot data to the second cluster is met includes: The total number of accesses to the data managed by the main control unit whose resource usage exceeds the first threshold is counted, and when the total number of accesses exceeds a third threshold, it is determined that the hotspot data managed by the main control unit whose resource usage exceeds the first threshold meets the migration condition.

8. The method according to any one of claims 1, 2, 5 and 7, It is characterized in that The method further comprises: monitoring the second cluster, where the second cluster manages hot data migrated from the first cluster to the second cluster; The number of accesses to the hot data within a preset time is counted; when the number of accesses to the hot data is less than a fourth threshold, the hot data is marked as non-hot data, the non-hot data is migrated to the first cluster, and an update of the access path to the non-hot data is triggered.

9. A method for accessing hotspot data. It is characterized in that include: According to a keyword in the service request, query whether a keyword set includes the keyword, the keyword set includes keywords corresponding to hot data migrated from the first cluster to the second cluster, the first cluster and the second cluster include at least one single-threaded main control unit, the single-threaded main control unit processes one access request at a time, the access request includes an access request for data, and the data accessed by the access request is jointly managed by the first cluster and the second cluster; Determine a target cluster according to the query result, where the target cluster includes the first cluster or the second cluster; Sending an access request to the target cluster, wherein the access request includes a keyword in the service request; The method further comprises: Receive migration information sent by a database server, the migration information including keywords corresponding to the hotspot data, and record the keywords corresponding to the hotspot data into the keyword set.

10. The method according to claim 9, It is characterized in that receiving first migration information sent by a database server, wherein the first migration information includes the hotspot data and a keyword corresponding to the hotspot data, and recording the hotspot data and the keyword corresponding to the hotspot data into a first set; receiving second migration information sent by the database server, the second migration information including non-hotspot data and keywords corresponding to the non-hotspot data, and recording the non-hotspot data and the keywords corresponding to the non-hotspot data into a second set; The first set and the second set are respectively used to cache data that has not been subjected to data consistency judgment.

11. The method according to claim 10, It is characterized in that Determining the target cluster based on the query results includes: If the keyword set and the second set include the keyword in the service request, and the data corresponding to the keyword in the second set is consistent with the data corresponding to the keyword managed by the second cluster, determining the first cluster as the target cluster; if they are inconsistent, determining the second cluster as the target cluster; If the keyword set does not include the keyword in the business request, the first set includes the keyword in the business request, and the data corresponding to the keyword in the first set is consistent with the data corresponding to the keyword managed by the first cluster, the second cluster is determined to be the target cluster; if they are inconsistent, the first cluster is determined to be the target cluster.

12. A network device, It is characterized in that include: A processing module, configured to determine hotspot data managed by a first cluster, wherein the first cluster includes at least one single-threaded main control unit, and the single-threaded main control unit processes one access request at a time, wherein the access request includes an access request for data; A judgment module, used to judge whether a migration condition for migrating the hot data to a second cluster is met, the second cluster includes at least one single-threaded main control unit, the data accessed by the access request is jointly managed by the first cluster and the second cluster, and the second cluster is used to manage the hot data that meets the migration condition; the main control unit is responsible for managing one or more hash slots, each of the hash slots includes one or more data, the identifier of the hash slot is determined based on a keyword of the stored data, and the identifiers of the hash slots managed by the first cluster and the second cluster are the same; A migration module, configured to migrate the hotspot data to the second cluster and trigger an update of an access path to the hotspot data when the migration condition is met; The processing module is further configured to read the hot data and a keyword corresponding to the hot data from the first cluster, and determine a first hash slot in the second cluster according to the keyword corresponding to the hot data; The migration module is further used to migrate the hot data and the keywords corresponding to the hot data to the first hash slot.

13. The network device according to claim 12, It is characterized in that The first main control unit in the first cluster and the first main control unit in the second cluster are deployed on different physical devices.

14. The network device according to claim 12 or 13, It is characterized in that The main control unit manages at least one hash slot, the identifier of the hash slot is an integer obtained by mapping data through a preset algorithm, and the hash slot includes at least one data; The hash slot managed by the first main control unit in the first cluster and the hash slot managed by the first main control unit in the second cluster have the same identifier.

15. The network device according to claim 12 or 13, It is characterized in that The network device also includes a sending module, The sending module is used to send migration information to a client communicating with the first cluster and the second cluster, the migration information including keywords corresponding to the hotspot data, the migration information being used to instruct the client to record the keywords corresponding to the hotspot data into a keyword set, the keyword set being used by the client to determine a target cluster corresponding to the access request, the target cluster including the first cluster and the second cluster.

16. The network device according to claim 12, It is characterized in that The processing module is further configured to read the hotspot data managed by the first cluster after the migration is completed, and verify whether the hotspot data is consistent with the hotspot data managed by the first cluster before the migration.

17. The network device according to any one of claims 12, 13 and 16, It is characterized in that The processing module is also used to calculate the resource usage of each main control unit in the first cluster, monitor the main control unit whose resource usage exceeds a first threshold; count the number of accesses to each data managed by the main control unit whose resource usage exceeds the first threshold within a preset time, and determine the data whose access times exceed a second threshold as hot data.

18. The network device according to claim 17, It is characterized in that The judgment module is also used to count the total number of accesses to the data managed by the main control unit whose resource usage exceeds the first threshold. When the total number of accesses exceeds a third threshold, it is determined that the hotspot data managed by the main control unit whose resource usage exceeds the first threshold meets the migration condition.

19. The network device according to any one of claims 12, 13, 16 and 18, It is characterized in that The processing module is further used to monitor the second cluster, the second cluster manages the hot data migrated from the first cluster to the second cluster; counts the number of accesses to the hot data within a preset time, and when the number of accesses to the hot data is less than a fourth threshold, marks the hot data as non-hot data; The migration module is further used to migrate the non-hotspot data to the first cluster and trigger an update of an access path to the non-hotspot data.

20. A client device, It is characterized in that include: A processing module, configured to query whether a keyword set includes the keyword according to a keyword in a business request, wherein the keyword set includes keywords corresponding to hot data migrated from a first cluster to a second cluster, wherein the first cluster and the second cluster include at least one single-threaded main control unit, and the single-threaded main control unit processes one access request at a time, wherein the access request includes a request for access to data, and the data accessed by the access request is jointly managed by the first cluster and the second cluster; and determine a target cluster according to the query result, wherein the target cluster includes the first cluster or the second cluster; The sending module is used to send an access request to the target cluster, where the access request includes the keyword in the service request.

21. The client device of claim 20, It is characterized in that The client device also includes a receiving module, The receiving module is used to receive the migration information sent by the database server, wherein the migration information includes the keywords corresponding to the hotspot data; The processing module is further used to record the keywords corresponding to the hot data into the keyword set.

22. The client device of claim 20, It is characterized in that The client device also includes a receiving module, The receiving module is used to receive first migration information sent by the database server, where the first migration information includes the hotspot data and keywords corresponding to the hotspot data; The processing module is further used to record the hotspot data and the keywords corresponding to the hotspot data into a first set; The receiving module is further used to receive second migration information sent by the database server, where the second migration information includes non-hotspot data and keywords corresponding to the non-hotspot data; The processing module is further used to record the non-hotspot data and the keywords corresponding to the non-hotspot data into a second set; The first set and the second set are respectively used to cache data that has not been subjected to data consistency judgment.

23. The client device of claim 22, It is characterized in that The processing module is further configured to determine the first cluster as the target cluster when the keyword set and the second set include the keyword in the service request, and the data corresponding to the keyword in the second set is consistent with the data corresponding to the keyword managed by the second cluster; and when they are inconsistent, determine the second cluster as the target cluster; The processing module is further configured to determine that the second cluster is a target cluster when the keyword set does not include the keyword in the service request, the first set includes the keyword in the service request, and the data corresponding to the keyword in the first set is consistent with the data corresponding to the keyword managed by the first cluster; When they are inconsistent, the first cluster is determined to be the target cluster.

24. A network device, It is characterized in that The network device comprises a memory and a processor, and the processor executes computer instructions stored in the memory, so that the network device performs the method according to any one of claims 1 to 8.

25. A client device, It is characterized in that The client device includes a memory and a processor, and the processor executes computer instructions stored in the memory, so that the client device performs the method according to any one of claims 9 to 11.