Cluster type data management system and node management method

Abstract

The invention discloses a cluster type data management system and a node management method, the cluster type data management system manages each data node through a registration center, including filter deployment, capacity expansion and contraction and the like, that is to say, cluster management can be carried out on Bloom filters deployed on the data nodes through the registration center; bloom filters on all data nodes can be added, modified or deleted in a one-stop mode, service requirements can be quickly responded, and mass data can be processed more efficiently compared with a Bloom filter carried by a single machine; according to the node management method, each data node is mapped on a Hash ring by constructing the Hash ring and corresponding Hash calculation, and node management and Bloom filter management are performed on the basis, so that uniform storage of data can be realized, more storage resources are saved, and the problem of data synchronization among the nodes is avoided.

Description

Technical Field

[0001] The present invention relates to cluster data management technology, and in particular to a cluster data management system and a node management method. Background Art

[0002] Bloom filter is a data structure used for querying set members. It is implemented through hash functions and bit arrays. It has the characteristics of fast query, allowing a certain false positive rate, but no false negative. At present, most Bloom filter implementations are mainly based on stand-alone memory and are installed on the business service itself. Data synchronization requires full synchronization of all clusters, which will affect the stability of business services and is not easy to manage. When new business needs are connected, the interface needs to be redeveloped. When the data synchronization of the filter is complex in a distributed cluster scenario and the data volume is huge, the stand-alone memory filter will have memory pressure. Summary of the invention

[0003] In view of this, the present invention aims to propose a clustered data management system and a node management method for clustered management of Bloom filters. The system can configure Bloom filters for different scenarios and different needs, support higher data storage capacity, and thus solve the problem of insufficient memory of Bloom filters based on single-machine memory.

[0004] In a first aspect, the present invention provides a cluster data management system, including a registration center, a management terminal, a business service terminal and a data node;

[0005] The data nodes are configured with Bloom filters; the registration center is connected to the business service end and the data nodes respectively, and is configured to manage the Bloom filters on each data node according to the request of the management end or the business service end.

[0006] Furthermore, a backup Bloom filter is configured on the data node.

[0007] Furthermore, the above system also includes an API interface connected to a registration center.

[0008] In a second aspect, the present invention provides a node management method, which is executed at a registration center and includes:

[0009] Receive node registration request from data node;

[0010] Build a hash ring, perform hash calculation on the node registration request, and obtain the mapping position of the node on the hash ring;

[0011] The mapping relationship between storage mapping location and data node;

[0012] receiving a Bloom filter management request, the Bloom filter management request including a node identifier and a Bloom filter instance identifier;

[0013] The mapping position on the data node hash ring is determined according to the node identifier, and the Bloom filter instance management is performed on the data node corresponding to the mapping position according to the Bloom filter instance identifier.

[0014] Furthermore, the node registration request includes the IP address of the data node, and the node registration request is hashed to obtain the mapping position of the node on the hash ring, including:

[0015] The consistent hashing algorithm is used to hash the IP address to obtain the mapping position of the node on the hash ring.

[0016] Further, the Bloom filter management request includes a request identifier, determining a mapping position on the data node hash ring according to the node identifier, and performing Bloom filter instance management in the data node corresponding to the mapping position according to the Bloom filter instance identifier includes:

[0017] The mapping position of the data node on the hash ring is determined according to the request identifier, and the Bloom filter instance is created, deleted or modified on the determined data node according to the Bloom filter instance identifier.

[0018] Furthermore, the above method also includes:

[0019] Receive node management requests from the management end or business service end;

[0020] Perform hash calculation on the node management request to determine the mapping position of the node on the hash ring;

[0021] The consistent hashing algorithm is used to transfer the data of the management node to other data nodes.

[0022] In a third aspect, the present invention provides a node management device, which is deployed in a registration center and includes:

[0023] A first receiving module, used to receive a node registration request from a data node;

[0024] The first hash calculation module is used to build a hash ring, perform hash calculation on the node registration request, and obtain the mapping position of the node on the hash ring;

[0025] A storage module, used for storing the mapping relationship between the mapping position and the data node;

[0026] A second receiving module, configured to receive a Bloom filter management request, the Bloom filter management request including a node identifier and a Bloom filter instance identifier;

[0027] The filter management module is used to determine the mapping position of the data node on the hash ring according to the node identifier, and perform Bloom filter instance management in the data node corresponding to the mapping position according to the Bloom filter instance identifier.

[0028] Furthermore, the above device also includes:

[0029] A third receiving module is used to receive a node management request from a management terminal or a business service terminal;

[0030] A second hash calculation module, used to perform hash calculation on the node management request to determine the mapping position of the node on the hash ring;

[0031] The data transfer module is used to transfer the data of the management node to other data nodes using the consistent hashing algorithm.

[0032] In a fourth aspect, the present invention provides an electronic device, comprising a memory storing computer executable instructions and a processor, wherein when the computer executable instructions are executed by the processor, the device executes each step of the node management method provided in the second aspect.

[0033] In a fifth aspect, the present invention provides a readable storage medium storing a computer executable program, which, when executed, can implement the various steps of the node management method provided in the second aspect.

[0034] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0035] The present invention provides a cluster data management system and a node management method, wherein the cluster data management system manages each data node through a registration center, including filter deployment, expansion and contraction, etc., that is, the Bloom filters deployed on the data nodes can be clustered and managed through the registration center, and the Bloom filters on each data node can be added, modified or deleted in a one-stop manner, so as to quickly respond to business needs and process massive data more efficiently than the Bloom filters installed on a single machine; the node management method maps each data node on the hash ring by constructing a hash ring and corresponding hash calculation, and performs node management and Bloom filter management based on this, so as to realize uniform data storage, save more storage resources, and avoid the problem of data synchronization between nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0037] Figure 1 is an architecture diagram of a cluster data management system provided by an embodiment of the present invention;

[0038] Figure 2is a flowchart of a node management method provided by an embodiment of the present invention;

[0039] Figure 3 It is a schematic diagram of adding a new node to a hash ring provided by an embodiment of the present invention;

[0040] Figure 4 It is a schematic diagram of deregistering a node on a hash ring provided by an embodiment of the present invention;

[0041] Figure 5 is a schematic diagram of writing data on a hash ring provided by an embodiment of the present invention;

[0042] Figure 6 is a schematic diagram of the structure of a node management device provided by an embodiment of the present invention;

[0043] Figure 7 An electronic device architecture diagram provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Cluster management refers to the unified management and coordination of each node in a cluster through centralized or distributed control in a distributed system. Its goal is to ensure the reliability, efficiency and flexibility of the cluster, so that the system can automatically adapt to load changes and dynamic changes of nodes. Cluster management usually includes functions such as node monitoring, load balancing, resource allocation, and fault recovery.

[0046] Taking into account the business needs of querying different data sources, in order to realize fast query of data based on a certain field condition in different data sources, the following embodiments of the present invention are based on the cluster management concept, provide a cluster data management system and a node management method, and cluster management of Bloom filters. The system can configure Bloom filters for different scenarios and different needs, support higher data storage capacity, and thus solve the problem of insufficient memory of Bloom filters based on single-machine memory.

[0047] See also Figure 1 An embodiment of the present invention provides a cluster data management system 100 , including a registration center 102 , a management terminal 104 , a business service terminal 106 and a data node 108 .

[0048] Among them, a Bloom filter 1081 is configured on each data node 108, and the registration center 102 is connected to the business service end 106 and the data node 108 respectively. The registration center 102 is configured to manage the Bloom filter 1081 on each data node 108 according to the request of the management end 104 or the business service end 106.

[0049] In this embodiment, the registration center 102 can provide proxy, parsing, node management and API capabilities to complete the registration of each data node. It can also receive read and write requests from the business service end 106 for parsing and forward the request to the data node where the data is located. At the same time, the data node can be dynamically expanded or reduced to cope with failures or emergency needs. As the core coordination module of the system, the registration center is responsible for managing and coordinating the communication between data nodes, business service ends and management ends, maintaining the registration information of all data nodes, including their status, health status and Bloom filter configuration information, and dynamically adjusting the configuration of nodes or Bloom filters on nodes according to the request of the management end or business service end.

[0050] The management end 104 can be configured as a visual and interactive operating platform, which can add, delete, modify and query the Bloom filters on the data nodes according to business needs. It can also directly send a request to the registration center to instruct the registration center to make configuration changes to the Bloom filters of certain data nodes, such as adding new Bloom filters, modifying the parameters of existing filters, or deleting certain Bloom filters that are no longer needed; it can also actively expand or reduce the capacity of nodes.

[0051] The business service end 106 is the user interface of the system and is responsible for submitting business requests to the registration center 102. These requests include data query requests, Bloom filter configuration change requests, etc. Specifically, the business service end can request to add, delete or modify Bloom filters on specific data nodes through the registration center according to current business needs. For example, when the amount of certain business queries increases, the business service end can request to increase the Bloom filter on the corresponding node to improve query efficiency. The business service end is mainly used to process user requests from the outside, so its behavior is often dynamic and can adjust the configuration requirements of nodes or Bloom filters according to business changes.

[0052] The data node 108 maintains a connection with the registration center 102, and performs operations such as adding, deleting, and modifying the Bloom filter according to the commands issued by the registration center 102. A Bloom filter can be configured on each data node to quickly determine the existence of data items to improve query efficiency. The data node needs to ensure that the Bloom filter configuration is updated in a timely manner to meet business needs. For example, when new data is received, the target data can be quickly filtered and located according to the configuration information of the Bloom filter.

[0053] In a further embodiment, Figure 1 As shown, a backup Bloom filter 1082 is also configured on the data node 108. Data synchronization will be performed between the main Bloom filter 1081 and the backup Bloom filter 1082, and the data processing logic of the backup Bloom filter 1082 is consistent with the main Bloom filter 1081. When the main Bloom filter 1081 fails, the backup Bloom filter 1082 can be started to process data.

[0054] In a further embodiment, the system 100 further includes an API interface 110 connected to the registration center 102 for interaction between the management end, the business service end and the registration center, and communication between the registration center and the data nodes.

[0055] The cluster data management system and its internal modules mentioned above can be servers corresponding to the systems in the embodiments of the present invention. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0056] See also Figure 2~Figure 6 Based on the cluster data management system provided in the above embodiments, the following embodiment of the present invention provides a node management method, including the following steps:

[0057] Step S210: Receive a node registration request from a data node.

[0058] Step S220: Build a hash ring, perform hash calculation on the node registration request, and obtain the mapping position of the node on the hash ring.

[0059] Step S230: Store the mapping relationship between the mapping position and the data node.

[0060] Step S240: Receive a Bloom filter management request, where the Bloom filter management request includes a node identifier and a Bloom filter instance identifier.

[0061] Step S250: Determine the mapping position of the data node on the hash ring according to the node identifier, and perform Bloom filter instance management in the data node corresponding to the mapping position according to the Bloom filter instance identifier.

[0062] Specifically, steps S210~S230 can be considered as the data node registration process. When the system starts, the data node registers with the registration center and reports its node information (such as IP, load status, Bloom filter configuration, etc.). The registration center performs hash calculation on the node registration request to determine the mapping position of the node on the hash ring.

[0063] In a further embodiment, a consistent hashing algorithm is used to calculate the mapping position of the data node on the hash ring. Specifically, a series of hash functions in the consistent hashing algorithm are used to perform hash calculations on the IP address of the data node to obtain a hash value, and the hash value is 2 32 The result of the modulo operation is an integer. The position of the integer on the hash ring is the mapping position of the data node. The registration center will store the mapping relationship between the mapping position and the data node.

[0064] In a further embodiment, for the system, the completion of node registration means that a new node has been added to the system. Considering the data storage characteristics of the consistent hash ring, in order to avoid problems with data synchronization between nodes, the data on the successor node on the hash ring relative to the new node needs to be synchronized to the new node.

[0065] For example, Figure 3 As shown in the figure, node 1 and node 2 are the currently mapped nodes on the hash ring. According to the data storage characteristics of the consistent hash ring, the hash value calculated by the hash algorithm for the data is mapped on the hash ring. Going clockwise along the ring, the first data node relative to the location of the data is the location where the data is stored or processed. Figure 3 In the figure, data 1 is stored in node 2, and data 2 is stored in node 1. When node 3 requests registration from the registration center and completes it, it becomes a newly added data node. In the clockwise direction, node 1 is the successor node of node 3. The first data node of data 2 originally stored in node 1 becomes node 3 in the clockwise direction. Therefore, a copy of the current data 2 needs to be made to node 3. After that, if a new request is mapped between nodes 2 and 3, it will be stored or processed in node 3.

[0066] The above steps S240 to S250 can be considered as the management process of the Bloom filter by the registration center, that is, when the management end or the business service end transmits a management request for the Bloom filter, the registration center will notify the data node to perform instance addition, deletion, modification and query operations.

[0067] Specifically, the request usually carries a request identifier, a Bloom filter instance identifier, a data key, etc. The request can come from the management end or the business service end, or from an external request accessed by an API. After the request enters the registration center, the content is parsed and mapped to a specific data node.

[0068] The node identifier is used to determine the location of the data node on the hash ring, and the request is directed to the data node. The consistent hash ring allows the Bloom filter instances to be evenly distributed among multiple nodes to achieve load balancing of the nodes. Therefore, if the node identifier is not directly mapped to a node on the hash ring, the node closest to it in the clockwise direction can be found to take over the request. After finding the target data node, the target Bloom filter instance is located on the data node according to the Bloom filter instance identifier in the request.

[0069] The instance is added, deleted or updated according to the request type. For new and updated requests, the request content also carries the configuration parameters of the Bloom filter, such as type, error rate, capacity, name, etc. Specifically, if the management request is to add a Bloom filter, a new Bloom filter instance is created on the target node according to the configuration parameters carried in the request; if the management request is to delete a Bloom filter, it is deleted on the target node according to the Bloom filter instance identifier carried in the request to release resources; if the management request is to change a Bloom filter, the parameters of the Bloom filter instance on the target node are modified according to the configuration parameters carried in the request, such as increasing the size of the filter or adjusting the error rate.

[0070] In a further embodiment, to ensure the fault tolerance of the system, the registration center needs to regularly synchronize the status of all nodes to ensure that the distribution information of the Bloom filter instances is consistent. When a node fails or exits, the registration center finds the next node mapped to it on the hash ring and reallocates the Bloom filter instances on it to ensure data accessibility. When a new node joins, the registration center will recalculate the hash ring and transfer some Bloom filter instances to the new node to achieve load balancing.

[0071] In a further embodiment, the registration center can receive external triggers from the management end, or the business service end, or the fault emergency strategy, or the API interface to manage the nodes, including adding and canceling the nodes. The newly added nodes can refer to the node registration process of the aforementioned embodiment, which will not be repeated here.

[0072] Node deregistration is the process of system scaling down, which can include the following steps:

[0073] Step S260: Receive a node management request from the management end or the business service end;

[0074] Step S270: Perform hash calculation on the node management request to determine the mapping position of the node on the hash ring;

[0075] Step S280. Use the consistent hashing algorithm to transfer the data of the management node to other data nodes.

[0076] Specifically, similar to the node registration process, the mapping position of the node on the hash ring is located through hash calculation. The registration center uses the consistent hashing algorithm to perform hash calculation on the node identifier of the deregistered node, such as the node IP address, to find the mapping position of the node on the hash ring, so as to accurately determine the virtual position of the deregistered node so that data can be smoothly transferred in the ring. In the consistent hashing algorithm, when a node is removed from the hash ring, the data it is responsible for will be taken over by the next node in the hash ring in a clockwise direction (called the successor node). The successor node will receive and store the data on the deregistered node, thereby ensuring the continuity and availability of the data.

[0077] For example, Figure 4 As shown, there are currently three nodes on the hash ring, namely node 1, node 2, and node 3. Data 1 is stored in node 2, and data 2 is stored in node 3. Now node 3 needs to be deleted. After completing the node verification step, data 2 needs to be copied and synchronized to the successor node in the clockwise direction, that is, node 1. The registration center removes the deregistered node in the hash ring and updates the mapping status of each node in the system to ensure that subsequent requests will no longer point to the deregistered data node.

[0078] In a further embodiment, based on the aforementioned node management and Bloom filter management, data written by the management end or the business service end can be read and written by hashing the nodes on the ring.

[0079] Specifically, the consistent hashing algorithm is used to calculate the hash value of the data key, and the hash value is mapped to the hash ring and the corresponding processing node is determined in a clockwise direction. After the node is determined, data is read or written. For data reading, if the read is successful, it means that the data requested to be read is stored on the located data node. After the read is completed, the read state is returned. If the read fails, the redundant data on the successor node of the current node is read, and then the read state is returned. Similarly, for data writing, if the write is successful, the data can be written to the successor node of the current node as redundant data at the same time, so that it can be obtained from the successor node when the current node fails. If the write fails, the write state is directly returned.

[0080] For example, Figure 5 As shown, data 1 is instructed to be written to node 2 and is simultaneously written to node 3 as redundant data; data 2 is instructed to be written to node 3 and is simultaneously written to node 1 as redundant data.

[0081] The above-mentioned embodiments provide a clustered data management system and a node management method, wherein the clustered data management system manages each data node through a registration center, including filter deployment, expansion and contraction, etc., that is, the Bloom filters deployed on the data nodes can be clustered and managed through the registration center, and the Bloom filters on each data node can be added, modified or deleted in one stop, so as to quickly respond to business needs. Compared with the Bloom filters installed on a single machine, it can process massive data more efficiently; the node management method maps each data node to the hash ring by constructing a hash ring and the corresponding hash calculation, and performs node management and Bloom filter management based on this, which can achieve uniform data storage, save more storage resources, and avoid the problem of data synchronization between nodes.

[0082] The above disclosed method can be implemented by using various forms of equipment, so the present invention also discloses a node management device corresponding to the above method, and a specific embodiment is given below for detailed description.

[0083] like Figure 6 As shown, an embodiment of the present invention provides a node management device, which is deployed in a registration center and includes:

[0084] The first receiving module 601 is used to receive a node registration request from a data node;

[0085] A first hash calculation module 602 is used to construct a hash ring, perform hash calculation on the node registration request, and obtain a mapping position of the node on the hash ring;

[0086] The storage module 603 is used to store the mapping relationship between the mapping position and the data node;

[0087] A second receiving module 604 is used to receive a Bloom filter management request, where the Bloom filter management request includes a node identifier and a Bloom filter instance identifier;

[0088] The filter management module 605 is used to determine the mapping position on the data node hash ring according to the node identifier, and perform Bloom filter instance management in the data node corresponding to the mapping position according to the Bloom filter instance identifier.

[0089] In a further embodiment, the above device further comprises:

[0090] The third receiving module 606 is used to receive a node management request from the management end or the business service end;

[0091] A second hash calculation module 607, used to perform hash calculation on the node management request to determine the mapping position of the node on the hash ring;

[0092] The data transfer module 608 is used to transfer the data of the management node to other data nodes using a consistent hashing algorithm.

[0093] The implementation principle and technical effects of the device provided in the embodiments of the present application are the same as those of the aforementioned method embodiments. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiments.

[0094] The methods and related devices mentioned in the above embodiments are described with reference to the method flow charts and / or structural diagrams provided in the embodiments of the present application. Specifically, each process and / or block in the method flow charts and / or structural diagrams, as well as the combination of processes and / or blocks in the flow charts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0095] The following embodiments are described by taking the method applied to a computer device as an example. It can be understood that the computer device can be any device with computing and processing functions, and can be but not limited to a server or a personal laptop computer, etc. In one embodiment, the computer device can be an application server, and the application server can be a server for running an application to be tested.

[0096] See also Figure 7, which shows a hardware block diagram of an electronic device, the electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0097] like Figure 7 As shown, the electronic device includes: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0098] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0099] The processor 1 may be a central processing unit CPU, or an application-specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

[0100] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), etc., such as at least one disk memory;

[0101] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to: implement each processing flow of the aforementioned node management solution.

[0102] An embodiment of the present invention further provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processing flow of the node management solution provided in the above embodiment and / or any possible implementation method in combination with the embodiment is implemented.

[0103] The above-mentioned embodiments have described the present invention in particular detail with respect to possible scenarios, and those skilled in the art will recognize that the present invention can be practiced through other embodiments. The specific naming of components, the capitalization of terms, attributes, data structures, or any other programming or structural aspects are not mandatory or important, and the mechanisms or features of the present invention may have different names, forms, or procedures. The system may be implemented by a combination of hardware and software (as described), entirely by hardware elements, or entirely by software elements. The specific division of functions between the various system components described herein is exemplary only and not mandatory; on the contrary, the functions performed by a single system component may be performed by multiple components, or the functions performed by multiple components may be performed by a single component.

[0104] Those skilled in the art should understand that the various steps of the above disclosed method can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0105] These computing device executable programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors, and these computing programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0106] Certain aspects of the present invention include process steps and instructions described herein in the form of algorithms. It should be noted that the process steps and instructions of the present invention can be implemented in software, firmware and / or hardware, and when implemented by software, it can be downloaded, stored on different platforms used by various operating systems and operated from the platforms.

[0107] Those skilled in the art will understand that the structures shown in the accompanying drawings are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the terminal device to which the scheme of the present application is applied. The specific terminal device may include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

[0108] Some embodiments may be described using "coupled" and "connected" and their derivatives. It should be understood that these terms are not intended to be synonymous with each other. For example, some embodiments may be described using the word "connected" to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the word "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. The embodiments are not limited to this context.

[0109] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "possible design" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0110] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0111] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cluster data management system, characterized in that: Including registration center, management terminal, business service terminal and data node; The data nodes are configured with Bloom filters; the registration center is connected to the business service end and the data nodes respectively, and is configured to manage the Bloom filters on each data node according to the request of the management end or the business service end.

2. The system according to claim 1, characterized in that The data node is also configured with a standby Bloom filter.

3. The system according to claim 1, characterized in that The system also includes an API interface connected to a registration center.

4. A node management method, characterized in that: Executed at the registry, including: Receive node registration request from data node; Build a hash ring, perform hash calculation on the node registration request, and obtain the mapping position of the node on the hash ring; The mapping relationship between storage mapping location and data node; receiving a Bloom filter management request, the Bloom filter management request including a node identifier and a Bloom filter instance identifier; The mapping position of the data node on the hash ring is determined according to the node identifier, and the Bloom filter instance management is performed on the data node corresponding to the mapping position according to the Bloom filter instance identifier.

5. The method according to claim 4, characterized in that The node registration request includes the IP address of the data node, and performing hash calculation on the node registration request to obtain the mapping position of the node on the hash ring includes: The consistent hashing algorithm is used to hash the IP address to obtain the mapping position of the node on the hash ring.

6. The method according to claim 4, characterized in that The Bloom filter management request includes a request identifier, and determining a mapping position of a data node on a hash ring according to the node identifier, and performing Bloom filter instance management on a data node corresponding to the mapping position according to the Bloom filter instance identifier includes: The mapping position on the data node hash ring is determined according to the request identifier, and the Bloom filter instance is created, deleted or modified on the determined data node according to the Bloom filter instance identifier.

7. The method according to claim 4, characterized in that The method further comprises: Receive node management requests from the management end or business service end; Perform hash calculation on the node management request to determine the mapping position of the node on the hash ring; The consistent hashing algorithm is used to transfer the data of the management node to other data nodes.

8. A node management device, characterized in that: Deployed in the registration center, including: A first receiving module, used to receive a node registration request from a data node; The first hash calculation module is used to build a hash ring, perform hash calculation on the node registration request, and obtain the mapping position of the node on the hash ring; A storage module, used for storing the mapping relationship between the mapping position and the data node; A second receiving module, configured to receive a Bloom filter management request, the Bloom filter management request including a node identifier and a Bloom filter instance identifier; The filter management module is used to determine the mapping position of the data node on the hash ring according to the node identifier, and perform Bloom filter instance management on the data node corresponding to the mapping position according to the Bloom filter instance identifier.

9. The device according to claim 8, characterized in that The device also includes: A third receiving module is used to receive a node management request from a management terminal or a business service terminal; A second hash calculation module, used to perform hash calculation on the node management request to determine the mapping position of the node on the hash ring; The data transfer module is used to transfer the data of the management node to other data nodes using the consistent hashing algorithm.

10. An electronic device, comprising a memory storing computer executable instructions and a processor, wherein when the computer executable instructions are executed by the processor, the device executes the node management method according to any one of claims 4 to 7.

Images