Data distribution information updating method and system

Abstract

The invention relates to the technical field of databases, and particularly provides a data distribution information updating method and system, and the method comprises the following steps: S1, a client communicates with a network; s2, initial triggering of data distribution information updating; and S3, data distribution information diffusion based on a message mechanism. Compared with the prior art, the method can avoid load pressure of a central node or a server due to the fact that each client carries out distributed information updating, and avoid system pressure fluctuation possibly occurring when data distribution changes.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and specifically provides a method and system for updating data distribution information. Background Art

[0002] In a distributed database, distributed data system, or distributed data solution, data distribution information is often not completely transparent to the client. In specific application scenarios or for specific business purposes, the client often needs to perceive data distribution information. A typical case is which computing or storage node a certain data shard belongs to (such as sharding information). If it is cached on the client, direct routing can be performed during data import and local transactions, thereby achieving better performance.

[0003] In a system where the client caches data distribution information, when the data storage location changes, such as the migration of the primary replica due to a node failure, or the system redirection for load balancing purposes, the client usually updates the local data distribution information in two ways:

[0004] 1) Each client relies on a globally unique metadata service or registration center to refresh data distribution information and then retries its request;

[0005] 2) For systems without a central metadata service, after receiving feedback from the server about a specific data distribution update, the client refreshes its cache information and retries its request.

[0006] These two mechanisms for updating distribution information will bring obvious system load pressure. If the client size is M, and the size of the data distribution information change caused by the cluster status change is N, the above methods require M*N requests to fully refresh all outdated distribution information. If an application accesses K data shards, the delay caused by failed retries may be K times. For heavy distributed systems, the order of magnitude of M*N may be 10,000,000. This order of magnitude of requests will bring obvious pressure to the system and also cause obvious delays to client applications.

[0007] Another problem is that the second situation above is more common in current distributed database implementations. When each node of the distributed database does not store all the distribution information, updating metadata of M*N scale will cause communication between service nodes of the same scale, which will amplify the request traffic and latency brought by this architecture. Summary of the invention

[0008] The present invention aims at solving the above-mentioned deficiencies of the prior art and provides a data distribution information updating method with strong practicability.

[0009] A further technical task of the present invention is to provide a data distribution information updating system that is reasonably designed, safe and applicable.

[0010] The technical solution adopted by the present invention to solve its technical problem is:

[0011] A method for updating data distribution information comprises the following steps:

[0012] S1, the client's Unicom network;

[0013] S2, initial triggering of data distribution information update;

[0014] S3. Data distribution information diffusion based on message mechanism.

[0015] Further, in step S1, when each client is started, the existence of the same client is detected in the neighbor network and the local machine according to the default port and protocol or the configurable port and protocol;

[0016] After detection, verify whether the other party's version information is compatible and can collaborate, cache the information confirming the compatible and collaborative client in the client, and perform heartbeat detection at a certain time interval;

[0017] Eventually, each client will have zero or more neighbor client information, and all clients will be divided into multiple client clusters that are directly or indirectly connected to each other according to their distributed network locations.

[0018] Furthermore, in step S2, the mechanism for triggering the update of data distribution information is still that when the server-side distribution information changes, when the first client holding outdated distribution information accesses the original data holding node, the server feedbacks that its request is invalid and points out that the accessed data belongs to another data node.

[0019] Furthermore, in step S3, when a client receives update information of data distribution information from the server, it will refresh the local data distribution information cache, and load the serial number, source node, and timestamp propagation related information for this update information based on the time and version key information of the last update information it received, and send update messages to other clients it has cached.

[0020] Furthermore, each client maintains a list of recent message sequence numbers that it has processed and node information of messages with this sequence number that it has received;

[0021] When a client receives data distribution information update information from other clients, it first determines whether the sequence number and source node key information of this update message are already processed update messages;

[0022] If it is an old message, it is ignored; otherwise, its own data distribution information cache is updated, and a data distribution information update message is sent to each assisting client in its cache except the end that sent the information this time.

[0023] A data distribution information update system. First, the client's network connection, all clients are divided into multiple client clusters directly or indirectly connected to each other according to their distributed network locations. Then, the initial trigger of the data distribution information update, the data distribution information is diffused based on the message mechanism.

[0024] Furthermore, in the client's connected network, when each client is started, the existence of the same client is detected in the neighbor network and the local machine according to the default port and protocol or the configurable port and protocol;

[0025] After detection, verify whether the other party's version information is compatible and can collaborate, cache the information confirming the compatible and collaborative client in the client, and perform heartbeat detection at a certain time interval;

[0026] Eventually, each client will have zero or more neighbor client information, and all clients will be divided into multiple client clusters that are directly or indirectly connected to each other according to their distributed network locations.

[0027] Furthermore, the mechanism for triggering the update of data distribution information is still that when the server-side distribution information changes, when the first client holding outdated distribution information accesses the original data holding node, the server feedbacks that its request is invalid and points out that the accessed data belongs to another data node.

[0028] Furthermore, in the data distribution information diffusion based on the message mechanism, when a client receives the update information of the data distribution information from the server, it will refresh the local data distribution information cache, and load the serial number, source node, timestamp propagation related information for this update information according to the time and version key information of the last update information it received, and send update messages to other clients it caches.

[0029] Furthermore, each client maintains a list of recent message sequence numbers that it has processed and node information of messages with this sequence number that it has received;

[0030] When a client receives data distribution information update information from other clients, it first determines whether the sequence number and source node key information of this update message are already processed update messages;

[0031] If it is an old message, it is ignored; otherwise, its own data distribution information cache is updated, and a data distribution information update message is sent to each assisting client in its cache except the end that sent the information this time.

[0032] Compared with the prior art, the data distribution information updating method and system of the present invention have the following outstanding beneficial effects:

[0033] The present invention enables the client to obtain updated data distribution information with lower latency when the data distribution of a distributed database or a distributed system changes, thereby reducing the number of error retries and the latency in responding to application requests.

[0034] At the same time, this invention can avoid the load pressure on the central node or server caused by each client updating the distribution information, and avoid the system pressure fluctuation that may occur when the data distribution changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Attached Figure 1 It is a schematic diagram of a client connection network in a data distribution information update method;

[0037] Attached Figure 2 It is a schematic diagram of initial triggering of updating data distribution information in a data distribution information updating method;

[0038] Attached Figure 3 The present invention is a schematic diagram of data distribution information diffusion based on a message mechanism in a data distribution information update method. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] A best embodiment is given below:

[0041] A data distribution information updating method in this embodiment has the following steps:

[0042] S1, the client's Unicom network;

[0043] When each client starts, it detects the presence of the same client in neighboring networks and on the local machine based on the default port and protocol or configurable ports and protocols.

[0044] After detection, verify whether the other party's version and other information are compatible and collaborative, cache the information confirming compatible and collaborative clients in the client, and perform heartbeat detection at certain time intervals.

[0045] Eventually, each client will have zero or more neighbor client information. From a global perspective, all clients will be divided into multiple client clusters that are directly or indirectly connected to each other according to their distributed network locations.

[0046] like Figure 1 As shown in the figure, this figure illustrates the process of a new client forming a connected network with other clients after it is started. The circle with a dotted border on the top represents the client that has just been started; the dotted line indicates that it is exploring similar clients in the local area network by broadcasting. It means that the exploration is completed and all the cooperative clients form a new connected network.

[0047] S2, initial triggering of data distribution information update;

[0048] The mechanism for triggering the update of data distribution information is still that when the server-side distribution information changes, when the first client holding outdated distribution information accesses the original data holding node, the server responds that its request is invalid and points out that the data it is accessing belongs to another data node.

[0049] like Figure 2 As shown in the figure, when the server node svc4 fails and cannot provide services, due to data rebalancing, part of the data of the svc3 node is transferred to svc2. At this time, the request sent by client 1 to svc3 according to the previous distribution information will receive feedback from svc3, requiring the client to redirect the request to svc2.

[0050] S3, data distribution information diffusion based on message mechanism;

[0051] When a client receives updated data distribution information from the server, it refreshes the local data distribution information cache, and loads the serial number, source node, timestamp and other propagation related information for this update information based on the time, version and other key information of the last update information it received, and sends update messages to other clients in its cache.

[0052] Each client maintains a list of the sequence numbers of recent messages it has processed and the node information of the messages received with this sequence number. When a client receives updated data distribution information from other clients, it first determines whether the key information such as the sequence number and source node of this update message is an updated message that has been processed. If it is an old message, it will be ignored; otherwise, it will update its own data distribution information cache and send data distribution information update messages to each client that can assist except the one that sent this message.

[0053] like Figure 3 As shown in the figure, when client 1 receives feedback from the server, it will first update the distribution information of chunk n data in its own cache.

[0054] Afterwards, client 1 will package this update information together with the timestamp, source node and other information into a message such as {seq1,c1,n:svc2} and send it to the nodes it knows (as represented by the connected network marked by the dotted arrows in the figure).

[0055] Assuming that client 2 and client 3 receive the {seq1,c1,n:svc2} message at the same time, they will refresh their own node caches of chunk n distribution information, and update the source node information with this message and send it to known nodes other than client 1. The message bodies are {seq1,c2,n:svc2} and {seq1,c3,n:svc2} respectively.

[0056] All clients that receive the message will authenticate based on the sequence number seq1 in the message. If the message with this sequence number has been processed by the client, it will be ignored. For client 2 in the figure, suppose it receives the message from client 3 later than the message from client 1. When it finds that it has already processed the message from client 3, it will not process it anymore.

[0057] The above is only one implementation method, which is optimized to send update messages to only a part of cached cooperative clients each time.

[0058] Other possible optimizations include sending full or partial data distribution information for updates each time, or spreading the update message after a certain period of time has passed since the client received it.

[0059] Based on the above method, a data distribution information update system in this embodiment, first, the client's connected network, all clients will be divided into multiple client clusters that are directly or indirectly connected to each other according to their distributed network locations, and then, the initial trigger of the data distribution information update, the data distribution information is diffused based on the message mechanism.

[0060] Among them, in the client's Unicom network, when each client is started, the existence of the same client is detected in the neighbor network and the local machine according to the default port and protocol or the configurable port and protocol;

[0061] After detection, verify whether the other party's version information is compatible and can collaborate, cache the information confirming the compatible and collaborative client in the client, and perform heartbeat detection at a certain time interval;

[0062] Eventually, each client will have zero or more neighbor client information, and all clients will be divided into multiple client clusters that are directly or indirectly connected to each other according to their distributed network locations.

[0063] The mechanism for triggering the update of data distribution information is still that when the server-side distribution information changes, when the first client holding outdated distribution information accesses the original data holding node, the server responds that its request is invalid and points out that the accessed data belongs to another data node.

[0064] In the data distribution information diffusion based on the message mechanism, when a client receives the update information of the data distribution information from the server, it will refresh the local data distribution information cache, and load the serial number, source node, timestamp and other related information for this update information according to the time and version key information of the last update information it received, and send update messages to other clients in its cache.

[0065] Each client maintains a list of recent message sequence numbers it has processed and node information for messages with the same sequence numbers it has received;

[0066] When a client receives data distribution information update information from other clients, it first determines whether the sequence number and source node key information of this update message are already processed update messages;

[0067] If it is an old message, it is ignored; otherwise, its own data distribution information cache is updated, and a data distribution information update message is sent to each assisting client in its cache except the end that sent the information this time.

[0068] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that comply with the claims of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0069] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for updating data distribution information, characterized in that: The steps are as follows: S1, the client's Unicom network; S2, initial triggering of data distribution information update; S3. Data distribution information diffusion based on message mechanism.

2. A method for updating data distribution information according to claim 1, characterized in that: In step S1, when each client starts, it detects the existence of the same client in the neighbor network and the local machine according to the default port and protocol or the configurable port and protocol; After detection, verify whether the other party's version information is compatible and can collaborate, cache the information confirming the compatible and collaborative client in the client, and perform heartbeat detection at a certain time interval; Eventually, each client will have zero or more neighbor client information, and all clients will be divided into multiple client clusters that are directly or indirectly connected to each other according to their distributed network locations.

3. A method for updating data distribution information according to claim 2, characterized in that: In step S2, the mechanism for triggering the update of data distribution information is still that when the server-side distribution information changes, when the first client holding outdated distribution information accesses the original data holding node, the server feedbacks that its request is invalid and points out that the accessed data belongs to another data node.

4. A method for updating data distribution information according to claim 3, characterized in that: In step S3, when a client receives update information of data distribution information from the server, it refreshes the local data distribution information cache, and loads the serial number, source node, and timestamp propagation related information for this update information based on the time and version key information of the last update information it received, and sends update messages to other clients in its cache.

5. A method for updating data distribution information according to claim 4, characterized in that: Each client maintains a list of recent message sequence numbers it has processed and node information for messages with the same sequence numbers it has received; When a client receives data distribution information update information from other clients, it first determines whether the sequence number and source node key information of this update message are already processed update messages; If it is an old message, it is ignored; otherwise, its own data distribution information cache is updated, and a data distribution information update message is sent to each assisting client in its cache except the end that sent the information this time.

6. A data distribution information updating system, characterized in that: First, the client's interconnected network, all clients will be divided into multiple client clusters that are directly or indirectly connected to each other according to their distributed network locations. Then, the initial trigger of data distribution information update, the data distribution information diffusion based on the message mechanism.

7. A data distribution information updating system according to claim 6, characterized in that: In the client's connected network, when each client starts, it detects the existence of the same client in the neighbor network and the local machine according to the default port and protocol or the configurable port and protocol; After detection, verify whether the other party's version information is compatible and can collaborate, cache the information confirming the compatible and collaborative client in the client, and perform heartbeat detection at a certain time interval; Eventually, each client will have zero or more neighbor client information, and all clients will be divided into multiple client clusters that are directly or indirectly connected to each other according to their distributed network locations.

8. A data distribution information updating system according to claim 7, characterized in that: The mechanism for triggering the update of data distribution information is still that when the server-side distribution information changes, when the first client holding outdated distribution information accesses the original data holding node, the server responds that its request is invalid and points out that the accessed data belongs to another data node.

9. A data distribution information updating system according to claim 8, characterized in that: In the data distribution information diffusion based on the message mechanism, when a client receives the update information of the data distribution information from the server, it will refresh the local data distribution information cache, and load the serial number, source node, timestamp and other related information for this update information according to the time and version key information of the last update information it received, and send update messages to other clients in its cache.

10. A data distribution information updating system according to claim 9, characterized in that: Each client maintains a list of recent message sequence numbers it has processed and node information for messages with the same sequence numbers it has received; When a client receives data distribution information update information from other clients, it first determines whether the sequence number and source node key information of this update message are already processed update messages; If it is an old message, it is ignored; otherwise, its own data distribution information cache is updated, and a data distribution information update message is sent to each assisting client in its cache except the end that sent the information this time.

Images