Data synchronization processing method and its device, equipment, and medium

Abstract

The present application discloses a data synchronization processing method and its apparatus, equipment, and medium, the method comprising: responding to a data update event of a target publishing topic in a local proxy database, determining a target topic group to which the target publishing topic belongs according to a topic grouping configuration; determining a synchronization implementation group associated with the target topic group according to a synchronization implementation grouping configuration, so as to query a target synchronization implementation object set in the synchronization implementation group that is responsible for synchronizing the target publishing topic; driving each synchronization implementation object in the target synchronization implementation object set to obtain updated data from the corresponding partition in the target distribution topic, so that these synchronization implementation objects update the updated data obtained by their partitions to the target subscription topics in the local and remote proxy databases. The present application groups synchronization implementation objects and topics to perform full data synchronization in the form of diversion and reduce the impact of rebalancing, and constructs a topic decoupling consumption business and production business with a publish-subscribe mechanism.

Description

Technical Field

[0001] The present application relates to the field of servers, and in particular to a data synchronization processing method, and also to corresponding devices, equipment and non-volatile storage media of the method. Background Art

[0002] Existing Internet platforms usually use distributed proxy databases to store data on the platform. For example, Internet live broadcast platforms will distribute and store the data of the platform's live broadcast rooms, anchor users, and audience users in various proxy databases to ensure that various types of users and management users on the platform can quickly obtain relevant data in the proxy database and use various types of online live broadcast services provided by the live broadcast platform. Existing Internet platforms generally build distributed proxy databases based on the Kafka concept, so that their distributed proxy databases have the advantages of Kafka, such as high throughput, persistent data storage, and easy system scalability.

[0003] In the existing Internet platform, a distributed proxy database system built on Kafka, when any proxy server adds or deletes a topic or modifies the topic partition structure, a rebalancing mechanism will be triggered to reallocate the topics consumed by the synchronization implementation objects responsible for consumption and production of each topic. Of course, rebalancing will also be performed when the synchronization implementation object is started or any synchronization implementation object stops running. However, when performing rebalancing, it is usually necessary to reallocate the consumer objects of all topics in the proxy database, that is, to reallocate the topics they need to consume to all synchronization implementation objects responsible for the proxy database. That is, the time complexity of performing rebalancing is high, and the proxy database that performs rebalancing will stop data processing, affecting the overall data processing efficiency of the distributed proxy database system.

[0004] In addition, when the platform executes online services that require full data, it needs to consume the full data of each proxy database for data aggregation, and users who apply this online service generally need to perform multiple writes to produce their own data. This makes the coupling between the consumption business and the production business in the proxy database highly coupled, making it inconvenient for system expansion and making the distributed proxy database system difficult to maintain.

[0005] In view of the problems existing in the existing distributed proxy database system built based on Kakfa, the applicant has made corresponding explorations in order to solve the problem. Summary of the Invention

[0006] The purpose of this application is to provide a data synchronization processing method to meet user needs, and also relates to corresponding devices, equipment, non-volatile storage media and computer program products of this method.

[0007] In order to achieve the purpose of this application, the following technical solutions are adopted:

[0008] A data synchronization processing method proposed for the purpose of this application includes the following steps:

[0009] In response to the data update event of the target publishing topic in the local proxy database, the target publishing topic is determined according to the topic grouping configuration to which the target publishing topic belongs;

[0010] Determine the synchronization implementation group associated with the target topic group according to the synchronization implementation group configuration, and query the target synchronization implementation object set in the synchronization implementation group that is responsible for synchronizing the target publication topic;

[0011] Drive each synchronization implementation object in the target synchronization implementation object set to obtain update data from the corresponding partition in the target distributed topic, so that these synchronization implementation objects can update the update data obtained by their partitions to the target subscription topics in the local and remote proxy databases.

[0012] In a further embodiment, before the step of responding to the data update event of the target publication topic in the local proxy database, the following steps are performed by the server:

[0013] Respond to the distributed topic creation instruction pushed by the management end and obtain the topic name contained in the creation instruction;

[0014] Create a local publishing topic and a local subscription topic named with the topic name in the local proxy database;

[0015] Push topic creation instructions to all remote proxy databases to control these proxy databases to create publishing topics and subscription topics with the same names and the same partition structures as the local publishing topics and local subscription topics.

[0016] In a further embodiment, the step of updating the target synchronization implementation group with the updated data corresponding to the target subscription topics in the local and remote proxy databases includes the following steps performed by the synchronization module:

[0017] In response to the update instruction pointed to the target publishing topic pushed by the server, according to the consumption partition identifiers corresponding to each synchronization implementation object, the consumption partitions corresponding to the consumption partition identifiers in the target publishing topic are determined;

[0018] The synchronization implementation objects are driven to obtain update data from their corresponding consumption partitions respectively, so that the synchronization implementation objects update their update data to the corresponding production partitions in the target subscription topic of the local proxy database or the remote proxy database.

[0019] In a further embodiment, the synchronization implementation objects updating their updated data to the corresponding production partitions in the target subscription topics of the local proxy database or the remote proxy database include the following steps performed by the synchronization implementation objects:

[0020] Drives the consumer objects contained in it to obtain updated data from the consumer partition;

[0021] Determine the database identifiers corresponding to each of the one or more production objects contained in the system, so as to drive these production objects to update the update data to the production partitions in the target subscription topic of the proxy database corresponding to the database identifiers.

[0022] In a further embodiment, after the step of allowing the synchronization implementation objects to update the updated data obtained by their partitions to the target subscription topics in the local and remote proxy databases, the following steps are performed by the synchronization implementation objects:

[0023] In response to a modification event of a target subject group associated with the local proxy database, an object information list corresponding to the current synchronization implementation object group is called, wherein the object information list stores object information of each synchronization implementation object in the running synchronization implementation object group, and the object information is sorted according to the object name of the synchronization implementation object to which it belongs;

[0024] Call the preset consumption rebalancing rule to obtain the consumption partition identifier corresponding to one or more target topics according to the sorting of each object information in the object information list and the topic information of each topic in the target topic group, so as to determine the consumption partition that needs to be consumed in these target topics according to the partition identifier.

[0025] In a further embodiment, after the step of allowing the synchronization implementation objects to update the updated data obtained by their partitions to the target subscription topics in the local and remote proxy databases, the following steps are performed by the synchronization implementation objects:

[0026] According to the preset synchronization protocol, monitor the object running events of the current synchronization implementation object group;

[0027] When it is monitored that the running state of any synchronous implementation object in the synchronous implementation object group is characterized as stopped running, the object information list corresponding to the synchronous implementation object group will be called, and the target subject group associated with the synchronous implementation object will be determined. The object information list stores the object information of each synchronous implementation object in the running synchronous implementation object group, and the object information is sorted according to the object name of the synchronous implementation object to which it belongs;

[0028] When a new synchronization implementation object start event acting on the synchronization implementation object group is monitored, the object information list is called and a target subject group associated with the synchronization implementation object is determined;

[0029] Call the preset consumption rebalancing rule to obtain the consumption partition identifier corresponding to one or more target topics according to the sorting of each object information in the object information list and the topic information of each topic in the target topic group, so as to determine the consumption partition that needs to be consumed in these target topics according to the partition identifier.

[0030] In a preferred embodiment, the step of calling a preset consumption rebalancing rule to obtain consumption partition identifiers corresponding to one or more target topics based on the order of each object information in the object information list and the topic information of each topic in the target topic group, and determining the consumption partitions to be consumed in these target topics based on the partition identifiers, includes the following steps performed by the synchronization implementation object:

[0031] Performing hash calculation on the subject information of any subject in the target subject group to obtain a hash value corresponding to the subject;

[0032] According to the number of object information in the object list, performing a remainder operation on the hash value and the number of information to obtain a corresponding remainder;

[0033] According to the number of partitions contained in the topic information of the topic and the residual value, the consumer object range corresponding to the topic is obtained to determine whether the sorting position of the current synchronization implementation object in the object information list is within the consumer object range. If so, the position of the sorting position in the consumer object range is the consumer partition identifier of the current synchronization implementation object acting on the topic.

[0034] A data synchronization processing device proposed for the purpose of this application includes:

[0035] A target topic group determination module is used to respond to data update events of target publishing topics in the local proxy database and determine the target topic group to which the target publishing topic belongs according to the topic group configuration;

[0036] A target object group determination module is used to determine the synchronization implementation group associated with the target topic group according to the synchronization implementation group configuration, so as to query the target synchronization implementation object set in the synchronization implementation group that is responsible for synchronizing the target publication topic;

[0037] The subscription topic update module is used to drive each synchronization implementation object in the target synchronization implementation object set to obtain updated data from the corresponding partition in the target distribution topic, so that these synchronization implementation objects can update the updated data obtained by their partitions to the target subscription topics in the local and remote proxy databases.

[0038] In a further embodiment, the subscription theme update module includes:

[0039] The consumption partition determination submodule is used to respond to the update instruction pointing to the target publishing topic pushed by the server, and determine the consumption partitions corresponding to the consumption partition identifiers in the target publishing topic according to the consumption partition identifiers corresponding to the synchronization implementation objects;

[0040] The production partition update submodule is used to drive each of the synchronization implementation objects to obtain update data from its corresponding consumption partition, so that these synchronization implementation objects update their update data to the corresponding production partition in the target subscription topic of the local proxy database or the remote proxy database.

[0041] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above-mentioned data synchronization processing method.

[0042] In order to solve the above technical problems, an embodiment of the present application further provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above-mentioned data synchronization processing method.

[0043] In order to solve the above technical problems, an embodiment of the present application further provides a computer program product, including a computer program and computer instructions. When the computer program and computer instructions are executed by a processor, the processor executes the steps of the above-mentioned data synchronization processing method.

[0044] Compared with the prior art, the advantages of this application are as follows:

[0045] This application groups the topics and synchronization implementation objects responsible for data synchronization in the distributed proxy database built based on Kafka, and associates the topic group with the synchronization implementation object group, so that the synchronization implementation object only performs data synchronization on the topics in its associated topic group, and the synchronization implementation object generally only performs synchronization on the data stored in one partition in the topic, so that the distributed proxy database system can achieve the effect of classification when synchronizing data, and when the topic of the proxy database is modified, the synchronization implementation object is started and stopped, and a new object is run, rebalancing can be performed as a group without rebalancing all the topics and synchronization implementation objects in the proxy database, so as to reduce the time complexity of rebalancing and reduce the impact on the overall data processing efficiency of the distributed proxy database system when rebalancing is performed.

[0046] Secondly, this application constructs topics in the proxy database with a subscription-publishing mechanism. Each proxy database stores publishing topics and subscription topics associated with other proxy databases. When any publishing topic of a proxy database generates new data, the synchronization implementation object set responsible for data synchronization for the publishing topic will update the newly produced data to the local and all remote proxy databases to ensure that each proxy database stores the full amount of data in the distributed proxy database system. Through the subscription-publishing mechanism, the consumption business and production business in the distributed proxy database system are decoupled, which improves the scalability of the system and prevents the mutual influence of consumption business and production business.

[0047] In addition, each synchronous implementation object of the present application will perform rebalancing business independently instead of central rebalancing, so as to prevent the inability to perform rebalancing business due to the collapse of the central method, resulting in data confusion. When the synchronous implementation object performs rebalancing, it only needs to perform hash operation based on the order of each synchronous implementation object in the same group and the feature code of the subject, thereby improving the processing efficiency of the synchronous implementation object performing rebalancing. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0049] Figure 1 A schematic diagram of a typical network deployment architecture for implementing the technical solution of this application;

[0050] Figure 2 This is a flowchart of a typical embodiment of the data synchronization processing method of the present application;

[0051] Figure 3 This is a flowchart of the specific implementation method of this application for creating publishing topics and subscribing to topics;

[0052] Figure 4 This is a flowchart of a specific implementation method for synchronously implementing an object set corresponding to an update target subscription topic in this application;

[0053] Figure 5 A flowchart of a specific implementation method for consuming and updating corresponding consumer objects and production objects in a synchronous implementation object of this application;

[0054] Figure 6 A flowchart illustrating a specific implementation method of the present application for synchronously implementing an object response to a modification event in an associated subject group to perform consumption rebalancing;

[0055] Figure 7 A flowchart of a specific implementation method of the present application for performing consumption rebalancing by monitoring a synchronization implementation object group in which the synchronization implementation object is located;

[0056] Figure 8 A flowchart of a specific implementation method for synchronously implementing consumption rebalancing of objects in this application;

[0057] Figure 9 This is a principle block diagram of a typical embodiment of the data synchronization processing device of the present application;

[0058] Figure 10 This is a basic structural block diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limiting the present application.

[0060] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0061] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0062] It will be understood by those skilled in the art that the terms "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, i.e., devices that only have wireless signal receivers without transmission capabilities, and devices that have receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers and tablet computers, which have single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service), which may combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices, which have and / or include a radio frequency receiver. As used herein, the terms "client," "terminal," or "terminal device" may be portable, transportable, or installed in a vehicle (air, sea, and / or land), or may be adapted and / or configured to operate locally and / or in a distributed manner at any other location on Earth and / or in space. As used herein, the terms "client," "terminal," or "terminal device" may also refer to a communication terminal, an Internet terminal, or a music / video playback terminal, such as a PDA, an MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or may include a smart TV, a set-top box, or other device.

[0063] The hardware referred to by names such as "server", "client", and "work node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0064] It should be noted that the concept of "server" referred to in this application can also be extended to server clusters. Based on the network deployment principles understood by those skilled in the art, the servers described should be logically divided. In physical space, these servers can be independent of each other but callable through interfaces, or integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method of this application.

[0065] See also Figure 1 , the hardware foundation required for the implementation of the relevant technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 referred to in this application is deployed in the cloud. As an online server, it can be responsible for further connecting relevant data servers and other servers that provide relevant support, etc., to form a logically related service cluster to provide services for relevant terminal devices such as the smartphone 81 and personal computer 82 shown in the figure or a third-party server (not shown). The smartphone and personal computer can both access the Internet through a well-known network access method and establish a data communication link with the server 80 in the cloud to run terminal applications related to the services provided by the server.

[0066] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions in this application that are suitable for running on the server can be implemented in the server in this way.

[0067] The application mentioned above refers to an application running on a server or terminal device. This application implements the relevant technical solutions of the present application in a programming manner. Its program code can be stored in a non-volatile storage medium that can be recognized by the computer in the form of computer-executable instructions, and can be loaded into the memory by the central processing unit for execution. The relevant device of the present application is constructed by the operation of this application on the computer.

[0068] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions in this application that are suitable for running on the server can be implemented in the server in this way.

[0069] Those skilled in the art should be aware that although the various methods of this application are described based on the same concept and thus exhibit commonality, unless otherwise specified, these methods can be independently executed. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept. Therefore, concepts with the same expression, as well as concepts that are appropriately transformed for convenience despite different expression, should be understood as equivalent.

[0070] See also Figure 2 In a typical embodiment of the present invention, a data synchronization processing method includes the following steps:

[0071] Step S11, responding to the data update event of the target publishing topic in the local proxy database, and determining the target topic group to which the target publishing topic belongs according to the topic group configuration:

[0072] The server responds to the data update event of the target publishing topic in the local proxy database to determine the target topic group corresponding to the target publishing topic according to the topic grouping configuration.

[0073] The proxy database is generally a distributed proxy database built based on Kafka technology, and the local proxy database is a proxy database located in the local server. Correspondingly, the proxy server located in other servers is generally called a remote proxy database.

[0074] The proxy database stores multiple publishing topics and subscription topics. The publishing topics store locally generated data, which are stored in partitions of the publishing topics respectively. Subscription topics are generally associated with publishing topics stored in the remote proxy database and the local proxy database. The partition structure of the subscription topic is the same as that of the associated publishing topic, and the subscription topic stores data from all associated publishing topics.

[0075] The topic generally refers to the topic in Kafka, and the topic has one or more partitions. The partition is the partiton in Kafka, and each partition in the topic stores corresponding data.

[0076] It can be understood that there is a publish-subscribe relationship between the publishing topic and the subscription topic. The synchronization implementation object in the subsequent steps is responsible for synchronizing the data produced in the publishing topic to the corresponding subscription topic. Specifically, taking the local proxy database and the remote proxy database as an example, the data and partition structure of the publishing topics and subscription topics stored in the local proxy database and the remote proxy database are the same. If there is a subscription topic A associated with the publishing topic A in the local proxy database, then there is also a publishing topic A and subscription topic A with the same name and partition structure as the local proxy database in the remote proxy database. However, when the publishing topic A in the local proxy database or the remote proxy database is updated, the updated data in the publishing topic A is updated to the subscription topic A of both the local proxy database and the remote proxy database to ensure that the subscription topic A stores the global data stored in the publishing topic A in the local and remote proxy databases associated with it.

[0077] When the local proxy database creates a new publishing topic and associated subscription topics, the server will push the topic creation instruction to all remote proxy databases so that the servers to which these remote proxy databases belong will respond to the topic creation instruction and create publishing topics and subscription topics with the same name and partition structure as the newly created publishing topics and subscription topics in the local proxy database in the remote proxy database.

[0078] The topic grouping configuration configures the topic groups to which each topic in the local proxy database belongs. The topic grouping configuration generally stores mapping relationship data consisting of topic identifiers and topic group names in the form of a table. The server uses the target topic identifier corresponding to the target publishing topic to query the target topic group name stored corresponding to the target topic identifier in the topic grouping configuration to determine the target topic group to which the target publishing topic belongs.

[0079] In addition to grouping the topics in the proxy database, the synchronization implementation objects in the server will also be grouped so that by associating the topic group with the synchronization implementation object group, the topics in the topic group will be handed over to the associated synchronization implementation object group for data synchronization, so as to prevent the need to rebalance all topics and synchronization implementation objects stored in the proxy database when rebalancing is performed in the proxy database built based on Kafka. Through grouping, when the topics in any topic group are modified, such as adding or deleting topics and modifying the partition structure of the topic, or when the synchronization implementation object of its associated synchronization implementation object group is started or stopped, it is only necessary to rebalance both parties without rebalancing all topics and synchronization implementation objects in the proxy database, so as to improve the execution efficiency of rebalancing of the proxy database and reduce the computing pressure on the server.

[0080] Step S12: Determine the synchronization implementation group associated with the target topic group according to the synchronization implementation group configuration, and query the target synchronization implementation object set in the synchronization implementation group that is responsible for synchronizing the target publication topic.

[0081] After the server determines the target topic group to which the target publishing topic belongs, it will determine the synchronization implementation group associated with the target topic based on the local synchronization implementation configuration, and then query the target synchronization implementation object set responsible for data synchronization for the target publishing topic from the synchronization implementation group. The synchronization implementation object set is composed of one or more synchronization implementation objects.

[0082] The synchronization implementation group configuration is configured with each synchronization implementation group responsible for synchronizing data for each subject group in the local database. The synchronization implementation group configuration can record each subject group and its corresponding synchronization implementation group in a table storage structure. The group name of each synchronization implementation group and the group name of the subject group associated with it are stored in the table accordingly. When the server needs to determine the synchronization implementation group associated with any subject group, it will query the synchronization implementation group name stored corresponding to the subject group name from the synchronization implementation group configuration based on the group name of the subject group to determine the synchronization implementation group associated with the subject group.

[0083] After the server determines the synchronization implementation group associated with the target topic group, it will query the synchronization implementation group for one or more synchronization implementation objects responsible for synchronizing the target publication topic. These synchronization implementation objects constitute the target synchronization implementation object set. Specifically, the synchronization implementation object generally has the topic identifier of the publication topic it is responsible for synchronizing. The server queries the synchronization implementation group for the topic identifier corresponding to the target publication topic to determine these synchronization implementation objects as the synchronization implementation objects responsible for synchronizing data for the target publication topic.

[0084] The number of synchronization implementation objects responsible for data synchronization for the publishing topic is generally determined by the number of partitions of the publishing topic. That is, one synchronization implementation object is generally responsible for data synchronization for one partition in the publishing topic to which it corresponds, so as to realize the diversion and synchronization processing of the synchronization data of the publishing topic.

[0085] The synchronization implementation object is generally composed of a consumer object and one or more production objects. The consumer object is responsible for consuming the updated data stored in a corresponding target partition in the corresponding publication topic, while the production object is responsible for updating the updated data consumed by the consumer object to the partition associated with the target partition in the subscription topic associated with the publication topic in the corresponding remote proxy database. For the specific implementation method of data synchronization for the publication topic by the synchronization implementation object, please refer to the subsequent steps. This step will not be repeated here.

[0086] Step S13: Drive each synchronization implementation object in the target synchronization implementation object set to obtain updated data from the corresponding partition of the target distributed topic, so that these synchronization implementation objects update the updated data obtained by their partitions to the target subscription topics in the local and remote proxy databases.

[0087] After determining the target synchronization implementation object set responsible for performing data synchronization for the target publishing topic, the server drives each synchronization implementation object included in the target synchronization implementation object set to perform data synchronization on the target publishing topic.

[0088] The synchronization module responsible for driving the target synchronization implementation object set receives the update instructions pointing to the target publication topic pushed by the server, so as to determine the consumption partitions corresponding to each of the consumption partition identifiers in the target publication topic according to the consumption partition identifiers corresponding to each of the synchronization implementation objects in the target synchronization implementation object set, and then drive each of the synchronization implementation objects to obtain update data from these consumption partitions respectively, so that these synchronization implementation objects update the updated data they obtain to the corresponding production partitions in the target subscription topic of the local proxy database or the remote proxy database; of course, the server can also directly drive each synchronization implementation object in the target synchronization implementation object set to perform data synchronization processing on the target publication topic.

[0089] The specific implementation method for the synchronous implementation object to update the updated data of the corresponding consumption partition in the target publication topic to the corresponding production partition in the target subscription topic in the local or remote proxy database is as follows: the consumer object in the synchronous implementation object consumes and obtains updated data from the consumption partition corresponding to its consumption partition identifier, and then drives one or more production objects contained in the synchronous implementation object to update the updated data obtained by the consumer object to the corresponding production partition in the target subscription topic in its corresponding local proxy database and remote proxy database. For example, when a production object obtains the updated data obtained by the consumer object from the consumption partition corresponding to the consumption group identifier, the production object determines the proxy database corresponding to the database identifier based on its corresponding database identifier, queries the target subscription topic with the same topic name as the target publication topic contained in the proxy database, and stores the updated data in the production partition corresponding to the consumption partition identifier in the target subscription topic.

[0090] Through the typical implementation of this method, it can be seen that this method groups the topics and synchronization implementation objects responsible for data synchronization in the distributed proxy database built based on Kafka, and associates the topic group with the synchronization implementation object group, so that the synchronization implementation object only performs data synchronization processing on the topics in its associated topic group, and the synchronization implementation object generally only performs synchronization processing on the data stored in one partition in the topic, so that the distributed proxy database system can achieve the effect of classification when synchronizing data, and when the topic of the proxy database is modified, the synchronization implementation object is started and stopped, and a new object is run, rebalancing processing can be performed as a group without rebalancing all the topics and synchronization implementation objects in the proxy database, so as to reduce the time complexity of rebalancing and reduce the impact on the overall data processing efficiency of the distributed proxy database system when rebalancing is performed.

[0091] Secondly, this method constructs topics in the proxy database with a subscription-publishing mechanism. Each proxy database stores publishing topics and subscription topics associated with other proxy databases. When any publishing topic of a proxy database generates new data, the synchronization implementation object set responsible for data synchronization for the publishing topic will update the newly produced data to the local and all remote proxy databases to ensure that each proxy database stores the full amount of data in the distributed proxy database system. In addition, through the subscription-publishing mechanism, the consumption business and production business in the distributed proxy database system are decoupled, which improves the scalability of the system and prevents the mutual influence of consumption business and production business.

[0092] The above exemplary embodiments and their variations fully disclose the implementation scheme of the data synchronization processing method of the present application. However, various variations of the method can be derived by transforming and amplifying certain technical means. Other embodiments are briefly described below:

[0093] In one embodiment, please refer to Figure 3 , before the step of responding to the data update event of the target publishing topic in the local proxy database, the server may perform the following steps:

[0094] Step S08: Respond to the distributed topic creation instruction pushed by the management terminal and obtain the topic name contained in the creation instruction:

[0095] The server receives the distributed topic instruction pushed by the management terminal, responds to the creation instruction, and obtains the topic name included in the creation instruction.

[0096] Step S09: Create a local publishing topic and a local subscription topic named after the topic name in the local proxy database:

[0097] After obtaining the topic name, the server creates the local publishing topic and the local subscription topic named with the topic name in the local proxy database. The partition structure of the local publishing topic is the same as that of the local subscription topic.

[0098] Step S10: Push topic creation instructions to all remote proxy databases to control these proxy databases to create publishing topics and subscription topics with the same name and the same partition structure as the local publishing topics and local subscription topics:

[0099] The server pushes the topic creation instruction to all remote proxy databases associated with it, so that the server with the remote proxy database responds to the topic creation instruction and creates a publishing topic and a subscription topic named after the topic name of the local publishing topic and the local subscription topic in the remote proxy database, and the partition structure of the publishing topic and the subscription topic is the same as that of the local publishing topic and the local subscription topic.

[0100] In this embodiment, when any proxy database creates a new publishing topic and subscription topic, it will push the topic creation instruction to other proxy databases associated with it to control these proxy databases to create publishing topics and subscription topics with the same name and partition structure, so that each proxy database can obtain the global data in the publishing topics of other proxy databases.

[0101] In one embodiment, please refer to Figure 4The step of updating the target synchronization implementation group to update the updated data to the target subscription topics in the local and remote proxy databases includes the following steps performed by the synchronization module:

[0102] Step S131, responding to the update instruction pointed to the target publishing topic pushed by the server, determining the consumption partitions corresponding to the consumption partition identifiers in the target publishing topic according to the consumption partition identifiers corresponding to the synchronization implementation objects;

[0103] The synchronization module responds to the update instruction pointed to the target publishing topic pushed by the server to drive each synchronization implementation object in the target synchronization implementation object to determine its consumption partition corresponding to the consumption partition identifier in the target publishing topic according to its corresponding consumption partition identifier.

[0104] Step S132: Drive each synchronization implementation object to obtain updated data from its corresponding consumption partition, so that the synchronization implementation object updates its updated data to the corresponding production partition in the target subscription topic of the local proxy database or the remote proxy database:

[0105] Each of the synchronization implementation objects determines that it publishes the corresponding consumption partition in the publishing topic, and each will obtain updated data from its corresponding consumption partition, and then these synchronization implementation objects will update the updated data they obtain to the corresponding production partition in the target subscription topic in the local proxy database or remote proxy database.

[0106] In this embodiment, the synchronization module is responsible for driving each synchronization implementation object in the target synchronization implementation object set to perform data synchronization processing acting on the publishing topic, so as to collect and manage each synchronization implementation object in the server.

[0107] In one embodiment, please refer to Figure 5 The synchronization implementation objects update their updated data to the corresponding production partitions in the target subscription topics of the local proxy database or the remote proxy database, including the following steps performed by the synchronization implementation objects:

[0108] Step S1321: Drive the consumer objects contained in it to obtain updated data from the consumer partition:

[0109] The current synchronization implementation object drives the consumer object contained in it to obtain updated data from the consumer partition corresponding to the consumer partition identifier in the target publishing topic.

[0110] Step S1322: Determine the database identifiers corresponding to the one or more production objects contained in the system, so as to drive these production objects to update the update data to the production partitions in the target subscription topic of the proxy database corresponding to their database identifiers.

[0111] After the consumer object of the current synchronization implementation object obtains the updated data, the current synchronization implementation object will drive one or more production objects contained in it to query the target subscription topic with the same topic name as the target publishing topic in the proxy database corresponding to its database identifier according to their respective corresponding database identifiers, so as to update the updated data to the production partition corresponding to the consumer partition in the target subscription topic.

[0112] In this embodiment, the synchronization implementation object provides the consumer object with updated data, and then drives the production object to update the updated data to the corresponding subscription topic, so as to achieve data synchronization between the publishing topic and the subscription topic.

[0113] In one embodiment, please refer to Figure 6 After the step of updating the update data obtained by the synchronization implementation objects in their partitions to the target subscription topics in the local and remote proxy databases, the following steps are performed by the synchronization implementation objects:

[0114] Step S14: In response to the modification event of the target subject group associated with the local proxy database, the object information list corresponding to the current synchronization implementation object group is called. The object information list stores the object information of each synchronization implementation object in the running synchronization implementation object group, and the object information is sorted according to the object name of the synchronization implementation object to which it belongs:

[0115] The current synchronization implementation object responds to the modification event of the target subject group associated with the synchronization implementation object group in the local database. The modification event is generally monitored by the server. When the server monitors the modification event acting on the target subject group, it will notify the current synchronization implementation object to respond to the modification event. The modification event is generally an event of adding a new subject, deleting an existing subject, and performing partition modification on an existing subject in the target subject group.

[0116] After the current synchronization implementation object responds to the modification event, it will call the object information list called by the current synchronization implementation object group. The object information list stores the object information of each synchronization implementation object currently running in the synchronization implementation object group, and these object information are sorted according to the object name of the synchronization implementation object to which they belong.

[0117] Step S15: Calling a preset consumption rebalancing rule to obtain consumption partition identifiers corresponding to one or more target topics based on the order of each object information in the object information list and the topic information of each topic in the target topic group, and determining the consumption partitions to be consumed in these target topics based on the partition identifiers:

[0118] After the current synchronization implementation object calls the object information list, it will call the consumption rebalancing rule of the style. The consumption rebalancing rule generally refers to the rebalance in kafka. The current synchronization object calls the consumption rebalancing rule to redetermine its consumption partition of the target topic in the target topic group.

[0119] The current synchronization implementation object determines the consumption partition identifier of one or more target topics corresponding to it according to the consumption rebalancing rule, in accordance with the order of each object information in the object information list and the topic information of each topic in the target topic group, and then determines its consumption partition in each target topic based on these consumption partition identifiers.

[0120] In this embodiment, when a modification event occurs in any topic group that affects the topics it contains, each implementation object in the synchronous implementation object group associated with it will reallocate the topics and corresponding partitions it is responsible for consuming in the topic group through the consumption rebalancing rules, forming a diversion process in a grouped and individually processed rebalancing manner.

[0121] In one embodiment, please refer to Figure 7 After the step of updating the update data obtained by the synchronization implementation objects in their partitions to the target subscription topics in the local and remote proxy databases, the following steps are performed by the synchronization implementation objects:

[0122] Step S14': monitor the object running events of the current synchronization implementation object group according to the preset synchronization protocol:

[0123] The current synchronization implementation object monitors the object running events of the current synchronization implementation object group in real time according to the synchronization protocol of the style. The object running events refer to the running status of the synchronization implementation objects in the synchronization implementation object group and the start events of the synchronization implementation objects in the synchronization implementation object group.

[0124] The synchronization protocol generally refers to the gossip, paxios or zab synchronization protocol. The synchronization implementation object perceives the running status of each synchronization implementation object in the same synchronization implementation object group and the startup event of the new synchronization implementation object in the group through the synchronization protocol.

[0125] Step S15': When it is detected that the running state of any synchronized implementation object in the synchronized implementation object group is stopped, the object information list corresponding to the synchronized implementation object group is called, and the target subject group associated with the synchronized implementation object is determined. The object information list stores the object information of each synchronized implementation object in the running synchronized implementation object group, and the object information is sorted according to the object name of the synchronized implementation object to which it belongs:

[0126] When the current synchronization implementation object monitors that the running status of any synchronization implementation object in its synchronization implementation object group is stopped, it will call the object information list called by the synchronization implementation object group and determine the target subject group associated with the synchronization implementation object group.

[0127] The object information list stores the object information of each currently running synchronous implementation object in the synchronous implementation object group, and the object information is sorted according to the object name of the synchronous implementation object to which it belongs.

[0128] Step S16': when a new synchronization implementation object start event acting on the synchronization implementation object group is monitored, the object information list is called and the target subject group associated with the synchronization implementation object is determined:

[0129] When the current synchronization implementation object monitors that a new synchronization implementation object group is started in the synchronization implementation object group in which it is located, it will also call the object information list and determine the associated target subject group.

[0130] Step S17': Call the preset consumption rebalancing rule to obtain the consumption partition identifier corresponding to one or more target topics according to the order of each object information in the object information list and the topic information of each topic in the target topic group, and determine the consumption partition to be consumed in these target topics according to the partition identifier:

[0131] After the current synchronization implementation object calls the object information list, it will call the consumption rebalancing rule of the style. The consumption rebalancing rule generally refers to the rebalance in kafka. The current synchronization object calls the consumption rebalancing rule to redetermine its consumption partition of the target topic in the target topic group.

[0132] The current synchronization implementation object determines the consumption partition identifier of one or more target topics corresponding to it according to the consumption rebalancing rule, in accordance with the order of each object information in the object information list and the topic information of each topic in the target topic group, and then determines its consumption partition in each target topic based on these consumption partition identifiers.

[0133] In this embodiment, when a modification event occurs in any topic group that affects the topics it contains, each implementation object in the synchronous implementation object group associated with it will reallocate the topics and corresponding partitions it is responsible for consuming in the topic group through the consumption rebalancing rules, forming a diversion process in a grouped and individually processed rebalancing manner.

[0134] In this embodiment, the synchronization implementation object perceives the running status of each synchronization implementation object in the same group and the newly started synchronization implementation object in the monitoring group through the synchronization protocol, so as to reallocate the topics to be consumed and the consumption partitions in the topics for itself according to the consumption rebalancing protocol.

[0135] In one embodiment, please refer to Figure 8 The step of calling the preset consumption rebalancing rule to obtain the consumption partition identifiers corresponding to one or more target topics according to the order of each object information in the object information list and the topic information of each topic in the target topic group, and determining the consumption partitions to be consumed in these target topics according to the partition identifiers, includes the following steps executed by the synchronization implementation object:

[0136] Step S151: Perform hash calculation on the topic information of any topic in the target topic group to obtain the hash value corresponding to the topic:

[0137] After the current synchronization implementation object calls the consumption rebalancing rule, it will perform a hash calculation on the subject information of any subject in the target subject group to obtain the hash value corresponding to the subject. Specifically, the current synchronization implementation object generally performs a hash calculation based on the md5 feature code or other feature code in the subject information of the subject, and uses the result of the hash calculation as the hash value of the subject.

[0138] Step S152: Perform a remainder operation on the hash value and the number of object information in the object list to obtain a corresponding remainder value:

[0139] The current synchronous implementation object performs a remainder operation on the implementation hash value and the number of object information existing in the implementation object list and the number of running synchronous implementation objects in the synchronous implementation object group where the current synchronous implementation object is located, and uses the hash value as the remainder divided by the number, and uses the remainder obtained by the operation as the remainder.

[0140] Step S153: Based on the number of partitions and the residual value contained in the topic information of the topic, the consumer object range corresponding to the topic is obtained to determine whether the sort position of the current synchronization implementation object in the object information list is within the consumer object range. If so, the position of the sort position in the consumer object range is the consumer partition identifier of the current synchronization implementation object acting on the topic:

[0141] After the current synchronization implementation object obtains the residual value, it will obtain the consumer object range corresponding to the topic based on the number of partitions represented in the topic information of the topic and the residual value. For example, when the residual value is 1 and the number of partitions of the topic is 3, the corresponding consumer object range is [1,4). If the current synchronization implementation object is ranked 2 in the object information list, the current synchronization implementation object will be responsible for consuming the topic, and the consumer partition identifier of the current synchronization implementation object acting on the topic is 2.

[0142] In this embodiment, the synchronization implementation object determines the topic it is responsible for consuming and the partition responsible for consumption of the topic in the associated topic group through hash calculation and residual operation. The corresponding consumption partition can be determined by the sorting in the object information list and the feature code and partition of the topic, thereby improving the efficiency of the synchronization implementation object in executing consumption rebalancing.

[0143] Furthermore, by functionalizing the various steps in the methods disclosed in the above embodiments, a data synchronization processing device of the present application can be constructed. Figure 9 In a typical embodiment, the device includes: a target topic group determination module 11, which is used to respond to the data update event of the target publishing topic in the local proxy database, and determine the target topic group to which the target publishing topic belongs according to the topic grouping configuration; a target object group determination module 12, which is used to determine the synchronization implementation group associated with the target topic group according to the synchronization implementation grouping configuration, so as to query the target synchronization implementation object set responsible for synchronizing the target publishing topic in the synchronization implementation group; a subscription topic update module 13, which is used to drive each synchronization implementation object in the target synchronization implementation object set to obtain update data from the corresponding partitions in the target distribution topic, so that these synchronization implementation objects can update the update data obtained by their partitions to the target subscription topics in the local and remote proxy databases.

[0144] In one embodiment, the subscription topic update module 13 includes: a consumption partition determination submodule, which is used to respond to the update instruction pointing to the target publishing topic pushed by the server, and determine the consumption partitions corresponding to each of the consumption partition identifiers in the target publishing topic according to the consumption partition identifiers corresponding to each synchronization implementation object; a production partition update submodule, which is used to drive each of the synchronization implementation objects to obtain update data from its corresponding consumption partition respectively, so that these synchronization implementation objects will update their update data to the corresponding production partitions in the target subscription topic of the local proxy database or the remote proxy database.

[0145] In order to solve the above technical problems, the embodiment of the present application further provides a computer device for running a computer program implemented according to the data synchronization processing method. Figure 10 , Figure 10 This is a basic structural block diagram of the computer device in this embodiment.

[0146] like Figure 10 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a non-volatile storage medium, a memory and a network interface connected via a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database and computer-readable instructions, and the database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a data synchronization processing method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute a data synchronization processing method. The network interface of the computer device is used to connect and communicate with the terminal. Those skilled in the art will understand that Figure 10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0147] In this embodiment, the processor is used to execute the specific functions of each module / submodule in the data synchronization processing device of this application, and the memory stores the program code and various data required to execute these modules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the data synchronization processing device, and the server can call the server's program code and data to execute the functions of all submodules.

[0148] The present application also provides a non-volatile storage medium, in which the data synchronization processing method is written into a computer program and stored in the storage medium in the form of computer-readable instructions. When the computer-readable instructions are executed by one or more processors, it means that the program is running in the computer, thereby enabling one or more processors to execute the steps of the data synchronization processing method of any of the above embodiments.

[0149] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed, the program can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0150] In summary, this application groups synchronization implementation objects and topics to synchronize full data in the form of diversion and reduce the impact of rebalancing, and uses a publish-subscribe mechanism to build topics to decouple consumer services and production services.

[0151] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0152] Those skilled in the art will appreciate that the steps, measures, and schemes in the various operations, methods, and processes discussed in this application may be interchanged, modified, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be interchanged, modified, rearranged, decomposed, combined, or deleted.

[0153] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data synchronization processing method, characterized in that: The steps include: In response to the data update event of the target publishing topic in the local proxy database, the target publishing topic is determined according to the topic grouping configuration to which the target publishing topic belongs; Determine the synchronization implementation group associated with the target topic group according to the synchronization implementation group configuration, and query the target synchronization implementation object set in the synchronization implementation group that is responsible for synchronizing the target publication topic; Drive each synchronization implementation object in the target synchronization implementation object set to obtain updated data from the corresponding partition in the target publication topic, so that these synchronization implementation objects update the updated data obtained by their partitions to the target subscription topics in the local and remote proxy databases; Before the step of responding to the data update event of the target publishing topic in the local proxy database, the following steps are performed by the server: Respond to the distributed topic creation instruction pushed by the management end and obtain the topic name contained in the creation instruction; Create a local publishing topic and a local subscription topic named with the topic name in the local proxy database; Push topic creation instructions to all remote proxy databases to control these proxy databases to create publishing topics and subscription topics with the same names and the same partition structures as the local publishing topics and local subscription topics.

2. The method according to claim 1, characterized in that The step of updating the target synchronization implementation group with the updated data to the target subscription topics in the local and remote proxy databases includes the following steps performed by the synchronization module: In response to the update instruction pointed to the target publishing topic pushed by the server, according to the consumption partition identifiers corresponding to each synchronization implementation object, the consumption partitions corresponding to the consumption partition identifiers in the target publishing topic are determined; The synchronization implementation objects are driven to obtain update data from their corresponding consumption partitions respectively, so that the synchronization implementation objects update their update data to the corresponding production partitions in the target subscription topic of the local proxy database or the remote proxy database.

3. The method according to claim 2, characterized in that The synchronization implementation objects update their updated data to the corresponding production partitions in the target subscription topics of the local proxy database or the remote proxy database, including the following steps performed by the synchronization implementation objects: Drives the consumer objects contained in it to obtain updated data from the consumer partition; Determine the database identifiers corresponding to each of the one or more production objects contained in the system, so as to drive these production objects to update the update data to the production partitions in the target subscription topic of the proxy database corresponding to the database identifiers.

4. The method according to claim 1, wherein After the step of updating the update data obtained by the synchronization implementation objects in their partitions to the target subscription topics in the local and remote proxy databases, the following steps are performed by the synchronization implementation objects: In response to a modification event of a target subject group associated with the local proxy database, an object information list corresponding to the current synchronization implementation object group is called, wherein the object information list stores object information of each synchronization implementation object in the running synchronization implementation object group, and the object information is sorted according to the object name of the synchronization implementation object to which it belongs; Call the preset consumption rebalancing rule to obtain the consumption partition identifier corresponding to one or more target topics according to the sorting of each object information in the object information list and the topic information of each topic in the target topic group, so as to determine the consumption partition that needs to be consumed in these target topics according to the partition identifier.

5. The method according to claim 1, wherein After the step of updating the update data obtained by the synchronization implementation objects in their partitions to the target subscription topics in the local and remote proxy databases, the following steps are performed by the synchronization implementation objects: According to the preset synchronization protocol, monitor the object running events of the current synchronization implementation object group; When it is monitored that the running state of any synchronous implementation object in the synchronous implementation object group is characterized as stopped running, the object information list corresponding to the synchronous implementation object group will be called, and the target subject group associated with the synchronous implementation object will be determined. The object information list stores the object information of each synchronous implementation object in the running synchronous implementation object group, and the object information is sorted according to the object name of the synchronous implementation object to which it belongs; When a new synchronization implementation object start event acting on the synchronization implementation object group is monitored, the object information list is called and a target subject group associated with the synchronization implementation object is determined; Call the preset consumption rebalancing rule to obtain the consumption partition identifier corresponding to one or more target topics according to the sorting of each object information in the object information list and the topic information of each topic in the target topic group, so as to determine the consumption partition that needs to be consumed in these target topics according to the partition identifier.

6. The method according to claim 4 or 5, characterized in that The step of calling the preset consumption rebalancing rule to obtain the consumption partition identifiers corresponding to one or more target topics according to the order of each object information in the object information list and the topic information of each topic in the target topic group, and determining the consumption partitions to be consumed in these target topics according to the partition identifiers, includes the following steps performed by the synchronization implementation object: Performing hash calculation on the subject information of any subject in the target subject group to obtain a hash value corresponding to the subject; According to the number of object information in the object information list, performing a remainder operation on the hash value and the number of information to obtain a corresponding remainder; According to the number of partitions contained in the topic information of the topic and the residual value, the consumer object range corresponding to the topic is obtained to determine whether the sorting position of the current synchronization implementation object in the object information list is within the consumer object range. If so, the position of the sorting position in the consumer object range is the consumer partition identifier of the current synchronization implementation object acting on the topic.

7. A data synchronization processing device, characterized in that: include: A target topic group determination module is used to respond to data update events of target publishing topics in the local proxy database and determine the target topic group to which the target publishing topic belongs according to the topic group configuration; A target object group determination module is used to determine the synchronization implementation group associated with the target topic group according to the synchronization implementation group configuration, so as to query the target synchronization implementation object set in the synchronization implementation group that is responsible for synchronizing the target publication topic; A subscription topic update module is used to drive each synchronization implementation object in the target synchronization implementation object set to obtain update data from the corresponding partition of the target publishing topic, so that these synchronization implementation objects update the update data obtained by their partitions to the target subscription topics in the local and remote proxy databases; Before responding to the data update event of the target publishing topic in the local proxy database, the following steps are included and executed by the server: responding to the distributed topic creation instruction pushed by the management end, obtaining the topic name contained in the creation instruction; creating a local publishing topic and a local subscription topic named with the topic name in the local proxy database; pushing the topic creation instruction to all remote proxy databases to control these proxy databases to create publishing topics and subscription topics with the same name and the same partition structure as the local publishing topic and local subscription topic.

8. The device according to claim 7, characterized in that The subscription theme update module includes: The consumption partition determination submodule is used to respond to the update instruction pointing to the target publishing topic pushed by the server, and determine the consumption partitions corresponding to the consumption partition identifiers in the target publishing topic according to the consumption partition identifiers corresponding to the synchronization implementation objects; The production partition update submodule is used to drive each of the synchronization implementation objects to obtain update data from its corresponding consumption partition, so that these synchronization implementation objects update their update data to the corresponding production partition in the target subscription topic of the local proxy database or the remote proxy database.

9. An electronic device comprising a central processing unit and a memory, characterized in that: The central processing unit is configured to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 6.

10. A non-volatile storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 6 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

Images