Data synchronization method and device, computer equipment and storage medium

Abstract

The invention provides a data synchronization method and device, computer equipment and a storage medium. The method comprises the following steps: when a data updating event in a first storage cluster is detected, acquiring library table information corresponding to the data updating event; performing synchronization rule searching in a preset synchronization rule base based on the base table information to obtain a target synchronization rule; extracting a synchronization field from the target synchronization rule, and extracting update content from a library table corresponding to the library table information based on the synchronization field; generating a synchronization event message based on the update content and the rule label of the target synchronization rule; and sending the synchronization event message to a message queue, so that the synchronization execution component calls a write data interface of at least one second storage cluster indicated by the target synchronization rule based on the synchronization event message after subscribing to the synchronization event message in the message queue, and synchronizes the update content to the second storage cluster. According to the method, the synchronization of the other part of data among the storage clusters can be realized on the basis of keeping the part of data in the storage clusters independent.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data synchronization method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Data synchronization is the process of copying data from one system to another. Data synchronization can be performed between different data storage nodes in the same data storage cluster, or between different data storage clusters.

[0003] Different systems can share data through data synchronization, but not all data needs to be shared across different systems. For example, to ensure data security and compliance, data from multiple financial institutions (such as banks and securities companies) cannot be linked or accessed by each other. However, for wealth management platforms, multiple financial institutions need to share data to ensure the platform's normal operation. Therefore, in different data clusters, some data needs to be stored independently, while other data needs to be shared across different data storage clusters.

[0004] In related technologies, there is a lack of methods to achieve synchronization of another part of the data between storage clusters while maintaining the independent storage of some data in the data storage cluster. Summary of the Invention

[0005] This disclosure provides a data synchronization method, apparatus, computer device, and storage medium. The method can achieve data synchronization between storage clusters while maintaining the independence of some data in the storage cluster.

[0006] According to one aspect of this disclosure, a data synchronization method is provided, the method comprising:

[0007] When a data update event is detected in the first storage cluster, the database table information corresponding to the data update event is obtained;

[0008] Based on the information in the database table, a synchronization rule is searched in a preset synchronization rule base to obtain the target synchronization rule;

[0009] Extract the synchronization field from the target synchronization rule, and extract the update content from the database table corresponding to the database table information based on the synchronization field;

[0010] A synchronization event message is generated based on the rule tags of the updated content and the target synchronization rule;

[0011] The synchronization event message is sent to a message queue so that after the synchronization execution component subscribes to the synchronization event message in the message queue, it calls the write data interface of at least one second storage cluster indicated by the target synchronization rule based on the synchronization event message to synchronize the updated content to the at least one second storage cluster.

[0012] According to one aspect of this disclosure, a data synchronization device is provided, the data synchronization device comprising:

[0013] The acquisition unit is used to acquire the database table information corresponding to the data update event when a data update event is detected in the first storage cluster.

[0014] The search unit is used to search for synchronization rules in a preset synchronization rule base based on the information in the database table to obtain the target synchronization rule;

[0015] The extraction unit is used to extract the synchronization field from the target synchronization rule and extract the update content from the database table corresponding to the database table information based on the synchronization field.

[0016] A generation unit is used to generate a synchronization event message based on the rule tags of the updated content and the target synchronization rule;

[0017] The sending unit is used to send the synchronization event message to the message queue, so that after the synchronization execution component subscribes to the synchronization event message in the message queue, it calls the write data interface of at least one second storage cluster indicated by the target synchronization rule based on the synchronization event message to synchronize the updated content to the at least one second storage cluster.

[0018] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0019] The first acquisition subunit is used to acquire the first tag information of the database tables contained in the multiple storage clusters to be synchronized, as well as the second tag information of the fields contained in each database table.

[0020] The second acquisition subunit is used to acquire the data management specifications associated with the plurality of storage clusters;

[0021] A sub-unit is constructed to build multiple synchronization rules based on the data management specifications, the first tag information, and the second tag information.

[0022] Optionally, in some embodiments, the building subunit includes:

[0023] The first building module is used to build preset synchronization rule building prompts;

[0024] The second construction module is used to call a preset large language model to construct synchronization rules based on the synchronization rule construction prompts, the data management specifications, the first tag information, and the second tag information, thereby obtaining multiple synchronization rules output by the preset large language model.

[0025] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0026] The display subunit is used to display multiple synchronization rules output by the preset large language model;

[0027] A determining subunit is configured to receive a filtering operation on the plurality of synchronization rules, and determine a plurality of selected synchronization rules among the plurality of synchronization rules based on the filtering operation;

[0028] Add a sub-unit to add the selected synchronization rules to a preset synchronization rule library.

[0029] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0030] The third acquisition subunit is used to acquire updated data, which includes the first sub-update data of the first tag information, the second sub-update data of the second tag information, and the third sub-update data of the data management specification.

[0031] The update subunit is used to generate a rule update instruction to update the preset synchronization rule base based on the update data, and to update the preset synchronization rule base according to the rule update instruction.

[0032] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0033] The parsing subunit is used to receive synchronization event messages in the message queue and parse the synchronization event messages to obtain the rule tag and synchronization event attribute data of the target synchronization rule, wherein the synchronization event attribute data includes the update content.

[0034] The first verification subunit is used to determine the target synchronization rule in the preset synchronization rule base based on the rule tag, and to verify the synchronization event attribute data according to the target synchronization rule to obtain the first verification result.

[0035] The first extraction subunit is used to extract at least one second storage cluster information to be synchronized from the target synchronization rule when the first verification result indicates that the verification is passed.

[0036] The first synchronization subunit is used to call at least one write data interface corresponding to the at least one second storage cluster information to synchronize the updated content to at least one second storage cluster corresponding to the at least one second storage cluster information.

[0037] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0038] The fourth acquisition subunit is used to acquire the historical synchronization rule corresponding to the historical version of the rule tag when the first verification result indicates that the verification fails.

[0039] The second verification subunit is used to verify the updated content based on the historical synchronization rules and obtain a second verification result;

[0040] The second extraction subunit is used to extract at least one second storage cluster information to be synchronized from the target synchronization rule when the second verification result indicates that the verification is passed.

[0041] The second synchronization subunit is used to call at least one write data interface corresponding to the at least one second storage cluster information to synchronize the updated content to at least one second storage cluster corresponding to the at least one second storage cluster information.

[0042] Optionally, in some embodiments, the first verification subunit includes:

[0043] The extraction module is used to extract reference source storage cluster information and reference database table information from the target synchronization rule, and to extract first storage cluster information and database table information from the synchronization event attribute data;

[0044] The matching module is used to match the reference source storage cluster information with the first storage cluster information to obtain a first matching result, and to match the reference library table information with the library table information to obtain a second matching result;

[0045] The determination module is used to determine a first verification result based on the first matching result and the second matching result.

[0046] Optionally, in some embodiments, the extraction unit includes:

[0047] A generation subunit is used to extract synchronization fields from the target synchronization rule and generate a read data request based on the synchronization fields;

[0048] The receiving subunit is used to call the read data interface of the first storage cluster based on the read data request, and to receive the updated content returned by the read data interface. The read data interface returns the updated content after the permission verification of the read data request is qualified.

[0049] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0050] The lookup subunit is used to obtain the database table information and the synchronization field from the read data request, and to perform a synchronization rule lookup in the preset synchronization rule base based on the database table information and the synchronization field.

[0051] The instruction subunit is used to indicate that the permission verification of the read data request has passed when the corresponding target synchronization rule is found in the preset synchronization rule base according to the database table information and the synchronization field.

[0052] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0053] The verification subunit is used to obtain preset synchronization data verification rules, verify the updated content synchronized to the at least one second storage cluster according to the synchronization data verification rules, and obtain the verification result.

[0054] The third synchronization subunit is used to call the write data interface of at least one of the second storage clusters indicated by the target synchronization rule based on the synchronization event message when the verification result indicates that the updated content synchronized to any of the second storage clusters is inconsistent with the updated content of the database tables in the first storage cluster, or when the verification result indicates that there is updated content in the message queue that has not been synchronized to the second storage clusters, to synchronize the unsynchronized updated content to the corresponding second storage cluster.

[0055] According to one aspect of this disclosure, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the data synchronization method described above.

[0056] According to one aspect of this disclosure, a storage medium is provided that stores a computer program, which, when executed by a processor, implements the data synchronization method described above.

[0057] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the data synchronization method as described above.

[0058] The data synchronization method provided in this disclosure, when a data update event is detected in a first storage cluster, obtains the database table information corresponding to the data update event; searches for synchronization rules in a preset synchronization rule base based on the database table information to obtain a target synchronization rule; extracts synchronization fields from the target synchronization rule, and extracts updated content from the database table corresponding to the database table information based on the synchronization fields; generates a synchronization event message based on the updated content and the rule tag of the target synchronization rule; and sends the synchronization event message to a message queue, so that after the synchronization execution component subscribes to the synchronization event message in the message queue, it calls the write data interface of at least one second storage cluster indicated by the target synchronization rule based on the synchronization event message to synchronize the updated content to at least one second storage cluster.

[0059] This disclosure utilizes message queues for data synchronization between different data clusters, decoupling the storage cluster sending and receiving synchronization data. This avoids modifying the architecture of the data storage clusters, thus resolving issues such as increased architectural complexity, higher maintenance costs, and lower data synchronization accuracy caused by modifications to the storage cluster architecture during data synchronization in related technologies. The synchronization rule base provided in this disclosure stores synchronization rules that define the data that can be synchronized between different storage clusters. Therefore, when a data update event is detected, the synchronization rules can be used to determine whether data synchronization is possible. This allows for the synchronization of another portion of data between storage clusters while maintaining the independence of some data within each cluster.

[0060] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0061] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0062] Figure 1 A system architecture diagram for the data synchronization method used in embodiments of this disclosure;

[0063] Figure 2 A flowchart illustrating the data synchronization method provided in this disclosure;

[0064] Figure 3 A schematic diagram of the data synchronization request curve provided in this disclosure;

[0065] Figure 4Another flowchart illustrating the data synchronization method provided in this disclosure;

[0066] Figures 5A to 5D A schematic diagram illustrating the specific process of the data synchronization method provided in this disclosure;

[0067] Figures 6A to 6B A schematic diagram illustrating the application of the data synchronization method provided in this disclosure to a wealth management platform;

[0068] Figure 7 This is a schematic diagram of the reconciliation interface provided in this disclosure;

[0069] Figure 8 A schematic diagram of the structure of the data synchronization device provided in the embodiments of this disclosure;

[0070] Figure 9 This is a server structure diagram illustrating the implementation of various methods according to an embodiment of the present disclosure. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0072] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0073] Database (DB) instance: A database instance is a running instance of a database management system used to manage and provide read, write, and access to data stored in the database.

[0074] Database clusters: A database cluster is a group of database instances that can coordinate and cooperate to improve the availability and scalability of the database. Each database cluster can be a complete business unit, independently providing access to a type of business data, and the data between multiple database clusters is usually kept independent. Common database cluster types include master-slave replication, multi-master replication, and sharded clusters.

[0075] Master-Slave Replication: In a master-slave replication cluster, one database instance acts as the master node, and the other database instances act as slave nodes. The master node handles all write operations and replicates the data to the slave nodes; the slave nodes typically handle read operations, thus reducing the load on the master node. This clustering approach can improve read performance, but write operations remain a single point of failure.

[0076] Multi-master replication: Multi-master replication clusters allow multiple database instances to handle read and write operations simultaneously. Each instance can act as a master node, and data is replicated across all nodes. While this approach can improve read and write performance, it also introduces complexities related to data consistency and conflict resolution.

[0077] Message Queues (MQ): Message queues are containers that store messages during message transmission, serving as a communication method for passing messages between applications. The typical model for message queues is the producer-consumer model. Message queues are characterized by decoupling and asynchronicity. Producers continuously produce messages into the message queue, and consumers continuously retrieve messages from the queue. Because message production and consumption are asynchronous, there is no intrusion of business logic during this process, thus achieving decoupling between producers and consumers.

[0078] Data consistency: In a distributed system, data consistency refers to the requirement that data replicas stored across multiple storage nodes maintain the same state and values ​​to ensure data accuracy and reliability. In distributed systems, inconsistencies may arise due to factors such as data replication, sharding, and concurrent execution of distributed transactions, necessitating the guarantee of data consistency. Database data consistency is categorized by timeliness into strong consistency, weak consistency, and eventual consistency. Strong consistency means that after a write operation, all storage nodes must immediately synchronize, ensuring that all storage nodes possess the same data. This consistency guarantees complete data consistency and is typically used in applications with extremely high data consistency requirements, such as financial transactions and e-commerce.

[0079] Data synchronization enables data sharing across different databases and data storage clusters, ensuring data reliability and consistency. It is widely used in financial services and e-commerce. However, current data synchronization solutions only support data synchronization between different storage nodes (e.g., databases) within a single data storage cluster. For example, a synchronization solution between different database instances in a relational database management system (MySQL) can only perform data synchronization within a single MySQL data storage cluster. This solution uses master-slave replication, where one master node handles write operations, and multiple slave nodes handle read operations. The master node synchronizes updated data to the slave nodes via a binary log (binlog) file. The binlog records SQL statements used by the service object to update the database, such as SQL statements that modify database tables and their contents (excluding queries that modify table content). These statements are stored in the binlog and synchronized to other slave nodes. This approach only supports data synchronization within a single data storage cluster (a group of database instances), maintaining data consistency through master-slave replication. It cannot support data synchronization across multiple data storage clusters, limiting its application scenarios.

[0080] Another example is the data synchronization scheme in the Hadoop Distributed File System (HDFS). This scheme divides the data being written into multiple blocks, with each block having multiple replicas stored on different data storage nodes within the HDFS. By default, the first replica is stored on the node where the data was written, while the others are stored on different racks. Essentially, this data synchronization method uses master-slave replication among multiple database instances for data synchronization.

[0081] Therefore, data synchronization solutions in related technologies only support data synchronization between different storage nodes in a single data storage cluster. However, using master-slave replication for data synchronization across multiple database instances leads to multi-source replication problems. Specifically, multi-source replication requires adjustments to the database architecture, with slave nodes needing to connect to multiple master nodes (i.e., source databases). Each slave node requires additional configuration before synchronizing data with a master node, such as network and routing configurations. When two or more source database instances update the same data record, processing logic conflicts may occur, ultimately leading to increased database architecture complexity, higher maintenance costs, poor configuration flexibility, and difficulty in guaranteeing data accuracy.

[0082] However, with the continuous advancement of technology and daily life, an increasing number of business services are emerging, and complex business scenarios require the synchronization of different business data across different data storage clusters. For example, multinational corporations need to share data among their subsidiaries worldwide to support business operations in different regions. Similarly, e-commerce platforms need to synchronize inventory data across logistics warehouses in multiple regions to ensure the accuracy and efficiency of order processing. With the continuous exploration of researchers in this field, some technical solutions for data synchronization between different data storage clusters have emerged, expanding the application scope of data synchronization. However, not all data needs to be shared across different data storage clusters. For example, to ensure data security and compliance, data from multiple financial institutions (such as banks and securities companies) cannot be linked or accessed by each other. However, for wealth management platforms, multiple financial institutions need to share data to ensure the platform's normal operation. Therefore, in different data clusters, some data needs to be stored independently, while other data needs to be shared. Currently, there is a lack of methods to synchronize another portion of data between storage clusters while maintaining the independent storage of some data within a single cluster.

[0083] Based on this, this disclosure provides a data synchronization method that aims to achieve data synchronization between storage clusters while maintaining the independence of some data within the storage cluster.

[0084] System architecture and scenario description of the embodiments disclosed herein

[0085] Figure 1 This is a system architecture diagram of a data synchronization method applied according to an embodiment of the present disclosure. It includes a first storage cluster 140, a second storage cluster 130, a data synchronization control device 120, a server 110, etc.

[0086] Server 110 refers to a computer system capable of providing certain services to terminals. Compared to ordinary terminals, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0087] The first storage cluster 140 may include multiple data storage nodes, each of which can be a server 110. The second storage cluster 130 may include multiple data storage nodes, each of which can be a server 110. The data synchronization control device 120 may be a server 110 or a proxy server. The data synchronization control device 120 may include a diffusion component and a synchronization execution component. The data synchronization control device 120 can realize data synchronization between the first storage cluster 140 and the second storage cluster 130.

[0088] The data synchronization method provided in this embodiment can be performed in a data synchronization control device 120. When the data synchronization control device 120 detects a data update event in the first storage cluster 140 (source storage cluster), it obtains the database table information corresponding to the data update event. Then, the data synchronization control device 120 can perform a synchronization rule search in a preset synchronization rule library based on the database table information to obtain a target synchronization rule. Further, the data synchronization control device 120 extracts a synchronization field from the target synchronization rule and extracts the updated content from the database table corresponding to the database table information based on the synchronization field. A synchronization event message is generated based on the updated content and the rule tag of the target synchronization rule. Finally, the data synchronization control device 120 sends the synchronization event message to a message queue, so that the synchronization execution component in the data synchronization control device 120, after subscribing to the synchronization event message in the message queue, calls the write data interface of at least one second storage cluster 130 (target storage cluster) indicated by the target synchronization rule based on the synchronization event message to synchronize the updated content to at least one second storage cluster 130.

[0089] The data synchronization method provided in this disclosure can be specifically applied to data synchronization scenarios between different data storage clusters. For example, it can be used to synchronize data from various financial institutions involved in a wealth management platform, or to synchronize inventory and order data from warehouses in different regions of an e-commerce platform.

[0090] When the data synchronization method provided in this disclosure is applied to a wealth management platform, the account information of wealth management product holders stored on the platform can be synchronized to the various financial institutions involved in the wealth management (such as banks, securities companies, and fund companies). For example, if a wealth management investor purchases wealth management products from securities company A and fund company B on the platform, and makes a remittance through bank C to purchase the corresponding products, the investor's account information (such as account name, age, and identity information) can be synchronized among securities company A, fund company B, and bank C. Each financial institution is an independently deployed data storage cluster. When a change occurs in a wealth management investor's account information, such as a change in identity information, the updated data can be synchronized to the corresponding financial institution. Specifically, when a data update event is detected in the first storage cluster (e.g., Bank C), the database table information corresponding to the data update event can be obtained; based on the database table information, a synchronization rule is searched in a preset synchronization rule base to obtain the target synchronization rule; the synchronization field is extracted from the target synchronization rule, and the updated content (e.g., updated identity information) is extracted from the database table corresponding to the database table information based on the synchronization field; a synchronization event message is generated based on the updated content and the rule tag of the target synchronization rule; the synchronization event message is sent to a message queue, so that after the synchronization execution component subscribes to the synchronization event message in the message queue, it calls the write data interface of at least one second storage cluster (e.g., Securities Company A and Fund Company B) indicated by the target synchronization rule based on the synchronization event message to synchronize the updated content to at least one second storage cluster.

[0091] For example, when this embodiment of the disclosure is applied to an e-commerce platform, data from warehouses in different regions can be synchronized. This includes synchronizing consumer e-commerce account information and order information (product name, quantity, and shipping address, etc.), as well as statistically analyzing inventory data for each warehouse. Therefore, upon receiving an order, the inventory information of the warehouse closest to the shipping address can be quickly located, and goods can be picked up from the warehouse with available stock, improving logistics speed. Furthermore, if a particular product is out of stock in one warehouse, goods can be quickly picked up from another warehouse with sufficient stock, further improving delivery speed. Specifically, when a data update event is detected in the first storage cluster (e.g., warehouse A), the database table information corresponding to the data update event can be obtained; based on the database table information, a synchronization rule is searched in a preset synchronization rule base to obtain the target synchronization rule; the synchronization field is extracted from the target synchronization rule, and the updated content (e.g., updated order information or account information) is extracted from the database table corresponding to the database table information based on the synchronization field; a synchronization event message is generated based on the updated content and the rule tag of the target synchronization rule; the synchronization event message is sent to a message queue, so that after the synchronization execution component subscribes to the synchronization event message in the message queue, it calls the write data interface of at least one second storage cluster (e.g., warehouse B and warehouse C) indicated by the target synchronization rule based on the synchronization event message, and synchronizes the updated content to at least one second storage cluster.

[0092] The above examples do not limit the scope of protection in this case.

[0093] General Description of Embodiments in this Disclosure

[0094] According to one embodiment of this disclosure, a data synchronization method is provided, which can be used in the aforementioned data synchronization scenarios. For example... Figure 2 The diagram shown is a flowchart of a data synchronization method provided in this disclosure. This method can be applied to a data synchronization device, which can be integrated into a computer device, specifically a server. The data synchronization method may include:

[0095] Step 210: When a data update event is detected in the first storage cluster, obtain the database table information corresponding to the data update event.

[0096] As disclosed above, the data synchronization method provided in this disclosure can be applied to various business scenarios to synchronize data between different entities, with each entity corresponding to a data storage cluster. In different business scenarios, different entities have different meanings. For example, in a wealth management business scenario, different entities can represent different financial companies, each of which is an independently deployed data storage cluster; in an e-commerce platform, different entities can represent different online shopping platforms, each of which is an independently deployed data storage cluster; different entities can also represent logistics warehouses in different regions, each of which can independently deploy a data storage cluster.

[0097] In this embodiment, the first storage cluster can represent either the storage cluster that generates the business update event or the storage cluster that provides the data to be synchronized. In a business scenario, a business update event can be generated in any entity, meaning that a business update event can be generated in any storage cluster. For example, in a financial platform, if a financial institution generates a business update event and initiates a data synchronization request, and can provide the updated data to be synchronized, then the storage cluster maintained by that financial institution is the first storage cluster. The first storage cluster can include multiple data storage nodes, and each data storage node can include multiple database instances. When the first storage cluster detects an event update, it can initiate a data synchronization request; please refer to [reference needed]. Figure 3 ,like Figure 3 As shown, the data synchronization request curve contains 21 records. The number of records displayed per page can be set, with 18 records displayed per page as shown in the figure. The horizontal axis represents the time when a subject initiates a data synchronization request, and the vertical axis represents the number of times a data synchronization request is initiated within different time periods. The solid line represents the data synchronization request curve recorded on year 'aa', month 'b', day 'c', and the dashed line represents the data synchronization request curve recorded on year 'aa', month 'b', day 'd'. Specifically, it includes the number of data synchronization requests in several time periods between "xx:x1:x2", "xx:x3:x4", "xx:x5:x6", and "xx:x7:x8". The figure only shows the relationship curve between time and the corresponding number of data synchronization requests. It can also record the subjects that initiate data synchronization requests within the same time period and the corresponding number of data synchronization requests. The data dimensions displayed on the data synchronization request curve can be set according to actual needs and are not limited here.

[0098] Data update events represent events that occur within a business process and can be captured and processed. These events trigger business logic to handle them, further altering the business status or data. For example, in an e-commerce platform, a data update event can represent an order status change (such as a change in the delivery address or quantity of goods in an order), allowing the changed order to be synchronized across various logistics warehouses. A data update event can also represent a warehouse inventory change, allowing the inventory quantity of a specific product in warehouse A to be synchronized to other warehouses. As another example, in a wealth management platform, a data update event can represent a change in the identity information of a wealth management product holder.

[0099] Furthermore, data update events can trigger operations on database tables, and changes to database table information can also trigger data update events. When a data update event is detected in the first storage cluster, the corresponding database table can be located in the first storage cluster based on the event type and content, and the corresponding database operation, such as inserting, updating, or deleting data, can be performed. For example, when a data update event is detected indicating a change in the account information (e.g., account ID or contact information) of a wealth management product holder, the database storing the account information can be located in the first storage cluster, then the table containing the holder's information can be found, and the table information can be updated. As another example, when a data update event is detected indicating a change in the wealth management risk assessment result of a wealth management product holder in the first storage cluster, the database storing the wealth management risk assessment result can be located in the first storage cluster, then the table containing the holder's wealth management risk assessment result can be found, and the table information can be updated.

[0100] Furthermore, when a data update event is detected in the first storage cluster, the corresponding database table information can be obtained. This information can include the database name, the table name (alias or actual name) within the database, and the table's unique primary key. The unique primary key can consist of one or more fields and is used to uniquely identify a row of data in the table. All fields in a database table with a unique primary key can be synchronized. First, the type and content of the update event, as well as the database type, can be determined based on the data update event. Then, the corresponding database can be located based on the database type and its name. Further, the corresponding database table information can be searched within the database based on the content of the data update event and the table's index information.

[0101] Step 220: Based on the information in the database table, search for synchronization rules in the preset synchronization rule database to obtain the target synchronization rule.

[0102] After obtaining the database table information corresponding to the data update events in the first storage cluster through the aforementioned steps, synchronization rules can be searched in a preset synchronization rule base based on the database table information. The synchronization rule base can include multiple synchronization rules, which can specify the data that can be synchronized in different data storage clusters, as well as information such as the source database of the data to be synchronized and its corresponding target database list. Each synchronization rule corresponds to a database table, and the target synchronization rule can represent the corresponding synchronization rule found based on the database table information. Before introducing the specific attributes of the synchronization rules, the construction process of the synchronization rules in the synchronization rule base will be explained in detail.

[0103] In some embodiments, the process of constructing synchronization rules in a preset synchronization rule base specifically includes the following steps:

[0104] Obtain the first tag information of the database tables contained in the multiple storage clusters to be synchronized, as well as the second tag information of the fields contained in each database table;

[0105] Obtain data management specifications associated with multiple storage clusters;

[0106] Multiple synchronization rules are constructed based on data management specifications, first tag information, and second tag information.

[0107] Specifically, the system can obtain the first tag information of the tables in multiple storage clusters to be synchronized, and the second tag information of the fields in each table. Each storage cluster can include multiple databases, and each database can contain multiple tables. The tag information helps to quickly query and identify the tables and fields in the storage cluster. The first tag information of a table represents its basic metadata, including the table name, creation time, database to which the table belongs, table type, number of rows, and table type. The second tag information of each field represents its basic metadata, including the field name, field length, data type, and index information in the table. Both the first and second tag information can be obtained through database queries, database maintenance tools, or by writing program code to connect to the database and obtain the tag information; there are no restrictions here.

[0108] Furthermore, data management specifications associated with multiple storage clusters can be obtained. These specifications can define data synchronization standards between multiple storage clusters in a specific business scenario. Specifically, this includes the data stored and maintained independently by each storage cluster, at least two storage clusters that can synchronize data, the data that can be synchronized between any two or more storage clusters, the synchronization direction (e.g., synchronization from storage cluster A to storage cluster B), and the industry standards and legal regulations that must be met for data synchronization between multiple storage clusters. For example, in wealth management, data management specifications can include financial regulatory standards and wealth management platform regulatory standards. Financial regulatory standards can stipulate that multiple financial institutions should store data independently, with each financial institution being a financial entity and each corresponding to a data storage cluster. This means that data between multiple financial entities cannot be directly linked, nor can data be accessed across entities. Wealth management platform regulatory standards stipulate that the account information of wealth management clients from multiple financial institutions should be linked within the wealth management platform, and non-sensitive data between multiple storage clusters should be synchronized in real time.

[0109] After obtaining the first tag information of the database tables in the multiple storage clusters to be synchronized, the second tag information of the fields in each database table, and the corresponding data management specifications, multiple synchronization rules can be constructed based on the data management specifications, the first tag information, and the second tag information.

[0110] In some embodiments, multiple synchronization rules are constructed based on data management specifications, first tag information, and second tag information, specifically including the following steps:

[0111] Construct preset synchronization rules and generate prompt words;

[0112] The preset large language model is invoked to construct synchronization rules based on the prompt words for the synchronization rules, the data management specifications, the first tag information, and the second tag information, resulting in multiple synchronization rules output by the preset large language model.

[0113] In this embodiment, after obtaining the information required for constructing synchronization rules, a preset large language model can be invoked to construct the synchronization rules. The preset large language model can be any deep learning model with abundant training data that performs well on various natural language processing tasks; no limitation is imposed here. Synchronization rule construction prompts can be used to initiate synchronization rule construction instructions to the large language model. These prompts may include first label information, second label information, and prompts corresponding to data management specifications from multiple storage clusters, as well as the configuration format and attribute content of the synchronization rules. The synchronization rule construction prompts can provide an attribute template for constructing synchronization rules, which may include identifiers of the fillable attribute content in the synchronization rules. Synchronization rule construction prompts can be recommended to the rule maker based on historical synchronization rule construction records, or the rule maker can input and modify them themselves. The rule maker can provide more synchronization rule construction prompts to the large language model, guiding it to generate more complete and accurate synchronization rules. For example, the following synchronization rule can be recommended to build prompt words based on historical records: "You can build synchronization rules about account names and account IDs based on the fields 'name' and 'user ID' in the database table 'user1' and Article 12 of the Financial Regulatory Standards." Another example is that the rule maker can manually input synchronization rules to build prompt words in a preset large language model, as shown in the following example: "Please output the synchronization rule as a JSON string based on the fields 'order ID' and 'order Status' in the database table 'order1' and Article 2 of the Financial Regulatory Standards."

[0114] When constructing synchronization rules using a large language model, you can input an attribute template file for the synchronization rules. This allows the large language model to construct synchronization rules based on the attribute template file and the prompt words for the synchronization rules. The attribute template file can include the attribute content of the synchronization rules.

[0115] Furthermore, after inputting synchronization rule construction prompts into the large language model, the large language model can output corresponding synchronization rules, which can be one or more. This embodiment of the present disclosure can construct data synchronization rules between multiple storage clusters through a preset large language model, greatly improving the speed of synchronization rule construction and thus enhancing the efficiency of data synchronization between multiple storage clusters.

[0116] In some embodiments, after constructing synchronization rules by invoking a preset large language model to construct synchronization rules based on the synchronization rule construction prompts, data management specifications, first tag information, and second tag information, and obtaining multiple synchronization rules output by the preset large language model, the data synchronization method provided in this disclosure embodiment further includes the following steps:

[0117] Displays multiple synchronization rules output by the preset large language model;

[0118] Receive filtering operations for multiple synchronization rules, and determine multiple selected synchronization rules from the multiple synchronization rules based on the filtering operations;

[0119] Add multiple selected synchronization rules to the preset synchronization rule library.

[0120] In this embodiment, after the large language model outputs multiple synchronization rules, these rules can be reviewed and filtered to improve their correctness and rationality. Specifically, the multiple synchronization rules output by the large language model can be displayed on the terminal device for review by developers or reviewers. Furthermore, the terminal device can receive filtering operations on the multiple synchronization rules, which can be performed via mouse events or voice commands; no limitation is made here. Then, multiple selected synchronization rules can be determined from the selected rules based on the filtering operation. If the large language model outputs only one synchronization rule, the selection result of that rule is determined based on the review result; if the review is passed, it is selected. After determining multiple selected synchronization rules, they can be added to a preset synchronization rule library, thus completing the construction of the synchronization rule library.

[0121] Next, we will introduce the specific attributes of the synchronization rules. Specifically, after each synchronization rule is constructed, a name, or command word, is automatically generated. This command word uniquely identifies a synchronization rule. Each synchronization rule can include the following attributes: database table alias, list of unique database fields, interface name for entity data synchronization, entity synchronization rule, whether to update existing records, field blacklist list, grayscale fields, grayscale rules, routing type, and routing key. These attributes can be stored in an attribute template file so that the large language model can construct synchronization rules based on the attribute template file and the synchronization rule construction prompts. Here, a database table alias represents a temporary name specified for a database table, simplifying query statements; a database unique field list represents database tables with unique primary keys, ensuring data synchronization only for fields in tables with unique primary keys, preventing duplicate data writing in case of network timeouts or retries; and the entity data synchronization interface name identifies the application programming interface (API) used to synchronize data between different storage clusters or databases. In this embodiment, the entity data synchronization interface name can be a valid remote procedure call (RPC). The complete method or function name of a Call (RPC) is used; entity synchronization rules can specify the source entity (source storage cluster) and the corresponding target entity (target storage cluster) for data synchronization. The format of entity synchronization rules can be a JSON dictionary containing a list of source and target entities, which can be used to specify that the data to be synchronized should be synchronized to a specific target entity; the attribute of whether to update existing records is a performance optimization strategy for synchronization rules. This attribute can reduce the update frequency of data records that are not changed after being written; the field blacklist can specify fields that do not participate in data synchronization; grayscale fields can be used for database fields to be grayscale controlled; grayscale rules can be used to control business objects that can access the new version of the function. Grayscale rules are used to promote synchronization rules across different storage clusters. Based on the feedback results of the business objects specified by the grayscale rules, it can be further decided to modify the synchronization rules or promote the synchronization rules to the remaining business objects; the route type and route key are used to identify dynamic routes.

[0122] This disclosure embodiment constructs synchronization rules for data synchronization between different storage clusters, which can specify the data stored independently by each storage cluster, as well as the data that can be synchronized between different storage clusters, thus realizing that storage clusters independently store part of the data, while the other part is synchronized between different storage clusters.

[0123] In some embodiments, after adding multiple selected synchronization rules to a preset synchronization rule base, the data synchronization method provided in this disclosure may further include the following steps:

[0124] Get updated data;

[0125] Based on the updated data, a rule update instruction is generated to update the preset synchronization rule base, and the preset synchronization rule base is updated according to the rule update instruction.

[0126] After constructing the synchronization rule base, the synchronization rules can be detected according to a preset update frequency to determine whether they need to be updated. This improves the applicability and correctness of the synchronization rules. The preset frequency can be set by the developers themselves. Specifically, in this embodiment, the updated data may include first sub-update data of the first tag information, second sub-update data of the second tag information, and third sub-update data of the data management specification. When an update to any of the first tag information, second tag information, or data management specification is detected, the synchronization rules can be updated.

[0127] Furthermore, rule update instructions can be generated based on the updated data to update the synchronization rule base. These rule update instructions can include the synchronization rule attributes that need to be updated. Then, the synchronization rules in the synchronization rule base can be updated according to the generated rule update instructions to improve the rationality and correctness of the synchronization rules, thereby ensuring the correctness of data synchronization and data consistency.

[0128] After constructing the synchronization rule base through the aforementioned steps, the synchronization rule base can be updated. The update of the synchronization rule base and the data synchronization process can be performed simultaneously. After determining the database table information corresponding to the data update event, a rule lookup can be performed in the synchronization rule base based on the database table information to obtain the target synchronization rule corresponding to the database table information. Since the synchronization rule stores the unique primary key of the database table, the inclusion of a unique primary key defined in the synchronization rule base in the database table information indicates that the data in that database table can be synchronized to other storage clusters. If no synchronization rule corresponding to the database table information is found in the synchronization rule base, it means that the data in that database table cannot be synchronized.

[0129] Step 230: Extract the synchronization field from the target synchronization rule, and extract the updated content from the database table corresponding to the database table information based on the synchronization field.

[0130] After finding the target synchronization rule corresponding to the table information in the synchronization rule base through the aforementioned steps, the synchronization field can be extracted from the target synchronization rule. Specifically, in this embodiment, the synchronization field can represent the unique primary key of the table in the target synchronization rule. Then, the table can be searched in the source database based on the synchronization field to find the target table corresponding to the table information. The unique primary key can identify a row of data in the table, and a row of data in the table is called a record. Further, the table corresponding to the table information can be determined based on the synchronization field, and then the updated content can be extracted from the table. The updated content can represent a complete and latest record in the table (a record refers to a row in the table), and the record can include the unique primary key of the table and other fields that can be synchronized.

[0131] In some embodiments, extracting synchronization fields from the target synchronization rule and extracting updated content from the database table corresponding to the database table information based on the synchronization fields may include the following steps:

[0132] Extract the synchronization field from the target synchronization rule and generate a read data request based on the synchronization field;

[0133] The system calls the read data interface of the first storage cluster based on the read data request and receives the updated content returned by the read data interface. After the read data request passes the permission verification, the read data interface returns the updated content.

[0134] In this embodiment, after extracting the synchronization field from the target synchronization rule, a read data request can be generated based on the synchronization field. Specifically, when a data update event is detected in the first storage cluster, the diffusion component can obtain the database table information corresponding to the data update event. The diffusion component is a set of general services that can perform main routing and data generalization RPC calls to synchronize data according to the diffusion protocol. The diffusion component can be integrated into the aforementioned data synchronization control device. Further, the diffusion component can perform rule lookup in the synchronization rule base based on the database table information. After finding the target synchronization rule corresponding to the database table information, the diffusion component can extract the synchronization field from the target synchronization rule and generate a read data request based on the synchronization field. The read data request can represent a request event initiated by the diffusion component to the first storage cluster to read the updated content from the database table corresponding to the database table information. Since the diffusion component and the first storage cluster also maintain a state of mutual independence and distrust, the diffusion component can initiate a read data request to the first storage cluster before extracting the updated content from the database table in the first storage cluster.

[0135] Furthermore, the read data interface of the first storage cluster can be invoked according to the read data request, and the updated content returned by the read data interface can be received. Specifically, the read data interface of the first storage cluster can obtain the updated content in the corresponding database table based on the information in the read data request, and send the obtained updated content to the diffusion component for further data synchronization. To ensure the data security of the data in the first storage cluster, when the read data interface of the first storage cluster receives a read data request sent by the diffusion component, it can first verify the permissions of the read data request. After the permission verification is successful, the read data interface can read the updated content from the first storage cluster and return the updated content to the diffusion component. The permissions for the read data request can include permissions to only access data and permissions to access and synchronize data. By verifying the permissions of the read data requests initiated by the diffusion component, insecure data access and reading of the first storage cluster after the diffusion component has been modified can be prevented, and direct reading of data from the first storage cluster by external parties can also be prevented, thereby ensuring the data security of the first storage cluster.

[0136] In some embodiments, the permission verification process for read data requests by the read data interface includes the following steps:

[0137] Retrieve database table information and synchronization fields from the read data request, and search for synchronization rules in the preset synchronization rule base based on the database table information and synchronization fields;

[0138] When the corresponding target synchronization rule is found in the preset synchronization rule base based on the database table information and synchronization fields, the read data interface indicates that the permission verification of the read data request has passed.

[0139] Specifically, a read data request may include database table information corresponding to a data update event, as well as synchronization fields obtained from the target synchronization rule. The read data interface of the first storage cluster can read the database table information and synchronization fields from the read data request sent by the diffusion component. Furthermore, the read data interface can perform a synchronization rule lookup in the synchronization rule base based on the database table information. If the read data interface finds the target synchronization rule corresponding to the database table information in the synchronization rule base, it can match the synchronization fields in the target synchronization rule with the synchronization fields in the read data request. If the synchronization fields in the target synchronization rule are the same as the synchronization fields in the read data request, it indicates that the read data request permission verification has passed. Furthermore, the read data interface can issue a prompt indicating that the read data request permission verification has passed to the first storage cluster, extract the corresponding update content from the database table corresponding to the database table information in the first storage cluster, and send the obtained update content to the diffusion component for the subsequent data synchronization process.

[0140] This embodiment of the disclosure verifies the permissions of read data requests initiated by the diffusion component through the read data interface of the first storage cluster, thereby achieving both the independence of the diffusion component and the first storage cluster and ensuring the data security of the first storage cluster.

[0141] Step 240: Generate a synchronization event message based on the rule tags of the updated content and the target synchronization rule.

[0142] In this embodiment of the disclosure, through the aforementioned steps, the diffusion component can extract updated content from the database tables corresponding to the database table information in the first storage cluster. Further, a synchronization event message can be generated based on the updated content and the rule tag of the target synchronization rule. The rule tag of the target synchronization rule can include a unique command word generated after the synchronization rule is constructed. Then, the diffusion component can package the updated content and the rule tag to obtain the synchronization event message. The synchronization event message can be sent to the storage cluster to be synchronized via a message queue. The message queue can decouple the first storage cluster and the storage cluster to be synchronized, ensuring that business processes in different storage clusters are not interrupted during data synchronization, and that the original architecture of the storage clusters is not modified.

[0143] Step 250: Send the synchronization event message to the message queue so that after the synchronization execution component subscribes to the synchronization event message in the message queue, it calls the write data interface of at least one second storage cluster indicated by the target synchronization rule based on the synchronization event message to synchronize the updated content to at least one second storage cluster.

[0144] In this embodiment, the diffusion component can send the packaged synchronization event message to a message queue, and the synchronization execution component can subscribe to the synchronization event message in the message queue. Further, the synchronization execution component can receive the synchronization event message in the message queue, then determine at least one second storage cluster to be synchronized by the first storage cluster based on the entity synchronization rule in the target synchronization rule of the synchronization event message, and determine the write data interface of the second storage cluster based on the target synchronization rule, wherein the second storage cluster can represent the target storage cluster for synchronization by the first storage cluster. Further, updated content can be synchronized to the second storage cluster through the write data interface of the second storage cluster.

[0145] In some embodiments, after the synchronization execution component subscribes to synchronization event messages in the message queue, it calls the write data interface of at least one second storage cluster indicated by the target synchronization rule based on the synchronization event messages to synchronize the updated content to at least one second storage cluster. The process includes the following steps:

[0146] Receive synchronization event messages from the message queue, parse the synchronization event messages, and obtain the rule label and synchronization event attribute data of the target synchronization rule;

[0147] Based on the rule tags, the target synchronization rule is determined in the preset synchronization rule base. The synchronization event attribute data is then verified according to the target synchronization rule to obtain the first verification result.

[0148] When the first verification result indicates that the verification is successful, extract at least one second storage cluster information to be synchronized from the target synchronization rule;

[0149] Call at least one write data interface corresponding to at least one second storage cluster information to synchronize the updated content to at least one second storage cluster corresponding to at least one second storage cluster information.

[0150] Specifically, the synchronization execution component can act as a medium for data synchronization between the message queue and the second storage cluster. It subscribes to synchronization event messages in the message queue and concurrently calls the entity data synchronization interfaces of multiple second storage clusters according to the target synchronization rules configured for the second storage cluster to write the updated content.

[0151] In this embodiment, the synchronization execution component can receive synchronization event messages from the message queue and parse them to obtain the rule tag and synchronization event attribute data of the target synchronization rule. The synchronization event attribute data may include the updated content of the database table, and may also include other attribute content in the target synchronization rule corresponding to the database table information. For example, the synchronization execution component can parse the source entity and corresponding target entity of the updated content, the primary key field in the updated content, and the database table information corresponding to the database table where the updated content is located from the synchronization event message. Since the second storage cluster, the diffusion component, the synchronization execution component, and the message queue are mutually untrusted, the data obtained from each execution entity can be verified before the updated content reaches the second storage cluster to ensure the accuracy and consistency of the data in the second storage cluster.

[0152] Furthermore, the synchronization execution component can determine the target synchronization rule in the synchronization rule base constructed in the aforementioned steps based on the rule tags, and verify the synchronization event attribute data parsed from the synchronization event message according to the attribute content in the target synchronization rule to obtain the first verification result. By verifying the updated content in the synchronization event message in the message queue, the correctness and consistency of the data synchronized to the second storage cluster can be further ensured. When the first verification result indicates that the verification passed, the synchronization execution component can extract at least one second storage cluster information to be synchronized from the target synchronization rule. The second storage cluster information can represent the information included in the entity synchronization rule of the target synchronization rule, specifically including the correspondence between the source entity and the target entity of the database table, that is, the correspondence between the first storage cluster and the second storage cluster to be synchronized, the cluster name corresponding to each storage cluster, the type of the storage cluster, and the configuration information of the write data interface of each second storage cluster (the interface name for entity data synchronization in the target synchronization rule), and may also include information about some nodes within the storage cluster, etc. The write data interface configuration information (interface name, interface protocol, etc.) of the second storage cluster can be used to configure the write data interface. The write data interface can respond to the synchronization event messages in the message queue and call the entity write data service to put the finally verified update content into the second storage cluster.

[0153] In this embodiment of the disclosure, the interface protocol for the write data interface is defined as follows:

[0154] message KeyValue{ / / Define a data structure to represent key-value pairs

[0155] optional string key = 1; / / Define a string to represent the key

[0156] optional string value = 2; / / Define a string to represent the value

[0157] }

[0158] message EntitySyncReq{ / / Define a data structure to represent an entity synchronization request.

[0159] optional int32 src_entity = 1; / / The source entity for entity data synchronization.

[0160] optional int32 dst_entity = 2; / / Target entity for entity data synchronization request

[0161] optional string cmd = 3; / / Rule tag, used to identify a synchronization rule, such as table_sample

[0162] optional string table_name = 4; / / Define a string to represent the actual name of the database table, for example, dbtabase_sample.table_sample

[0163] repeated KeyValue uniq_data_kv = 5; / / A list of key-value pairs (field name, field value) for the database's unique primary key field.

[0164] repeated KeyValue data_kv = 6; / / List of key-value pairs (field name, field value) for the database table fields that need to be inserted or updated.

[0165] optional int32 update_exist = 7; / / Whether to perform an update operation on existing records, 0 - no, 1 - yes

[0166] optional string fmodify_time = 8; / / Modification time of database table records (duration of time for updated content)

[0167] optional string sign = 9; / / Defines a string to represent the signature, using an internationally standard hash algorithm (such as SM3) to sign the fields in the synchronization event message.

[0168] }

[0169] In this embodiment, the specific process of writing data through the write data interface of the second storage cluster is as follows: The write data interface can register an RPC interface name, then respond to the synchronization event message that has passed verification in the message queue, and parse the synchronization event message. The parsing process can specifically include obtaining the rule tag of the target synchronization rule and the attribute information of the database table to be synchronized. The attribute information of the database table to be synchronized can specifically include the unique primary key of the database table, the first storage cluster and database where the database table is located, and the corresponding update content, etc. Further, the write data interface can perform a synchronization rule query in the synchronization rule library according to the rule tag of the target synchronization rule, thereby verifying whether the target synchronization rule corresponding to the rule tag exists in the synchronization rule library, and whether the current second storage cluster is in the list of second storage clusters indicated in the target synchronization rule. If any of the verification conditions are not met, the write data interface can return the synchronization event message to the message queue.

[0170] Furthermore, if both of the aforementioned checks pass, the write data interface can query the second storage cluster to see if the update content corresponding to the unique primary key of the database table already exists (to check if an updated record exists). If the update content to be synchronized already exists, it can check whether another update is needed based on the target synchronization rule. If the target synchronization rule indicates that an existing record needs to be updated, and the field values ​​of the update content to be synchronized are inconsistent with the field values ​​of the existing record in the database of the second storage cluster, an update operation can be triggered to update the existing record; if the update content to be synchronized does not exist, an insert operation is triggered to insert the record to be updated.

[0171] After extracting at least one second storage cluster information to be synchronized from the target synchronization rules, the write data interface corresponding to the write data interface configuration information in the second storage cluster information can be called. Furthermore, the updated content can be written to the second storage cluster corresponding to the second storage cluster information through the write data interface.

[0172] In some embodiments, after determining the target synchronization rule in a preset synchronization rule base based on the rule tag, verifying the synchronization event attribute data according to the target synchronization rule, and obtaining the first verification result, the data synchronization method provided in this disclosure embodiment further includes the following steps:

[0173] When the first verification result indicates that the verification fails, retrieve the historical synchronization rule corresponding to the historical version of the rule tag;

[0174] The updated content is validated based on historical synchronization rules to obtain a second validation result;

[0175] When the second verification result indicates that the verification passed, extract at least one second storage cluster information to be synchronized from the target synchronization rule;

[0176] Call at least one write data interface corresponding to at least one second storage cluster information to synchronize the updated content to at least one second storage cluster corresponding to at least one second storage cluster information.

[0177] As mentioned above, the synchronization rule base is updated according to a preset update frequency, or when updated data is detected, the synchronization rule base is updated based on the updated data. The synchronization rules before the update can be stored to obtain historical synchronization rules. Each synchronization rule can correspond to a version number and a build time.

[0178] When the first verification result indicates that the verification fails, the reason for the failure could be that the attribute content indicated in the target synchronization rule is inconsistent with the synchronization event attribute data parsed from the synchronization event message, or that the synchronization rule was updated to a new version during the process of sending the synchronization event message to the message queue. This causes a mismatch between the synchronization rule included in the synchronization event message sent to the message queue before the update and the updated synchronization rule. Specifically, this could be a mismatch in the version number of the synchronization rule, thus causing the synchronization event attribute data verification to fail. Therefore, when the first verification result indicates that the verification fails, the historical synchronization rule corresponding to the rule tag can be obtained, thereby obtaining the historical version of the synchronization rule with the same version number as the target synchronization rule in the synchronization event message. This embodiment of the disclosure improves the correctness of the verification by finding the historical version of the historical synchronization rule to verify the synchronization event attribute data, thereby improving the accuracy of data synchronization.

[0179] Furthermore, the updated content in the synchronization event message can be validated according to historical synchronization rules to obtain a second validation result. When the second validation result indicates that the validation passed, at least one second storage cluster information to be synchronized can be extracted from the target synchronization rule. Then, at least one write data interface corresponding to at least one second storage cluster information can be called to synchronize the updated content to at least one second storage cluster corresponding to at least one second storage cluster information. When the second validation result indicates that the validation failed, the data can be returned to the message queue, terminating the data synchronization process.

[0180] This embodiment of the disclosure improves the accuracy of updating content verification by performing secondary verification on the attribute data of synchronization events, thereby improving the accuracy of data synchronization.

[0181] In some embodiments, a target synchronization rule is determined in a preset synchronization rule base based on rule tags, and the synchronization event attribute data is validated according to the target synchronization rule to obtain a first validation result, including the following steps:

[0182] Extract reference source storage cluster information and reference database table information from the target synchronization rules, and extract first storage cluster information and database table information from the synchronization event attribute data;

[0183] The reference source storage cluster information is matched with the first storage cluster information to obtain the first matching result, and the reference database table information is matched with the database table information to obtain the second matching result.

[0184] The first verification result is determined based on the first matching result and the second matching result.

[0185] Specifically, reference source storage cluster information and reference database table information can be extracted from the target synchronization rules. The reference source storage cluster information can represent the cluster information of the first storage cluster corresponding to the database table information specified in the synchronization rules, such as its name. The reference database table information can include the update content determined according to the synchronization fields in the synchronization rules. Furthermore, the first storage cluster information and database table information can be extracted from the synchronization event attribute data. The first storage cluster information can include information such as the name of the first storage cluster, and the database table information can include the database table name and the update content in the database table.

[0186] Then, the reference source storage cluster information and the first storage cluster information can be matched to obtain the first matching result, and the reference database table information can be matched to obtain the second matching result. If the first matching result indicates that the reference source storage cluster information and the first storage cluster information match, and the second matching result indicates that the reference database table information and the database table information match, then the first verification result indicates that the synchronization event attribute data verification passes. If either matching result is a mismatch, the first verification result indicates that the synchronization event attribute data verification fails.

[0187] In some embodiments, after synchronizing the updated content to at least one second storage cluster, the method provided in this disclosure further includes:

[0188] Obtain the preset synchronization data verification rules, verify the updated content synchronized to at least one second storage cluster according to the synchronization data verification rules, and obtain the verification results;

[0189] When the verification result indicates that the updated content synchronized to any second storage cluster is inconsistent with the updated content of the database table in the first storage cluster, or when the verification result indicates that there is updated content in the message queue that has not been synchronized to the second storage cluster, the write data interface of at least one second storage cluster indicated by the target synchronization rule is called based on the synchronization event message to synchronize the unsynchronized updated content to the corresponding second storage cluster.

[0190] Specifically, the data synchronization verification rules can include reconciliation configuration rules and order replenishment configuration rules. The updated content of the database tables in the first storage cluster and the updated content synchronized to the second storage cluster can be verified according to the reconciliation configuration rules to obtain the verification results and discover inconsistent data synchronized in the first and second storage clusters.

[0191] The order replenishment configuration rules can handle data that failed to synchronize in synchronization event messages and inconsistent data found during reconciliation. When the verification result indicates that the updated content synchronized to any second storage cluster is inconsistent with the updated content of the database tables in the first storage cluster, or when the verification result indicates that there is updated content in the message queue that has not been synchronized to the second storage cluster, the source database table and the fields for replenishment can be determined based on the replenishment task configured in the order replenishment configuration rules. Then, based on the synchronization event message, the write data interface of at least one second storage cluster indicated by the target synchronization rule is called to synchronize the unsynchronized updated content to the corresponding second storage cluster. This embodiment of the disclosure can ensure data consistency by verifying the data synchronized to the second storage cluster.

[0192] This disclosure utilizes message queues for data synchronization between different data clusters, decoupling the storage cluster sending and receiving synchronization data. This avoids modifying the architecture of the data storage clusters, thus resolving issues such as increased architectural complexity, higher maintenance costs, and lower data synchronization accuracy caused by modifications to the storage cluster architecture during data synchronization in related technologies. The synchronization rule base provided in this disclosure stores synchronization rules that define the data that can be synchronized between different storage clusters. Therefore, when a data update event is detected, the synchronization rules can be used to determine whether data synchronization is possible. This allows for the synchronization of another portion of data between storage clusters while maintaining the independence of some data within each cluster.

[0193] This disclosure provides a detailed description of the embodiments in conjunction with specific application scenarios.

[0194] like Figure 4 The diagram shown is another flowchart illustrating the data synchronization method provided in this disclosure. This embodiment will use a financial management platform as an example to describe the data synchronization method in detail, focusing on the executing entities of each step. The method specifically includes the following steps:

[0195] Step 401: The diffusion component detects data update events in the source storage cluster and obtains the database table information corresponding to the data update events.

[0196] As described in the aforementioned application scenario of this disclosure, in a wealth management platform, to ensure data security and compliance, each financial institution within the platform should store its data independently. However, data management regulations stipulate that the account information of service recipients of different financial entities should be interconnected, requiring the synchronization of some data across different storage clusters. Based on this, this disclosure provides a data synchronization method to enable some data to be stored independently by each financial entity, while other non-sensitive data can be synchronized among the various financial entities.

[0197] In this embodiment of the disclosure, the diffusion component can detect data update events in the source storage cluster and obtain the database table information corresponding to the data update events. Specifically, please refer to... Figure 5A As shown in the figure, you can first configure the synchronization rules needed for data synchronization. The synchronization management service allows you to enter the command words and attribute content of the synchronization rules, and then send the complete synchronization rules to the preset synchronization rule library for storage. After the synchronization rule library successfully stores the rules, it will prompt the synchronization management service that the rules have been successfully stored. In the wealth management platform, each financial entity can maintain some data, or a financial entity can obtain updated data and synchronize it to other entities through the data synchronization channel. For details, please refer to [link to relevant documentation]. Figure 6A As shown in the figure, the data in the wealth management platform can be divided into four parts: public identifier data 601, entity-divergible data 602, payment institution data 603, and entity-independent data 604. Public identifier data 601 can include xxxx mapping table 1, xxxx mapping table 2, and xxxx table; entity-divergible data 602 specifically includes information such as xxxx cards and xxx assessments; payment institution data 603 specifically includes xxx element information; and entity-independent data 604 specifically includes information such as xxx accounts (e.g., wealth management accounts) and institutional xxx accounts. Public identifier data 601 and entity-divergible data 602 can be uniformly maintained by the first financial entity, then distributed to other financial entities for storage. Payment institution data 603 and entity-independent data 604 can be maintained by individual financial entities. Based on this, when configuring synchronization rules, the source and target entities of the database tables can be determined and converted into the target and constraints of the synchronization rules. That is, the synchronization rules specify which part of the data can be synchronized and which part of the data can be maintained independently by each entity.

[0198] Further, please refer to Figure 6BThe wealth management platform can include a first financial entity, a second financial entity, and a third financial entity. The first financial entity can include entity-specific maintained data 620 and public identifiers and disseminated data 610. Similarly, the second and third financial entities also include entity-specific maintained data 620 and public identifiers and disseminated data 610. Entity-specific maintained data 620 includes payment institution data 603 and entity-specific data 604; public identifiers and disseminated data 610 includes public identifier data 601 and entity disseminated data 602. The entity-specific maintained data 620 can be managed and maintained through the first financial entity object information management service. Similarly, the entity-specific maintained data 620 can be managed and maintained through the second financial entity object information management service, and the entity-specific maintained data 620 can be managed and maintained through the third financial entity object information management service.

[0199] Step 402: The diffusion component searches for synchronization rules in the synchronization rule base based on the database table information. When a target synchronization rule corresponding to the database table information is found, the synchronization field is extracted from the target synchronization rule.

[0200] The diffusion component can search for the target synchronization rule in the synchronization rule base based on the database table name obtained from the database table information. If the diffusion component can find the target synchronization rule corresponding to the database table information in the synchronization rule base, it indicates that the updated content in the database table can be synchronized. Furthermore, the synchronization rule base can return the target synchronization rule to the diffusion component. The obtained target synchronization rule can include the rule's command word and attribute content. This allows checking whether a target synchronization rule exists for the database table, and checking whether the source storage cluster of the database table is consistent with the source storage cluster specified in the rule. If both conditions are met, data synchronization can continue; otherwise, it returns.

[0201] Step 403: The diffusion component extracts the updated content from the database table corresponding to the database table information based on the synchronization field, generates a synchronization event message based on the updated content and the rule tag of the target synchronization rule, and sends the synchronization event message to the message queue.

[0202] After confirming through the aforementioned steps that the updated data in the database tables can be synchronized, the diffusion component can extract the updated content from the database tables corresponding to the synchronized data based on the synchronization fields. For details, please refer to [link to documentation / reference]. Figure 5AIn step 403, the diffusion component can find the unique primary key of the updated database table according to the target synchronization rules, and then send the primary key field to the source storage cluster to find the latest record corresponding to the database table. The source storage cluster can perform record lookup based on the primary key field of the database table sent by the diffusion component, and return the latest record found to the diffusion component.

[0203] Furthermore, after receiving the update records returned by the source storage cluster, the diffusion component can verify the returned update records according to the target synchronization rules, checking whether the source storage cluster and primary key of the update records are consistent with those in the rules. The diffusion component can also check whether the returned update records are unique and whether the modification time is less than N seconds to ensure the accuracy of the data to be synchronized.

[0204] After checking the update records, the diffusion component can package the command word of the target synchronization rule and the latest record of the database table to obtain a synchronization event message. Then, it sends the packaged synchronization event message to the message queue and calls the message queue interface to push the message to the diffusion forwarding service for subsequent data synchronization steps.

[0205] Please continue to refer to Figure 6B Step 403, in a specific financial platform application scenario, specifically includes: accessing the public identifier data 601 and entity diffusible data 602 in the first financial entity through the public identifier and diffusible data management service; and the first financial entity synchronizing data with the second and third financial entities through the first financial entity data diffusive processing service. Specifically, data synchronization can be performed through a data synchronization channel, which can be a Fast Message Queue (FMQ). The second and third financial entities can synchronize data by subscribing to event messages in the data synchronization channel.

[0206] This disclosure embodiment uses a message queue to decouple the first storage cluster and the storage cluster to be synchronized, ensuring that business processes in different storage clusters are not interrupted during data synchronization. Furthermore, this disclosure embodiment uses a common business service for data synchronization between storage clusters, without modifying the original architecture of the storage clusters, thus offering greater configuration flexibility.

[0207] Step 404: The diffusion forwarding service receives the synchronization event message and parses it to obtain the rule tag and update content.

[0208] In the embodiments disclosed herein, such as Figure 5BAs shown, the diffusion forwarding service can subscribe to and receive synchronization event messages in the message queue. After receiving a synchronization event message, the diffusion forwarding service can parse it to obtain the command word of the target synchronization rule, database table information, and the latest record in the database table. It can also obtain other attributes in the rule, such as the specified source storage cluster and the corresponding target storage cluster.

[0209] Specifically, in the wealth management platform, the second financial entity data diffusion processing service can receive synchronization event messages from the data synchronization channel, parse the synchronization event messages, and obtain the command words of the synchronization rules and the latest records of the database tables. The third financial entity data diffusion processing service can also receive synchronization event messages from the data synchronization channel, parse the synchronization event messages, and obtain the command words of the synchronization rules and the latest records of the database tables. The two processes are performed concurrently.

[0210] Step 405: The diffusion forwarding service searches for the target synchronization rule in the synchronization rule base based on the rule tag, and verifies the updated content based on the target synchronization rule.

[0211] After the diffusion forwarding service parses the rule command word from the received synchronization event message, please continue to refer to... Figure 5B In step 405, the diffusion forwarding service can search for the target synchronization rule in the synchronization rule base based on the parsed command word. If a synchronization rule matching the command word exists, the synchronization rule base can return the matching target synchronization rule to the diffusion forwarding service. Furthermore, the diffusion forwarding service can verify the parsed update content based on the returned target synchronization rule.

[0212] Specifically, the diffusion and forwarding of each entity verifies the updated content. It can check whether the database table information in the synchronization event message is consistent with the database table information in the target synchronization rule, and it can also check whether the source storage cluster information in the synchronization event message is consistent with the source storage cluster information in the synchronization rule. If all the check results meet the conditions, it means that the updated content has passed the verification. If any check result does not meet the conditions, the updated content will be returned to the data synchronization channel.

[0213] Step 406: When the updated content verification passes, the diffusion forwarding service finds the target storage cluster to be synchronized from the target synchronization rules and calls the entity write data interface corresponding to the target storage cluster. The entity write data interface writes the updated content to the target storage cluster.

[0214] The diffusion and forwarding service verifies the update through the aforementioned steps. Once verification is successful, refer to... Figure 5B In step 406, the diffusion forwarding service can find the target storage cluster to be synchronized based on the target synchronization rules. For example, in... Figure 6B In this context, the target financial entities corresponding to the first financial entity are the second and third financial entities. After retrieving the target storage cluster to be synchronized, the diffusion forwarding service can concurrently call the entity write data services corresponding to multiple target storage clusters to perform data synchronization. Upon receiving the call from the diffusion forwarding service, the entity write data service can call the write data interface of the target storage cluster to write the latest records from the database tables into the target storage cluster. For example, in... Figure 6B In the process, the second financial entity's data diffusion processing service can call the second financial entity's write data interface to append the updated public identifier data 601 or entity diffuseable data 602 from the first financial entity to the entity's public identifier and diffuseable data 610. Similarly, the third financial entity's data diffusion processing service can call the third financial entity's write data interface to append the updated public identifier data 601 or entity diffuseable data 602 from the first financial entity to the entity's public identifier and diffuseable data 610. After the entity write data interface writes the updated content to the target storage cluster, the diffusion forwarding service can return a message indicating that the synchronization event message has been correctly processed to the message queue, signifying that the data synchronization between different storage clusters has been successfully completed.

[0215] In some embodiments, please refer to Figure 5C , Figure 5C It shows Figure 5B The detailed processing procedure of step 4061 is as follows: Specifically, the entity write data service can parse the synchronization event message again, obtain the database table information and rule command words from the synchronization event message, and then search for the target synchronization rule in the synchronization rule base according to the command words. If a target synchronization rule matching the command words exists, it is returned to the entity write data service.

[0216] Furthermore, the entity write data service can check the existence of the target synchronization rule through a synchronization rule query. Then, it can check if the target storage cluster to which the data is to be written exists within the target storage cluster specified by the target synchronization rule. If both conditions are met, the entity write data service can query the target storage cluster to see if there is a record to be updated. The entity write data service can then determine the operation to be performed on the record to be updated based on the target synchronization rule. If a record to be updated exists, the target storage cluster will return the record to be updated, and the entity write data service will execute the update operation. If no record to be updated exists, the entity write data service will not receive the returned record and will instead perform an insert operation.

[0217] After the entity write data service completes the corresponding write data operation, the target storage cluster can return the updated result to the entity write data service. Furthermore, the entity write data service can return the data synchronization result to the diffusion forwarding service.

[0218] Step 407: The reconciliation management service obtains the reconciliation configuration file and configures the reconciliation configuration database based on the reconciliation configuration file.

[0219] To improve data consistency, reconciliation and order replenishment can be performed after data synchronization. For details, please refer to [link to relevant documentation]. Figure 5D , Figure 5D The diagram illustrates the reconciliation timeline. The reconciliation management service can access the reconciliation configuration file for configuration. Specifically, it can configure source storage cluster link information (e.g., the database handle of the source storage cluster) and target storage cluster link information (e.g., the database handle of the target storage cluster). Then, it can configure the Structured Query Language (SQL) configuration, such as configuring the reconciliation SQL for the source and target storage clusters, as well as the matching SQL and reconciliation fields. The reconciliation SQL is used to batch retrieve a batch of data to be reconciled and compare it based on the reconciliation fields. The reconciliation SQL can be batch-queried using `modify_time`. The matching SQL is used to handle data with inconsistent `modify_time` values ​​that cannot be batch-queried or reconciled. Furthermore, the scheduling period and time offset can be configured. Specifically, it retrieves data from the most recent N+M minutes every N minutes for reconciliation, where N is typically 5 and M is typically 1. Retrieving more than M minutes of data helps avoid missing data within a given range. Finally, reconciliation alarms and retries can be configured. This includes configuring alarm reporting IDs and parameters to promptly notify the responsible party when data anomalies are detected. Records with failed reconciliations will be periodically reconciled according to a time-based strategy, with a maximum of N reconciliations (N defaults to 10). After the reconciliation management service successfully configures the reconciliation configuration based on the reconciliation configuration file, it can synchronize the reconciliation configuration to the reconciliation configuration database. Once the reconciliation configuration database synchronizes the reconciliation configuration file, it can return a synchronization success message to the reconciliation management service.

[0220] Step 408: The reconciliation management service parses the target synchronization rules, verifies the update content to be synchronized in multiple storage clusters and the target synchronization rules based on the reconciliation configuration file, and stores the verification results in the reconciliation configuration database.

[0221] In this embodiment, the reconciliation management service can parse the target synchronization rules and periodically schedule data for verification over a certain period. Then, it can verify the update content to be synchronized across multiple storage clusters and the target synchronization rules according to the reconciliation configuration file. Please refer to... Figure 5DThe detailed process of step 408 involves checking the rules to be reconciled, and using the `modify_time` function to query data in batches from both the source and target storage clusters. The source and target storage clusters then return the queried data to the reconciliation management service. After receiving the queried data, the reconciliation management service verifies the data against the reconciliation fields, reconciling any discrepancies between the source and target storage clusters. For details, please refer to [link to relevant documentation]. Figure 7 As shown in the figure, the reconciliation view displays the menu bar 701, the missing reconciliation interval column 702, the data anomaly record column 703, and the error record - pending record column 704 for the multi-entity reconciliation table. The menu bar 701 can include multiple controls, such as "Summary," "List," "Events," and "Information." The missing reconciliation interval column 702 can display a prompt box indicating "Total () missing reconciliations" to show the current number of missing records. The missing reconciliation interval column 702 can also display the start time, end time, and duration of the reconciliation. The figure shows the result as "No missing reconciliation interval records √." The data anomaly record column 703 can display a prompt box indicating "Total () records" to show the current number of data anomaly records. The data anomaly record column 703 can also display the start and end times of the data, check the start and end times, the number of pending records, and the number of error records. Including the amount of mixing, etc., the result shown in the figure is "No abnormal data records √"; the "Error Record - Outstanding Record" column 704 can display the "Error Record" column and the "Outstanding Record" column respectively. The "Error Record" column can display a prompt box of "Total () Error Records" to indicate how many errors there are, and can also display "Text 1", "Text 2" and "Text 3", each of which can represent an error record. The "Outstanding Record" column can display a prompt box of "Total () Outstanding Records" to indicate how many outstanding records there are, and can also display "Text 4", "Text 5" and "Text 6", each of which can represent an outstanding record. It should be noted that the above-displayed content does not limit the embodiments of this disclosure.

[0222] Furthermore, the reconciliation management service can store the reconciliation results in the verification result table of the reconciliation configuration database. After receiving and storing the verification results sent by the reconciliation management service, the reconciliation configuration database can return a message indicating successful verification update to the reconciliation management service. If the reconciliation management service fails to reconcile, it will upload alarm information.

[0223] Step 409: When the verification result indicates that the update content to be synchronized in multiple storage clusters is inconsistent, or when the verification result indicates that there is update content that failed to be synchronized, the reconciliation management service initiates a supplementary task and calls the entity write data interface to supplement the order.

[0224] After reconciliation through the aforementioned steps, missing data from failed synchronization by the diffusion forwarding service and discrepancies discovered during reconciliation can be handled through order replenishment. Specifically, the reconciliation management service can configure and initiate order replenishment tasks. These tasks can specify the source database table and the fields for initiating the replenishment, which may include the diffusion result table and the reconciliation result table. Order replenishment tasks can be scheduled to be initiated every five minutes. The reconciliation management service then retrieves the records to be replenished. It can initiate replenishment for records from the diffusion result table that failed to diffuse, and for records from the reconciliation result table that have reconciliation errors, by calling the entity write data interface.

[0225] Description of apparatus and devices according to embodiments of this disclosure

[0226] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0227] It should be noted that in the various specific embodiments of this disclosure, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with the relevant laws, regulations, and standards of the relevant regions. In addition, when this application embodiment needs to obtain target object attribute information, separate permission or consent from the target object will be obtained through pop-up windows or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of this application embodiment be obtained.

[0228] Figure 8 A schematic diagram of the structure of a data synchronization device 800 provided in an embodiment of this disclosure. The device includes:

[0229] The acquisition unit 801 is used to acquire the database table information corresponding to the data update event when a data update event is detected in the first storage cluster.

[0230] The lookup unit 802 is used to perform a synchronization rule lookup in a preset synchronization rule base based on the database table information to obtain the target synchronization rule;

[0231] Extraction unit 803 is used to extract synchronization fields from the target synchronization rule and extract update content from the database table corresponding to the database table information based on the synchronization fields;

[0232] The generation unit 804 is used to generate a synchronization event message based on the rule tags of the updated content and the target synchronization rule;

[0233] The sending unit 805 is used to send the synchronization event message to the message queue, so that after the synchronization execution component subscribes to the synchronization event message in the message queue, it calls the write data interface of at least one second storage cluster indicated by the target synchronization rule based on the synchronization event message to synchronize the updated content to the at least one second storage cluster.

[0234] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0235] The first acquisition subunit is used to acquire the first tag information of the database tables contained in the multiple storage clusters to be synchronized, as well as the second tag information of the fields contained in each database table.

[0236] The second acquisition subunit is used to acquire the data management specifications associated with the plurality of storage clusters;

[0237] A sub-unit is constructed to build multiple synchronization rules based on the data management specifications, the first tag information, and the second tag information.

[0238] Optionally, in some embodiments, the building subunit includes:

[0239] The first building module is used to build preset synchronization rule building prompts;

[0240] The second construction module is used to call a preset large language model to construct synchronization rules based on the synchronization rule construction prompts, the data management specifications, the first tag information, and the second tag information, thereby obtaining multiple synchronization rules output by the preset large language model.

[0241] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0242] The display subunit is used to display multiple synchronization rules output by the preset large language model;

[0243] A determining subunit is configured to receive a filtering operation on the plurality of synchronization rules, and determine a plurality of selected synchronization rules among the plurality of synchronization rules based on the filtering operation;

[0244] Add a sub-unit to add the selected synchronization rules to a preset synchronization rule library.

[0245] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0246] The third acquisition subunit is used to acquire updated data, which includes the first sub-update data of the first tag information, the second sub-update data of the second tag information, and the third sub-update data of the data management specification.

[0247] The update subunit is used to generate a rule update instruction to update the preset synchronization rule base based on the update data, and to update the preset synchronization rule base according to the rule update instruction.

[0248] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0249] The parsing subunit is used to receive synchronization event messages in the message queue and parse the synchronization event messages to obtain the rule tag and synchronization event attribute data of the target synchronization rule, wherein the synchronization event attribute data includes the update content.

[0250] The first verification subunit is used to determine the target synchronization rule in the preset synchronization rule base based on the rule tag, and to verify the synchronization event attribute data according to the target synchronization rule to obtain the first verification result.

[0251] The first extraction subunit is used to extract at least one second storage cluster information to be synchronized from the target synchronization rule when the first verification result indicates that the verification is passed.

[0252] The first synchronization subunit is used to call at least one write data interface corresponding to the at least one second storage cluster information to synchronize the updated content to at least one second storage cluster corresponding to the at least one second storage cluster information.

[0253] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0254] The fourth acquisition subunit is used to acquire the historical synchronization rule corresponding to the historical version of the rule tag when the first verification result indicates that the verification fails.

[0255] The second verification subunit is used to verify the updated content based on the historical synchronization rules and obtain a second verification result;

[0256] The second extraction subunit is used to extract at least one second storage cluster information to be synchronized from the target synchronization rule when the second verification result indicates that the verification is passed.

[0257] The second synchronization subunit is used to call at least one write data interface corresponding to the at least one second storage cluster information to synchronize the updated content to at least one second storage cluster corresponding to the at least one second storage cluster information.

[0258] Optionally, in some embodiments, the first verification subunit includes:

[0259] The extraction module is used to extract reference source storage cluster information and reference database table information from the target synchronization rule, and to extract first storage cluster information and database table information from the synchronization event attribute data;

[0260] The matching module is used to match the reference source storage cluster information with the first storage cluster information to obtain a first matching result, and to match the reference library table information with the library table information to obtain a second matching result;

[0261] The determination module is used to determine a first verification result based on the first matching result and the second matching result.

[0262] Optionally, in some embodiments, the extraction unit includes:

[0263] A generation subunit is used to extract synchronization fields from the target synchronization rule and generate a read data request based on the synchronization fields;

[0264] The receiving subunit is used to call the read data interface of the first storage cluster based on the read data request, and to receive the updated content returned by the read data interface. The read data interface returns the updated content after the permission verification of the read data request is qualified.

[0265] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0266] The lookup subunit is used to obtain the database table information and the synchronization field from the read data request, and to perform a synchronization rule lookup in the preset synchronization rule base based on the database table information and the synchronization field.

[0267] The indicator word unit is used to indicate that the permission verification of the read data request has passed when the corresponding target synchronization rule is found in the preset synchronization rule base according to the database table information and the synchronization field.

[0268] Optionally, in some embodiments, the data synchronization apparatus provided in this disclosure further includes:

[0269] The verification subunit is used to obtain preset synchronization data verification rules, verify the updated content synchronized to the at least one second storage cluster according to the synchronization data verification rules, and obtain the verification result.

[0270] The third synchronization subunit is used to call the write data interface of at least one of the second storage clusters indicated by the target synchronization rule based on the synchronization event message when the verification result indicates that the updated content synchronized to any of the second storage clusters is inconsistent with the updated content of the database tables in the first storage cluster, or when the verification result indicates that there is updated content in the message queue that has not been synchronized to the second storage clusters, to synchronize the unsynchronized updated content to the corresponding second storage cluster.

[0271] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0272] Figure 9 This is a partial structural block diagram of a server 110 implementing the data synchronization method of this disclosure embodiment. The server 110 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 922 (e.g., one or more processors) and storage devices 932, and one or more storage media 930 (e.g., one or more mass storage devices) storing application programs 942 or data 944. The storage devices 932 and storage media 930 can be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the server 110. Furthermore, the CPU 922 may be configured to communicate with the storage media 930 and execute the series of instruction operations in the storage media 930 on the server 110.

[0273] Server 110 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input / output interfaces 958, and / or one or more operating systems 941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0274] The central processing unit 922 in server 110 can be used to execute the data synchronization method of the embodiments of this disclosure.

[0275] This disclosure also provides a storage medium for storing program code for executing the data synchronization methods of the foregoing embodiments.

[0276] This disclosure also provides a computer program product comprising a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the data synchronization method described above.

[0277] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0278] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0279] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0280] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0281] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0282] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0283] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0284] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0285] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A data synchronization method, characterized in that, The method includes: When a data update event is detected in the first storage cluster, the database table information corresponding to the data update event is obtained; Based on the information in the database table, a synchronization rule is searched in a preset synchronization rule base to obtain the target synchronization rule; Extract the synchronization field from the target synchronization rule, and extract the update content from the database table corresponding to the database table information based on the synchronization field; A synchronization event message is generated based on the rule tags of the updated content and the target synchronization rule; The synchronization event message is sent to a message queue so that after the synchronization execution component subscribes to the synchronization event message in the message queue, it calls the write data interface of at least one second storage cluster indicated by the target synchronization rule based on the synchronization event message to synchronize the updated content to the at least one second storage cluster.

2. The method according to claim 1, characterized in that, The synchronization rules in the preset synchronization rule base are constructed using the following method: Obtain the first tag information of the database tables contained in the multiple storage clusters to be synchronized, as well as the second tag information of the fields contained in each database table; Obtain the data management specifications associated with the multiple storage clusters; Multiple synchronization rules are constructed based on the data management specifications, the first tag information, and the second tag information.

3. The method according to claim 2, characterized in that, The construction of multiple synchronization rules based on the data management specifications, the first tag information, and the second tag information includes: Construct preset synchronization rules and generate prompt words; The preset large language model is invoked to construct synchronization rules based on the prompt words for the synchronization rules, the data management specifications, the first tag information, and the second tag information, resulting in multiple synchronization rules output by the preset large language model.

4. The method according to claim 3, characterized in that, After invoking a preset large language model to construct synchronization rules based on the synchronization rule construction prompts, the data management specifications, the first tag information, and the second tag information, and obtaining multiple synchronization rules output by the preset large language model, the method further includes: Displays multiple synchronization rules output by the preset large language model; Receive filtering operations on the plurality of synchronization rules, and determine a plurality of selected synchronization rules from the plurality of synchronization rules based on the filtering operations; Add the selected synchronization rules to the preset synchronization rule library.

5. The method according to claim 4, characterized in that, After adding the selected synchronization rules to the preset synchronization rule library, the method further includes: Obtain updated data, which includes first sub-update data of the first tag information, second sub-update data of the second tag information, and third sub-update data of the data management specification; Based on the updated data, a rule update instruction is generated to update the preset synchronization rule base, and the preset synchronization rule base is updated according to the rule update instruction.

6. The method according to claim 1, characterized in that, After subscribing to synchronization event messages in the message queue, the synchronization execution component calls the write data interface of at least one second storage cluster indicated by the target synchronization rule based on the synchronization event messages to synchronize the updated content to the at least one second storage cluster. The process includes the following steps: Receive synchronization event messages from the message queue and parse the synchronization event messages to obtain the rule tag and synchronization event attribute data of the target synchronization rule, wherein the synchronization event attribute data includes the update content; Based on the rule tags, a target synchronization rule is determined in a preset synchronization rule base. The synchronization event attribute data is then verified according to the target synchronization rule to obtain a first verification result. When the first verification result indicates that the verification is successful, at least one second storage cluster information to be synchronized is extracted from the target synchronization rule; Call at least one write data interface corresponding to the at least one second storage cluster information to synchronize the updated content to at least one second storage cluster corresponding to the at least one second storage cluster information.

7. The method according to claim 6, characterized in that, After determining the target synchronization rule in a preset synchronization rule base based on the rule tag, and verifying the synchronization event attribute data according to the target synchronization rule to obtain a first verification result, the method further includes: When the first verification result indicates that the verification fails, retrieve the historical synchronization rule of the historical version corresponding to the rule tag; The updated content is verified based on the historical synchronization rules to obtain a second verification result; When the second verification result indicates that the verification is successful, at least one second storage cluster information to be synchronized is extracted from the target synchronization rule; Call at least one write data interface corresponding to the at least one second storage cluster information to synchronize the updated content to at least one second storage cluster corresponding to the at least one second storage cluster information.

8. The method according to claim 6, characterized in that, The process of determining a target synchronization rule in a preset synchronization rule base based on the rule tag, and verifying the synchronization event attribute data according to the target synchronization rule to obtain a first verification result includes: Extract reference source storage cluster information and reference database table information from the target synchronization rule, and extract first storage cluster information and database table information from the synchronization event attribute data; The reference source storage cluster information is matched with the first storage cluster information to obtain a first matching result, and the reference library table information is matched with the library table information to obtain a second matching result; The first verification result is determined based on the first matching result and the second matching result.

9. The method according to claim 1, characterized in that, The step of extracting synchronization fields from the target synchronization rule and extracting update content from the database table corresponding to the database table information based on the synchronization fields includes: Extract the synchronization field from the target synchronization rule, and generate a read data request based on the synchronization field; The system calls the read data interface of the first storage cluster based on the read data request and receives the updated content returned by the read data interface. The read data interface returns the updated content after verifying the permission of the read data request.

10. The method according to claim 9, characterized in that, The permission verification process for the read data request by the read data interface includes the following steps: Obtain the database table information and the synchronization field from the read data request, and perform a synchronization rule search in the preset synchronization rule base based on the database table information and the synchronization field; When the corresponding target synchronization rule is found in the preset synchronization rule base based on the database table information and the synchronization field, the read data interface indicates that the permission verification of the read data request has passed.

11. The method according to claim 1, characterized in that, After synchronizing the updated content to the at least one second storage cluster, the method further includes: Obtain preset synchronization data verification rules, verify the updated content synchronized to the at least one second storage cluster according to the synchronization data verification rules, and obtain the verification result; When the verification result indicates that the updated content synchronized to any of the second storage clusters is inconsistent with the updated content of the database tables in the first storage cluster, or when the verification result indicates that there is updated content in the message queue that has not been synchronized to the second storage clusters, the write data interface of at least one of the second storage clusters indicated by the target synchronization rule is invoked based on the synchronization event message to synchronize the unsynchronized updated content to the corresponding second storage cluster.

12. A data synchronization device, characterized in that, The device includes: The acquisition unit is used to acquire the database table information corresponding to the data update event when a data update event is detected in the first storage cluster. The search unit is used to search for synchronization rules in a preset synchronization rule base based on the information in the database table to obtain the target synchronization rule; The extraction unit is used to extract the synchronization field from the target synchronization rule and extract the update content from the database table corresponding to the database table information based on the synchronization field. A generation unit is used to generate a synchronization event message based on the rule tags of the updated content and the target synchronization rule; The sending unit is used to send the synchronization event message to the message queue, so that after the synchronization execution component subscribes to the synchronization event message in the message queue, it calls the write data interface of at least one second storage cluster indicated by the target synchronization rule based on the synchronization event message to synchronize the updated content to the at least one second storage cluster.

13. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data synchronization method according to any one of claims 1 to 11.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the data synchronization method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the data synchronization method according to any one of claims 1 to 11.

Images