Data synchronization method and related equipment
By processing data update operations and flow update operations as the same transaction and monitoring the log records of the business database, the data inconsistency caused by the loss of flow records in the business database is solved, and the data synchronization and inspection scope is reduced, which improves efficiency.
Patent Information
- Application Number
- CN202410965595.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-15
AI Technical Summary
In the prior art, when the flow records of business database changes are lost, it is difficult to check problems, resulting in inconsistent data on the business side and server side, and it is difficult to narrow the scope of investigation, affecting efficiency.
By processing data update operations and flow update operations as the same transaction, we ensure atomicity, and when business flow records are lost, we monitor the log records of the business database, send flow update messages to the consumer side, generate service flow records, and realize data synchronization and Reduce the scope of troubleshooting.
It has achieved the narrowing of the inspection scope when the flow records are lost, improved the inspection efficiency, ensured the consistency of the data on the business side and the server side, and avoided the simultaneous inspection of all business flow records and consumer side flow records.
Smart Images

Figure CN118964479A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of data processing, and in particular, to a data synchronization method and related equipment. Background Art
[0002] E-commerce systems face a large influx of orders every day, and these orders are processed in the ERP system. However, with the change of order status, such as product distribution, returns or inventory adjustments, inventory changes will be triggered. Since these changes generally occur on the business side, in order to ensure data consistency between the business side and the service side, it is necessary to accurately record every change in the business database.
[0003] The solution adopted by the prior art is as follows: after the business database of the business end is changed, the business end will send an open notification service (ONS) message to the message middleware. The ONS message contains a running record of the business database change, such as product ID, change quantity and change type. Subsequently, the message middleware will forward the ONS message to the corresponding consumer end to ensure data consistency between the business end and the server end.
[0004] In actual applications, the log records of business database changes may be lost for various reasons, resulting in inconsistent data between the business end and the server end. At this time, if the message middleware does not retain the log records of business database changes, it will be difficult to troubleshoot the problem in the face of a large number of log records on the business end and the server end. Summary of the invention
[0005] The embodiments of the present application provide a data synchronization method and related equipment for narrowing the scope of investigation when transaction records are lost, eliminating the need to simultaneously investigate all business transaction records corresponding to the business database and all transaction records on the consumer side, thereby improving investigation efficiency.
[0006] A first aspect of an embodiment of the present application provides a data synchronization method, including:
[0007] Receiving a data change instruction of a business database, the data change instruction is used to instruct to perform a pipeline update operation and a data update operation of the business database, the pipeline update operation is used to generate a business pipeline record corresponding to the data update operation;
[0008] Processing the data update operation and the pipeline update operation as the same transaction;
[0009] Based on the same transaction, synchronously updating the business database and business flow record;
[0010] If the business flow record is monitored, a flow update message corresponding to the business flow record is sent to the consumer end, so that the consumer end generates a corresponding service flow record according to the business flow record.
[0011] In a specific implementation, the business transaction record has a corresponding transaction update identifier, and updating the business database includes:
[0012] Updating the business data in the business database, and adding a log record in a log file of the business database, wherein the log record has a corresponding data update identifier;
[0013] The method further comprises:
[0014] If an update of the log file is monitored, determining a data update identifier corresponding to the newly added log record;
[0015] Determine the business transaction record whose corresponding transaction update identifier is consistent with the data update identifier, and determine that the business transaction record is monitored.
[0016] In a specific implementation, the sending of a transaction flow update message corresponding to the service transaction flow record to the consumer end includes:
[0017] Obtaining the business transaction record;
[0018] Extracting the fields and field values updated by the corresponding data update operation from the business flow record;
[0019] The transaction update message is sent to the consumer end, where the transaction update message includes the fields updated by the data update operation and the field values.
[0020] In a specific implementation, the sending the flow update message to the consumer end includes:
[0021] The pipeline update message and the business type to which the business database belongs are sent to the message middleware, wherein the business type is used to determine the consumer end belonging to the same cluster, so that the message middleware sends the pipeline update message to the consumer end of the same cluster.
[0022] A second aspect of an embodiment of the present application provides a data synchronization method, which is applied to a consumer end and includes:
[0023] The receiving business end sends a transaction update message after monitoring the business transaction record. The transaction update message corresponds to the business transaction record. The business transaction record corresponding to the data update operation is generated by the transaction update operation. The data change instruction is used to instruct the execution of the transaction update operation and the data update operation of the business database. The data update operation and the transaction update operation are processed as the same transaction, and the same transaction is used to synchronously update the business database and the business transaction record.
[0024] In a specific implementation, the data change instruction is initiated by a user, and the method further includes:
[0025] Update the service database of the consumer end based on the running water update message;
[0026] Determine a service flow table associated with the user based on a preset partitioning algorithm;
[0027] Generate a service flow record corresponding to the business flow record;
[0028] The service flow record is stored in the service flow table associated with the user.
[0029] A third aspect of an embodiment of the present application provides a service end, including:
[0030] A receiving unit, configured to receive a data change instruction of a business database, wherein the data change instruction is used to instruct execution of a pipeline update operation and a data update operation of the business database, wherein the pipeline update operation is used to generate a business pipeline record corresponding to the data update operation;
[0031] A processing unit, used for processing the data update operation and the pipeline update operation as a same transaction;
[0032] The processing unit is further used to synchronously update the business database and the business flow record based on the same transaction;
[0033] A sending unit is used to send a flow update message corresponding to the business flow record to the consumer end if the business flow record is monitored, so that the consumer end generates a corresponding service flow record according to the business flow record
[0034] In a specific implementation, the business flow record has a corresponding flow update identifier, and the processing unit is specifically used to update the business data in the business database and add a log record in a log file of the business database, and the log record has a corresponding data update identifier;
[0035] The sending unit is also used to determine the data update identifier corresponding to the newly added log record if an update of the log file is monitored; determine the business flow record whose corresponding flow update identifier is consistent with the data update identifier, and determine that the business flow record is monitored.
[0036] In a specific implementation, the sending unit is specifically used to obtain the business flow record;
[0037] Extracting the fields and field values updated by the corresponding data update operation from the business flow record;
[0038] The transaction update message is sent to the consumer end, where the transaction update message includes the fields updated by the data update operation and the field values.
[0039] In a specific implementation, the sending unit is specifically used to send the pipeline update message and the business type to which the business database belongs to the message middleware, wherein the business type is used to determine the consumer end belonging to the same cluster, so that the message middleware sends the pipeline update message to the consumer end of the same cluster.
[0040] A fourth aspect of the embodiments of the present application provides a consumer end, including:
[0041] A receiving unit is used to receive a transaction update message sent by a business end after monitoring a business transaction record, wherein the transaction update message corresponds to the business transaction record, and the business transaction record corresponding to the data update operation is generated by the transaction update operation. The data change instruction is used to instruct the execution of the transaction update operation and the data update operation of the business database, and the data update operation and the transaction update operation are processed as the same transaction, and the same transaction is used to synchronously update the business database and the business transaction record.
[0042] In a specific implementation, the data change instruction is initiated by a user, and the consumer end further includes: a processing unit;
[0043] The processing unit is used to update the service database of the consumer end based on the running water update message;
[0044] The processing unit is further used to determine the service flow table associated with the user based on a preset partitioning algorithm;
[0045] The processing unit is further used to generate a service flow record corresponding to the business flow record;
[0046] The processing unit is further configured to store the service flow record in a service flow table associated with the user.
[0047] A fifth aspect of an embodiment of the present application provides a computer device, including:
[0048] CPU, memory and input / output interface;
[0049] The memory is a short-term storage memory or a persistent storage memory;
[0050] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first aspect or the second aspect.
[0051] A sixth aspect of the embodiments of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect or the second aspect.
[0052] A seventh aspect of an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the method described in the first aspect or the second aspect.
[0053] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages: by treating the data update operation and the flow update operation as a transaction, the atomicity of the data update operation and the flow update operation is guaranteed. In other words, the data update operation and the flow update operation will only be updated and executed successfully at the same time or failed at the same time. Secondly, by continuously monitoring the business flow records, the business flow records are updated in time to the consumer end that needs to maintain data consistency with the business database, thereby achieving data synchronization. Under the above design, if the business flow record is lost, it can be determined whether the problem comes from the business end by whether there is a corresponding business flow record in the business database, thereby narrowing the scope of investigation, and there is no need to investigate all business flow records corresponding to the business database and all flow records on the consumer end at the same time. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 A system architecture diagram of the data synchronization method disclosed in the embodiment of the present application;
[0055] Figure 2 A schematic diagram of a flow chart of the data synchronization method disclosed in an embodiment of the present application;
[0056] Figure 3 A schematic diagram of the structure of the service end disclosed in the embodiment of the present application;
[0057] Figure 4 A schematic diagram of the structure of the consumer end disclosed in the embodiment of the present application;
[0058] Figure 5Another structural schematic diagram of the computer device disclosed in the embodiment of the present application. DETAILED DESCRIPTION
[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0060] The embodiments of the present application provide a data synchronization method and related equipment for narrowing the scope of investigation when transaction records are lost, eliminating the need to simultaneously investigate all business transaction records corresponding to the business database and all transaction records on the consumer side, thereby improving investigation efficiency.
[0061] See also Figure 1 In order to better implement the data synchronization method of the embodiment of the present application, the embodiment of the present application provides the following Figure 1 The data synchronization architecture shown includes: at least one business end, a message middleware, and at least one consumer end, wherein the business end stores a modifiable business database.
[0062] First, the business end needs to execute data update operations and pipeline update operations on the business database in response to the data change instructions. In particular, in order to ensure the atomicity of both data update operations and pipeline update operations, this application places the data update operations and pipeline update operations in the same transaction for processing; if the business end successfully executes the pipeline update operation, the corresponding business pipeline records will be generated.
[0063] Then, the message middleware monitors the newly generated business flow records on the business side and sends the corresponding flow update message to the consumer side, which is equivalent to the business side monitoring the flow update operation performed by the business side. The business side data update operation and the flow update operation are atomic, which is equivalent to monitoring the data update operation of the business side.
[0064] Finally, after receiving the flow update message, the consumer end that needs to obtain the business flow changes of the aforementioned business database will generate a corresponding service flow record to synchronize the business flow update of the business end to the consumer end.
[0065] It should be noted that the business end referred to in this application refers to a server or computer device that performs database updates, and the consumer end refers to a server or computer device that needs to synchronize data updates from the business end. In other words, a computer device or server can be a business end or a consumer end, depending on the role it plays in each data processing process.
[0066] Based on the aforementioned data synchronization architecture, an embodiment of the present application provides a data processing method, which can be deployed on a business end by software, including the following steps:
[0067] 201. Receive a data change instruction of a business database, where the data change instruction is used to instruct execution of a flow update operation and a data update operation of the business database, where the flow update operation is used to generate a business flow record corresponding to the data update operation.
[0068] When the business data in the business database needs to be changed, the business end to which the business database belongs will receive a data change instruction instructing to execute a flow update operation and a data update operation of the business database. The data update operation is used to instruct to update the business database, and the flow update operation is used to generate a business flow record corresponding to the data update operation.
[0069] In order to facilitate users to accurately and quickly query the data update of the business database, the embodiment of the present application also includes a flow update operation that generates a business flow record corresponding to the data update operation. Users can accurately know each data update of the business database through the business flow record, and thus better understand the data changes of the business database.
[0070] 202. Process the data update operation and the pipeline update operation as the same transaction.
[0071] In computer science, especially in database management systems, a transaction is a sequence of operations performed as a single logical unit of work. The main purpose of a transaction is to ensure the integrity and consistency of data, even in the event of a system crash or other errors. Transactions are used when business logic requires a series of operations to either all succeed or all fail (i.e., the "all or nothing" principle).
[0072] Based on the above content, the embodiment of the present application processes the data update operation and the pipeline update operation in the same transaction to ensure the atomicity between the data update operation and the pipeline update operation.
[0073] Transactions are used when business operations need to ensure that a series of operations are either all successful or all failed (i.e., the "all or nothing" principle). For example, in a bank transfer system, the two operations of deducting money from one account and depositing money into another account must be processed as a transaction to ensure the integrity and accuracy of the funds. Specifically, in terms of code implementation, the technical implementation of this step can be ensured by defining the relevant software code for data update operations and transaction update operations in one transaction.
[0074] 203. Based on the same transaction, the business database and business flow records are updated synchronously.
[0075] That is, the data update operation and the flow update operation are processed as one transaction, but the data update operation and the flow update operation have different functions. Specifically, the data update operation is used to update the business database, while the flow update operation is used to update the corresponding business flow record.
[0076] 204. If a business flow record is monitored, a flow update message corresponding to the business flow record is sent to the consumer end, so that the consumer end generates a corresponding service flow record according to the business flow record.
[0077] When the business end detects the generation of a new business flow record, the business end will send a flow update message corresponding to the business flow record to the consumer end, so that the consumer end generates a corresponding service flow record based on the business flow record. After the consumer end generates the corresponding service flow record, the data synchronization between the business end and the service end is completed. Among them, the consumer end that needs to receive the flow update message in this step is any consumer end that has a data synchronization relationship with the aforementioned business end.
[0078] In the embodiment of the present application, the atomicity of the data update operation and the flow update operation is guaranteed by treating the data update operation and the flow update operation as a transaction. In other words, the data update operation and the flow update operation will only be updated and executed successfully at the same time or failed at the same time. Secondly, by continuously monitoring the business flow records, the business flow records are updated in time to the consumer end that needs to maintain data consistency with the business database, thereby achieving data synchronization. Under the above design, if the business flow record is lost, it can be determined whether the problem comes from the business end by whether there is a corresponding business flow record in the business database, thereby narrowing the scope of investigation, and there is no need to simultaneously investigate all business flow records corresponding to the business database and all flow records on the consumer end.
[0079] In some specific implementations, in order to distinguish the business flow records corresponding to different data update operations, each business flow record has a corresponding flow update identifier, and the "updating the business database" in the aforementioned step 203 includes: updating the business data in the business database, and adding a log record in the log file of the business database, and the log record has a corresponding data update identifier. On this basis, if the update of the log file is monitored, the data update identifier corresponding to the newly added log record is determined; the business flow record whose corresponding flow update identifier is consistent with the data update identifier is determined, and it is determined that the business flow record is monitored.
[0080] In actual applications, the log file of the business database will be synchronously updated after the business data is updated (or after the business database performs a data update operation), and a log record corresponding to the business data update will be generated. The log record can contain the data update identifier of the corresponding data update operation. Similarly, if the message middleware monitors the update of the log file, it can determine that the business flow record is monitored based on the data update identifier corresponding to the newly added log record, that is, the update of the business flow record is monitored.
[0081] Specifically, the data update identifier can be any identifier that can uniquely identify the data update operation, such as the time when the corresponding data update operation is completed or executed; the flow update identifier can be any identifier that can uniquely identify the flow update operation, such as the time when the corresponding flow update operation is completed or executed, and is not limited here. It should be noted that the data update identifier of the data update operation corresponding to a certain data update and the flow update identifier corresponding to the flow update operation should be consistent, so that the message middleware can quickly find the corresponding business flow record.
[0082] Furthermore, the aforementioned step 205 can be specifically implemented according to the following steps: obtaining business transaction records; extracting the fields and field values updated by the corresponding data update operation from the business transaction records; sending a transaction update message to the consumer end, the transaction update message including the fields and field values updated by the data update operation.
[0083] Specifically, the business transaction record will record the update object of the corresponding data update operation, that is, at least one field updated by the corresponding data update operation and the field value of each updated field. In this way, when the message middleware provides a transaction update message to the consumer, the consumer can accurately obtain the data update actually performed by the business end from the fields and field values updated by the data update operation included in the transaction update message, so as to accurately complete the data synchronization with the business end.
[0084] Furthermore, in addition to sending the flow update message directly from the business end to the consumer end, "sending the flow update message to the consumer end" also includes: sending the flow update message and the business type of the business database to the message middleware, wherein the business type is used to determine the consumer end belonging to the same cluster, so that the message middleware sends the flow update message to the consumer end of the same cluster.
[0085] Specifically, the databases of the consumer end and the business end that need to receive messages from the business end of the embodiment of the present application are likely to have the same business type. Therefore, in order to accurately and quickly select the business end that needs to receive the aforementioned flow update message from the optional consumer end of the message middleware. The embodiment of the present application allocates corresponding business types to each consumer end and business end, or allocates corresponding business types to the service database of each consumer end and the business database of the business end, which is not limited here.
[0086] Based on the business type corresponding to each consumer end and the business end, or the business type corresponding to the service database of each consumer end and the business database of the business end, the present application can quickly determine the consumer ends belonging to the same cluster, and the databases of these consumer ends have the same business type as the business types to which the business databases belong. Specifically, the configuration of the business type can be configured for the business end and each consumer end separately, or a cluster can be constructed for the business type, and the cluster includes all consumer ends, which is not specifically limited here.
[0087] On the basis of the foregoing embodiments, in some specific implementations, in order to better meet the needs of users to check transaction records, the embodiments of the present application store all service transaction records generated by the same user in the same service transaction table, and generate corresponding service transaction records based on the business transaction records, including: updating the service database of the consumer end based on the transaction update message; determining the service transaction table associated with the user based on a preset partitioning algorithm; generating a service transaction record corresponding to the business transaction record; and storing the service transaction record in the service transaction table associated with the user.
[0088] For any consumer end that receives the flow update message, it can update the service database of the current consumer end after receiving the flow update message. In addition, similar to the business end, after the data is updated, the flow update is also required. Therefore, the embodiment of the present application also needs to generate a service flow record corresponding to the business flow record. Since different business ends will generate a large amount of flow data, in order to improve the efficiency of data query, the service flow record of each user can be stored in a designated service flow table. Specifically, "determining the service flow table associated with the user based on a preset partitioning algorithm" can be: taking the remainder of the quotient of the user identifier and the number of service flow tables, and using the obtained remainder as the basis for partitioning, or randomly specifying the service flow table associated with different users, which is not limited here. It should be noted that no matter what method is used to determine the service flow table associated with the user, the service flow table associated with the same user determined at different times should be the same, that is, the uniqueness of the service flow table associated with the user should be guaranteed.
[0089] A variety of implementation methods have been described above. The following describes the data synchronization method of an embodiment of the present application in a specific business scenario.
[0090] When, for example, goods are put into storage, taken out of storage, or sold, the inventory data will change. The inventory business end will perform data update operations and flow update operations to store the changed data in the warehouse. This may be achieved through database triggers, business logic code, or other methods, which are not limited here.
[0091] Apache Flink is used as the message middleware to monitor the inventory business end's flow table. Flink captures the updated business flow records of the inventory business end in real time through the binlog monitoring function provided by MySQL. Once new data is generated, Flink will generate corresponding flow update messages based on the business flow records (such as extracting necessary fields, formatting, etc.), and send the flow update messages to the consumer end of the corresponding business type.
[0092] Set up a cluster of consumer ends corresponding to the inventory business end. Multiple consumer ends in the cluster can consume the flow update messages related to the inventory business end in parallel from Flink, ensuring high availability and scalability of data processing. The consumer end obtains the flow update message from the message middleware, updates the service data and generates the corresponding service flow record. Finally, the consumer end can perform various processing according to business needs, such as inventory calculation, inventory status update, etc. These processing and changes can be further persisted to the database or other storage system for subsequent query and analysis.
[0093] In the embodiment of the present application, the technical means of Flink CDC is used to consider factors such as data skew and partition strategy to optimize the data processing process. Understand the fault tolerance mechanism provided by Flink to ensure the consistency and reliability of data in the event of a failure. Use the savepoint and checkpoint mechanisms to ensure data consistency and recoverability.
[0094] See also Figure 3 , the embodiment of the present application provides a service end, including:
[0095] A receiving unit 301 is used to receive a data change instruction of a business database, the data change instruction is used to instruct to perform a pipeline update operation and a data update operation of the business database, the pipeline update operation is used to generate a business pipeline record corresponding to the data update operation;
[0096] The processing unit 302 is used to process the data update operation and the pipeline update operation as the same transaction;
[0097] The processing unit 302 is further used to synchronously update the business database and the business flow record based on the same transaction;
[0098] The sending unit 303 is used to send a flow update message corresponding to the business flow record to the consumer end if a business flow record is monitored, so that the consumer end generates a corresponding service flow record according to the business flow record.
[0099] In a specific implementation, the business flow record has a corresponding flow update identifier, and the processing unit 302 is specifically used to update the business data in the business database and add a log record in the log file of the business database, and the log record has a corresponding data update identifier;
[0100] The sending unit 303 is also used to determine the data update identifier corresponding to the newly added log record if the update of the log file is monitored; determine the business flow record whose corresponding flow update identifier is consistent with the data update identifier, and determine that the business flow record is monitored.
[0101] In a specific implementation, the sending unit 303 is specifically used to obtain a business flow record;
[0102] Extract the fields and field values updated by the corresponding data update operation from the business flow record;
[0103] Send a running update message to the consumer, which includes the fields updated by the data update operation and the field values.
[0104] In a specific implementation, the sending unit 303 is specifically used to send a pipeline update message and a business type to which the business database belongs to the message middleware, wherein the business type is used to determine the consumer end belonging to the same cluster, so that the message middleware sends a pipeline update message to the consumer end of the same cluster.
[0105] See also Figure 4 , the embodiment of the present application provides a consumer end, including:
[0106] Receiving unit 401 is used to receive a transaction update message sent by the business end after monitoring the business transaction record. The transaction update message corresponds to the business transaction record. The business transaction record corresponding to the data update operation is generated by the transaction update operation. The data change instruction is used to indicate the execution of the transaction update operation and the data update operation of the business database. The data update operation and the transaction update operation are processed as the same transaction, and the same transaction is used to synchronously update the business database and the business transaction record.
[0107] In a specific implementation, the data change instruction is initiated by a user, and the consumer end further includes: a processing unit;
[0108] A processing unit, used to update a service database of a consumer end based on a running update message;
[0109] The processing unit is further used to determine a service flow table associated with the user based on a preset partitioning algorithm;
[0110] The processing unit is further used to generate a service flow record corresponding to the business flow record;
[0111] The processing unit is further used to store the service flow record in the service flow table associated with the user.
[0112] Figure 5 It is a schematic diagram of a computer device structure provided in an embodiment of the present application. The computer device 500 may include one or more central processing units (CPU) 501 and a memory 505. The memory 505 stores one or more application programs or data.
[0113] The memory 505 may be a volatile storage or a persistent storage. The program stored in the memory 505 may include one or more modules, each of which may include a series of instruction operations in the computer device. Furthermore, the central processing unit 501 may be configured to communicate with the memory 505 and execute a series of instruction operations in the memory 505 on the computer device 500.
[0114] The computer device 500 may also include one or more power supplies 502, one or more wired or wireless network interfaces 503, one or more input and output interfaces 504, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0115] The CPU 501 can execute the aforementioned Figures 1 to 4 The operations performed by the business end and / or the consumer end in the illustrated embodiment will not be described in detail here.
[0116] It should be noted that, although the steps in the flowcharts involved in the embodiments are drawn in sequence according to the instructions of the arrows, unless otherwise clearly stated in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0117] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0118] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0121] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.
[0122] An embodiment of the present application also provides a computer program product comprising instructions, and when the computer program product is run on a computer, the computer executes the data synchronization method as described above.
Claims
1. A data synchronization method, characterized in that: include: Receiving a data change instruction of a business database, the data change instruction is used to instruct to perform a pipeline update operation and a data update operation of the business database, the pipeline update operation is used to generate a business pipeline record corresponding to the data update operation; Processing the data update operation and the pipeline update operation as the same transaction; Based on the same transaction, synchronously updating the business database and business flow record; If the business flow record is monitored, a flow update message corresponding to the business flow record is sent to the consumer end, so that the consumer end generates a corresponding service flow record according to the business flow record.
2. The data synchronization method according to claim 1, characterized in that: The business transaction record has a corresponding transaction update identifier, and the updating of the business database includes: Updating the business data in the business database, and adding a log record in a log file of the business database, wherein the log record has a corresponding data update identifier; The method further comprises: If an update of the log file is monitored, determining a data update identifier corresponding to the newly added log record; Determine the business transaction record whose corresponding transaction update identifier is consistent with the data update identifier, and determine that the business transaction record is monitored.
3. The data synchronization method according to claim 1 or 2, characterized in that: The sending of a transaction update message corresponding to the service transaction record to the consumer end includes: Obtaining the business flow record; Extracting the fields and field values updated by the corresponding data update operation from the business flow record; The transaction update message is sent to the consumer end, where the transaction update message includes the fields updated by the data update operation and the field values.
4. The data synchronization method according to claim 3, characterized in that: The sending the flow update message to the consumer end includes: The pipeline update message and the business type to which the business database belongs are sent to the message middleware, wherein the business type is used to determine the consumer end belonging to the same cluster, so that the message middleware sends the pipeline update message to the consumer end of the same cluster.
5. A data synchronization method, characterized in that: Applied to the consumer side, including: A flow update message sent by a receiving service end after monitoring a service flow record, wherein the flow update message corresponds to the service flow record, the service flow record corresponding to the data update operation is generated by the flow update operation, the data change instruction is used to instruct the execution of the flow update operation and the data update operation of the service database, the data update operation and the flow update operation are processed as the same transaction, and the same transaction is used to synchronously update the service database and the service flow record; Generate a corresponding service flow record based on the business flow record.
6. The data synchronization method according to claim 5, characterized in that: The data change instruction is initiated by a user, and the generating of a corresponding service flow record based on the business flow record includes: Update the service database of the consumer end based on the running water update message; Determine a service flow table associated with the user based on a preset partitioning algorithm; Generate a service flow record corresponding to the business flow record; The service flow record is stored in the service flow table associated with the user.
7. A service terminal, characterized in that: include: A receiving unit, configured to receive a data change instruction of a business database, wherein the data change instruction is used to instruct execution of a pipeline update operation and a data update operation of the business database, wherein the pipeline update operation is used to generate a business pipeline record corresponding to the data update operation; A processing unit, used for processing the data update operation and the pipeline update operation as a same transaction; The processing unit is further used to synchronously update the business database and the business flow record based on the same transaction; The sending unit is used to send a flow update message corresponding to the business flow record to the consumer end if the business flow record is monitored, so that the consumer end generates a corresponding service flow record according to the business flow record.
8. A consumer terminal, characterized in that: include: A receiving unit is used to receive a transaction update message sent by a business end after monitoring a business transaction record, wherein the transaction update message corresponds to the business transaction record, and the business transaction record corresponding to the data update operation is generated by the transaction update operation. The data change instruction is used to instruct the execution of the transaction update operation and the data update operation of the business database, and the data update operation and the transaction update operation are processed as the same transaction, and the same transaction is used to synchronously update the business database and the business transaction record.
9. A computer device, characterized in that: include: CPU, memory and input / output interface; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 4 or 5 to 6.
10. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 4 or 5 to 6.
11. A computer storage medium, characterized in that: The computer storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 4 or 5 to 6.
