Data processing method, data server, electronic device and storage medium

Abstract

The present application discloses a data processing method, a data server, an electronic device and a storage medium, and relates to the field of big data technology. The specific implementation scheme is: when it is detected that the data in the data table of the source database has changed, the timestamp of the data change is recorded in the changed data; when the data table is subscribed, based on the identifier of the data table, the operation information of the data that has changed in the data table, and the data after the change, a data synchronization message is generated; the data synchronization message is added to the message queue of the message processing device, so that the target data server that subscribes to the data table can synchronize the changed data. The technical solution of the present application can effectively improve the data synchronization effect between the target data server side and the source data server side, and ensure the consistency of data between the target data service side and the source data service side.

Description

Technical Field

[0001] The present application relates to the field of computer technology, in particular to the field of big data technology, and specifically to a data processing method, a data server, an electronic device and a storage medium. Background Art

[0002] In actual applications, data in a database often needs to be shared among different business terminals, and data sharing is achieved through data synchronization, that is, a copy of the same set of data can be stored on each terminal.

[0003] In the prior art, data synchronization can usually be performed by transferring data from the source data service side to the target data service side through a message queue, which can decouple the data of both parties and separate the businesses of both parties. For example, in a specific implementation, the source data service side can write data-related messages into a message queue, and the message queue can push the data-related messages in the message queue to the target data service side that has subscribed to the data table to which the data belongs in turn based on the first-in-first-out principle, so that the target data service side can synchronize the data based on the data-related messages.

[0004] However, in the above mode, the timing of messages in the message queue can be guaranteed, but for distributed systems, the order of messages entering the message queue from the source data service side cannot be guaranteed, which corresponds to the order of business operations on the source data service side, resulting in inconsistent data between the target data service side and the source data service side. Summary of the invention

[0005] In order to solve the above problems, the present application provides a data processing method, a data server, an electronic device and a storage medium.

[0006] According to one aspect of the present application, a data processing method is provided, wherein the method comprises:

[0007] When it is detected that the data in the data table of the source database has changed, the timestamp of the data change is recorded in the changed data;

[0008] In the case where the data table is subscribed, generating a data synchronization message based on the identifier of the data table, the operation information of the data changed in the data table, and the changed data;

[0009] The data synchronization message is added to the message queue of the message processing device so that the target data server subscribed to the data table can synchronize the changed data.

[0010] According to another aspect of the present application, a data processing method is provided, wherein the method comprises:

[0011] Acquire a data synchronization message from a message queue of a message processing device, wherein the data synchronization message includes changed data when the data changes, and the changed data records a timestamp of the data change;

[0012] Data synchronization processing is performed based on the timestamp of the data change.

[0013] According to another aspect of the present application, a source data server is provided, wherein the source data server comprises:

[0014] A recording module is used to record the timestamp of the data change in the changed data when detecting that the data in the data table of the source database has changed;

[0015] A generating module, configured to generate a data synchronization message based on an identifier of the data table, operation information of the data changed in the data table, and the changed data when the data table is subscribed;

[0016] The adding module is used to add the data synchronization message to the message queue of the message processing device so that the target data server subscribed to the data table can synchronize the changed data.

[0017] According to another aspect of the present application, a target data server is provided, wherein the target data server comprises:

[0018] An acquisition module, used for acquiring a data synchronization message from a message queue of a message processing device, wherein the data synchronization message includes the changed data when the data changes, and the changed data records a timestamp of the data change;

[0019] The synchronization processing module is used to perform data synchronization processing based on the timestamp of the data change.

[0020] According to yet another aspect of the present application, there is provided an electronic device, including:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0024] According to yet another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0025] According to yet another aspect of the present application, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0026] According to the technology of the present application, by recording the timestamp of the data change in the changed data, the timestamp can uniquely identify the business operation time on the source data server side, so that the target data server that subscribes to the corresponding data table can perform data synchronization processing based on the timestamp of the data change. This can effectively improve the data synchronization effect between the target data server side and the source data server side, and ensure the consistency of data between the target data service side and the source data service side.

[0027] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0029] Figure 1 is an application architecture diagram of an embodiment of the present application;

[0030] Figure 2 is a schematic diagram according to the first embodiment of the present application;

[0031] Figure 3 is a schematic diagram according to a second embodiment of the present application;

[0032] Figure 4 is a schematic diagram according to a third embodiment of the present application;

[0033] Figure 5 is a schematic diagram according to a fourth embodiment of the present application;

[0034] Figure 6 is a schematic diagram according to a fifth embodiment of the present application;

[0035] Figure 7 is a schematic diagram according to a sixth embodiment of the present application;

[0036] Figure 8 is a schematic diagram according to a seventh embodiment of the present application;

[0037] Fig. 9 is a schematic diagram according to an eighth embodiment of the present application;

[0038] Fig.10 is a schematic diagram according to a ninth embodiment of the present application;

[0039] Fig.11 It is a block diagram of an electronic device used to implement the data processing method of an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0041] Figure 1 This is an application architecture diagram of an embodiment of the present application. Figure 1 As shown, taking the data in the database (DataBase; DB) on the source data server side as an example, data can be shared on multiple target data service sides. Figure 1 As shown, take the target data server 1, target data server 2 and target data server... as an example. Under this architecture, the principle of data sharing is: the source data server monitors the DB on the source data server side. When data synchronization is required, the data synchronization message is written into the message queue of the message processing device. When each target data server needs to synchronize data, it obtains the data synchronization message from the message queue and updates the corresponding target data server side DB according to the data synchronization message, thereby achieving data synchronization with the source data server side DB. Figure 1 The architecture shown can decouple the data on the source data server side and each target data server side, and can also separate the services on the source data server side and each target data server side.

[0042] It should be noted that the source data server side can manage and maintain countless data tables. Each target server can subscribe to the data table it needs in the message processing device to obtain data synchronization messages related to the subscribed data table in the message queue, thereby realizing synchronization processing of the data in the subscribed data table.

[0043] Figure 2 is a schematic diagram according to the first embodiment of the present application; Figure 2 As shown, this embodiment provides a data processing method, which may specifically include the following steps:

[0044] S101, when it is detected that data in a data table of a source database has changed, a timestamp of the data change is recorded in the changed data;

[0045] S102: When a data table is subscribed, a data synchronization message is generated based on an identifier of the data table, operation information of data changed in the data table, and the changed data;

[0046] S103: Add a data synchronization message to the message queue of the message processing device so that the target data server subscribing to the data table can synchronize the changed data.

[0047] The data processing method of this embodiment, Figure 1 The source data server side in the architecture diagram shown describes the technical solution of the present application.

[0048] The source database of this embodiment refers to the database on the source data server side, and multiple data tables can be managed and maintained in the source database. When the source data server detects that the data in the data table of the source database is inserted, modified or deleted, the data in the corresponding data table changes at this time. At this time, the source data server will synchronously record the operation information of the data change. Moreover, in this embodiment, it is also necessary to record the timestamp of the data change in the changed data. Optionally, the unit of the timestamp of this embodiment can reach milliseconds or even microseconds to accurately record the moment when the data changes. Then, based on the identifier of the data table, the operation information of the data that has changed in the data table, and the data after the change, a data synchronization message is generated, that is, in the data synchronization message, it is recorded what kind of operation has occurred on the data in which data table, what the data after the operation is, and what the timestamp of the operation causing the data to change is. Then the data synchronization message is added to the message queue of the message output device. According to the method of this embodiment, since the timestamp of the data change is recorded in the changed data, the timestamp can uniquely identify the moment of the business operation on the source data server side. Therefore, when the same target server obtains multiple data synchronization messages, the order of operations for changing each data can be determined based on the timestamps of the data changes in the multiple changed data. At this time, the identification information of the data synchronization message can be removed from the message queue. When the target data server that subsequently subscribes to the data table synchronizes the changed data, it can accurately complete the data synchronization based on the timestamp of the changed data and according to other messages in the data synchronization message.

[0049] In addition, optionally, in this embodiment, when the source data server detects that the data in the data table of the source database has changed, it records the timestamp of the data change in the changed data. In specific implementation, the information in the time information field in the data can be updated based on the timestamp of the data change.

[0050] That is to say, in this embodiment, a time information field can be added to the data table of the source database to record the timestamp of the corresponding data change. And every time the data in the data table changes, the time information field corresponding to the data is updated to ensure that the time information field always records the latest timestamp of the data change.

[0051] Optionally, in this embodiment, before generating a data synchronization message, it is also possible to detect whether the data table to which the changed data belongs is subscribed. If it is subscribed, a data synchronization message needs to be generated at this time for the subscribed target data server to update; otherwise, if the data table is not subscribed, there is no need to generate a corresponding data synchronization message at this time, because no target data server needs to update the data table.

[0052] In actual applications, the target data server subscribes to the data table in the database of the source data server to the message processing device. In the message processing device, a corresponding subscription relationship between the data table identifier and the target data server identifier can be established based on the subscription request on the target data server side. Furthermore, the message processing device can obtain the identifiers of all subscribed data tables based on the subscription relationships of all target data servers and send them to the source data server so that the source data server can record which data tables are subscribed. In this way, the corresponding data synchronization message is generated only when the data in the subscribed data table is detected to have changed; otherwise, it does not need to be generated, and the implementation is very flexible.

[0053] For example, according to the existing processing method, on the source data server side, if business operation 1 is executed first, but business operation 1 has more execution processes, and there are other related operations that have not been executed after the data changes, and although business operation 2 is initiated after business operation 1, its operation process is simple and is executed quickly, so when writing to the message queue, the data synchronization message corresponding to business operation 2 is written first, and the data synchronization message of business operation 1 is written later, and the queue follows the first-in-first-out rule, which causes the order of messages written to the message queue to be inconsistent with the actual business operation order. If business operations 1 and 2 involve processing the same piece of data, then on the target data server side, the target data server side is based on the order of executing business operation 2 first and then executing business operation 1, and it does not follow the business operation order of the source data server side, which leads to inconsistent data between the target data service side and the source data service side.

[0054] By adopting the technical solution of this embodiment, the identifier of the message is removed from the message queue, and the timestamp of the data change is added to the changed data corresponding to each of business operation 1 and business operation 2. At this time, when data synchronization is performed on the target data server side, data synchronization processing is performed based on the timestamp of the data change. Even if business operation 1 and business operation 2 correspond to the processing of the same data, data synchronization processing is performed according to the timestamp of the data change. If the business operation with a later timestamp is executed first, the business operation with an earlier timestamp can be discarded to ensure the absolute synchronization of the data on the target data server side with the data on the source data server side.

[0055] Through the above comparative analysis, it can be known that in the existing scheme, after the data synchronization message is added to the message queue, the message queue is used to ensure the timing of the data synchronization message. This scheme has certain problems. For example, for a distributed system, it is often impossible to ensure that the timing of entering the queue is the order of the business end operation. For example, the business party performs two business operations on the same data in sequence. Business operation 1 is to update its field status = 1, and business operation 2 updates its status = 2. The theoretical order should be that business operation 1 enters the queue first, and business operation 2 enters the queue later. However, due to the resource scheduling problem of the distributed system, the actual order may be that operation 2 enters the queue first, and business operation 1 enters the queue later. In this way, the receiver will first execute business operation 2 according to the queue order, that is, update status to 2, and then execute business operation 1, that is, update status to 1, which leads to data inconsistency between the source data server and the target data server. The scheme of this embodiment can avoid this problem, that is, the timing of the data is guaranteed by the business party, rather than by the identifier d of the data synchronization message of the message queue. For example, a time information field is added to each source data table structure, which can be named data_version field, and its attribute is set to datetime, and is "updated according to the timestamp of the operation time", where datetime is in milliseconds or microseconds. Similarly, for the above operation, although the receiver is sent to the receiver in the order of business operation 2 and business operation 1, the receiver will first execute business operation 2. When business operation 1 comes, the receiver will not update the data_version field in the changed data if it finds that it is earlier, thus ensuring the consistency of the data on the source data server and the target data server.

[0056] The data processing method of this embodiment records the timestamp of the data change in the changed data when detecting the data change in the data table of the source database; generates a message based on the identifier of the data table, the operation information of the changed data in the data table and the changed data; and adds a message to the message queue of the message processing device when the data table is subscribed, so that the target data server subscribed to the data table can synchronize the changed data. In this embodiment, by recording the timestamp of the data change in the changed data, the timestamp can uniquely identify the business operation time on the source data server side, so that the target data server subscribed to the corresponding data table can perform data synchronization processing based on the timestamp of the data change, which can effectively improve the data synchronization effect between the target data server side and the source data server side, and ensure the consistency of data between the target data service side and the source data service side.

[0057] Figure 3 is a schematic diagram according to the second embodiment of the present application; Figure 3 As shown, the data processing method of this embodiment, in the above Figure 2 Based on the technical solutions of the illustrated embodiments, the technical solutions of the present application are further described in more detail. Figure 3 As shown, the data processing method of this embodiment may specifically include the following steps:

[0058] S201, receiving a data synchronization request carrying a data table identifier and a time period identifier sent by a target data server;

[0059] S202, obtaining each piece of data operation information within a corresponding time period and the changed data in the corresponding data table;

[0060] S203, generating a corresponding data synchronization message group based on each piece of data operation information in a time period and the changed data in the corresponding data table; the data synchronization message group includes at least one data synchronization message;

[0061] S204: Add a data synchronization message group to the message queue of the message processing device, so that the target data server can obtain the data synchronization message group from the message queue, and synchronize the data in the data table within the time period based on the data synchronization message group in the message queue.

[0062] Above Figure 2The data processing method of the illustrated embodiment can ensure that the data on the source data server side can be synchronized to the subscribed target data server side in a timely manner when the data changes, so that the data on the target data server side can be synchronized with the data on the source data server side. However, in actual applications, there may still be an abnormality in the message queue or the message processing device where the message queue is located, resulting in the target data server being unable to obtain the data synchronization message in the message queue, resulting in the failure to perform data synchronization processing in a timely manner.

[0063] Therefore, based on this scenario, in this embodiment, the target data server can also perform data synchronization detection regularly to detect whether the data table it maintains is consistent with the data table on the source data server side. At this time, the target data server side can directly access the source data server to request to detect and query the data in a certain data table, and compare it with the data in the corresponding data table in the target data server. If it is not synchronized, the target data server can initiate a data synchronization request to the source data server, and carry the data table identifier and time period identifier in the data synchronization request. Since the target data server performs regular detection and ensures that the data after each detection has reached synchronization. Therefore, the target data server performs regular detection, which can be understood as detecting the data within the time period from the last detection to the current detection. The time period can be a determined time interval based on the last detection of the local detection, for example, it can be identified by the start time and the end time, or a day can be pre-divided into multiple time segments, and regular detection is performed at the start time of each time segment. At this time, the identifier of the time segment can also be used to identify the corresponding time period.

[0064] Correspondingly, on the source data server side, the source data server can receive the data synchronization request carrying the data table identifier and the time period identifier sent by the target data server, and obtain each piece of data operation information within the corresponding time period and the changed data in the corresponding data table, and then generate a corresponding data synchronization message group based on each piece of data operation information in the time period and the changed data in the corresponding data table, wherein the data synchronization message group includes at least one data synchronization message; wherein each data synchronization message is consistent with the above Figure 1 The method for generating the data synchronization message in the illustrated embodiment is the same. The data synchronization message group here includes all data synchronization messages within a time period. Then, the data synchronization message group is added to the message queue of the message processing device, so that the target data server can obtain the data synchronization message group from the message queue, and synchronize the data in the data table within the time period based on the data synchronization message group in the message queue. The implementation principle is the same as the above Figure 1The principle of synchronizing a data synchronization message in the embodiment shown is the same. In a similar manner, data synchronization is performed on the target data server side based on each data synchronization message in the data synchronization message group, so that the data table on the target data server side can be consistent with the data table on the source data server side.

[0065] In this embodiment, the target data server uses data tables and time periods as detection units for synchronization detection. If it is detected that multiple maintained data tables are inconsistent with the data on the source data server side within the same time period, at this time, a data synchronization request can be initiated based on each data table, or a data synchronization request including multiple data table identifiers and time period identifiers can be initiated simultaneously. For each data table, a corresponding data synchronization message group can be generated, so that the target data server can synchronize the data in the data table based on the data synchronization message group. Of course, the data synchronization message groups of all data tables can also be combined together as a large data synchronization message group. At this time, it is necessary to determine the corresponding data table for each data synchronization message, and then perform the corresponding data synchronization operation. In actual applications, the corresponding operation processing method can be selected based on specific needs, and it is not limited here.

[0066] In the prior art, the related scheme is also implemented by a timed task, which polls and detects the difference (diff) between the data on the target data server and the source data server at regular intervals, issues an alarm for inconsistent data, and notifies maintenance personnel to perform manual synchronization. However, the existing scheme requires manual synchronization every time the data is inconsistent, which is labor-intensive, and maintenance personnel are unlikely to be online 24 hours a day, so that the data inconsistency exists online for a certain period of time. With the technical solution of this embodiment, the target data server performs the timed monitoring task, and when it finds that the data on both sides is diff, it notifies the source data server, which synchronizes the difference data again, completes the automation of data synchronization, and can effectively reduce labor costs.

[0067] The data processing method of this embodiment, by adopting the above technical solution, can automatically start data synchronization processing when asynchrony is detected periodically, so as to timely perform data synchronization processing between the source data server and the target data server, thereby ensuring data synchronization between the source data server and the target data server. Compared with the prior art, it can effectively reduce the cost of manual data synchronization processing and ensure data consistency between the source data server and the target data server.

[0068] Figure 4 is a schematic diagram according to the third embodiment of the present application; Figure 4 As shown, this embodiment also provides a data processing method, which may specifically include the following steps:

[0069] S301, obtaining a data synchronization message from a message queue of a message processing device, wherein the data synchronization message includes the changed data when the data changes, and the changed data records a timestamp of the data change;

[0070] S302: Perform data synchronization based on the timestamp of the data change.

[0071] With the above Figure 2 The difference between the embodiment shown is that the data processing method of this embodiment, in Figure 1 Any target data server side in the architecture diagram shown describes the technical solution of the present application.

[0072] In this embodiment, the message queue of the message processing device stores data synchronization messages of all data tables in the database of the source data server, but each target data server may only share one, two or more data tables. Therefore, when performing data synchronization processing, each target data server only needs to obtain the data synchronization message corresponding to the data table it maintains, and does not need to obtain the data synchronization message of the data table that is not related to itself. Therefore, before step S301, each target data server needs to subscribe to the data synchronization message of the required data table from the message processing device. For example, in the message processing device, a corresponding subscription relationship between the data table identifier and the target data server identifier can be established based on the subscription request on the target data server side. In this way, the message processing device can push the corresponding data synchronization message to the target data server that subscribes to the data table corresponding to the data synchronization message based on the subscription relationship. Or based on the request of each target data server, the corresponding target data server pulls the data synchronization message corresponding to the subscribed data table from the message queue. Of course, before the pull, the message processing device needs to detect that the target data server has the authority to pull the data synchronization message based on the subscription relationship.

[0073] In this embodiment, when the target data server obtains the data synchronization message from the message queue of the message processing device, since the data synchronization message includes the changed data when the data changes, and the changed data records the timestamp of the data change; thus, the timestamp of the data change can be further obtained from the data synchronization message; refer to the above Figure 2 According to the relevant records of the embodiment, the timestamp of the data change can uniquely identify the timing of the business operation on the source data server side. In this embodiment, based on the timestamp of the data change, data synchronization processing is performed with reference to other messages in the data synchronization message, which can ensure the consistency of the data on the target data server side and the data on the source data server side.

[0074] For example, if multiple data synchronization messages are obtained at the same time, each data synchronization message can be executed in sequence based on the order of the timestamps of the data changes in each data synchronization message to ensure that the order of data operations is consistent with the business operation order of the source data server, thereby ensuring data consistency.

[0075] The data processing method of this embodiment obtains a data synchronization message from a message queue of a message processing device, wherein the data synchronization message includes the changed data when the data is changed, and the changed data records the timestamp of the data change; based on the timestamp of the data change, data synchronization processing is performed. Since the timestamp can uniquely identify the business operation time on the source data server side, the target data server performs data synchronization processing based on the timestamp of the data change, which can effectively improve the data synchronization effect between the target data server side and the source data server side, and ensure the consistency of data between the target data service side and the source data service side.

[0076] Figure 5 is a schematic diagram according to the fourth embodiment of the present application; Figure 5 As shown, the data processing method of this embodiment, in the above Figure 4 Based on the technical solutions of the illustrated embodiments, the technical solutions of the present application are further described in more detail. Figure 5 As shown, the data processing method of this embodiment may specifically include the following steps:

[0077] S401, obtaining a data synchronization message from a message queue of a message processing device, wherein the data synchronization message includes an identifier of a corresponding data table, operation information of data changed in the data table, and data after the data changes, and the changed data records a timestamp of the data change;

[0078] S402, detecting whether the processed message involves the same piece of data in the data synchronization message; if so, executing step S403; otherwise, executing step S405;

[0079] S403, detecting whether the timestamp of the data change recorded in the data synchronization message is earlier than the timestamp of the data change recorded in the processed message; if so, executing step S404; otherwise, executing step S405;

[0080] S404: discard the data synchronization message to ensure data synchronization, and end.

[0081] S405: Based on the identifier of the data table in the data synchronization message and the operation information of the changed data in the data table, synchronize the changed data in the data table, and end.

[0082] In this embodiment, it is taken as an example that a data synchronization message is obtained on the target data server side each time. However, in order to ensure the synchronization of data with the source data server side, for each data synchronization message obtained, before the data synchronization processing is performed, it is first detected whether the data synchronization message is the same data involved in the processing of the processed message. If so, in order to ensure the accuracy of the operation sequence, it is first detected whether the timestamp of the data change recorded in the data synchronization message is earlier than the timestamp of the data change recorded in the processed message. If so, it means that the data synchronization message should be received earlier and processed earlier, but now it is later than the timestamp of the data change recorded in the processed message. If the operation is executed at this time, it will cause the operation sequence to be disordered. At this time, the data synchronization message can be discarded and not executed to ensure data synchronization. If the timestamp of the data change recorded in the data synchronization message is later than the timestamp of the data change recorded in the processed message, or the data involved in the data synchronization message is not the same data involved in the processed message, at this time, the data synchronization message and the processed message do not affect each other, and the changed data in the data table can be synchronized directly based on the identifier of the data table in the data synchronization message and the operation information of the changed data in the data table.

[0083] The data processing method of this embodiment, by adopting the above-mentioned technical scheme, can perform data synchronization processing based on the timestamp of the data change, and can discard the data synchronization message when the timestamp of the data change recorded in the data synchronization message is earlier than the timestamp of the data change recorded in the processed message to ensure data synchronization; and when the processed message and the data synchronization message do not involve the same data, or the timestamp of the data change recorded in the data synchronization message is later than the timestamp of the data change recorded in the processed message, based on the identifier of the data table in the data synchronization message and the operation information of the data changed in the data table, the changed data in the data table is synchronized, which can effectively improve the data synchronization effect between the target data server side and the source data server side, and ensure the consistency of data between the target data service side and the source data service side.

[0084] Figure 6 is a schematic diagram according to the fifth embodiment of the present application; Figure 6 As shown, the data processing method of this embodiment, in the above Figure 4 or Figure 5 Based on the technical solutions of the illustrated embodiments, the technical solutions of the present application are further described in more detail. Figure 6 As shown, the data processing method of this embodiment may specifically include the following steps:

[0085] S501, periodically checking whether the data table within the time period is synchronized with the data table in the source database on the subscribed source data server side; if not, executing step S502; otherwise, returning to step S501 to continue checking;

[0086] S502, sending a data synchronization request carrying a data table identifier and a time period identifier to a source data server;

[0087] S503: Obtain a data synchronization message group from a message queue, where the data synchronization message group is a message group including at least one data synchronization message generated by a source data server based on a data table identifier and a time period identifier;

[0088] S504: Based on the data synchronization message group, synchronize the data in the data table within the time period in the database on the target data server side.

[0089] The above steps S501-S504 are the same as the above steps S501-S504. Figure 3 The difference of the embodiment shown is only that: Figure 3 The embodiment shown describes the technical solution of the present application on the source data server side; while this embodiment describes the technical solution of the present application on the target data server side, and the remaining implementation principles are the same as those described above. Figure 3 The embodiments shown are the same, and the details can be referred to above Figure 3 The description of the illustrated embodiments will not be repeated here.

[0090] S505: When a new data table needs to be synchronized, a configuration file of the new data table is generated based on the structures of the new data table and the synchronized data table, and the logical function implementation file of the synchronized data table.

[0091] It should be noted that when a new data table needs to be synchronized on the target data server side, the structure of the new data table needs to be configured in advance on the target data server side, and then various logical functions of the new data table need to be configured on the target data server side, such as how to find the data table, how to insert, modify or delete data in the data table, etc., to ensure that all functions of the data table can be used on the target data server side. In actual applications, the implementation of these logical functions of the data table is through writing program code, completing the test, and then going online. This process requires a certain amount of manpower cost to write program code, and it is easy to cause errors in old functions.

[0092] In order to overcome this technical problem, in this application, the same set of program codes can be deployed. When a new data table needs to be synchronized, based on the structures of the new data table and the synchronized data table, and the logical function implementation file of the synchronized data table, only the changed part of the structure of the new data table relative to the structure of the synchronized data table is abstracted into a configuration file with reference to the logical function implementation file of the synchronized data table, and output in json format, such as:

[0093]

[0094]

[0095] Among them, mirror_ral and to_ral represent the service configuration of the data in the synchronized data table and the new data table respectively;

[0096] Topic_name represents the name of the synchronized database table;

[0097] table_name represents the name of the new database table; pattern is the name of the new data table, handler is the table sharding strategy, and field is the field of the sharded table.

[0098] allow_topic_types represents the type of data synchronization operation. 1, 2, and 3 are insert, update, and delete operations, respectively.

[0099] fields represents the fields of the database table, name is the field name, type is the field type, required indicates whether the data source must be transmitted, and primary_key indicates whether it is the primary key of the table.

[0100] This step S505 is a supplementary solution provided by this embodiment. This step S505 can also be respectively Figure 4 or Figure 5 The illustrated embodiments are combined to form an optional technical solution of the present application.

[0101] The solution of step S505 can effectively improve development efficiency, save costs, and improve development accuracy by abstracting the configuration file of the new data table without making any program code changes for the newly added data table.

[0102] The data processing method of this embodiment, by adopting the above technical solution, can automatically start data synchronization processing when asynchrony is detected periodically, so as to timely perform data synchronization processing between the source data server and the target data server, thereby ensuring data synchronization between the source data server and the target data server. Compared with the prior art, it can effectively reduce the cost of manual data synchronization processing and ensure data consistency between the source data server and the target data server.

[0103] Figure 7 is a schematic diagram according to the sixth embodiment of the present application; Figure 7 As shown, this embodiment provides a source data server 700, and the source data server 700 includes:

[0104] The recording module 701 is used to record the timestamp of the data change in the changed data when detecting that the data in the data table of the source database has changed;

[0105] A generating module 702 is used to generate a data synchronization message based on the identifier of the data table, the operation information of the changed data in the data table, and the changed data when the data table is subscribed;

[0106] The adding module 703 is used to add a data synchronization message to the message queue of the message processing device so that the target data server subscribing to the data table can synchronize the changed data.

[0107] The source data server 700 of this embodiment realizes the data processing by adopting the above modules, and the implementation principle and technical effect are the same as those of the above related method embodiments. For details, please refer to the records of the above related method embodiments, which will not be repeated here.

[0108] Figure 8 is a schematic diagram according to the seventh embodiment of the present application; Figure 8 As shown, the source data server 700 of this embodiment, in the above Figure 7 Based on the technical solution of the illustrated embodiment, the technical solution of the present application is further described in more detail.

[0109] The source data server 700 and the recording module 701 of this embodiment are used to:

[0110] When it is detected that the data in the data table of the source database has changed, the information of the time information field in the data is updated based on the timestamp of the data change.

[0111] Further optionally, if Figure 8 As shown, the source data server 700 of this embodiment further includes:

[0112] The adding module 704 is used to add a time information field to the data table of the source database.

[0113] Further optionally, if Figure 8 As shown, the source data server 700 of this embodiment further includes:

[0114] The detection module 705 is used to detect and determine whether the data table is subscribed.

[0115] Further optionally, if Figure 8 As shown, the source data server 700 of this embodiment further includes a receiving module 706 and an acquiring module 707;

[0116] The receiving module 706 is used to receive a data synchronization request carrying a data table identifier and a time period identifier sent by a target data server;

[0117] An acquisition module 707 is used to acquire each piece of data operation information within a corresponding time period and the changed data in the corresponding data table;

[0118] The generating module 702 is further used to generate a corresponding data synchronization message group based on each piece of data operation information in a time period and the changed data in the corresponding data table, wherein the data synchronization message group includes at least one data synchronization message;

[0119] The adding module 703 is also used to add a data synchronization message group to the message queue of the message processing device, so that the target data server can obtain the data synchronization message group from the message queue, and synchronize the data in the data table within the time period based on the data synchronization message group in the message queue.

[0120] The source data server 700 of this embodiment realizes the data processing by adopting the above modules, and the implementation principle and technical effect are the same as those of the above related method embodiments. For details, please refer to the records of the above related method embodiments, which will not be repeated here.

[0121] Fig. 9 is a schematic diagram according to the eighth embodiment of the present application; Figure 8 As shown, this embodiment provides a target data server 900, and the target data server 900 includes:

[0122] The acquisition module 901 is used to acquire a data synchronization message from a message queue of a message processing device, wherein the data synchronization message includes the changed data when the data changes, and the changed data records a timestamp of the data change;

[0123] The synchronization processing module 902 is used to perform data synchronization processing based on the timestamp of the data change.

[0124] The target data server 900 of this embodiment realizes the data processing by adopting the above modules, and the implementation principle and technical effect are the same as those of the above related method embodiments. For details, please refer to the records of the above related method embodiments, which will not be repeated here.

[0125] Fig.10 is a schematic diagram according to the ninth embodiment of the present application; Fig.10 As shown, the target data server 900 of this embodiment, in the above Fig. 9 Based on the technical solution of the illustrated embodiment, the technical solution of the present application is further described in more detail.

[0126] like Fig.10 As shown, in the target data server 900 of this embodiment, the synchronization processing module 902 includes:

[0127] An information detection unit 9021 is used to detect whether the processed message involves the same piece of data in the data synchronization message;

[0128] The information detection unit 9021 is further used to detect whether the timestamp of the data change recorded in the data synchronization message is earlier than the timestamp of the data change recorded in the processed message;

[0129] The processing unit 9022 is configured to discard the data synchronization message if yes, so as to ensure data synchronization.

[0130] Further optionally, the data synchronization message also includes an identifier of a data table to which the data belongs and operation information of the data changed in the data table; the processing unit 9022 is further configured to:

[0131] If the processed message does not involve the same data in the data synchronization message, or the timestamp of the data change recorded in the data synchronization message is later than the timestamp of the data change recorded in the processed message, the changed data in the data table is synchronized based on the identifier of the data table and the operation information of the changed data in the data table.

[0132] Further optionally, if Fig.10 As shown, the target data server 900 of this embodiment further includes a synchronization detection module 903 and a sending module 904:

[0133] A synchronization detection module 903 is used to periodically detect whether a data table within a time period is synchronized with a data table in a source database on a subscribed source data server side;

[0134] A sending module 904 is used to send a data synchronization request carrying a data table identifier and a time period identifier to a source data server if the data is not synchronized;

[0135] The acquisition module 901 is further used to acquire a data synchronization message group from the message queue, where the data synchronization message group is a message group including at least one data synchronization message generated by the source data server based on the data table identifier and the time period identifier;

[0136] The synchronization processing module 902 is also used to synchronize the data in the data table within a time period in the database on the target data server side based on the data synchronization message group.

[0137] For example, the processing unit 9022 may synchronize the data in the data table within a time period in the database on the target data server side based on the data synchronization message group.

[0138] Further optionally, if Fig.10 As shown, the target data server 900 of this embodiment further includes:

[0139] The generation module 905 is used to generate a configuration file of the new data table based on the structures of the new data table and the synchronized data table, and the logical function implementation file of the synchronized data table when a new data table needs to be synchronized.

[0140] The target data server 900 of this embodiment realizes the data processing by adopting the above modules, and the implementation principle and technical effect are the same as those of the above related method embodiments. For details, please refer to the records of the above related method embodiments, which will not be repeated here.

[0141] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0142] like Fig.11 , is a block diagram of an electronic device for implementing a data processing method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein. The electronic device of this embodiment may be used as a source data server or a target data server, respectively,

[0143] like Fig.11As shown, the electronic device includes: one or more processors 1101, a memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.11 A processor 1101 is taken as an example.

[0144] The memory 1102 is a non-transient computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the data processing method provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the data processing method provided in the present application.

[0145] The memory 1102 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the data processing method in the embodiment of the present application (for example, the attached Figure 7 And attached Figure 8 , or attach Fig. 9 And attached Fig.10 The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, that is, implements the data processing method in the above method embodiment.

[0146] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of an electronic device that implements the data processing method, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device that implements the data processing method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The electronic device implementing the data processing method may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0148] The input device 1103 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device implementing the data processing method, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0149] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0150] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0152] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0153] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0154] According to the technical solution of the embodiment of the present application, when the data in the data table of the source database is detected to have changed, the timestamp of the data change is recorded in the changed data; based on the identifier of the data table, the operation information of the changed data in the data table and the changed data, a message is generated; when the data table is subscribed, a message is added to the message queue of the message processing device so that the target data server subscribed to the data table can synchronize the changed data. In this embodiment, by recording the timestamp of the data change in the changed data, the timestamp can uniquely identify the business operation moment on the source data server side, so that the target data server subscribed to the corresponding data table can perform data synchronization processing based on the timestamp of the data change, which can effectively improve the data synchronization effect between the target data server side and the source data server side, and ensure the consistency of data between the target data service side and the source data service side.

[0155] According to the technical solution of the embodiment of the present application, when asynchrony is detected periodically, data synchronization processing can be automatically started to perform data synchronization processing between the source data server and the target data server in a timely manner, thereby ensuring data synchronization between the source data server and the target data server. Compared with the prior art, the cost of manual data synchronization processing can be effectively reduced, and data consistency between the source data server and the target data server can be ensured.

[0156] According to the technical solution of the embodiment of the present application, a data synchronization message is obtained from the message queue of a message processing device, wherein the data synchronization message includes the changed data when the data changes, and the changed data records the timestamp of the data change; data synchronization processing is performed based on the timestamp of the data change. Since the timestamp can uniquely identify the business operation moment on the source data server side, the target data server performs data synchronization processing based on the timestamp of the data change, which can effectively improve the data synchronization effect between the target data server side and the source data server side, and ensure the consistency of data between the target data service side and the source data service side.

[0157] According to the technical solution of the embodiments of the present application, data synchronization processing can be performed based on the timestamp of the data change. When the timestamp of the data change recorded in the data synchronization message is earlier than the timestamp of the data change recorded in the processed message, the data synchronization message can be discarded to ensure data synchronization. When the processed message and the data synchronization message do not involve the same data, or the timestamp of the data change recorded in the data synchronization message is later than the timestamp of the data change recorded in the processed message, the changed data in the data table is synchronized based on the identifier of the data table in the data synchronization message and the operation information of the changed data in the data table, which can effectively improve the data synchronization effect between the target data server side and the source data server side and ensure the consistency of data between the target data service side and the source data service side.

[0158] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0159] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A data processing method, wherein: The method comprises: When a change in data in a data table of a source database is detected, a timestamp of the data change is recorded in the changed data; the timestamp of the data change can uniquely identify the time of the corresponding business operation when the data changes on the source data server side; In the case where the data table is subscribed, generating a data synchronization message based on the identifier of the data table, the operation information of the data changed in the data table, and the changed data; Adding the data synchronization message to a message queue of a message processing device so that a target data server subscribed to the data table can synchronize the changed data; In the case where the data table is subscribed, before generating a data synchronization message based on the identifier of the data table, the operation information of the data changed in the data table, and the changed data, the method further includes: The message processing device receives the identifier of the subscribed data table sent by the message processing device, and the message processing device establishes a corresponding subscription relationship between the data table identifier requested for subscription and the corresponding target data server identifier based on the subscription request on the target data server side; and determines the identifier of the subscribed data table based on the corresponding subscription relationship of each target data server.

2. The method according to claim 1, wherein: When a change is detected in the data table of the source database, the timestamp of the data change is recorded in the changed data, including: When it is detected that the data in the data table of the source database is changed, the information of the time information field in the data is updated based on the timestamp of the data change.

3. The method according to claim 2, wherein: When it is detected that the data in the data table of the source database has changed, before updating the information of the time information field in the data based on the timestamp of the data change, the method further includes: The time information field is added to the data table of the source database.

4. The method according to claim 1, wherein: Before generating a data synchronization message based on the identifier of the data table, the operation information of the changed data in the data table, and the changed data, the method further includes: Detect and determine that the data table is subscribed.

5. The method according to any one of claims 1 to 4, wherein: The method further comprises: Receiving a data synchronization request sent by the target data server and carrying the data table identifier and the time period identifier; Acquire each piece of data operation information within the corresponding time period and the changed data in the corresponding data table; Based on each piece of the data operation information in the time period and the changed data in the corresponding data table, generating a corresponding data synchronization message group, wherein the data synchronization message group includes at least one data synchronization message; The data synchronization message group is added to the message queue of the message processing device, so that the target data server obtains the data synchronization message group from the message queue, and synchronizes the data in the data table within the time period based on the data synchronization message group in the message queue.

6. A data processing method, wherein: The method comprises: Subscribe a subscription request for a required data table to a message processing device, so that the message processing device establishes a corresponding subscription relationship between a data table identifier and a target data server identifier based on the subscription request; Obtaining a data synchronization message from a message queue of a message processing device, wherein the data synchronization message includes the changed data when the data changes, and the changed data records a timestamp of the data change; the timestamp of the data change can uniquely identify the time of the corresponding business operation when the data changes on the source data server side; the data synchronization message also includes an identifier of the data table subscribed by the target data server; Data synchronization processing is performed based on the timestamp of the data change.

7. The method according to claim 6, wherein: Based on the timestamp of the data change, data synchronization processing is performed, including: Detecting whether the processed message involves the same piece of data in the data synchronization message; If so, detecting whether the timestamp of the data change recorded in the data synchronization message is earlier than the timestamp of the data change recorded in the processed message; If yes, the data synchronization message is discarded to ensure data synchronization.

8. The method according to claim 7, wherein: The data synchronization message also includes an identifier of the data table to which the data belongs and operation information of the data changed in the data table; the method also includes: If the processed message does not involve the same data in the data synchronization message, or the timestamp of the data change recorded in the data synchronization message is later than the timestamp of the data change recorded in the processed message, the changed data in the data table is synchronized based on the identifier of the data table and the operation information of the changed data in the data table.

9. The method according to any one of claims 6 to 8, further comprising: Periodically checking whether the data table within the time period is synchronized with the data table in the source database on the subscribed source data server side; If not synchronized, a data synchronization request carrying a data table identifier and a time period identifier is sent to the source data server; Acquire a data synchronization message group from the message queue, where the data synchronization message group is a message group including at least one data synchronization message generated by the source data server based on the data table identifier and the time period identifier; Based on the data synchronization message group, the data in the data table within the time period is synchronized in the database on the target data server side.

10. The method according to any one of claims 6 to 8, further comprising: When a new data table needs to be synchronized, a configuration file of the new data table is generated based on the structures of the new data table and the synchronized data table, and the logical function implementation file of the synchronized data table.

11. A source data server, wherein: The source data server comprises: A recording module, for recording a timestamp of the data change in the changed data when detecting a data change in the data table of the source database; the timestamp of the data change can uniquely identify the time of the corresponding business operation when the data changes on the source data server side; A generating module, configured to generate a data synchronization message based on an identifier of the data table, operation information of the data changed in the data table, and the changed data when the data table is subscribed; An adding module, used for adding the data synchronization message to the message queue of the message processing device, so that the target data server subscribed to the data table can synchronize the changed data; The receiving module is used to receive the identifier of the subscribed data table sent by the message processing device. The message processing device establishes a corresponding subscription relationship between the data table identifier requested for subscription and the corresponding target data server identifier based on the subscription request on the target data server side; and determines the identifier of the subscribed data table based on the corresponding subscription relationship of each target data server.

12. The source data server according to claim 11, wherein: The recording module is used to: When it is detected that the data in the data table of the source database is changed, the information of the time information field in the data is updated based on the timestamp of the data change.

13. The source data server according to claim 12, wherein: The source data server also includes: An adding module is used to add the time information field to the data table of the source database.

14. The source data server according to claim 11, wherein: The source data server also includes: The detection module is used to detect and determine that the data table is subscribed.

15. The source data server according to any one of claims 11 to 14, wherein: The source data server also includes an acquisition module: The receiving module is used to receive the data synchronization request sent by the target data server and carrying the data table identifier and the time period identifier; The acquisition module is used to acquire each piece of data operation information within the corresponding time period and the changed data in the corresponding data table; The generating module is further configured to generate a corresponding data synchronization message group based on each piece of the data operation information in the time period and the changed data in the corresponding data table, wherein the data synchronization message group includes at least one data synchronization message; The adding module is also used to add the data synchronization message group to the message queue of the message processing device, so that the target data server can obtain the data synchronization message group from the message queue, and synchronize the data in the data table within the time period based on the data synchronization message group in the message queue.

16. A target data server, wherein: The target data server comprises: A sending module, used for sending a subscription request for a required data table to a message processing device, so that the message processing device can establish a corresponding subscription relationship between a data table identifier and a target data server identifier based on the subscription request; An acquisition module is used to acquire a data synchronization message from a message queue of a message processing device, wherein the data synchronization message includes the changed data when the data changes, and the changed data records a timestamp of the data change; the timestamp of the data change can uniquely identify the time of the corresponding business operation when the data changes on the source data server side; the data synchronization message also includes an identifier of the data table subscribed by the target data server; The synchronization processing module is used to perform data synchronization processing based on the timestamp of the data change.

17. The target data server according to claim 16, wherein: The synchronization processing module comprises: An information detection unit, used to detect whether the processed message involves the same piece of data in the data synchronization message; The information detection unit is further configured to detect whether the timestamp of the data change recorded in the data synchronization message is earlier than the timestamp of the data change recorded in the processed message; The processing unit is configured to discard the data synchronization message if yes, so as to ensure data synchronization.

18. The target data server according to claim 17, wherein: The data synchronization message also includes an identifier of a data table to which the data belongs and operation information of the data changed in the data table; the processing unit is further used to: If the processed message does not involve the same data in the data synchronization message, or the timestamp of the data change recorded in the data synchronization message is later than the timestamp of the data change recorded in the processed message, the changed data in the data table is synchronized based on the identifier of the data table and the operation information of the changed data in the data table.

19. The target data server according to any one of claims 16 to 18, further comprising a synchronization detection module: The synchronization detection module is used to periodically detect whether the data table within a time period is synchronized with the data table in the source database on the subscribed source data server side; The sending module is used to send a data synchronization request carrying a data table identifier and a time period identifier to the source data server if there is no synchronization; The acquisition module is further used to acquire a data synchronization message group from the message queue, wherein the data synchronization message group is a message group including at least one data synchronization message generated by the source data server based on the data table identifier and the time period identifier; The synchronization processing module is further used to synchronize the data in the data table within the time period in the database on the target data server side based on the data synchronization message group.

20. The target data server according to any one of claims 16 to 18, further comprising: The generation module is used to generate a configuration file of the new data table based on the structures of the new data table and the synchronized data table, and the logical function implementation file of the synchronized data table when a new data table needs to be synchronized.

21. An electronic device, wherein: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5 or 6-10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5 or 6 to 10.

23. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1-5, or 6-10.

Images