Data synchronization method and device, computer equipment, readable storage medium and program product

By acquiring and parsing the messages to be synchronized in the synchronization message queue, and determining the target data and message queues based on the preset transmission strategy, the problems of low data synchronization efficiency and poor flexibility in the prior art are solved, and efficient and flexible data synchronization is achieved.

CN118964490APending Publication Date: 2024-11-15SUZHOU LANGDONG NET TEC CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411119171.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing technology has low efficiency and poor flexibility in data processing, making it difficult to meet the real-time data synchronization needs in financial, e-commerce and other scenarios.

Method used

By obtaining the messages to be synchronized in the synchronization message queue, parsing and determining the target data and target message queues based on the preset transmission strategy, data synchronization is achieved, data processing is reduced, and data processing is improved.

Benefits of technology

It improves data processing efficiency and flexibility, is suitable for a variety of application scenarios, reduces resource waste and data processing workload, and realizes unaware synchronization of downstream data changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118964490A_ABST
    Figure CN118964490A_ABST
Patent Text Reader

Abstract

The invention relates to a data synchronization method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps that a to-be-synchronized message in a synchronization message queue is acquired, and the synchronization message queue is used for receiving a change message generated by data change in a preset database; analyzing the to-be-synchronized message to obtain to-be-synchronized data; according to a preset transmission strategy, target data is determined from the to-be-synchronized data, a target message queue corresponding to the target data is determined, and the preset transmission strategy comprises a mapping relation between the data and the message queue; and synchronizing the target data to the target message queue. By adopting the method, the data processing efficiency and flexibility can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data synchronization method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of the information age, the requirements for data real-time are becoming higher and higher in other scenarios that require rapid response, such as finance, e-commerce, social media, and the Internet of Things. Therefore, it is necessary to synchronize data changes to downstream systems in a timely manner.

[0003] In the related art, the query data is first written into the database, and each time the data changes, the data in the database is uniformly updated, which requires a large amount of data processing work, poor flexibility, and low efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a data synchronization method, device, computer equipment, computer-readable storage medium and computer program product that can improve data processing efficiency and flexibility in response to the above technical problems.

[0005] In a first aspect, the present application provides a data synchronization method, comprising:

[0006] Obtaining a message to be synchronized in a synchronization message queue, wherein the synchronization message queue is used to receive a change message generated by a data change in a preset database;

[0007] Parsing the message to be synchronized to obtain the data to be synchronized;

[0008] According to a preset transmission strategy, determining target data from the data to be synchronized, and determining a target message queue corresponding to the target data, wherein the preset transmission strategy includes a mapping relationship between data and message queues;

[0009] The target data is synchronized to the target message queue.

[0010] In one of the embodiments, before determining the target data from the data to be synchronized according to the preset transmission strategy, the method further includes:

[0011] Obtaining the data change type corresponding to the data to be synchronized;

[0012] In response to the data change type being consistent with a preset synchronization type, a preset transmission strategy is obtained, wherein the preset synchronization type includes at least one of a new addition type and a modification type.

[0013] In one embodiment, in response to the data change type being consistent with the preset synchronization type, obtaining the preset transmission strategy includes:

[0014] In response to the data change type being a modification type, obtaining data description fields before and after the change corresponding to the data to be synchronized;

[0015] In response to the data description fields before and after the change being different, a preset transmission strategy is obtained.

[0016] In one embodiment, determining the target data from the data to be synchronized according to a preset transmission strategy includes:

[0017] Get the preset transmission strategy;

[0018] Determine the target transmission field according to the preset transmission strategy;

[0019] The data corresponding to the target transmission field is determined from the data to be synchronized to obtain the target data.

[0020] In one embodiment, synchronizing the target data to the target message queue includes:

[0021] Determining a target transmission format according to the preset transmission strategy;

[0022] Converting the data format of the target data into the target transmission format to obtain converted target data;

[0023] The converted target data is synchronized to the target message queue.

[0024] In one embodiment, the method for obtaining the preset transmission strategy includes:

[0025] Determine the target transmission interface;

[0026] The preset transmission strategy is periodically obtained through the target transmission interface.

[0027] In a second aspect, the present application further provides a data synchronization device, including:

[0028] An acquisition module, used to acquire the to-be-synchronized message in the synchronization message queue, wherein the synchronization message queue is used to receive the change message generated by the data change in the preset database;

[0029] A parsing module, used for parsing the message to be synchronized to obtain the data to be synchronized;

[0030] A determination module, configured to determine target data from the data to be synchronized and determine a target message queue corresponding to the target data according to a preset transmission strategy, wherein the preset transmission strategy includes a mapping relationship between data and message queues;

[0031] A synchronization module is used to synchronize the target data to the target message queue.

[0032] In one embodiment, the device further comprises:

[0033] A first acquisition submodule is used to acquire the data change type corresponding to the data to be synchronized;

[0034] The second acquisition submodule is used to acquire a preset transmission strategy in response to the data change type conforming to a preset synchronization type, wherein the preset synchronization type includes at least one of a new addition type and a modification type.

[0035] In one embodiment, the second acquisition submodule is further used for:

[0036] In response to the data change type being a modification type, obtaining data description fields before and after the change corresponding to the data to be synchronized;

[0037] In response to the data description fields before and after the change being different, a preset transmission strategy is obtained.

[0038] In one embodiment, the determining module is further configured to:

[0039] Get the preset transmission strategy;

[0040] Determine the target transmission field according to the preset transmission strategy;

[0041] The data corresponding to the target transmission field is determined from the data to be synchronized to obtain the target data.

[0042] In one embodiment, the synchronization module is further used for:

[0043] Determining a target transmission format according to the preset transmission strategy;

[0044] Converting the data format of the target data into the target transmission format to obtain converted target data;

[0045] The converted target data is synchronized to the target message queue.

[0046] In one embodiment, the device further includes a module for acquiring the preset transmission strategy, and the module for acquiring the preset transmission strategy includes:

[0047] A determination submodule, used to determine a target transmission interface;

[0048] The third acquisition submodule is used to periodically acquire a preset transmission strategy through the target transmission interface.

[0049] In a third aspect, an embodiment of the present disclosure further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0050] In a fourth aspect, the embodiments of the present disclosure further provide a computer-readable storage medium, wherein a computer program is stored thereon, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0051] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in the embodiments of the present disclosure are implemented.

[0052] The above-mentioned data synchronization method, device, computer equipment, computer-readable storage medium and computer program product, when performing data synchronization, obtain the to-be-synchronized message in the synchronization message queue, parse the to-be-synchronized message, obtain the to-be-synchronized data, and determine the target data and the target message queue corresponding to the target data from the to-be-synchronized data according to the preset transmission strategy, and synchronize the target data to the target message queue, thereby realizing data synchronization. The synchronization message queue is used to receive the change message generated by the data change in the preset database. Therefore, when performing data synchronization, the to-be-synchronized message is obtained from the synchronization message queue, and there is no need to obtain all the data in the preset database for change query, which effectively reduces the data processing volume and improves the data processing efficiency; and a preset transmission strategy is set, and the mapping relationship between the data and the message queue is set, so that the to-be-synchronized data can be further screened, and the message synchronization is performed according to the actual application requirements, which further reduces the data processing volume. Through the setting of the preset transmission strategy, the data to be synchronized can be flexibly configured, and the downstream data change non-perception setting is realized, which improves the flexibility of data synchronization, performs data synchronization according to the actual application scenario, improves resource utilization, and is suitable for more application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 A schematic diagram of a flow chart of a data synchronization method in an embodiment;

[0055] Figure 2 A flowchart of a data synchronization method in another embodiment;

[0056] Figure 3 A flowchart of a data synchronization method in another embodiment;

[0057] Figure 4 A schematic diagram of a flow chart of a method for obtaining a preset transmission strategy in an embodiment;

[0058] Figure 5 A flowchart of a data synchronization method in another embodiment;

[0059] Figure 6 is a structural block diagram of a data synchronization device in an embodiment;

[0060] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] In one embodiment, Figure 1 As shown, a data synchronization method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0063] Step S110, obtaining a message to be synchronized in a synchronization message queue, wherein the synchronization message queue is used to receive a change message generated by a data change in a preset database;

[0064] Exemplarily, the synchronization message queue is used to receive change messages generated by data changes in a preset database. The preset database may be a database corresponding to the data that needs to be synchronized according to the actual application scenario. For example, when the data of platform A needs to be synchronized, the corresponding preset database may be a database that stores data related to platform A.

[0065] In one example, in response to a user's operation on a preset database, the data in the preset database changes, a change message is generated, and the change message is stored in a synchronization message queue. Optionally, the change message can be captured by an incremental data synchronization tool. The user's operation on the preset database may include, but is not limited to, insert operations, update operations, delete operations, etc., which can be determined based on the actual application scenario. In one possible implementation, the preset database may be a TiDB database (TiDB Distributed Database, a distributed relational database), and the incremental data synchronization tool may be TiCDC (TiDB Change Data Capture, an incremental data synchronization tool for the TiDB ecosystem).

[0066] In one example, messages to be synchronized may be periodically acquired, and may include newly added messages in a synchronization message queue; in another example, in response to a message acquisition instruction, messages to be synchronized may be acquired from a synchronization message queue, and may include messages matching the acquisition instruction, which may be determined based on an actual application scenario.

[0067] Step S120, parsing the message to be synchronized to obtain the data to be synchronized;

[0068] Exemplarily, after obtaining the message to be synchronized, the message can be parsed to obtain the data to be synchronized. In one example, the message body of the message to be synchronized is parsed to obtain the data to be synchronized. The data to be synchronized may include but is not limited to database name, table name, etc., which can be determined based on the actual application scenario.

[0069] Step S130, determining target data from the data to be synchronized and determining a target message queue corresponding to the target data according to a preset transmission strategy, wherein the preset transmission strategy includes a mapping relationship between data and message queues;

[0070] Exemplarily, the preset transmission strategy may be a data synchronization strategy determined according to an actual application scenario. The preset transmission strategy may include a mapping relationship between data and message queues. Different data may correspond to the same or different message queues. In one example, data may be stored in tables, and the preset transmission strategy may include a mapping relationship between the table to which the data belongs and the message queue. Optionally, in the mapping relationship between data and message queues, the message queue is a downstream message queue that needs to be synchronized.

[0071] In one example, the preset transmission strategy also configures information such as data fields that need to be synchronized. According to the preset transmission strategy, the data to be synchronized can be screened to obtain the target data corresponding to the preset transmission strategy. Based on the mapping relationship between data and message queues, the target message queue that matches the target data can be determined.

[0072] Exemplarily, different target data may correspond to different target message queues; optionally, the preset transmission strategy may be obtained through platform configuration. In one example, when data is changed, a data change log (such as a changelog) is generated, and the preset transmission strategy may also include a mapping relationship between messages and change log queues.

[0073] Step S140: synchronizing the target data to the target message queue.

[0074] Exemplarily, after obtaining the target data, the target data is synchronized to the target message queue. In one example, the synchronization message queue and the target message queue can be a preset kafka message queue.

[0075] In an example, different target data corresponds to different target message queues. When synchronizing the target data to the target message queue, the different target data can be synchronized to the corresponding target message queues respectively.

[0076] In the disclosed embodiment, when data synchronization is performed, the to-be-synchronized message in the synchronization message queue is obtained, the to-be-synchronized message is parsed, the to-be-synchronized data is obtained, and according to the preset transmission strategy, the target data and the target message queue corresponding to the target data are determined from the to-be-synchronized data, and the target data is synchronized to the target message queue, thereby realizing data synchronization. The synchronization message queue is used to receive change messages generated by data changes in a preset database. Therefore, when data synchronization is performed, the to-be-synchronized message is obtained from the synchronization message queue, and there is no need to obtain all the data in the preset database for change query, which effectively reduces the amount of data processing and improves the data processing efficiency; and a preset transmission strategy is set, and the mapping relationship between data and message queue is set, so that the to-be-synchronized data can be further screened, and message synchronization is performed according to actual application requirements, further reducing the amount of data processing. Through the setting of the preset transmission strategy, the data to be synchronized can be flexibly configured, and the non-perception setting of downstream data changes is realized, which improves the flexibility of data synchronization, performs data synchronization according to actual application scenarios, improves resource utilization, and is applicable to more application scenarios.

[0077] In one embodiment, before determining the target data from the data to be synchronized according to the preset transmission strategy, the method further includes:

[0078] Obtaining the data change type corresponding to the data to be synchronized;

[0079] In response to the data change type being consistent with a preset synchronization type, a preset transmission strategy is obtained, wherein the preset synchronization type includes at least one of a new addition type and a modification type.

[0080] Exemplarily, the data change type corresponding to the data to be synchronized is obtained, wherein the data change type may include but is not limited to a new addition type, a modification type, a deletion type, etc., which may be specifically determined according to an actual application scenario.

[0081] The preset synchronization type may be a data change type that needs to be synchronized and is determined based on an actual application scenario. In one example, the preset synchronization type may include at least one of a new type (such as a create message) and a modification type (such as an update message). When the data change type corresponding to the data to be synchronized meets the preset synchronization type, the data to be synchronized may be considered to be data corresponding to the data change type that needs to be synchronized, and the preset transmission strategy is obtained.

[0082] In one example, when the data change type corresponding to the data to be synchronized does not conform to the preset synchronization type, it can be considered that the data to be synchronized does not need to be synchronized and can be skipped. For example, when the data change type corresponding to the data to be synchronized is a deletion type (such as a delete message), the data to be synchronized is skipped.

[0083] In the disclosed embodiment, before determining the target data from the data to be synchronized, the data change type of the data to be synchronized is first determined. When the data change type corresponding to the data to be synchronized meets the preset synchronization type, the preset transmission strategy is obtained, so that the data to be synchronized of the preset synchronization type can be synchronized, thereby further improving the flexibility of data synchronization, reducing the data processing workload, and improving data synchronization efficiency.

[0084] In one embodiment, in response to the data change type being consistent with the preset synchronization type, obtaining the preset transmission strategy includes:

[0085] In response to the data change type being a modification type, obtaining data description fields before and after the change corresponding to the data to be synchronized;

[0086] In response to the data description fields before and after the change being different, a preset transmission strategy is obtained.

[0087] Optionally, when the data change type is a modification type, further judgment is required. When the data change type of the data to be synchronized is a modification type, obtain the data description fields before and after the change corresponding to the data to be synchronized,

[0088] In one example, the data description field may include the attribute field and value field of the data to be synchronized. If the data description fields before and after the change are the same, it can be considered that the data before and after the change have not changed; if the data description fields before and after the change are different, it can be considered that the data before and after the change have changed.

[0089] In one example, when the data before and after the change changes, data synchronization is required, and therefore, in response to the difference in the data description fields before and after the change, a preset transmission strategy is obtained.

[0090] In the embodiment of the present disclosure, when the data change type is a modification type, the data description fields before and after the data to be synchronized are judged. If the data description fields are different, a preset transmission strategy is obtained, so that the data of the modification type and the changed data can be synchronized, thereby further improving the accuracy of data synchronization, filtering out the modification type and the unchanged data, reducing the data processing workload, improving the data synchronization efficiency, and being suitable for more application scenarios.

[0091] In one embodiment, Figure 2 As shown, determining the target data from the data to be synchronized according to the preset transmission strategy includes:

[0092] Step S131, obtaining a preset transmission strategy;

[0093] Step S132, determining a target transmission field according to the preset transmission strategy;

[0094] Step S133, determining data corresponding to the target transmission field from the data to be synchronized, and obtaining target data.

[0095] Exemplarily, when filtering data according to a preset transmission policy, the data can be filtered based on the target transmission field. Obtaining the preset transmission policy, in one example, the preset transmission policy can be obtained from the platform configuration. The target transmission field can be determined according to the preset transmission policy, and the target transmission field can include but is not limited to the data attribute field that needs to be synchronized.

[0096] According to the digital field corresponding to the data to be synchronized, the data corresponding to the target transmission field is determined from the data to be synchronized, which is the target data. In one example, the target transmission field may include a data attribute field that needs to be synchronized, and it is determined from the data to be synchronized whether there is data with the same data attribute field as the target transmission field. If so, the corresponding data is the data that needs to be synchronized, and is determined as the target data. In one example, if not, the data to be synchronized does not need to be synchronized and transmitted.

[0097] In the embodiments of the present disclosure, a target transmission field is determined according to a preset transmission strategy, and data corresponding to the target transmission field is determined from the data to be synchronized, so that the fields to be synchronized can be set according to the preset transmission strategy, and the data corresponding to the target transmission field can be transmitted, thereby improving the flexibility of data transmission. By modifying the configuration of the preset transmission strategy, data transmission with different requirements in different application scenarios can be achieved, thereby improving the accuracy and reliability of data synchronization.

[0098] In one embodiment, Figure 3 As shown, synchronizing the target data to the target message queue includes:

[0099] Step S141, determining a target transmission format according to the preset transmission strategy;

[0100] Step S142, converting the data format of the target data into the target transmission format to obtain the converted target data;

[0101] Step S143, synchronizing the converted target data to the target message queue.

[0102] Exemplarily, the target transmission format may be a format of data that needs to be synchronized to a downstream message queue, ie, a target message queue, determined according to an actual application scenario. In one example, different target data may correspond to different target transmission formats.

[0103] The data format of the target data is converted to obtain the converted target data. In one example, the data volume of the data before and after the conversion is different. Optionally, the data volume of the converted data is less than the data volume of the data before the conversion. The target data may include change intermediate data, etc. The converted target data may only include data corresponding to the change result, etc., thereby effectively reducing the data processing volume of subsequent data synchronous transmission. In one example, different target data corresponds to different target transmission formats, and different target data can be format converted respectively to obtain corresponding converted target data.

[0104] In the embodiments of the present disclosure, when synchronizing target data to a target message queue, format conversion is performed according to the target transmission format corresponding to a preset transmission strategy, the converted target data is obtained and data synchronization is performed. The transmission format is set by the preset transmission strategy, so that in different application scenarios, target data in the required format can be transmitted according to actual needs, thereby improving the flexibility of data synchronization, reducing resource waste, and reducing data processing workload.

[0105] In one embodiment, Figure 4 As shown, the method for obtaining the preset transmission strategy includes:

[0106] Step S410, determining the target transmission interface;

[0107] Step S420: periodically obtain a preset transmission strategy through the target transmission interface.

[0108] Exemplarily, the preset transmission strategy is obtained through a preset interface. The target transmission interface may be an interface corresponding to the platform configuration, which may be used to obtain the preset transmission strategy. In one example, the target transmission interface may be a preset REST API interface, through which data corresponding to the platform configuration may be obtained.

[0109] In one example, the preset transmission strategy may be periodic acquisition, and the acquisition period may be determined according to an actual application scenario. In one example, the preset transmission strategy may be updated and optimized according to an actual application scenario.

[0110] The disclosed embodiment periodically obtains the preset transmission strategy through the target transmission interface, and can quickly and accurately determine the corresponding preset transmission strategy and perform data synchronization during the data synchronization process, thereby improving the accuracy and real-time performance of data synchronization, ensuring the flexibility and efficiency of data transmission, and being suitable for more application scenarios.

[0111] Figure 5 A flowchart of a data synchronization method according to an exemplary embodiment is shown. Figure 5 As shown in the figure, the user operates the database, and the data in the database changes. The incremental data synchronization tool intercepts the data change and stores the corresponding change message in the synchronization message queue. The to-be-synchronized message in the to-be-synchronized message queue is obtained, and the target data is determined from the to-be-synchronized message according to the preset transmission strategy in the platform configuration, and the target data is synchronized to the downstream target message queue.

[0112] In one possible implementation, the Topic set (message queue) that receives the Changelog (data) corresponding to the database mapped by the data table is determined according to the preset transmission strategy in the internal platform configuration. The program consumes the data of these topics, and filters out the target data to be synchronized by parsing the data to be synchronized and obtaining the database and table name. According to all the field values ​​before and after the change corresponding to the target data, the field changes of the ignored fields are excluded and forwarded to the kafka (target message queue) configured in the platform.

[0113] Through this embodiment, the computing power and resource usage of the downstream program are reduced, and the stability of the program is improved; the configurable and automated transmission of data synchronization is achieved, the workload of upstream data workflow developers is reduced, and the efficiency and flexibility of data synchronization are improved.

[0114] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0115] Based on the same inventive concept, the embodiment of the present application also provides a data synchronization device for implementing the data synchronization method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data synchronization device embodiments provided below can refer to the limitations on the data synchronization method above, and will not be repeated here.

[0116] In an exemplary embodiment, Figure 6 As shown, a data synchronization device 600 is provided, comprising:

[0117] The acquisition module 610 is used to acquire the to-be-synchronized message in the synchronization message queue, wherein the synchronization message queue is used to receive the change message generated by the data change in the preset database;

[0118] The parsing module 620 is used to parse the message to be synchronized to obtain the data to be synchronized;

[0119] A determination module 630, configured to determine target data from the data to be synchronized and determine a target message queue corresponding to the target data according to a preset transmission strategy, wherein the preset transmission strategy includes a mapping relationship between data and message queues;

[0120] The synchronization module 640 is used to synchronize the target data to the target message queue.

[0121] In one embodiment, the device further comprises:

[0122] A first acquisition submodule is used to acquire the data change type corresponding to the data to be synchronized;

[0123] The second acquisition submodule is used to acquire a preset transmission strategy in response to the data change type conforming to a preset synchronization type, wherein the preset synchronization type includes at least one of a new addition type and a modification type.

[0124] In one embodiment, the second acquisition submodule is further used for:

[0125] In response to the data change type being a modification type, obtaining data description fields before and after the change corresponding to the data to be synchronized;

[0126] In response to the data description fields before and after the change being different, a preset transmission strategy is obtained.

[0127] In one embodiment, the determining module is further configured to:

[0128] Get the preset transmission strategy;

[0129] Determine the target transmission field according to the preset transmission strategy;

[0130] The data corresponding to the target transmission field is determined from the data to be synchronized to obtain the target data.

[0131] In one embodiment, the synchronization module is further used for:

[0132] Determining a target transmission format according to the preset transmission strategy;

[0133] Converting the data format of the target data into the target transmission format to obtain converted target data;

[0134] The converted target data is synchronized to the target message queue.

[0135] In one embodiment, the device further includes a module for acquiring the preset transmission strategy, and the module for acquiring the preset transmission strategy includes:

[0136] A determination submodule, used to determine a target transmission interface;

[0137] The third acquisition submodule is used to periodically acquire a preset transmission strategy through the target transmission interface.

[0138] Each module in the above data synchronization device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0139] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the method described in this embodiment, such as messages to be synchronized. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data synchronization method is implemented.

[0140] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0142] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0143] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0145] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0147] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: Obtaining a message to be synchronized in a synchronization message queue, wherein the synchronization message queue is used to receive a change message generated by a data change in a preset database; Parsing the message to be synchronized to obtain the data to be synchronized; According to a preset transmission strategy, determining target data from the data to be synchronized, and determining a target message queue corresponding to the target data, wherein the preset transmission strategy includes a mapping relationship between data and message queues; The target data is synchronized to the target message queue.

2. The method according to claim 1, characterized in that Before determining the target data from the data to be synchronized according to the preset transmission strategy, the method further includes: Obtaining the data change type corresponding to the data to be synchronized; In response to the data change type being consistent with a preset synchronization type, a preset transmission strategy is obtained, wherein the preset synchronization type includes at least one of a new addition type and a modification type.

3. The method according to claim 2, characterized in that In response to the data change type being consistent with the preset synchronization type, obtaining the preset transmission strategy includes: In response to the data change type being a modification type, obtaining data description fields before and after the change corresponding to the data to be synchronized; In response to the data description fields before and after the change being different, a preset transmission strategy is obtained.

4. The method according to claim 1, characterized in that: The step of determining target data from the data to be synchronized according to a preset transmission strategy includes: Get the preset transmission strategy; Determine the target transmission field according to the preset transmission strategy; The data corresponding to the target transmission field is determined from the data to be synchronized to obtain the target data.

5. The method according to claim 1, characterized in that The step of synchronizing the target data to the target message queue includes: Determining a target transmission format according to the preset transmission strategy; Converting the data format of the target data into the target transmission format to obtain converted target data; The converted target data is synchronized to the target message queue.

6. The method according to claim 1, characterized in that The method for obtaining the preset transmission strategy includes: Determine the target transmission interface; The preset transmission strategy is periodically obtained through the target transmission interface.

7. A data synchronization device, characterized in that: The device comprises: An acquisition module, used to acquire the to-be-synchronized message in the synchronization message queue, wherein the synchronization message queue is used to receive the change message generated by the data change in the preset database; A parsing module, used for parsing the message to be synchronized to obtain the data to be synchronized; A determination module, configured to determine target data from the data to be synchronized and determine a target message queue corresponding to the target data according to a preset transmission strategy, wherein the preset transmission strategy includes a mapping relationship between data and message queues; A synchronization module is used to synchronize the target data to the target message queue.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.