Business data synchronization method and device, non-volatile storage medium, and processor
By listening to business data updates in the source database and transferring them to the target message queue, performing data table type conversion processing and then pushing them to a third-party platform, the problem of the existing technology being unable to monitor and synchronously processed simultaneously, and improving data synchronization efficiency.
Patent Information
- Application Number
- CN202011282003.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-16
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-11-16
AI Technical Summary
The prior art cannot take into account both the monitoring of the source database and the synchronization processing of changing data, resulting in an increase in cost during data migration.
When listening to the update of business data in the source database, the data to be pushed is transferred to the target message queue, and the synchronous conversion service program is used to convert according to the data table type, and the converted data is pushed to the target third party through the push service program platform.
Real-time monitoring of the source database and synchronous processing of changing data is realized, the efficiency of synchronizing data from the source database to third-party platforms is improved, and synchronization and conversion problems caused by data structure differences are solved.
Smart Images

Figure CN112380229B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a business data synchronization method and device, a non-volatile storage medium, and a processor. Background Art
[0002] With the rapid development of third-party services, more and more situations require the introduction of services provided by third-party vendors or the provision of services for other vendors. Usually, the data structures of both parties may be quite different, which will cause a lot of manpower and material resources to be invested in data migration, increasing costs.
[0003] In a data conversion method and system in the prior art, a method of using a reading tool to read data from a source database is used to obtain the data to be converted. However, when the source database data is updated, it is impossible to simultaneously monitor the source database and synchronously process the conversion of the changed data.
[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0005] The embodiments of the present invention provide a business data synchronization method and device, a non-volatile storage medium, and a processor to at least solve the technical problem that the business data synchronization method in the prior art cannot simultaneously take into account the monitoring of the source database and the synchronization processing of the changed data.
[0006] According to one aspect of an embodiment of the present invention, a business data synchronization method is provided, including: when monitoring business data updates in a source database, transferring the acquired data to be pushed to a target message queue; when a synchronization conversion service program monitors the data to be pushed in the target message queue, converting the data to be pushed according to the data table type of the push data to obtain converted data to be pushed, wherein the data structure of the converted data to be pushed corresponds to the data structure of a target third-party platform; and using a push service program to push the converted data to the target third-party platform.
[0007] Optionally, a monitoring component is used to monitor the source data log file in the above-mentioned source database to determine whether the above-mentioned business data is updated, wherein the above-mentioned monitoring component includes: a database synchronization monitoring Canal component.
[0008] Optionally, the acquired data to be pushed is transferred to the target message queue, including: acquiring the updated business data in the above-mentioned source database, and the pre-update business data corresponding to the above-mentioned updated business data; converting the above-mentioned pre-update business data and the above-mentioned updated business data into the above-mentioned data to be pushed, wherein the data structure of the above-mentioned data to be pushed is the target message body structure.
[0009] Optionally, the data to be pushed is transformed according to the data table type of the push data, including: obtaining data identification information carried in the data to be pushed; determining the data table type based on the data identification information; judging whether the data to be pushed is suitable for local customized services according to the data table type, wherein the local customized service processes the data to be pushed based on a target event listening mechanism, and encapsulates the data to be pushed into a target event for broadcasting; if the judgment result is yes, determining the listening type corresponding to the data table type, wherein different listening types correspond to different event listeners, and the event listener is used to transform the data to be pushed after receiving the broadcast.
[0010] Optionally, if the judgment result is no, the above data to be pushed is discarded.
[0011] Optionally, a push service program is used to push the converted data to be pushed to the target third-party platform, including: using the push service program to convert the converted data to be pushed into a target character string; accessing a data receiving interface of the target third-party platform through a hypertext transfer protocol Http request, and using an interface request tool to push the target character string to the data receiving interface, wherein the target third-party platform is also used to store the target character string in a local database.
[0012] According to another aspect of an embodiment of the present invention, a business data synchronization device is also provided, including: a monitoring module, which is used to transfer the acquired data to be pushed to a target message queue when monitoring the business data update in the source database; a conversion processing module, which is used to convert the data to be pushed according to the data table type of the push data when the synchronization conversion service program monitors the data to be pushed in the target message queue, so as to obtain the converted data to be pushed, wherein the data structure of the converted data to be pushed corresponds to the data structure of the target third-party platform; a push module, which adopts a push service program to push the converted data to be pushed to the target third-party platform.
[0013] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, wherein the non-volatile storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing any one of the above-mentioned business data synchronization methods.
[0014] According to another aspect of an embodiment of the present invention, a processor is further provided, and the processor is used to run a program, wherein the program is configured to execute any one of the above-mentioned business data synchronization methods when running.
[0015] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any one of the above-mentioned business data synchronization methods.
[0016] In an embodiment of the present invention, when the business data in the source database is monitored for update, the acquired data to be pushed is transferred to the target message queue; when the synchronous conversion service program monitors the data to be pushed in the above-mentioned target message queue, the data to be pushed is converted according to the data table type of the above-mentioned push data to obtain the converted data to be pushed, wherein the data structure of the converted data to be pushed corresponds to the data structure of the target third-party platform; the push service program is used to push the converted data to the above-mentioned target third-party platform, thereby achieving the purpose of taking into account both the monitoring of the source database and the synchronization of the changed data, thereby realizing the technical effect of improving the efficiency of synchronizing data from the source database to the third-party platform, and further solving the technical problem that the business data synchronization method in the prior art cannot take into account both the monitoring of the source database and the synchronization of the changed data. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0018] Figure 1 is a flow chart of a business data synchronization method according to an embodiment of the present invention;
[0019] Figure 2 is a schematic diagram of an optional execution flow of synchronizing service data according to an embodiment of the present invention;
[0020] Figure 3 It is a structural diagram of a service data synchronization device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0023] Example 1
[0024] According to an embodiment of the present invention, an embodiment of a business data synchronization method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0025] Figure 1 is a flow chart of a method for synchronizing business data according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:
[0026] Step S102, when monitoring the business data update in the source database, transferring the acquired data to be pushed to the target message queue;
[0027] Step S104, when the synchronous conversion service program monitors the data to be pushed in the target message queue, the data to be pushed is converted according to the data table type of the push data to obtain converted data to be pushed, wherein the data structure of the converted data to be pushed corresponds to the data structure of the target third-party platform;
[0028] Step S106: using a push service program to push the converted data to be pushed to the target third-party platform.
[0029] In an embodiment of the present invention, when business data updates in the source database are monitored, the acquired data to be pushed are transferred to the target message queue; when the synchronous conversion service program monitors the data to be pushed in the above-mentioned target message queue, the data to be pushed is converted according to the data table type of the above-mentioned push data to obtain the converted data to be pushed, wherein the data structure of the converted data to be pushed corresponds to the data structure of the target third-party platform; the push service program is used to push the converted data to the above-mentioned target third-party platform, thereby achieving the purpose of taking into account both the monitoring of the source database and the conversion synchronization of the changed data, thereby realizing the technical effect of improving the efficiency of synchronizing data from the source database to the third-party platform, and further solving the technical problem that the business data synchronization method in the prior art cannot take into account both the monitoring of the source database and the synchronous processing of the changed data.
[0030] In an optional embodiment, a monitoring component is used to monitor the source data log file in the source database to determine whether the business data is updated, wherein the monitoring component includes: a database synchronization monitoring Canal component.
[0031] Optionally, the source database may be, but is not limited to, a MySQL business database; the target message queue may be a Kafka message queue or a RabbitMq message queue.
[0032] In the embodiment of the present application, it is also possible to determine whether corresponding processing is required in the target third-party platform according to the update type or change type (addition, modification, deletion) of the business data, such as Figure 2 As shown, if necessary, the data to be pushed is converted into the data structure required by the target third-party platform, and the converted data to be pushed is pushed to the target third-party platform in the target message body structure through the third-party platform push service program.
[0033] Through the embodiments of the present application, the data synchronization and conversion problems caused by the differences in the data structures of the two parties can be solved. The Canal component monitors the update of the source database data, and the updated data is converted through a customized data synchronization conversion service, and the changed business data is directly converted into the data structure required by the third-party platform and then pushed to the third-party platform.
[0034] In an optional embodiment, transferring the acquired data to be pushed to a target message queue includes:
[0035] Step S202, obtaining updated business data in the source database, and business data before updating corresponding to the updated business data;
[0036] Step S204: converting the pre-update service data and the post-update service data into the data to be pushed, wherein the data structure of the data to be pushed is a target message body structure.
[0037] In an embodiment of the present application, after the Canal component is used to monitor the changed business data in the MySQL business database, the business data before and after the update of the business data are obtained, and the business data before and after the update are converted into the data to be pushed of the target message body structure; then the data to be pushed is put into the Kafka message queue, and a synchronous conversion service program (i.e., a customized data synchronous conversion service program) is used to listen to the data to be pushed in the Kafka message queue. When the synchronous conversion service program listens to the data to be pushed in the Kafka message queue, the data to be pushed is converted according to the data table type of the push data to obtain the converted data to be pushed, wherein the data structure of the converted data to be pushed corresponds to the data structure of the target third-party platform.
[0038] In an optional embodiment, the data to be pushed is converted according to the data table type of the pushed data, including:
[0039] Step S302, obtaining data identification information carried in the data to be pushed;
[0040] Step S304, determining the data table type based on the data identification information;
[0041] Step S306, judging whether the data to be pushed is applicable to the local customized service according to the data table type, wherein the local customized service processes the data to be pushed based on the target event monitoring mechanism, and encapsulates the data to be pushed into a target event for broadcasting;
[0042] Step S308, if the judgment result is yes, determine the monitoring type corresponding to the above data table type, wherein different monitoring types correspond to different event listeners, and the above event listeners are used to convert the above data to be pushed after receiving the broadcast.
[0043] In the above optional embodiment, if the judgment result is no, the above data to be pushed is discarded.
[0044] Optionally, the above data identification information may be data table identification information and / or source database identification information.
[0045] Optionally, the above-mentioned target event can be a Spring event, and the above-mentioned target event monitoring mechanism can be a Spring event monitoring mechanism. The Spring event monitoring mechanism is implemented based on the observer pattern design pattern. It should be noted that, in addition to the above-mentioned Spring event monitoring mechanism, other event monitoring technical solutions based on the observer design pattern are also applicable to the embodiments of the present application.
[0046] As an optional embodiment, the workflow of the customized statistical service in the data synchronization method provided in the embodiment of the present application is described below from the perspective of real-time performance and customizability:
[0047] First, the source data log file is monitored through the Canal plug-in. When data is updated in the source database, the Canal plug-in converts an updated data into a Canal message body structure containing the data before and after the change. The Canal message body is pushed to the customized data synchronization conversion service through the Kafka message queue for synchronization conversion processing.
[0048] Then, when the Canal message body structure is transferred to the customized data synchronization conversion service, the customized data synchronization conversion service will determine whether the message is applicable to the local customized service based on the database name and indication in the message. If applicable, continue to execute; if not applicable, directly discard the Canal message body structure. The above-mentioned local customized service processes messages based on the Spring event monitoring mechanism, encapsulates the Canal message body structure into a Spring event and broadcasts it. For example, it can be divided into different monitoring types according to the data table identification information. Different monitoring types correspond to different listeners. After receiving the broadcast, the listener performs corresponding message conversion processing.
[0049] As an optional embodiment, a push service program is used to push the converted data to be pushed to the target third-party platform, including:
[0050] Step S402, using the push service program to convert the converted data to be pushed into a target character string;
[0051] Step S404, access the data receiving interface of the target third-party platform through a hypertext transfer protocol Http request, and use an interface request tool to push the target string to the data receiving interface, wherein the target third-party platform is also used to store the target string in a local database.
[0052] Optionally, the above-mentioned interface request tool can be a POST request. In an embodiment of the present application, after generating a data body in a third-party format, the data body is converted into a string in JSON format, and an HTTP request is used to access the data receiving interface of the third-party platform, wherein different data processing methods access the corresponding interface, and a POST request is used to push the JSON string to the target interface, which is processed by the third-party platform and saved in the third-party local database to achieve data synchronization between the two parties.
[0053] As an optional embodiment, assume that there are currently three event listeners A, B, and C. When the Canal message of a table is pushed to the customized service via the Kafka message queue, different data changes are processed accordingly. Combined with the event monitoring based on data table classification described in the previous part, the strategy mode is used to classify and process the monitored change data of the source database table. For example, if table A1 receives a Canal message, the Canal message is monitored by listener A; if table B1 receives a Canal message, the Canal message is monitored by listener B; if table C1 receives a Canal message, the Canal message is monitored by listener C.
[0054] It should be noted that if the data table corresponding to the predetermined event monitor does not receive the Canal message, that is, the Canal message is not applicable to the local customized service, the Canal message will be discarded.
[0055] Assume that a new data is added to the existing source database A1 table. The event listener A corresponding to the A1 table listens to and receives the newly added data, imports the changed data into the policy factory, obtains the change type (Insert) of the changed data, traverses the policy implementation class object array in the policy factory, imports the change type parameter into the change type verification method of the object, and when traversing to a policy implementation class object element that meets the type, executes the data conversion processing method implemented by the object as shown below: import the Canal message body structure into the policy factory to obtain the change type of the changed data, i=0, traverse the policy implementation class object array in the policy factory, obtain the i-th element in the array, verify whether the element meets the conditions, and process it if the verification result is yes. If the verification result is no, i=i+1, until the verification result is yes and ends after processing.
[0056] Each message processing strategy has a corresponding data conversion processing method. According to the type of different data tables, the corresponding data model is used to convert the incoming data into the data structure specified by the third-party platform. For example, when an A1 table data is updated in the source database, the data is received by the A1 table listener through the process, and the data processing strategy of the corresponding change type is selected and the data model of the A1 table of the third-party platform is used. The attributes in the source data body are converted into corresponding new attributes according to the data model, such as the data body format correspondence table shown in Table 1. The source data body contains attributes such as id and address, which correspond to dataId and address in the data model. The necessary attribute values in the source data body are written into the attributes of the corresponding new data body, and the redundant attributes in the source data body are discarded to obtain the new data body.
[0057] Table 1
[0058] Source data attributes Property Value New data volume attributes Property Value Id 001 dataId 001 address Beijing address Beijing
[0059] The embodiment of the present application is applicable to the access of most third-party platforms and has good universality. In addition, the event monitoring part and the data processing part in the embodiment of the present application have good scalability, and can add monitoring of different tables and new data processing methods according to the docking of different third-party platforms.
[0060] In addition, the embodiments of the present application can simultaneously realize real-time monitoring of source data and synchronous conversion of change data; the change data monitored by Canal uses the Kafka message queue to enable the processing program to obtain the change data. Compared with the read and write functions of the Redis cache, the use of the Kafka message queue avoids the read and write pressure on the cache database when the data volume is large.
[0061] Example 2
[0062] According to an embodiment of the present invention, there is also provided an apparatus embodiment for implementing the above-mentioned service data synchronization method. Figure 3 is a schematic diagram of the structure of a service data synchronization device according to an embodiment of the present invention. Figure 3 As shown, the above-mentioned service data synchronization device includes: a monitoring module 500, a conversion processing module 502 and a push module 504, wherein:
[0063] The monitoring module 500 is used to transfer the acquired data to be pushed to the target message queue when the business data update in the source database is monitored; the conversion processing module 502 is used to convert the above-mentioned data to be pushed according to the data table type of the above-mentioned push data when the synchronous conversion service program monitors the above-mentioned data to be pushed in the above-mentioned target message queue, so as to obtain the converted data to be pushed, wherein the data structure of the above-mentioned converted data to be pushed corresponds to the data structure of the target third-party platform; the push module 504 is used to use the push service program to push the above-mentioned converted data to be pushed to the above-mentioned target third-party platform.
[0064] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.
[0065] It should be noted that the above-mentioned monitoring module 500, conversion processing module 502 and push module 504 correspond to steps S102 to S106 in Example 1, and the examples and application scenarios implemented by the above-mentioned modules and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules as part of the device can be run in a computer terminal.
[0066] It should be noted that the optional or preferred implementation of this embodiment can refer to the relevant description in Example 1, which will not be repeated here.
[0067] The above-mentioned business data synchronization device may also include a processor and a memory. The above-mentioned monitoring module 500, conversion processing module 502 and push module 504 are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.
[0068] The processor includes a kernel, which retrieves the corresponding program unit from the memory. The kernel may be one or more. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory includes at least one memory chip.
[0069] According to an embodiment of the present application, an embodiment of a non-volatile storage medium is also provided. Optionally, in this embodiment, the non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute any of the above-mentioned service data synchronization methods.
[0070] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group, and the non-volatile storage medium includes a stored program.
[0071] Optionally, when the program is running, the device where the non-volatile storage medium is located is controlled to perform the following functions: when the business data update in the source database is monitored, the acquired data to be pushed is transferred to the target message queue; when the synchronous conversion service program monitors the above-mentioned data to be pushed in the above-mentioned target message queue, the above-mentioned data to be pushed is converted according to the data table type of the above-mentioned push data to obtain the converted data to be pushed, wherein the data structure of the above-mentioned converted data to be pushed corresponds to the data structure of the target third-party platform; the push service program is used to push the above-mentioned converted data to be pushed to the above-mentioned target third-party platform.
[0072] Optionally, when the program is running, the device where the non-volatile storage medium is located is controlled to perform the following functions: a monitoring component is used to monitor the source data log file in the above-mentioned source database to determine whether the above-mentioned business data is updated, wherein the above-mentioned monitoring component includes: a database synchronization monitoring Canal component.
[0073] Optionally, when the program is running, the device where the non-volatile storage medium is located is controlled to perform the following functions: obtain the updated business data in the above-mentioned source database, and the pre-update business data corresponding to the above-mentioned updated business data; convert the above-mentioned pre-update business data and the above-mentioned updated business data into the above-mentioned data to be pushed, wherein the data structure of the above-mentioned data to be pushed is the target message body structure.
[0074] Optionally, when the program is running, the device where the non-volatile storage medium is located is controlled to perform the following functions: obtain data identification information carried in the above-mentioned data to be pushed; determine the above-mentioned data table type based on the above-mentioned data identification information; judge whether the above-mentioned data to be pushed is suitable for local customized services based on the above-mentioned data table type, wherein the above-mentioned local customized service processes the data to be pushed based on the target event monitoring mechanism, and encapsulates the data to be pushed into a target event for broadcasting; if the judgment result is yes, determine the monitoring type corresponding to the above-mentioned data table type, wherein different above-mentioned monitoring types correspond to different event listeners, and the above-mentioned event listeners are used to convert and process the above-mentioned data to be pushed after receiving the broadcast.
[0075] Optionally, when the program is running, the device where the non-volatile storage medium is located is controlled to perform the following function: if the judgment result is no, the above-mentioned data to be pushed is discarded.
[0076] Optionally, when the program is running, the device where the non-volatile storage medium is located is controlled to perform the following functions: using the above-mentioned push service program to convert the above-mentioned converted data to be pushed into a target character string; accessing the data receiving interface of the above-mentioned target third-party platform through the hypertext transfer protocol Http request, and using the interface request tool to push the above-mentioned target character string to the above-mentioned data receiving interface, wherein the above-mentioned target third-party platform is also used to store the above-mentioned target character string in the local database.
[0077] According to an embodiment of the present application, an embodiment of a processor is also provided. Optionally, in this embodiment, the processor is used to run a program, wherein the program executes any one of the above-mentioned business data synchronization methods when it is run.
[0078] According to an embodiment of the present application, an embodiment of an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any one of the business data synchronization methods described above.
[0079] According to an embodiment of the present application, an embodiment of a computer program product is also provided, which, when executed on a data processing device, is suitable for executing a program that initializes any one of the steps of the above-mentioned business data synchronization method.
[0080] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0081] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0082] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above-mentioned units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0083] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0084] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0085] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a non-volatile storage medium, including a number of instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present invention. The aforementioned non-volatile storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.
[0086] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for synchronizing business data, characterized in that: include: When monitoring the business data update in the source database, the acquired data to be pushed is transferred to the target message queue; When the synchronous conversion service program monitors the data to be pushed in the target message queue, the data to be pushed is converted according to the data table type of the pushed data to obtain the converted data to be pushed, wherein the data structure of the converted data to be pushed corresponds to the data structure of the target third-party platform; Using a push service program to push the converted data to be pushed to the target third-party platform; The data to be pushed is converted according to the data table type of the pushed data, including: Determining whether the data to be pushed is applicable to the local customized service according to the data table type, wherein the local customized service processes the data to be pushed based on the target event monitoring mechanism, and encapsulates the data to be pushed into a target event for broadcasting; If the judgment result is yes, then determine the listening type corresponding to the data table type, wherein different listening types correspond to different event listeners, and the event listeners are used to convert the data to be pushed after receiving the broadcast.
2. The method according to claim 1, characterized in that A monitoring component is used to monitor the source data log file in the source database to determine whether the business data is updated, wherein the monitoring component includes: a database synchronization monitoring Canal component.
3. The method according to claim 1, characterized in that Transfer the acquired data to be pushed to the target message queue, including: Acquire updated business data in the source database, and pre-update business data corresponding to the updated business data; The pre-update service data and the post-update service data are converted into the data to be pushed, wherein the data structure of the data to be pushed is a target message body structure.
4. The method according to claim 1, characterized in that The data to be pushed is converted according to the data table type of the pushed data, including: Obtaining data identification information carried in the data to be pushed; The data table type is determined based on the data identification information.
5. The method according to claim 1, characterized in that If the judgment result is no, the data to be pushed is discarded.
6. The method according to claim 1, characterized in that The push service program is used to push the converted data to be pushed to the target third-party platform, including: The push service program is used to convert the converted data to be pushed into a target character string; The data receiving interface of the target third-party platform is accessed through a hypertext transfer protocol Http request, and the target character string is pushed to the data receiving interface using an interface request tool, wherein the target third-party platform is also used to store the target character string in a local database.
7. A service data synchronization device, characterized in that: include: The monitoring module is used to transfer the acquired data to be pushed to the target message queue when the business data in the source database is updated; a conversion processing module, for converting the data to be pushed according to the data table type of the pushed data when the synchronous conversion service program monitors the data to be pushed in the target message queue, to obtain the converted data to be pushed, wherein the data structure of the converted data to be pushed corresponds to the data structure of the target third-party platform; A push module, used to push the converted data to be pushed to the target third-party platform using a push service program; The conversion processing module is further used to determine whether the data to be pushed is applicable to the local customized service according to the data table type, wherein the local customized service processes the data to be pushed based on the target event monitoring mechanism and encapsulates the data to be pushed into a target event for broadcasting; If the judgment result is yes, then determine the listening type corresponding to the data table type, wherein different listening types correspond to different event listeners, and the event listeners are used to convert the data to be pushed after receiving the broadcast.
8. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the business data synchronization method described in any one of claims 1 to 6.
9. A processor, characterized in that: The processor is used to run a program, wherein the program is configured to execute the business data synchronization method according to any one of claims 1 to 6 when running.
10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the business data synchronization method according to any one of claims 1 to 6.
