Heterogeneous data real-time synchronization method and device and computer equipment

Abstract

The application belongs to the technical field of big data, and particularly relates to a heterogeneous data real-time synchronization method, device and computer equipment. The method comprises the following steps: obtaining change information of a data operation log file in a relational database of a source end business system, wherein the change information comprises at least one piece of change data; pre-processing the change information to generate a binary file; transmitting the binary file to a target file storage system; restoring the binary file to obtain the change data in the target file storage system; writing the change data into an incremental data table of the target file storage system; and merging the incremental data table with a stock data table of the target file storage system to complete heterogeneous data synchronization. The method, device and computer equipment can shorten the time of heterogeneous data synchronization, and better meet the data application scene with high timeliness requirement.

Description

Technical Field

[0001] This application relates to the field of big data technology, and more specifically, to a method, apparatus, and computer device for real-time synchronization of heterogeneous data. Background Technology

[0002] In recent years, the informatization of various industries has developed rapidly, and various types of information have gradually formed massive amounts of data. The aggregation, storage, and updating of massive amounts of data have become important technological breakthroughs for the digital transformation of various industries. In existing technologies, incremental data from relational databases from different sources is generally synchronized in real time to a relational data warehouse storing massive amounts of data through CDC (Change Data Capture) tools. During fixed periods when business operations are relatively idle, the data is then updated to a big data platform (such as Hadoop) for use in various business scenarios through offline full data synchronization. The traditional offline full data update method requires a lot of synchronization time, resulting in delayed data analysis results, which cannot meet the needs of data application scenarios with high timeliness and makes it difficult to reflect the value of the data. Summary of the Invention

[0003] The main purpose of this application is to provide a method, apparatus and computer device for real-time synchronization of heterogeneous data, which aims to solve the technical problems of long synchronization time and low timeliness of heterogeneous data in the prior art.

[0004] To achieve the above-mentioned objectives, this application provides a method for real-time synchronization of heterogeneous data, comprising:

[0005] Obtain change information from the data operation log file in the relational database of the source business system, wherein the change information includes at least one changed data record;

[0006] The changed information is preprocessed to generate a binary file;

[0007] The binary file is transferred to the target file storage system;

[0008] The binary file is restored, and the changed data is obtained from the target file storage system;

[0009] Write the changed data into the incremental data table of the target file storage system;

[0010] The incremental data table is merged with the existing data table of the target file storage system to complete the heterogeneous data synchronization.

[0011] In one embodiment, the step of transferring the binary file to the target file storage system includes:

[0012] All the aforementioned changed data are marked with primary keys, changed types are added, and changed time fields are added to generate preprocessed changed data;

[0013] All the preprocessed change data are packaged to generate a preprocessed change data file;

[0014] Convert the preprocessed change data file into a binary file.

[0015] In one embodiment, the step of writing the changed data into the incremental data table of the target file storage system includes:

[0016] All the changed data obtained from the target file storage system will be sent to the message queue component in a preset format;

[0017] The changed data received by the message queue component is marked with an offset, wherein the offset is used to query the changed data;

[0018] The changed data, which has been marked with offsets, is partitioned and stored in the message queue according to the primary key;

[0019] Consume data from the message queue component at preset time intervals;

[0020] The changed data is obtained by parsing the data in the message queue component according to preset rules;

[0021] The changed data is written into the incremental data table of the target file storage system.

[0022] In one embodiment, before the step of obtaining change information from the data operation log file in the relational database of the source business system, the method further includes:

[0023] Real-time monitoring of data operation log files in the relational database of the source business system;

[0024] Determine whether there is any change information in the data operation log file.

[0025] In one embodiment, the step of sending all the changed data obtained from the target file storage system to the message queue component in a preset format includes:

[0026] Based on the data transmission rules of the message queue component and each of the changed data, first data in a preset format is generated;

[0027] Send the first data to the message queue component;

[0028] Determine whether the message queue component returns an acknowledgment signal within a preset return time;

[0029] If the message queue component returns an acknowledgment signal within a preset return time and the first data has not been completely sent, then the next piece of first data will continue to be sent.

[0030] In one embodiment, the step of sending all the changed data obtained from the target file storage system to the message queue component in a preset format further includes:

[0031] If the information queue component does not return an acknowledgment signal within the preset return time, data transmission will be stopped and an error will be reported.

[0032] In one embodiment, prior to the step of real-time monitoring of the data operation log files in the relational database of the source business system, the method further includes:

[0033] A relational database that connects to the source business system.

[0034] This application also provides a heterogeneous data real-time synchronization device, including:

[0035] The change information acquisition module is used to acquire change information of data operation log files in the relational database of the source business system, wherein the change information includes at least one piece of changed data;

[0036] A binary file generation module is used to preprocess the change information to generate a binary file;

[0037] The transmission module is used to transmit the binary file to the target file storage system;

[0038] The restore module is used to restore the binary file and obtain the changed data from the target file storage system;

[0039] The data writing module is used to write the changed data into the incremental data table of the target file storage system;

[0040] The merging module is used to merge the incremental data table with the existing data table of the target file storage system to complete heterogeneous data synchronization.

[0041] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the heterogeneous data real-time synchronization method provided in any of the above embodiments.

[0042] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the heterogeneous data real-time synchronization method provided in any of the above embodiments.

[0043] This application provides a method, apparatus, and computer device for real-time heterogeneous data synchronization. The method involves acquiring change information from a data operation log file in a relational database of a source business system, wherein the change information includes at least one piece of changed data. The change information is preprocessed to generate a binary file. The binary file is then transmitted to a target file storage system. The binary file is restored, and the changed data is retrieved from the target file storage system. The changed data is written into an incremental data table in the target file storage system. The incremental data table is then merged with the existing data table in the target file storage system to complete the heterogeneous data synchronization. This heterogeneous data synchronization technology eliminates the need for traditional offline full-data synchronization, achieving real-time synchronization of heterogeneous data, thereby shortening the synchronization time and better meeting the needs of data application scenarios with high timeliness requirements. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the architecture of a big data platform according to an embodiment of this application;

[0045] Figure 2 This is a flowchart illustrating a method for real-time synchronization of heterogeneous data according to an embodiment of this application.

[0046] Figure 3 This is a flowchart illustrating step S20 of a heterogeneous data real-time synchronization method according to an embodiment of this application.

[0047] Figure 4 This is a flowchart illustrating a real-time heterogeneous data synchronization method according to another embodiment of this application.

[0048] Figure 5 This is a schematic diagram of the point-to-point mode operation of an information queue component according to an embodiment of this application;

[0049] Figure 6 This is a flowchart illustrating step S50 in a heterogeneous data real-time synchronization method according to another embodiment of this application.

[0050] Figure 7 This is a flowchart illustrating step S501 in a heterogeneous data real-time synchronization method according to an embodiment of this application.

[0051] Figure 8 This is a schematic diagram of the structure of a heterogeneous data real-time synchronization device according to an embodiment of this application;

[0052] Figure 9 This is a schematic diagram of the structure of a computer device according to an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] Please refer to Figure 1 , Figure 1 This paper illustrates a typical big data platform architecture (Hadoop big data platform). In existing technologies, incremental data from relational databases originating from different data sources (data from various business systems) is typically synchronized in real-time to a relational data warehouse storing massive amounts of data using CDC (Change Data Capture) tools. During relatively quiet periods, the data is then updated offline to the big data platform (such as Hadoop) for use in various scenarios. However, this offline full-data update method is time-consuming, leading to delayed data analysis results and failing to meet the time-sensitive needs of data usage scenarios, thus hindering the realization of data value. Therefore, a new heterogeneous data synchronization scheme is needed to shorten the synchronization time of heterogeneous data to meet the needs of various data application scenarios with high timeliness requirements.

[0055] Please refer to Figure 2 This application provides a method for real-time synchronization of heterogeneous data, including steps S10-S60. The detailed description of each step of the method is as follows.

[0056] In one embodiment, the heterogeneous data real-time synchronization method includes:

[0057] S10. Obtain change information from the data operation log file in the relational database of the source business system, wherein the change information includes at least one piece of changed data;

[0058] S20. Preprocess the change information to generate a binary file;

[0059] S30. Transfer the binary file to the target file storage system;

[0060] S40. Restore the binary file and obtain the changed data from the target file storage system;

[0061] S50. Write the changed data into the incremental data table of the target file storage system;

[0062] S60. Merge the incremental data table with the existing data table of the target file storage system to complete heterogeneous data synchronization.

[0063] As described in steps S10-S60 above, in this embodiment, the corresponding changed data is obtained by acquiring the change information of the data operation log file in the relational database of the source business system. The change information of the data operation log file includes at least one piece of changed data. In some embodiments, the changed data is newly added data.

[0064] To alleviate the pressure of data transmission, in this embodiment, the changed information is preprocessed to generate a binary file, which is then transmitted to the target file storage system (i.e., Hadoop's HDFS). This reduces the total amount of data transmission without compromising data integrity. The obtained binary file is then restored in the target file storage system to obtain the changed data. This changed data is then written to the incremental data table in the target file storage system, and the incremental data table is merged with the existing data table in real time, thus completing the real-time synchronization of heterogeneous data.

[0065] This application provides a method for real-time heterogeneous data synchronization. The method involves obtaining change information from a data operation log file in a relational database of a source business system, wherein the change information includes at least one changed data entry; preprocessing the change information to generate a binary file; transmitting the binary file to a target file storage system; restoring the binary file to retrieve the changed data in the target file storage system; writing the changed data into an incremental data table in the target file storage system; and merging the incremental data table with the existing data table in the target file storage system to complete the heterogeneous data synchronization. This heterogeneous data synchronization technology eliminates the need for traditional offline full-data synchronization, achieving real-time synchronization of heterogeneous data, thereby shortening the synchronization time and better meeting the needs of data application scenarios with high timeliness requirements.

[0066] In some embodiments, please refer to Figure 3 Step S20 above, namely the step of preprocessing the changed information to generate a binary file, includes:

[0067] S201. Generate preprocessed change data by marking all the changed data with primary keys, adding change types, and adding change time fields;

[0068] S202. Encapsulate all the preprocessed change data to generate a preprocessed change data file;

[0069] S203. Convert the preprocessed change data file into a binary file.

[0070] As described in steps S201-S203 above, after obtaining the changed data from the data operation log file, all changed data is marked with a primary key, changed type is added, and a changed time field is added to generate preprocessed changed data. The primary key (or primary key) is a unique key and part of the table definition. It is one or more fields in a database table whose value is used to uniquely identify a record in the table. In a relationship between two tables, the primary key is used to reference a specific record from another table. A table's primary key can consist of multiple keys, and the primary key column cannot contain null values. Changed types include data addition, deletion, and data type conversion, while the changed time field records the time of data change.

[0071] All the change data that has been marked with primary keys and whose change type has been added (i.e., the change time field has been added) (i.e., preprocessed change data) are encapsulated to generate a preprocessed change data file. In order to reduce the pressure of data transmission, the above-mentioned preprocessed change data file is converted into a binary format file for transmission.

[0072] In some embodiments, please refer to Figure 4 Step S50 above, which is the step of writing the changed data into the incremental data table of the target file storage system, includes:

[0073] S501. Send all the changed data obtained from the target file storage system to the message queue component in a preset format;

[0074] S502. Mark the changed data received by the message queue component with an offset, wherein the offset is used to query the changed data;

[0075] S503. The changed data that has been marked with offset is partitioned and stored in the message queue according to the primary key;

[0076] S504. Consume the data in the message queue component at preset time intervals;

[0077] S505. Obtain the changed data by parsing the data in the message queue component according to preset rules;

[0078] S506. Write the changed data into the incremental data table of the target file storage system.

[0079] As described in steps S501-S502 above, the message queue component is an important component of large-scale data processing systems and has gradually become a core means of internal communication within these systems. It possesses a series of functions such as loose coupling, asynchronous messaging, traffic shaping, reliable delivery, broadcasting, flow control, and eventual consistency. Common application scenarios for message queues in practice include application decoupling, asynchronous processing, traffic shaping and flow control, and log processing. The message queue component includes publish / subscribe and point-to-point modes. The publish / subscribe model is a push-based message delivery model. This model can have multiple different subscribers; temporary subscribers receive messages only when actively listening to the topic, while persistent subscribers listen to all messages on the topic, even if the current subscriber is offline. The point-to-point model is typically a pull or polling-based message delivery model. This model requests information from the queue rather than pushing messages to the client. A key characteristic of this model is that messages sent to the queue are received and processed by only one receiver. In this embodiment, the message queue component operates in point-to-point mode. Please refer to... Figure 5 , Figure 5 A schematic diagram of the peer-to-peer mode operation of the message queue component is shown.

[0080] In this embodiment, all changed data obtained from the target file storage system is sent to the message queue component in a preset format. This preset format is the data transmission format followed by the message queue component; different message queue components can have different data transmission formats. To more quickly retrieve the relevant changed data, the changed data received in the message queue component is marked with an offset, meaning the offset of each changed data item is set to a different value. After offset marking, the offset-marked changed data is partitioned and stored according to the primary key information marked in step S201. Data in the queue component is consumed at preset time intervals (e.g., 1ms). The system receives changed data transmitted from the target file storage system in a preset format; it then parses each piece of data in the message queue according to preset rules to obtain the original changed data. These preset rules correspond to the preset format, which is the relationship between data packet synthesis and parsing. Finally, the obtained original changed data is written into the incremental data table of the target file storage system. The incremental data table is set up to store changed data and can be initialized within a certain time or under certain conditions (i.e., clear all data in the table). For example, it can be initialized every 10 seconds or after merging the data in the incremental data table with the existing data table in the target file storage system.

[0081] In some embodiments, please refer to Figure 6Before step S10, i.e., the step of obtaining change information of data operation log files in the relational database of the source business system, the heterogeneous data real-time synchronization method further includes:

[0082] S01. Real-time monitoring of data operation log files in the relational database of the source business system;

[0083] S02. Determine whether there is any change information in the data operation log file.

[0084] As described in steps S01-S02 above, a monitoring program can be set up to monitor the data operation log file in the relational database of the source business system in real time and determine whether there is any change information in the data operation log file. If there is change information in the data operation log file, the steps S10-S60 above will continue to be executed. If there is no change information in the data operation log file, the real-time monitoring operation will continue to be executed.

[0085] In some embodiments, please refer again Figure 6 Before step S01 above, i.e., the step of obtaining change information of data operation log files in the relational database of the source business system, the heterogeneous data real-time synchronization method further includes:

[0086] S00, a relational database that connects to the source business system.

[0087] As described in step S00 above, a data connection program can be set up to connect to the relational database of the source business system, and the steps S01-S60 above can be executed sequentially.

[0088] In some embodiments, please refer to Figure 7 Step S501 above, which is the step of sending all the changed data obtained from the target file storage system to the message queue component in a preset format, includes:

[0089] S5011. Based on the data transmission rules of the message queue component and each of the changed data, generate first data in a preset format;

[0090] S5012, Send the first data to the message queue component;

[0091] S5013. Determine whether the message queue component returns an acknowledgment signal within a preset return time;

[0092] S5014. If the message queue component has returned an acknowledgment signal within a preset return time and the first data has not been completely sent, then continue to send the next piece of first data.

[0093] As described in steps S5011-S5014 above, in order to ensure that the transmission of changed data is accurate and orderly, after the operation of sending a first piece of data (i.e., a data packet generated based on the changed data in a preset format) to the message queue component is performed, it is determined whether the message queue component returns an acknowledgment signal within a preset return time (e.g., 1 second); if the message queue component has returned an acknowledgment signal within the preset return time and the first piece of data has not been completely sent, then the next piece of first data continues to be sent.

[0094] In some embodiments, please refer again Figure 7 The step of sending all the changed data obtained from the target file storage system to the message queue component in a preset format further includes:

[0095] S5015. If the information queue component does not return an acknowledgment signal within the preset return time, data transmission will be stopped and an error will be reported.

[0096] As described in step S5015 above, if the information queue component does not return an acknowledgment signal within the preset return time, it indicates that the data transmission may have encountered an error. In this case, data transmission is stopped and an error is reported for error investigation and repair.

[0097] This application provides a method for real-time heterogeneous data synchronization. The method involves obtaining change information from a data operation log file in a relational database of a source business system, wherein the change information includes at least one changed data entry; preprocessing the change information to generate a binary file; transmitting the binary file to a target file storage system; restoring the binary file to retrieve the changed data in the target file storage system; writing the changed data into an incremental data table in the target file storage system; and merging the incremental data table with the existing data table in the target file storage system to complete the heterogeneous data synchronization. This heterogeneous data synchronization technology eliminates the need for traditional offline full-data synchronization, achieving real-time synchronization of heterogeneous data, thereby shortening the synchronization time and better meeting the needs of data application scenarios with high timeliness requirements.

[0098] Please refer to Figure 8 This application also provides a heterogeneous data real-time synchronization device, including:

[0099] The change information acquisition module 801 is used to acquire change information of data operation log files in the relational database of the source business system, wherein the change information includes at least one piece of changed data;

[0100] The binary file generation module 802 is used to preprocess the change information to generate a binary file;

[0101] Transmission module 803 is used to transmit the binary file to the target file storage system;

[0102] The restoration module 804 is used to restore the binary file and obtain the changed data from the target file storage system.

[0103] The data writing module 805 is used to write the changed data into the incremental data table of the target file storage system;

[0104] The merging module 806 is used to merge the incremental data table with the existing data table of the target file storage system to complete the heterogeneous data synchronization.

[0105] In this embodiment, the change information acquisition module 801 acquires the change information of the data operation log file in the relational database of the source business system to obtain the corresponding change data. The change information of the data operation log file includes at least one change data. In some embodiments, the change data is newly added data.

[0106] To alleviate the pressure of data transmission, in this embodiment, the binary file generation module 802 preprocesses the change information to generate a binary file, which is then transmitted to the target file storage system via the transmission module 803, thereby reducing the total amount of data transmission without compromising data integrity. The restoration module 804 then restores the obtained binary file in the target file storage system to obtain the changed data. The data writing module 805 then writes the changed data into the incremental data table of the target file storage system, and the merging module 806 merges the incremental data table with the existing data table of the target file storage system in real time, thus completing the real-time synchronization of heterogeneous data.

[0107] In some embodiments, the binary file generation module 802 includes a preprocessing change data unit, a packaging unit, and a binary file conversion unit. The preprocessing change data unit is used to perform primary key tagging, change type addition, and change time field addition on all the change data to generate preprocessed change data; the packaging unit is used to package all the preprocessed change data to generate a preprocessed change data file; and the binary file conversion unit is used to convert the preprocessed change data file into a binary file.

[0108] In this embodiment, after obtaining changed data from the data operation log file, the preprocessing changed data unit performs primary key marking, adds change type, and adds a change time field to all changed data to generate preprocessed changed data. The primary key (i.e., primary key) is one or more fields in a database table whose value is used to uniquely identify a record in the table. In the relationship between two tables, the primary key is used to reference a specific record from another table. A primary key is a unique key and part of the table definition. A table's primary key can consist of multiple keys, and the primary key column cannot contain null values. Change type includes data addition, deletion, data type conversion, etc., and the change time field is used to record the time of data change.

[0109] The encapsulation unit encapsulates all the change data (i.e., preprocessed change data) that has been marked with primary keys and added with change type and change time fields, generating a preprocessed change data file. To reduce the pressure of data transmission, the binary file conversion unit converts the above-mentioned processed change data file into a binary format file for transmission.

[0110] In some embodiments, the data writing module 805 includes a data sending unit, an offset marking unit, a storage unit, a consumption unit, a parsing unit, and a data writing unit. Specifically, the data sending unit sends all the changed data obtained from the target file storage system to a message queue component in a preset format; the offset marking unit marks the changed data received by the message queue component with offsets, wherein the offsets are used to query the changed data; the storage unit stores the offset-marked changed data in the message queue according to the primary key; the consumption unit consumes data from the message queue component at preset time intervals; the parsing unit parses the data from the message queue component according to preset rules to obtain the changed data; and the data writing unit writes the changed data into an incremental data table of the target file storage system.

[0111] In this embodiment, the message queue component is a crucial component of large-scale data processing systems and has gradually become a core means of internal communication within these systems. It possesses a range of functionalities including loose coupling, asynchronous messaging, traffic shaping, reliable delivery, broadcasting, flow control, and eventual consistency. Common practical applications of message queues include application decoupling, asynchronous processing, traffic shaping and flow control, and log processing. The message queue component includes publish / subscribe and point-to-point modes. The publish / subscribe model is a push-based message delivery model. This model can have various subscribers; ephemeral subscribers receive messages only when actively listening to a topic, while persistent subscribers listen to all messages on the topic, even if the subscriber is offline. The point-to-point model is typically a pull or polling-based message delivery model. This model requests information from the queue rather than pushing messages to the client. A key characteristic of this model is that messages sent to the queue are received and processed by only one receiver. In this embodiment, the message queue component operates in point-to-point mode.

[0112] The data sending unit retrieves all changed data from the target file storage system and sends it to the message queue component in a preset format. This preset format is the data transmission format followed by the message queue component; different message queue components can have different data transmission formats. To retrieve relevant changed data more quickly, the offset marking unit marks the received changed data in the message queue component with offsets, setting a different offset value for each changed data. After offset marking, the storage unit partitions and stores the offset-marked changed data according to the primary key information. Then, the consumption unit consumes data from the queue component at preset time intervals (e.g., 1ms). The system receives changed data from the target file storage system transmitted in a preset format; and obtains the original changed data by parsing each data in the message queue according to preset rules through the parsing unit. The preset rules correspond to the preset format, i.e., the relationship between data packet synthesis and parsing. Finally, the data writing unit writes the obtained original changed data into the incremental data table of the target file storage system. The incremental data table is set to store changed data and can be initialized within a certain time or under certain conditions (i.e., clear all data in the table). For example, it can be initialized every 10 seconds or after merging the data in the incremental data table with the existing data table in the target file storage system.

[0113] In some embodiments, the heterogeneous data real-time synchronization device further includes a monitoring module 807 and a first judgment module 808. The monitoring module 807 is used to monitor the data operation log files in the relational database of the source business system in real time; the first judgment module 808 is used to determine whether there is change information in the data operation log files.

[0114] In this embodiment, a monitoring program can be set up through the monitoring module 807 to monitor the data operation log file in the relational database of the source business system in real time, and the first judgment module 808 can determine whether there is change information in the data operation log file. When there is change information in the data operation log file, the corresponding change information processing operation is performed, and when there is no change information in the data operation log file, the real-time monitoring operation continues.

[0115] In some embodiments, the heterogeneous data real-time synchronization device further includes a connection module 809. The connection module 809 is used to connect to the relational database of the source business system.

[0116] In this embodiment, a data connection program can be set up through the connection module 809 to connect to the relational database of the source business system, thereby enabling subsequent operations such as real-time synchronization of heterogeneous data.

[0117] In some embodiments, the data sending unit in the data writing module 805 includes a first data generation subunit, a data sending subunit, a judgment subunit, and a first execution subunit. The first data generation subunit generates first data in a preset format based on the data transmission rules of the message queue component and each modified data. The data sending subunit sends the first data to the message queue component. The judgment subunit determines whether the message queue component returns an acknowledgment signal within a preset return time. The first execution subunit continues sending the next piece of first data when the message queue component has returned an acknowledgment signal within the preset return time and the first data has not been fully sent.

[0118] In this embodiment, in order to ensure that the transmission of changed data is accurate and orderly, after the operation of sending a first piece of data (i.e., a data packet generated based on the changed data in a preset format) to the message queue component is performed, the judgment subunit determines whether the message queue component returns an acknowledgment signal within a preset return time (e.g., 1 second); the first execution subunit continues to send the next piece of first data when the message queue component has returned an acknowledgment signal within the preset return time and the first data has not been completely sent.

[0119] In some embodiments, the data sending unit in the data writing module 805 further includes a second execution subunit. The second execution subunit is used to stop data transmission and report an error when the information queue component does not return an acknowledgment signal within a preset return time.

[0120] In this embodiment, if the information queue component does not return an acknowledgment signal within a preset return time, it indicates that the data transmission may have encountered an error. In this case, the second execution subunit stops data transmission and reports an error to troubleshoot and repair the error.

[0121] Please refer to Figure 9 This application also provides a computer device whose internal structure can be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a storage medium and internal memory. The storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the storage medium. The database stores data related to a heterogeneous data real-time synchronization method. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the heterogeneous data real-time synchronization method provided in any of the above embodiments.

[0122] This application also provides a computer-readable storage medium, which can be non-volatile or volatile, storing a computer program thereon. When the computer program is executed by a processor, it implements the heterogeneous data real-time synchronization method provided in any of the above embodiments.

[0123] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media provided in this application and used in the embodiments can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-rate SDRAM (SSRSDRAM), expanded SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0124] This application provides a method, apparatus, and computer device for real-time heterogeneous data synchronization. The method involves acquiring change information from a data operation log file in a relational database of a source business system, wherein the change information includes at least one piece of changed data. The change information is preprocessed to generate a binary file. The binary file is then transmitted to a target file storage system. The binary file is restored, and the changed data is retrieved from the target file storage system. The changed data is written into an incremental data table in the target file storage system. The incremental data table is then merged with the existing data table in the target file storage system to complete the heterogeneous data synchronization. This heterogeneous data synchronization technology eliminates the need for traditional offline full-data synchronization, achieving real-time synchronization of heterogeneous data, thereby shortening the synchronization time and better meeting the needs of data application scenarios with high timeliness requirements.

[0125] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0126] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for real-time synchronization of heterogeneous data, characterized in that, include: Obtain change information from the data operation log file in the relational database of the source business system, wherein the change information includes at least one changed data record; The changed information is preprocessed to generate a binary file; The binary file is transferred to the target file storage system; The binary file is restored, and the changed data is obtained from the target file storage system; Write the changed data into the incremental data table of the target file storage system; The incremental data table is merged with the existing data table of the target file storage system to complete the heterogeneous data synchronization. in; The step of writing the changed data into the incremental data table of the target file storage system includes: All the changed data obtained from the target file storage system will be sent to the message queue component in a preset format; The changed data received by the message queue component is marked with an offset, wherein the offset is used to query the changed data; The changed data, which has been marked with offsets, is partitioned and stored in the message queue according to the primary key; Consume data from the message queue component at preset time intervals; The changed data is obtained by parsing the data in the message queue component according to preset rules; Write the changed data into the incremental data table of the target file storage system; in; The step of sending all the changed data obtained from the target file storage system to the message queue component in a preset format includes: Based on the data transmission rules of the message queue component and each of the changed data, first data in a preset format is generated; Send the first data to the message queue component; Determine whether the message queue component returns an acknowledgment signal within a preset return time; If the message queue component has returned an acknowledgment signal within a preset return time and the first data has not been completely sent, then the next piece of first data will continue to be sent. If the information queue component does not return an acknowledgment signal within the preset return time, data transmission will be stopped and an error will be reported.

2. The method for real-time synchronization of heterogeneous data according to claim 1, characterized in that, The step of transferring the binary file to the target file storage system includes: All the aforementioned changed data are marked with primary keys, changed types are added, and changed time fields are added to generate preprocessed changed data; All the preprocessed change data are packaged to generate a preprocessed change data file; Convert the preprocessed change data file into a binary file.

3. The method for real-time synchronization of heterogeneous data according to claim 1, characterized in that, Before the step of obtaining change information from the data operation log file in the relational database of the source business system, the method further includes: Real-time monitoring of data operation log files in the relational database of the source business system; Determine whether there is any change information in the data operation log file.

4. The method for real-time synchronization of heterogeneous data according to claim 3, characterized in that, Before the step of real-time monitoring of the data operation log files in the relational database of the source business system, the method further includes: A relational database that connects to the source business system.

5. A heterogeneous data real-time synchronization device, used to implement the method according to any one of claims 1-4, characterized in that, include: The change information acquisition module is used to acquire change information of data operation log files in the relational database of the source business system, wherein the change information includes at least one piece of changed data; A binary file generation module is used to preprocess the change information to generate a binary file; The transmission module is used to transmit the binary file to the target file storage system; The restore module is used to restore the binary file and obtain the changed data from the target file storage system; The data writing module is used to write the changed data into the incremental data table of the target file storage system; The merging module is used to merge the incremental data table with the existing data table of the target file storage system to complete heterogeneous data synchronization.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the heterogeneous data real-time synchronization method according to any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the heterogeneous data real-time synchronization method according to any one of claims 1-4.

Images