Data synchronization method and device
By setting up multiple data synchronization channels in real-time transmission equipment and selecting applicable channels according to the service type of the target system, the problem that the data synchronization needs of different service types in the existing technology is solved, and data synchronization of different service types is achieved , improves the flexibility and adaptability of data synchronization.
Patent Information
- Application Number
- CN202110634330.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-07
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-06-07
AI Technical Summary
The prior art is difficult to meet the diversified needs of target systems of different service types in the data synchronization process, resulting in the inability to effectively meet the data synchronization needs of certain service types.
Set up multiple data synchronization channels in real-time transmission equipment, and select applicable data synchronization channels according to the service type of the target system, thereby realizing data synchronization of different service types.
Through the real-time transmission of multiple data synchronization channels in the device, it is possible to select appropriate channels according to the service type of each target system for data synchronization, meeting the data synchronization needs of different service types, and improving the flexibility and adaptability of data synchronization.
Smart Images

Figure CN113239121B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of data processing technology, and in particular, to a data synchronization method and device. Background Art
[0002] In the field of data synchronization technology, it is usually necessary to synchronize data from a source system to a target system in real time so that the source system and the target system have the same data. For example, when new data is written or data is modified in the source system, the written new data or modified data also needs to be synchronized to the target system in real time. The source system and the target system store both data in the database, which is data synchronization between databases. Among them, the source system is used to generate data, and the target system is used to use data. This relationship between the target system and the source system can be called a subscription relationship, with the target system being the subscriber and the source system being the subscriber. The source system and the target system can be connected to a real-time transmission device, which is used to achieve data synchronization between the two. The subscription relationship between the source system and the target system can also be pre-configured in the real-time transmission device.
[0003] How to meet the data synchronization requirements of target systems of different service types is an urgent problem to be solved. Summary of the invention
[0004] The embodiments of the present disclosure provide a data synchronization method and device, which can meet the data synchronization requirements of target systems of different service types.
[0005] In a first aspect, an embodiment of the present disclosure provides a data synchronization method, which is applied to a real-time transmission device, wherein the real-time transmission device is connected to an active system and a target system, and at least two data synchronization channels are provided in the real-time transmission device, and different data synchronization channels are used to perform data synchronization under different service types, and the method includes:
[0006] Determining a service type of the target system;
[0007] Determining a corresponding target data channel from the at least two data synchronization channels according to a service type of the target system;
[0008] The data in the source system is synchronized to the target system in real time through the target data channel.
[0009] In a second aspect, an embodiment of the present disclosure provides a data synchronization device, which is applied to a real-time transmission device, wherein the real-time transmission device is connected to an active system and a target system, wherein at least two data synchronization channels are provided in the real-time transmission device, and different data synchronization channels are used to perform data synchronization under different service types, wherein the device includes:
[0010] A service type determination module, used to determine the service type of the target system;
[0011] a target data channel determination module, configured to determine a corresponding target data channel from the at least two data synchronization channels according to a service type of the target system;
[0012] The data synchronization module is used to synchronize the data in the source system to the target system in real time through the target data channel.
[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;
[0014] The memory stores computer-executable instructions;
[0015] The at least one processor executes the computer-executable instructions stored in the memory, so that the electronic device implements the method as described in the first aspect.
[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, a computing device implements the method described in the first aspect.
[0017] In a fifth aspect, an embodiment of the present disclosure provides a computer program, wherein the computer program is used to implement the method described in the first aspect.
[0018] The disclosed embodiment provides a data synchronization method and device, which can be applied to a real-time transmission device, wherein the real-time transmission device is connected to an active system and a target system, and at least two data synchronization channels are provided in the real-time transmission device, and different data synchronization channels are used to perform data synchronization under different service types, including: determining the service type of the target system; determining the corresponding target data channel from at least two data synchronization channels according to the service type of the target system; and synchronizing the data in the source system to the target system in real time through the target data channel. The real-time transmission device of the disclosed embodiment is provided with data synchronization channels applicable to a variety of service types, so that the applicable data synchronization channel is selected for data synchronization according to the service type of each target system. In this way, the data synchronization requirements of target systems of different service types can be met by the real-time transmission device. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0020] Figure 1 A schematic diagram of a data synchronization scenario between at least two systems is exemplarily shown;
[0021] Figure 2 A schematic diagram showing the relationship between a real-time transmission device provided by an embodiment of the present disclosure and a source system and a target system respectively;
[0022] Figure 3 The following is a flowchart showing a method for synchronizing data according to an embodiment of the present invention.
[0023] Figure 4 The following is an exemplary structural diagram of a data synchronization device provided by an embodiment of the present disclosure;
[0024] Figure 5 The structural block diagram of an electronic device provided by an embodiment of the present disclosure is exemplarily shown. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0026] The embodiments of the present disclosure may be applied to a data synchronization scenario between at least two systems. Figure 1 The following is a schematic diagram of a data synchronization scenario between at least two systems. Figure 1 As shown, systems SS1, SS2 and SS3 are source systems, SS4 and SS5 are target systems, and the source systems can synchronize data to the target systems. For example, after the data in SS1 is changed, the data in SS4 also needs to be updated accordingly to ensure that the data between SS1 and SS4 is always the same.
[0027] It should be noted that the above data synchronization process is carried out in real time, that is, after the data of SS1 changes, SS4 must update the data immediately to avoid the data of SS1 and SS4 being different at certain moments.
[0028] from Figure 1As can be seen in the figure, the data synchronization process can be performed through a real-time transmission device, which can also be called a real-time data bus. The real-time transmission device can be a server. When it is determined that the data of the systems SS1, SS2 and SS3 may need to be synchronized for use by other systems, the systems SS1, SS2 and SS3 can be connected to the server as source systems, that is, the systems SS1, SS2 and SS3 are respectively connected to the server for communication.
[0029] In addition, if at least one of the systems SS4 and SS5 needs to use the data of the other systems, the system SS4 and SS5 is connected to the real-time transmission device as the target system, that is, the system SS4 and SS5 are respectively connected to the server for communication. In this way, the real-time transmission device is connected to the active system and the target system at the same time, so that the data of the source system can be synchronized to the target system.
[0030] The relationship between the source system and the target system in the above real-time transmission device can be called a subscription relationship, which is stored in the real-time transmission device. Figure 1 SS1 and SS4 have a subscription relationship, and SS2 and SS3 both have a subscription relationship with SS5. Therefore, the real-time transmission device can synchronize the data of SS1 to SS4, the data of SS2 to SS5, and the data of SS3 to SS5.
[0031] The above subscription relationship can be configured after the target system is connected to the real-time transmission device.
[0032] The above data synchronization scenarios are widely used in practical applications. For example, after a user makes a payment in the consumption system, an incentive service corresponding to the payment content can be issued to the user's account in the banking service system. At this time, the consumption system can be used as the source system and the banking service system can be used as the target system.
[0033] For another example, after a user performs inquiries or transactions in a banking service system, the user can analyze the query performance and transaction performance of the banking service system in the performance analysis system, and can also analyze abnormal transactions of the banking service system in the risk monitoring system to block abnormal transactions. In this case, the banking service system can be used as the source system, and the performance analysis system and risk monitoring system can be used as the target system.
[0034] It should be noted that, refer to Figure 1 As shown, the data of the source system is stored in the database, and the data synchronized from the source system by the target system is also stored in the database.
[0035] In the prior art, the above-mentioned real-time transmission equipment usually adopts an inherent method to achieve data synchronization, for example, synchronizing data from a database of a source system to a database of a target system.
[0036] However, since the target systems connected to the real-time transmission equipment are diverse, the corresponding service types are also different. Different service types may have different requirements for data synchronization. Therefore, the inherent method may fail to meet the data synchronization requirements of target systems of certain service types.
[0037] In order to solve the above problems, the real-time transmission device of the embodiment of the present disclosure is provided with data synchronization channels applicable to multiple service types, so that the applicable data synchronization channel is selected for data synchronization according to the service type of each target system. In this way, the data synchronization requirements of target systems of different service types can be met through the real-time transmission device.
[0038] The following specific embodiments are used to describe in detail the technical solutions of the embodiments of the present disclosure and how the technical solutions of the present disclosure solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present disclosure will be described below in conjunction with the accompanying drawings.
[0039] Figure 2 The following is a schematic diagram showing the relationship between a real-time transmission device provided by an embodiment of the present disclosure and a source system and a target system. Figure 2 As shown, the real-time transmission device can be connected to the source system and the target system. Four data synchronization channels are set in the real-time transmission device. Different data synchronization channels are used for data synchronization under different service types.
[0040] It should be noted that access Figure 2 The source system and the target system of the real-time transmission device in can be one or more.
[0041] Figure 3 The following is a flowchart showing a method for synchronizing data according to an embodiment of the present disclosure. Figure 3 As shown, the data synchronization method includes:
[0042] S101: Determine the service type of the target system.
[0043] The service type can be determined based on the service content of the target system. The service type is used to indicate the degree of real-time performance required by the target system, whether the target system can tolerate a certain degree of data loss, whether the data required by the target system is in the same database table, and whether the source system to which the target system subscribes supports transformation.
[0044] S102: Determine a corresponding target data channel from at least two data synchronization channels according to a service type of the target system.
[0045] In the real-time transmission device, a correspondence between the service type and the data synchronization channel may be stored, and the correspondence may be pre-configured, so that the target data channel corresponding to the service type of the target system may be determined according to the correspondence.
[0046] S103: Synchronize the data in the source system to the target system in real time through the target data channel.
[0047] Specifically, when data in a source system changes, a target system that subscribes to the data of the source system is determined from a subscription relationship, so that the data is synchronized to the target system in real time.
[0048] It is understandable that one source system may correspond to one or more target systems, so that the data of the source system can be synchronized to the one or more target systems at the same time.
[0049] Reference Figure 2 As shown, the above-mentioned data synchronization channel includes: a first channel for data synchronization through the database of the source system and the database of the target system, a second channel for data synchronization through the message queue, a third channel for data synchronization through the first channel and the message queue, and a fourth channel for data synchronization through the mirror data packet in the network server and the message queue.
[0050] The following describes in detail how to achieve data synchronization between the source system and the target system when the target data channel is the first channel, the second channel, the third channel or the fourth channel.
[0051] When the databases of the source system and the target system are both Oracle databases, the first channel may be a channel implemented based on OGG (Oracle golden gate).
[0052] Among them, OGG is a structured data replication tool based on log files, which can capture, transform and deliver in real time, realize data synchronization between the database of the source system and the database of the target system, and maintain sub-second data latency.
[0053] Specifically, when the target data channel is the first channel, synchronizing the data in the source system to the target system in real time through the target data channel may include: first, obtaining a log file from the database of the source system through the first channel; then, sending the log file to the database of the target system through the first channel, and the database of the target system is used to parse the log file to obtain data for data synchronization.
[0054] The log file may be a binary file for storing data, which may be called a Trail file. When the source system generates data, the data may be stored in the log file. When the database of the target system receives the log file including the data, since the data stored in the log file is binary data, the database of the target system needs to parse the binary data from the log file into decimal data to store it in the database of the target system.
[0055] It can be understood that when the database of the source system is an open database, the first channel of the real-time transmission device can actively obtain the log file from the database of the source system.
[0056] The above-mentioned log file is collected by the database of the source system, and a data collection terminal may be provided on the database to be connected to the first channel. The data collection terminal may collect data from the database of the source system through a data collection process and store the data in the log file. When the real-time transmission device needs the log file, the data collection terminal may send the log file to the first channel of the real-time transmission device through a data transmission process.
[0057] After receiving the log file, the first channel of the real-time transmission device sends the log file to the database of the target system. The database of the target system may be provided with a data replication end, which may parse the log file through a data replication process to obtain data, so as to store the data in the database of the target system.
[0058] It should be noted that the first channel is the synchronization between databases achieved through log files, which usually does not cause data loss. In addition, in order to synchronize data from the database of the source system to the database of the target system at one time and avoid different elements of a piece of data being synchronized at different times, the first channel requires that each element of the data be stored in a database table.
[0059] Based on the above characteristics of the first channel, the first channel can be applied to service types that cannot tolerate data loss and where all data elements are stored in a data table, that is, the target system cannot tolerate data loss. For example, after a user makes a payment in a consumer system, an incentive service corresponding to the payment content can be issued to the user's account in a banking service system. The banking service system can be used as a target system and cannot tolerate the failure to issue an incentive service after payment.
[0060] When the target data channel is the second channel, synchronizing the data in the source system to the target system in real time through the target data channel may include: first, receiving the data sent by the source system through the second channel; and then, sending the data to the target system in a message queue through the second channel.
[0061] The data is generated by the source system according to the user operation after receiving the user operation. For example, when the source system is a banking service system, after the user makes a transfer, the banking service system can generate a piece of data according to the transfer operation, including the transfer target, transfer time, transfer amount, etc.
[0062] It should be noted that, refer to Figure 2 As shown, a data generation end may be provided in the source system. After the source system generates data, the data generation end may store the data in a database of the source system. The database of the source system may record the data through a log file.
[0063] At the same time, after the source system generates data, the data generation end can also send the data to the real-time transmission device in the form of a message, and the real-time transmission device can receive the data through the second channel. The second channel can add the data to the message queue, so that when the target system requests data, the message is sent to the target system in the message queue. The target system receives the data through the data consumption end and stores the data in the database of the target system. In this way, data synchronization between the source system and the target system is achieved.
[0064] It should be noted that since the time for the data consumer of the target system to obtain data from the message queue is uncertain, and it takes time to store the data in the database of the target system after obtaining it, the service type that the second channel can be applied to represents that the target system has lower real-time requirements.
[0065] In addition, since the source system needs to write data to the message queue and the target system also needs to fetch data from the message queue, the service type applicable to the second channel represents the target system and the corresponding source system to support transformation.
[0066] When the target data channel is the third channel, synchronizing the data in the source system to the target system in real time through the target data channel may include: first, parsing the data from the log file in the first channel through the third channel; and then, sending the data to the target system in a message queue through the third channel.
[0067] It can be understood that the above log file is a log file obtained when the first channel synchronizes databases.
[0068] When the third channel receives the log file, it can parse the data from the log file. The parsing principle is the same as the principle of the database of the target system parsing data from the log file. The disclosed embodiment will not be repeated here. The data in the log file can be the data generated by the source system after receiving the user operation, that is, new data.
[0069] It is understandable that since the source system does not need to write data to the message queue, but the target system needs to get data from the message queue, the service type applicable to the third channel represents that the target system supports transformation, but the corresponding source system does not support transformation.
[0070] In addition, since it takes time for the target system to obtain data from the message queue and store it in the database, the service type applicable to the third channel represents that the target system has lower real-time requirements.
[0071] When the target data channel is the fourth channel, synchronizing the data in the source system to the target system in real time through the target data channel may include: first, performing IP (internet protocol) fragment reorganization on at least two mirrored data packets of the network server through the fourth channel to obtain at least two first data packets; then, performing TCP (transfer control protocol) connection reconstruction on the at least two first data packets through the fourth channel to obtain a second data packet; then, performing message parsing on the second data packet through the fourth channel to obtain data; and finally, sending the data to the target system through the message queue through the fourth channel.
[0072] Among them, the fourth channel is a channel implemented based on bypass monitoring technology. The bypass monitoring technology is a technology that obtains network data packets from the network access layer through the "port mirroring" function of the network server and implements specific monitoring functions through network message analysis.
[0073] Among them, the network server is a server used to forward data, which can also be called a core switch. Figure 2 As shown, the core switch can forward the data of the application client to the application server, or forward the data of the application server to the application client.
[0074] Figure 2 The network splitter in the system can mirror the data forwarded by the core switch to obtain a mirrored data packet, and copy and distribute the mirrored data packet to the bypass server.
[0075] The bypass server is a virtual server located in the real-time transmission device. It can obtain mirror data packets from the network splitter, and capture the required data from the network mirror data based on the network packet capture program of the libpcap library, and send it to the message queue after processing.
[0076] The bypass server processes the captured required data in at least three steps: IP fragment reassembly, TCP connection reconstruction, and message parsing.
[0077] Among them, IP fragment reassembly is used to splice multiple IP fragments that arrive out of order into a complete IP data packet according to the offset of each IP fragment. IP fragments are generated according to the MTU (max transfer unit) of the link layer, which limits the maximum length of the data frame. The maximum length is different for different network types. For example, the MTU of Ethernet is 1500 bytes. Therefore, when the amount of data to be transmitted is greater than or equal to the MTU, the data to be transmitted will be split into multiple IP fragments for transmission, and the amount of data in each IP fragment is less than or equal to the MTU.
[0078] TCP connection reconstruction is used to stitch together IP packets belonging to the same TCP connection in order and extract useful data from the TCP connection. A TCP connection is a communication connection based on the TCP protocol, where TCP is a connection-oriented, byte stream-based transport layer communication protocol, including three-way handshakes to establish a connection, full-duplex data transmission through a sliding window mechanism, and four-way handshakes to tear down the connection.
[0079] Message parsing is performed at the application layer. The application layer protocol defines how application processes on different systems parse messages with each other. The main task of application layer message parsing is to extract the required fields according to the application layer protocol.
[0080] After the above processing, the network mirror data is processed into data that can be used by the target system, so that it can be sent to the target system through the message queue.
[0081] Optionally, sending the data to the target system through the message queue includes: sending the data satisfying a preset rule to the target system through the message queue.
[0082] The preset rules can be used to filter useless data and can be set according to the needs of the target system to synchronize some of the data required by the target system from the source system, which helps to reduce the amount of data in the target system and improve the performance of the target system.
[0083] It should be noted that the delay of data synchronization through the fourth channel is in milliseconds, while the delay of data synchronization through the first channel is in sub-seconds, so the first channel is more suitable for target systems with service types that have higher real-time requirements.
[0084] In addition, the fourth channel is data captured from the network server, and does not need the source system to capture data, so it will not consume the resources of the source system, saving the resources of the source system. Moreover, the fourth channel does not need the source system to be modified to achieve data synchronization, so the service type applicable to the fourth channel represents the resource shortage of the source system corresponding to the target system, and does not need to modify the source system corresponding to the target system. Finally, since the data synchronized by the fourth channel is obtained from the network server, it includes a variety of data and is suitable for the synchronization of data not recorded in the database, that is, the applicable service type represents that the data required by the target system may not be recorded in the database.
[0085] In summary, the fourth channel can be applied to the following scenarios. For example, in the risk monitoring system, abnormal transactions of the banking service system are analyzed to block abnormal transactions. At this time, the banking service system can be used as the source system and the risk monitoring system can be used as the target system. The risk monitoring system can accept a certain degree of data loss, and in order to accurately determine whether there are abnormal transactions, the risk monitoring system needs more comprehensive data. Since the fourth channel is data obtained from the network server, its data is relatively comprehensive, so the risk monitoring system is suitable for the fourth channel.
[0086] For another example, after a user performs a transaction in a banking service system, the user can analyze the transaction performance of the banking service system in the performance analysis system. At this time, the banking service system can be used as the source system and the performance analysis system can be used as the target system. The performance analysis system can tolerate a certain degree of data loss and has a low requirement for real-time performance, so the first channel or the fourth channel can be used to achieve data synchronization.
[0087] For example, after a user performs a query operation in a banking service system, the query performance of the banking service system can be analyzed in a performance analysis system. At this time, the banking service system can be used as a source system and the performance analysis system can be used as a target system. Since query operations are usually not stored in the database of the banking service system, a fourth channel is required for data synchronization.
[0088] Corresponding to the data synchronization method of the above embodiment, Figure 4 The structural block diagram of a data synchronization device provided by an embodiment of the present disclosure is exemplarily shown, and is applied to a real-time transmission device. For the sake of convenience, only the part related to the embodiment of the present disclosure is shown. Figure 4 The data synchronization device 200 includes: a service type determination module 201, a target data channel determination module 202 and a data synchronization module 203.
[0089] The service type determination module 201 is used to determine the service type of the target system.
[0090] The target data channel determination module 202 is configured to determine a corresponding target data channel from the at least two data synchronization channels according to the service type of the target system.
[0091] The data synchronization module 203 is used to synchronize the data in the source system to the target system in real time through the target data channel.
[0092] Optionally, the data synchronization channel includes: a first channel for data synchronization through log files in the database of the source system, a second channel for data synchronization through a message queue, a third channel for data synchronization through log files in the first channel and the message queue, and a fourth channel for data synchronization through mirrored data packets in a network server and the message queue.
[0093] Optionally, the database is an Oracle database, and the first channel is a channel implemented based on OGG.
[0094] Optionally, the service type is determined based on at least two of the following characteristics: the degree of real-time performance required by the target system, whether the target system requires no data loss, whether the data required by the target system is in the same database table, and whether the source system to which the target system subscribes supports transformation.
[0095] Optionally, when the target data channel is the first channel, the data synchronization module 203 is further used to:
[0096] The log file is obtained from the database of the source system through the first channel; the log file is sent to the database of the target system through the first channel, and the database of the target system is used to parse the log file to obtain data for data synchronization.
[0097] Optionally, when the target data channel is the second channel, the data synchronization module 203 is further used to:
[0098] The data sent by the source system is received through the second channel; and the data is sent to the target system in the message queue through the second channel.
[0099] Optionally, when the target data channel is the third channel, the data synchronization module 203 is further used to:
[0100] Parse data from the log file in the first channel through a third channel; and send the data to the target system in the message queue through the third channel.
[0101] Optionally, when the target data channel is the fourth channel, the data synchronization module 203 is further used to:
[0102] Perform IP fragment reorganization on at least two mirrored data packets of the network server through the fourth channel to obtain at least two first data packets; perform TCP connection reconstruction on at least two of the first data packets through the fourth channel to obtain a second data packet; perform message parsing on the second data packet through the fourth channel to obtain data; and send the data to the target system through the fourth channel in the message queue.
[0103] Optionally, the data synchronization module 203 is further configured to send the data satisfying a preset rule to the target system in the message queue via a fourth channel.
[0104] The data synchronization device provided in this embodiment can be used to perform the above Figure 3 The implementation principle and technical effect of the method shown are similar, and will not be repeated here in this embodiment.
[0105] Figure 5 The structure block diagram of an electronic device 600 provided by an embodiment of the present disclosure is exemplarily shown. The electronic device 600 includes a memory 602 and at least one processor 601 .
[0106] The memory 602 stores computer-executable instructions.
[0107] At least one processor 601 executes the computer execution instructions stored in the memory 602, so that the electronic device 601 implements the aforementioned Figure 3 The method in .
[0108] In addition, the electronic device may further include a receiver 603 and a transmitter 604, wherein the receiver 603 is used to receive information from other devices or equipment and forward it to the processor 601, and the transmitter 604 is used to send the information to other devices or equipment.
[0109] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0110] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0111] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A data synchronization method, characterized in that: Applied to a real-time transmission device, the real-time transmission device is connected to an active system and a target system, the real-time transmission device is provided with at least two data synchronization channels, and different data synchronization channels are used for data synchronization under different service types, the method comprises: Determine the service type of the target system; the service type is determined based on at least two of the following characteristics: the degree of real-time performance required by the target system, whether the target system requires no data loss, whether the data required by the target system is in the same database table, and whether the source system to be subscribed by the target system supports transformation; Determining a corresponding target data channel from the at least two data synchronization channels according to a service type of the target system; Synchronize the data in the source system to the target system in real time through the target data channel; The data synchronization channel includes: a first channel for data synchronization through a log file in a database of the source system, a second channel for data synchronization through a message queue, a third channel for data synchronization through a log file in the first channel and the message queue, and a fourth channel for data synchronization through a mirror data packet in a network server and the message queue; When the target data channel is the first channel, synchronizing the data in the source system to the target system in real time through the target data channel includes: Obtaining a log file from a database of the source system through the first channel; Sending the log file to a database of the target system through the first channel, the database of the target system being used to parse the log file to obtain data for data synchronization; When the target data channel is the second channel, synchronizing the data in the source system to the target system in real time through the target data channel includes: receiving data sent by the source system through the second channel; Sending the data to the target system in the message queue through the second channel; When the target data channel is the third channel, synchronizing the data in the source system to the target system in real time through the target data channel includes: Parsing data from the log file in the first channel through the third channel; Sending the data to the target system in the message queue through the third channel; When the target data channel is the fourth channel, synchronizing the data in the source system to the target system in real time through the target data channel includes: Reassembling IP fragments of at least two mirrored data packets of the network server through the fourth channel to obtain at least two first data packets; Reconstructing the TCP connection of at least two of the first data packets through the fourth channel to obtain a second data packet; Performing message parsing on the second data packet through the fourth channel to obtain data; The data is sent to the target system in the message queue through the fourth channel.
2. The method according to claim 1, characterized in that The database is an Oracle database, and the first channel is a channel implemented based on OGG.
3. The method according to claim 2, characterized in that The sending the data to the target system in the message queue through the fourth channel includes: The data satisfying the preset rule is sent to the target system in the message queue through the fourth channel.
4. A data synchronization device, characterized in that: Applied to a real-time transmission device, the real-time transmission device is connected to an active system and a target system, the real-time transmission device is provided with at least two data synchronization channels, and different data synchronization channels are used for data synchronization under different service types, the device comprises: A service type determination module is used to determine the service type of the target system; the service type is determined based on at least two of the following characteristics: the degree of real-time performance required by the target system, whether the target system requires no data loss, whether the data required by the target system is in the same database table, and whether the source system to be subscribed by the target system supports transformation; a target data channel determination module, configured to determine a corresponding target data channel from the at least two data synchronization channels according to a service type of the target system; A data synchronization module, used to synchronize the data in the source system to the target system in real time through the target data channel; The data synchronization channel includes: a first channel for data synchronization through a log file in a database of the source system, a second channel for data synchronization through a message queue, a third channel for data synchronization through a log file in the first channel and the message queue, and a fourth channel for data synchronization through a mirror data packet in a network server and the message queue; The data synchronization module is specifically configured to obtain log files from the database of the source system through the first channel; Sending the log file to a database of the target system through the first channel, the database of the target system being used to parse the log file to obtain data for data synchronization; The data synchronization module is specifically configured to receive data sent by the source system through the second channel; Sending the data to the target system in the message queue through the second channel; The data synchronization module is specifically configured to parse the log file in the first channel through the third channel to obtain data; Sending the data to the target system in the message queue through the third channel; The data synchronization module is specifically used to reassemble IP fragments of at least two mirror data packets of the network server through the fourth channel to obtain at least two first data packets; Reconstructing the TCP connection of at least two of the first data packets through the fourth channel to obtain a second data packet; Performing message parsing on the second data packet through the fourth channel to obtain data; The data is sent to the target system in the message queue through the fourth channel.
5. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the electronic device implements the method according to any one of claims 1 to 3.
6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the computing device implements the method according to any one of claims 1 to 3.
7. A computer program, characterized in that The computer program is used to implement the method according to any one of claims 1 to 3.
Citation Information
Patent Citations
Report data filling method and system
CN103268335A
Data synchronization method and device and terminal equipment
CN110750592A
Data synchronization method, device and system
CN110795496A
Data refreshing method and device, electronic equipment and computer readable storage medium
CN111880948A
Cloud-based device synchronization
US20120079095A1
