Data transmission method and system applied to data synchronization

Abstract

The invention discloses a data transmission method and system applied to data synchronization, belongs to the technical field of data synchronization, and aims to solve the technical problems of how to solve the universality of a data transmission format, and how to reduce the dependence on a data transmission layer and the binding of a programming language. Comprising the following steps: constructing a data tuple based on field attributes, and creating a library table in a target database based on the field attributes in the data tuple; based on the structure type of the synchronous data, filling the synchronous data into the corresponding data blocks, if the storage capacity of the current data block is smaller than the size of the synchronous data, creating a new data block to carry out cross-block storage on the synchronous data, and arranging the data blocks according to the block numbers of the data blocks; and sending the data block corresponding to the synchronous data to a data storage module through the message queue, analyzing the data block through the data storage module, and storing the synchronous data obtained through analysis into a library table corresponding to the target database.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a data transmission method and system applied to data synchronization. Background Art

[0002] With the rapid development of informatization today, database synchronization is widely used in various business scenarios as an important means to ensure data consistency and reliability. However, the data transmission format generated by database synchronization tools is usually related to the programming language to facilitate data encoding and decoding; MQ may be used to transmit data from the acquisition end to the destination end, because the data size supported by MQ has certain constraints, and the size of the synchronization data packet is likely to exceed the support capacity of MQ, so generally another layer of data unpacking and assembly is introduced. After the data is unpacked, it is transmitted to the destination end through MQ and then assembled into the original data packet, and then further parsed and used.

[0003] In the process of data synchronization, how to solve the universality of data transmission format, reduce dependence on the data transmission layer and binding to programming languages ​​are technical issues that need to be solved. Summary of the invention

[0004] The technical task of the present invention is to address the above deficiencies and provide a data transmission method and system for data synchronization to solve the technical problems of how to solve the universality of the data transmission format, reduce the dependence on the data transmission layer and the binding to the programming language.

[0005] In a first aspect, the present invention provides a data transmission method for data synchronization, which is applied between a data acquisition module and a data storage module, and comprises the following steps:

[0006] Data block design: For the synchronization data collected from the source database table through the data acquisition module, a fixed-size data block is configured for each structure type of synchronization data. Each data block is configured with verification information and a unique block number.

[0007] Data value design: For the synchronization data collected from the library table of the source database through the data collection module, the data storage format is configured for the synchronization data of each data format, and the synchronization data is filled into the data block through the pre-configured data storage format;

[0008] Data table construction: For the source database table, the data collection module parses the fields in the source database table, and constructs data tuples based on the field attributes, and sends the data tuples to the message queue. The data tuples are sent to the data input module through the message queue. After the data input module receives the data tuple, it creates a table in the target database based on the field attributes in the data tuple.

[0009] Data transmission: For the source database table, after creating the table in the target database through data table construction, the data acquisition module collects the synchronization data to be synchronized from the source database table, parses the synchronization data, and fills the synchronization data into the corresponding data block based on the structure type of the synchronization data. If the storage capacity of the current data block is smaller than the size of the synchronization data, a new data block is created to store the synchronization data across blocks, and the data blocks are arranged according to the block numbers of the data blocks. The data blocks corresponding to the synchronization data are sent to the data warehousing module through the message queue, and the data blocks are parsed by the data warehousing module and the parsed synchronization data are stored in the corresponding table of the target database.

[0010] Preferably, the content in the data block includes checksum information, block number, header information, Lower, Upper, data row index and data row content;

[0011] The check information is located at the head of the data block, including a check identifier and a check code. When the check identifier is 0, it means that the data block does not need to be checked, and the check code is a null value. When the check identifier is 1, it means that the data block needs to be checked, and the check code is added after the check identifier.

[0012] The block number is at the end of the verification information;

[0013] The header information is located at the end of the block number. The header information includes the synchronization data type identifier and data length added to the block;

[0014] Lower is located at the back end of the header information, pointing to the index header of the next data row;

[0015] Upper is located at the back end of Lower and points to the end position of the data of the next data row;

[0016] The data row index is located at the back end of Upper. For each row of synchronization data added to the data block, the data row index is added from the end of Upper from the front to the back.

[0017] The data row content is located at the back end of the data row index. For each row of synchronization data added to the data block, the data content is added from the end of the data block in a back-to-front manner.

[0018] Preferably, the structure types of the synchronized data include metadata, full data, incremental data, large data blocks, and a full end identifier, the type identifier corresponding to the metadata is A, the type identifier corresponding to the full data is B, the type identifier corresponding to the incremental data is C, the type identifier corresponding to the large data block is D, and the type identifier corresponding to the full end identifier is E;

[0019] The data storage format includes value type and data length. After the synchronization data to be synchronized is collected from the library table of the source database through the data acquisition module, the synchronization data is filled into the data block with the specified value type based on the pre-configured value type and data length. The data block is parsed through the data entry module and written into the library table of the target database according to the format of the synchronization data in the library table of the source database.

[0020] Preferably, for metadata, the header information includes type identifier, table unique identifier, schema name and table name, and the data tuple includes field name, whether it is a primary key, whether it is a null value, default value and remarks;

[0021] For full data, the header information includes the type identifier and table unique identifier, and the data tuple includes the data value of each column in the source database table;

[0022] For incremental data, the header information includes the type identifier, table unique identifier, timestamp, and transaction number, and the data tuple includes the data value of each column and the After value of each column;

[0023] For the full end identifier, the header information includes the type identifier, and the data tuple includes the schema name and table name;

[0024] For a large data block, the data tuple includes the length of the data in the block, the length of the remaining data, and the location of the large data block of the remaining data.

[0025] Preferably, after receiving the data block from the message queue, the data storage module verifies whether the received synchronization data contains missing or duplicate data based on the verification information and block number of the data block.

[0026] In a second aspect, the present invention provides a data transmission system for data synchronization, which is applied between a data acquisition module and a data storage module, and includes a data block design module, a data value design module, a data table construction module, and a data transmission module;

[0027] The data block design module is used to perform the following: for the synchronization data collected from the library table of the source database by the data collection module, a fixed-size data block is configured for each structure type of synchronization data, and each data block is configured with verification information and a unique block number;

[0028] The data value design module is used to perform the following: for the synchronization data collected from the library table of the source database by the data collection module, configure the data storage format for each data format of the synchronization data, and fill the synchronization data into the data block through the pre-configured data storage format;

[0029] The data table construction module is used to perform the following: for the source database library table, the data acquisition module parses the fields in the source database library table, and constructs data tuples based on the field attributes, sends the data tuples to the message queue, and sends the data tuples to the data storage module through the message queue. After the data storage module receives the data tuple, it creates a library table in the target database based on the field attributes in the data tuple;

[0030] The data transmission module is used to perform the following: for the source database library table, after creating the library table in the target database through data table construction, the synchronization data to be synchronized is collected from the source database table through the data acquisition module, the synchronization data is parsed, and based on the structure type of the synchronization data, the synchronization data is filled into the corresponding data block. If the storage capacity of the current data block is smaller than the size of the synchronization data, a new data block is created to store the synchronization data across blocks, and the data blocks are arranged according to the block numbers of the data blocks. The data blocks corresponding to the synchronization data are sent to the data warehousing module through the message queue, and the data blocks are parsed by the data warehousing module and the parsed synchronization data is stored in the library table corresponding to the target database.

[0031] Preferably, the content in the data block includes checksum information, block number, header information, Lower, Upper, data row index and data row content;

[0032] The check information is located at the head of the data block, including a check identifier and a check code. When the check identifier is 0, it means that the data block does not need to be checked, and the check code is a null value. When the check identifier is 1, it means that the data block needs to be checked, and the check code is added after the check identifier.

[0033] The block number is at the end of the verification information;

[0034] The header information is located at the end of the block number. The header information includes the data format and data length of the synchronization data added to the block;

[0035] Lower is located at the back end of the header information, pointing to the index header of the next data row;

[0036] Upper is located at the back end of Lower and points to the end position of the data of the next data row;

[0037] The data row index is located at the back end of Upper. For each row of synchronization data added to the data block, the data row index is added from the end of Upper from the front to the back.

[0038] The data row content is located at the back end of the data row index. For each row of synchronization data added to the data block, the data content is added from the end of the data block in a back-to-front manner.

[0039] Preferably, the structure types of the synchronized data include metadata, full data, incremental data, large data blocks, and a full end identifier, the type identifier corresponding to the metadata is A, the type identifier corresponding to the full data is B, the type identifier corresponding to the incremental data is C, the type identifier corresponding to the large data block is D, and the type identifier corresponding to the full end identifier is E;

[0040] The data storage format includes value type and data length. After the synchronization data to be synchronized is collected from the library table of the source database through the data acquisition module, the synchronization data is filled into the data block with the specified value type based on the pre-configured value type and data length. The data block is parsed through the data entry module and written into the library table of the target database according to the format of the synchronization data in the library table of the source database.

[0041] Preferably, for metadata, the header information includes type identifier, table unique identifier, mode name and table name, and the data tuple includes field name, whether it is a primary key, whether it is a null value, default value and remarks;

[0042] For full data, the header information includes the type identifier and table unique identifier, and the data tuple includes the data value of each column in the source database table;

[0043] For incremental data, the header information includes the type identifier, table unique identifier, timestamp, and transaction number, and the data tuple includes the data value of each column and the After value of each column;

[0044] For the full end identifier, the header information includes the type identifier, and the data tuple includes the schema name and table name;

[0045] For a large data block, the data tuple includes the length of the data in the block, the length of the remaining data, and the location of the large data block of the remaining data.

[0046] Preferably, after receiving the data block from the message queue, the data storage module verifies whether the received synchronization data contains missing or duplicate data based on the verification information and block number of the data block.

[0047] The data transmission method and system for data synchronization of the present invention have the following advantages:

[0048] 1. Synchronous data is transmitted to the destination in the form of data blocks and arranged in sequence, reducing the dependence on the transport layer;

[0049] 2. The data collection and storage end are not bound to the programming language, and can be implemented in a familiar language, which improves the flexibility of implementation;

[0050] 3. Reduce the complexity of warehousing and memory usage;

[0051] 3. Using standard data formats facilitates reuse and functional expansion;

[0052] 4. Only changes are made to the transport layer, and the existing data synchronization functions can be reused. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] The present invention is further described below in conjunction with the accompanying drawings.

[0055] Figure 1 This is a flowchart of a data transmission method for data synchronization in Example 1;

[0056] Figure 2 This is a schematic diagram of the format of a data block in a data transmission method for data synchronization in Example 1. DETAILED DESCRIPTION

[0057] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0058] The embodiments of the present invention provide a data transmission method and system for data synchronization, which are used to solve the technical problems of how to solve the universality of the data transmission format, reduce the dependence on the data transmission layer and the binding to the programming language.

[0059] Embodiment 1:

[0060] The present invention discloses a data transmission method for data synchronization, which is applied between a data acquisition module and a data storage module and includes four steps: data block design, data value design, data table construction and data transmission.

[0061] Step S100: Data block design: For the synchronization data collected from the library table of the source database by the data collection module, a fixed-size data block is configured for each structure type of synchronization data, and each data block is configured with verification information and a unique block number.

[0062] The content of the data block includes checksum information, block number, header information, Lower, Upper, data row index and data row content.

[0063] Among them, the verification information is located at the head end of the data block, including a verification identifier and a verification code. When the verification identifier is 0, it means that the data block does not need to be verified, and the verification code is a null value. When the verification identifier is 1, it means that verification is required, and the verification code is added after the verification identifier.

[0064] The block number is located at the back end of the verification information; the header information is located at the back end of the block number, and the header information includes the synchronization data type identifier and data length added in the block; Lower is located at the back end of the header information, pointing to the position of the header of the index of the next data row; Upper is located at the back end of Lower, pointing to the end position of the data of the next data row; the data row index is located at the back end of Upper, and for each row of synchronization data added to the data block, the data row index is added from the end of Upper in a front-to-back manner; the data row content is located at the back end of the data row index, and for each row of synchronization data added to the data block, the data content is added from the end of the data block in a back-to-front manner.

[0065] like Figure 2 The checksum shown is the checksum of the subsequent data in the block. 0 means no checksum. The sequence number is an 8-bit sequence number starting from 1 and increasing by one for each block. It is used by the destination to determine whether the data is retransmitted or lost. The header information contains information related to the data parsing in the block. The first byte is the data format. The subsequent data is parsed according to the format. Lower points to the position of the header of the index of the next data row. Upper executes the end position of the data of the next data row. 1 is the position index of the data row tuple1 (offset in the block), and 2 is the position index of the data row tuple2 (offset in the block). Tuple1 is the specific content of data row 1, and Tuple2 is the specific content of data row 2.

[0066] The index of the data row is added from the front to the back, and the specific content of the data row is added from the back to the front. The middle is an empty space. Data is added gradually until the space in the middle is not enough to store a piece of data.

[0067] In this embodiment, the overall structures of different types of data blocks remain consistent, and the differences are header information and specific data row content.

[0068] The structural types of synchronized data include metadata, full data, incremental data, large data blocks, and full end markers. The type marker corresponding to metadata is A, the type marker corresponding to full data is B, the type marker corresponding to incremental data is C, the type marker corresponding to large data blocks is D, and the type marker corresponding to the full end marker is E.

[0069] For metadata, the header information includes type identifier, table unique identifier, schema name and table name, and the data tuple includes field name, whether it is a primary key, whether it is a null value, default value and remarks.

[0070] When the metadata of a table is stored in this format, the data row is the attribute information of each column, such as field name, whether it is the primary key, whether it allows null, default value, remarks, etc. The header information includes the table name, mode name, and table metadata unique identifier. Because the table may add or delete fields, which may cause metadata changes, the table metadata unique identifier will also change. Use the new metadata unique identifier in subsequent data, otherwise data parsing errors will occur.

[0071] Table 1: Metadata header information

[0072] Type Identification Table unique identifier Mode Name Table name

[0073] Table 2: Metadata data tuple

[0074] Field Name Is it a primary key? Is it null? default value Remark

[0075] For full data, the header information includes the type identifier and the table unique identifier, and the data tuple includes the data value of each column in the source database table.

[0076] When the full data of a table is stored in this format, the data row is the information of each data value in a row, and the header information contains the unique identifier of the metadata of the table.

[0077] Table 3: Header information of the full data

[0078] Type Identification Table unique identifier

[0079] Table 4: Data tuples of full data

[0080] Column 1 data value Column 2 data value 。。。 Column N data value

[0081] For incremental data, the header information includes the type identifier, table unique identifier, timestamp, and transaction number, and the data tuple includes the data value of each column and the After value of each column.

[0082] When the incremental data of a table is stored in this format, the data row includes the addition, deletion, and modification information of a data row, and the header information includes the table unique identifier and transaction information.

[0083] Table 5: Header information of incremental data

[0084] Type Identification Table unique identifier Timestamp Transaction Number

[0085] Table 6: Data tuple of incremental data

[0086]

[0087] For the full end identifier, the header information includes the type identifier, and the data tuple includes the schema name and table name.

[0088] Table 7: Header information of the full end mark

[0089] Type Identification

[0090] Table 8: Data tuple of full volume end mark

[0091] Mode Name Table name

[0092] For a large data block, the data tuple includes the length of the data in the block, the length of the remaining data, and the location of the large data block of the remaining data.

[0093] The large data block is used to record values ​​that are too long to be recorded in a data tuple. The data tuple format is shown in Table 9.

[0094] Table 9: Data tuples of a large data block

[0095] Data length in block Remaining data length Remaining data big data block location

[0096] The above data formats can meet general needs. Other formats of data can be expanded as needed. For example, if a table creation statement needs to be passed, a type identifier is added. The data tuple records the table schema name and table name, and the table creation SQL statement. Incremental data can be synchronized using SQL, and a data type can be added in a similar way.

[0097] Step S200 data value design: for the synchronization data collected from the library table of the source database by the data collection module, a data storage format is configured for the synchronization data of each data format, and the synchronization data is filled into the data block through the pre-configured data storage format.

[0098] In this embodiment, the data storage format includes a value type and a data length. After the synchronization data to be synchronized is collected from the library table of the source database through the data acquisition module, the synchronization data is filled into the data block with the specified value type based on the preconfigured value type and data length. The data block is parsed through the data warehousing module and written into the library table of the target database according to the format of the synchronization data in the library table of the source database.

[0099] Supports common data types. The data type and attribute of the value may be different. For example, if the field is int, the value may be a string type. Values ​​are divided into fixed-length and non-fixed-length values. Fixed-length values ​​such as int, long, char, etc. do not need length in the value. Long-length values ​​such as varchar first record the data length in the value. The recording position of the value includes two categories: one is the value recorded in the tuple, the value length is relatively small, and the other is the value with a very large data length. The location of other data blocks is recorded in the data tuple. The data value format is shown in Table 10.

[0100] Table 10: Data value format:

[0101] Value Types Is it null?

Data length

[0102] Table 11: The block location format is:

[0103] Block number Index Number

[0104] For data with a large length (for example, data length exceeding 2k or a certain threshold), large data blocks are used for storage. Multiple shards of a large data are connected in series through the block position.

[0105] Step S300: Data table construction: For the source database table, the fields in the source database table are parsed through the data acquisition module, and data tuples are constructed based on the field attributes. The data tuples are sent to the message queue, and the data tuples are sent to the data entry module through the message queue. After the data entry module receives the data tuple, a table is created in the target database based on the field attributes in the data tuple.

[0106] Step S400 Data transmission: For the source database library table, after creating the library table in the target database through data table construction, the synchronization data to be synchronized is collected from the source database table through the data acquisition module, and the synchronization data is parsed. Based on the structure type of the synchronization data, the synchronization data is filled into the corresponding data block. If the storage capacity of the current data block is smaller than the size of the synchronization data, a new data block is created to store the synchronization data across blocks, and the data blocks are arranged according to the block numbers of the data blocks. The data blocks corresponding to the synchronization data are sent to the data warehousing module through the message queue, and the data blocks are parsed by the data warehousing module and the parsed synchronization data is stored in the library table corresponding to the target database.

[0107] In this embodiment, the data blocks are incremented in sequence and a checksum is set to verify whether the data is missing or duplicated at the destination end.

[0108] For the same batch of data, such as multiple blocks of full data, the first block is the block of full data, followed by any large data blocks.

[0109] The data on the destination end is arranged in files or memory according to the block number. It is recommended to use memory file mapping to avoid excessive memory usage due to inconsistent data reception and consumption progress.

[0110] When parsing, the starting sequence number of the parsing and the expected block sequence number are recorded. The data may index the data in a subsequent block, but it has not been received yet.

[0111] Data that is received and processed correctly by the destination can be cleared periodically.

[0112] Embodiment 2:

[0113] The present invention discloses a data transmission system for data synchronization, which is applied between a data acquisition module and a data storage module and comprises a data block design module, a data value design module, a data table construction module and a data transmission module.

[0114] The data block design module is used to perform the following: for the synchronization data collected from the library table of the source database by the data acquisition module, a fixed-size data block is configured for each structural type of synchronization data, and each data block is configured with verification information and a unique block number.

[0115] The content of the data block includes checksum information, block number, header information, Lower, Upper, data row index and data row content.

[0116] Among them, the verification information is located at the head end of the data block, including a verification identifier and a verification code. When the verification identifier is 0, it means that the data block does not need to be verified, and the verification code is a null value. When the verification identifier is 1, it means that verification is required, and the verification code is added after the verification identifier.

[0117] The block number is located at the back end of the verification information; the header information is located at the back end of the block number, and the header information includes the synchronization data type identifier and data length added in the block; Lower is located at the back end of the header information, pointing to the position of the header of the index of the next data row; Upper is located at the back end of Lower, pointing to the end position of the data of the next data row; the data row index is located at the back end of Upper, and for each row of synchronization data added to the data block, the data row index is added from the end of Upper in a front-to-back manner; the data row content is located at the back end of the data row index, and for each row of synchronization data added to the data block, the data content is added from the end of the data block in a back-to-front manner.

[0118] In this embodiment, the overall structures of different types of data blocks remain consistent, and the differences are header information and specific data row content.

[0119] The structural types of synchronized data include metadata, full data, incremental data, large data blocks, and full end markers. The type marker corresponding to metadata is A, the type marker corresponding to full data is B, the type marker corresponding to incremental data is C, the type marker corresponding to large data blocks is D, and the type marker corresponding to the full end marker is E.

[0120] For metadata, the header information includes type identifier, table unique identifier, schema name and table name, and the data tuple includes field name, whether it is a primary key, whether it is a null value, default value and remarks.

[0121] For full data, the header information includes the type identifier and the table unique identifier, and the data tuple includes the data value of each column in the source database table.

[0122] For incremental data, the header information includes the type identifier, table unique identifier, timestamp, and transaction number, and the data tuple includes the data value of each column and the After value of each column.

[0123] For the full end identifier, the header information includes the type identifier, and the data tuple includes the schema name and table name.

[0124] For a large data block, the data tuple includes the length of the data in the block, the length of the remaining data, and the location of the large data block of the remaining data.

[0125] The data value design module is used to perform the following: for the synchronization data collected from the library table of the source database through the data collection module, the data storage format is configured for the synchronization data of each data format, and the synchronization data is filled into the data block through the pre-configured data storage format.

[0126] In this embodiment, the data storage format includes a value type and a data length. After the synchronization data to be synchronized is collected from the library table of the source database through the data acquisition module, the synchronization data is filled into the data block with the specified value type based on the preconfigured value type and data length. The data block is parsed through the data warehousing module and written into the library table of the target database according to the format of the synchronization data in the library table of the source database.

[0127] Supports common data types. The data type and attributes of the value may be different. For example, if the field is int, the value may be a string type. Values ​​are divided into fixed-length and non-fixed-length values. Fixed-length values ​​such as int, long, char, etc. do not need length in the value. Long-length values ​​such as varchar first record the data length in the value. The recording position of the value includes two categories: one is the value recorded in the tuple, the value length is relatively small, and the other is the value with a very large data length. The position of other data blocks is recorded in the data tuple.

[0128] The data table construction module is used to perform the following: for the source database library table, the fields in the source database library table are parsed through the data acquisition module, and data tuples are constructed based on the field attributes, and the data tuples are sent to the message queue. The data tuples are sent to the data entry module through the message queue. After the data entry module receives the data tuple, the library table is created in the target database based on the field attributes in the data tuple.

[0129] The data transmission module is used to perform the following: for the source database library table, after creating the library table in the target database through data table construction, the synchronization data to be synchronized is collected from the source database table through the data acquisition module, the synchronization data is parsed, and based on the structure type of the synchronization data, the synchronization data is filled into the corresponding data block. If the storage capacity of the current data block is smaller than the size of the synchronization data, a new data block is created to store the synchronization data across blocks, and the data blocks are arranged according to the block numbers of the data blocks. The data blocks corresponding to the synchronization data are sent to the data warehousing module through the message queue, and the data blocks are parsed by the data warehousing module and the parsed synchronization data is stored in the library table corresponding to the target database.

[0130] The system of this embodiment can execute the method disclosed in Example 1 to realize the transmission of synchronous data. During the transmission process, the data blocks are incremented in sequence, and a checksum is set to verify whether the data is missing or duplicated at the destination. The same batch of data, such as multiple blocks of full data, starts with the block of full data, followed by possible large data blocks. The data at the destination is arranged in the file or memory according to the block sequence number. It is recommended to use the memory file mapping method to avoid excessive memory usage due to inconsistent data reception and consumption progress. During parsing, the starting sequence number of the parsing and the expected block sequence number are recorded. The data may be indexed and used in a subsequent block, but it has not been received yet. The data received and processed correctly by the destination can be cleared regularly.

[0131] The present invention has been shown and described in detail above through the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art can know that the means in the above different embodiments can be combined to obtain more embodiments of the present invention, and these embodiments are also within the protection scope of the present invention.

Claims

1. A data transmission method for data synchronization, characterized in that: Applied between the data acquisition module and the data storage module, including the following steps: Data block design: For the synchronization data collected from the source database table through the data acquisition module, a fixed-size data block is configured for each structure type of synchronization data. Each data block is configured with verification information and a unique block number. Data value design: For the synchronization data collected from the library table of the source database through the data collection module, the data storage format is configured for the synchronization data of each data format, and the synchronization data is filled into the data block through the pre-configured data storage format; Data table construction: For the source database table, the data collection module parses the fields in the source database table, and constructs data tuples based on the field attributes, and sends the data tuples to the message queue. The data tuples are sent to the data input module through the message queue. After the data input module receives the data tuple, it creates a table in the target database based on the field attributes in the data tuple. Data transmission: For the source database table, after creating the table in the target database through data table construction, the data acquisition module collects the synchronization data to be synchronized from the source database table, parses the synchronization data, and fills the synchronization data into the corresponding data block based on the structure type of the synchronization data. If the storage capacity of the current data block is smaller than the size of the synchronization data, a new data block is created to store the synchronization data across blocks, and the data blocks are arranged according to the block numbers of the data blocks. The data blocks corresponding to the synchronization data are sent to the data warehousing module through the message queue, and the data blocks are parsed by the data warehousing module and the parsed synchronization data are stored in the corresponding table of the target database.

2. The data transmission method for data synchronization according to claim 1, characterized in that: The content of the data block includes checksum information, block number, header information, Lower, Upper, data row index and data row content; The check information is located at the head of the data block, including a check identifier and a check code. When the check identifier is 0, it means that the data block does not need to be checked, and the check code is a null value. When the check identifier is 1, it means that the data block needs to be checked, and the check code is added after the check identifier. The block number is at the end of the verification information; The header information is located at the end of the block number. The header information includes the synchronization data type identifier and data length added to the block; Lower is located at the back end of the header information, pointing to the index header of the next data row; Upper is located at the back end of Lower and points to the end position of the data of the next data row; The data row index is located at the back end of Upper. For each row of synchronization data added to the data block, the data row index is added from the end of Upper from the front to the back. The data row content is located at the back end of the data row index. For each row of synchronization data added to the data block, the data content is added from the end of the data block in a back-to-front manner.

3. The data transmission method for data synchronization according to claim 1, characterized in that: The structure types of synchronized data include metadata, full data, incremental data, large data block, and full data end mark. The type mark corresponding to metadata is A, the type mark corresponding to full data is B, the type mark corresponding to incremental data is C, the type mark corresponding to large data block is D, and the type mark corresponding to the full data end mark is E; The data storage format includes value type and data length. After the synchronization data to be synchronized is collected from the library table of the source database through the data acquisition module, the synchronization data is filled into the data block with the specified value type based on the pre-configured value type and data length. The data block is parsed through the data entry module and written into the library table of the target database according to the format of the synchronization data in the library table of the source database.

4. The data transmission method for data synchronization according to claim 3, characterized in that: For metadata, the header information includes type identifier, table unique identifier, schema name, and table name. The data tuple includes field name, whether it is a primary key, whether it is a null value, default value, and remarks. For full data, the header information includes the type identifier and table unique identifier, and the data tuple includes the data value of each column in the source database table; For incremental data, the header information includes the type identifier, table unique identifier, timestamp, and transaction number, and the data tuple includes the data value of each column and the After value of each column; For the full end identifier, the header information includes the type identifier, and the data tuple includes the schema name and table name; For a large data block, the data tuple includes the length of the data in the block, the length of the remaining data, and the location of the large data block of the remaining data.

5. The data transmission method for data synchronization according to claim 1, characterized in that: After receiving the data block from the message queue, the data warehousing module verifies whether the received synchronization data contains missing or duplicate data based on the checksum information and block number of the data block.

6. A data transmission system for data synchronization, characterized in that: Applied between the data acquisition module and the data storage module, including the data block design module, data value design module, data table construction module and data transmission module; The data block design module is used to perform the following: for the synchronization data collected from the library table of the source database by the data collection module, a fixed-size data block is configured for each structure type of synchronization data, and each data block is configured with verification information and a unique block number; The data value design module is used to perform the following: for the synchronization data collected from the library table of the source database by the data collection module, configure the data storage format for each data format of the synchronization data, and fill the synchronization data into the data block through the pre-configured data storage format; The data table construction module is used to perform the following: for the source database library table, the data acquisition module parses the fields in the source database library table, and constructs data tuples based on the field attributes, sends the data tuples to the message queue, and sends the data tuples to the data storage module through the message queue. After the data storage module receives the data tuple, it creates a library table in the target database based on the field attributes in the data tuple; The data transmission module is used to perform the following: for the source database library table, after creating the library table in the target database through data table construction, the synchronization data to be synchronized is collected from the source database table through the data acquisition module, the synchronization data is parsed, and based on the structure type of the synchronization data, the synchronization data is filled into the corresponding data block. If the storage capacity of the current data block is smaller than the size of the synchronization data, a new data block is created to store the synchronization data across blocks, and the data blocks are arranged according to the block numbers of the data blocks. The data blocks corresponding to the synchronization data are sent to the data warehousing module through the message queue, and the data blocks are parsed by the data warehousing module and the parsed synchronization data is stored in the library table corresponding to the target database.

7. The data transmission system for data synchronization according to claim 6, characterized in that: The content of the data block includes checksum information, block number, header information, Lower, Upper, data row index and data row content; The check information is located at the head of the data block, including a check identifier and a check code. When the check identifier is 0, it means that the data block does not need to be checked, and the check code is a null value. When the check identifier is 1, it means that the data block needs to be checked, and the check code is added after the check identifier. The block number is at the end of the verification information; The header information is located at the end of the block number. The header information includes the data format and data length of the synchronization data added to the block; Lower is located at the back end of the header information, pointing to the index header of the next data row; Upper is located at the back end of Lower and points to the end position of the data of the next data row; The data row index is located at the back end of Upper. For each row of synchronization data added to the data block, the data row index is added from the end of Upper from the front to the back. The data row content is located at the back end of the data row index. For each row of synchronization data added to the data block, the data content is added from the end of the data block in a back-to-front manner.

8. The data transmission system for data synchronization according to claim 6, characterized in that: The structure types of synchronized data include metadata, full data, incremental data, large data block, and full data end mark. The type mark corresponding to metadata is A, the type mark corresponding to full data is B, the type mark corresponding to incremental data is C, the type mark corresponding to large data block is D, and the type mark corresponding to the full data end mark is E; The data storage format includes value type and data length. After the synchronization data to be synchronized is collected from the library table of the source database through the data acquisition module, the synchronization data is filled into the data block with the specified value type based on the pre-configured value type and data length. The data block is parsed through the data entry module and written into the library table of the target database according to the format of the synchronization data in the library table of the source database.

9. The data transmission system for data synchronization according to claim 5, characterized in that: For metadata, the header information includes type identifier, table unique identifier, schema name, and table name. The data tuple includes field name, whether it is a primary key, whether it is a null value, default value, and remarks. For full data, the header information includes the type identifier and table unique identifier, and the data tuple includes the data value of each column in the source database table; For incremental data, the header information includes the type identifier, table unique identifier, timestamp, and transaction number, and the data tuple includes the data value of each column and the After value of each column; For the full end identifier, the header information includes the type identifier, and the data tuple includes the schema name and table name; For a large data block, the data tuple includes the length of the data in the block, the length of the remaining data, and the location of the large data block of the remaining data.

10. The data transmission system for data synchronization according to claim 5, characterized in that: After receiving the data block from the message queue, the data warehousing module verifies whether the received synchronization data contains missing or duplicate data based on the checksum information and block number of the data block.

Images