Data transmission methods and systems applied to data synchronization

Abstract

This invention discloses a data transmission method and system for data synchronization, belonging to the field of data synchronization technology. The technical problem it addresses is how to achieve universality in data transmission formats, reduce dependence on the data transmission layer, and minimize binding to programming languages. The method includes: constructing data tuples based on field attributes; creating database tables in the target database based on the field attributes in the data tuples; filling the synchronized data into corresponding data blocks based on the structure type of the synchronized data; if the storage capacity of the current data block is less than the size of the synchronized data, creating new data blocks for cross-block storage of the synchronized data and arranging them according to their block numbers; sending the data blocks corresponding to the synchronized data to the data entry module via a message queue; and parsing the data blocks and storing the parsed synchronized data into the corresponding database tables in the target database.

Description

Technical Field

[0001] This invention relates to the field of data synchronization technology, specifically to a data transmission method and system applied to data synchronization. Background Technology

[0002] In today's rapidly developing information age, database synchronization, as a crucial means of ensuring data consistency and reliability, is widely used in various business scenarios. However, the data transmission formats generated by database synchronization tools are typically language-dependent to facilitate data encoding and decoding. While message queues (MQ) may be used for data transmission from the acquisition end to the destination, the size of the synchronization data packets may exceed MQ's capabilities. Therefore, an additional layer of data unpacking and repackaging is usually introduced. After unpacking, the data is transmitted via MQ to the destination where it is reassembled into the original data packets for further parsing and use.

[0003] During data synchronization, the technical challenges that need to be addressed include ensuring the universality of data transmission formats, reducing reliance on data transmission layers, and minimizing binding to programming languages. Summary of the Invention

[0004] The technical objective of this invention is to address the above-mentioned shortcomings by providing a data transmission method and system for data synchronization, thereby solving the technical problems of how to achieve universality of data transmission formats, reduce dependence on data transmission layers, and reduce binding to programming languages.

[0005] In a first aspect, the present invention provides a data transmission method for data synchronization, applied between a data acquisition module and a data storage module, comprising the following steps:

[0006] Data block design: For synchronized data collected from the source database tables through the data acquisition module, a fixed-size data block is configured for each type of synchronized data structure. Each data block is configured with verification information and a unique block number.

[0007] Data value design: For synchronized data collected from the source database tables through the data acquisition module, a data storage format is configured for each data format of synchronized data, and the synchronized data is filled into the data block according to the pre-configured data storage format;

[0008] Data table construction: For the source database tables, the data acquisition module parses the fields in the source database tables and constructs data tuples based on the field attributes. The data tuples are then sent to the message queue, which in turn sends them to the data entry module. After receiving the data tuples, the data entry module creates the database tables in the target database based on the field attributes in the data tuples.

[0009] Data transmission: For source database tables, after creating tables in the target database using data table construction, the data acquisition module collects the synchronization data to be synchronized from the source database tables. The synchronization data is parsed, and based on the structure type of the synchronization data, it is filled into the corresponding data blocks. If the storage capacity of the current data block is less than the size of the synchronization data, a new data block is created to store the synchronization data across blocks, and the data blocks are arranged according to their block numbers. The data blocks corresponding to the synchronization data are sent to the data entry module through a message queue. The data entry module parses the data blocks and stores the parsed synchronization data into the corresponding tables in the target database.

[0010] Preferably, the content of the data block includes verification information, block number, header information, Lower, Upper, data row index, and data row content;

[0011] The verification information is located at the beginning of the data block and includes a verification identifier and a verification code. When the verification identifier is 0, it means that the data block does not need to be verified and the verification code is empty. When the verification identifier is 1, it means that verification is required and the verification code is added after the verification identifier.

[0012] The block number is located at the back end of the verification information;

[0013] The header information is located after the block number and includes the identifier of the synchronization data type added to the block and the data length.

[0014] Lower is located at the end of the header information and points to the beginning of the index of the next data row;

[0015] Upper is located after Lower and points to the end of the data in the next data row;

[0016] The data row index is located at the back end of the Upper. For each row of synchronized data added to the data block, the data row index is added from the end of the Upper in a front-to-back manner.

[0017] The data row content is located at the back end of the data row index. For each row of synchronized data added to the data block, the data content is added from the end of the data block in a back-to-front manner.

[0018] As a preferred option, the structure types of synchronized data include metadata, full data, incremental data, large data blocks, and full data end marker. The type identifier corresponding to metadata is A, the type identifier corresponding to full data is B, the type identifier corresponding to incremental data is C, the type identifier corresponding to large data blocks is D, and the type identifier corresponding to full data end marker is E.

[0019] The data storage format includes value type and data length. After the data acquisition module collects the synchronization data to be synchronized from the source database tables, it fills the synchronization data into the data block with the specified value type based on the pre-configured value type and data length. The data entry module parses the data block and writes it into the target database tables according to the format of the synchronization data in the source database tables.

[0020] As a preferred option, for metadata, the header information includes type identifier, table unique identifier, schema name and table name, and the data tuple includes field name, whether it is a primary key, whether it is null, default value and remarks;

[0021] For the full dataset, the header information includes a type identifier and a table unique identifier, and the data tuples include the data values ​​of each column in the source database tables.

[0022] For incremental data, the header information includes a type identifier, a table unique identifier, a timestamp, and a transaction number, and the data tuple includes the data value of each column and the After value of each column;

[0023] For the full end marker, the header information includes a type identifier, and the data tuple includes the schema name and table name;

[0024] For large data blocks, the data tuple includes the length of the data within the block, the remaining data length, and the location of the remaining large data block.

[0025] Preferably, after receiving a data block from the message queue, the data entry module verifies whether the received synchronized data is missing or duplicated based on the data block's verification information and block number.

[0026] Secondly, the present invention provides a data transmission system for data synchronization, which is applied between a data acquisition module and a data storage module, including a data block design module, a data value design module, a data table construction module, and a data transmission module;

[0027] The data block design module is used to perform the following: For synchronized data collected from the source database tables through the data acquisition module, a fixed-size data block is configured for each type of synchronized data structure. Each data block is configured with verification information and a unique block number.

[0028] The data value design module is used to perform the following: For synchronized data collected from the source database tables through the data acquisition module, configure the data storage format for synchronized data of each data format, and fill the data blocks with the synchronized data through the pre-configured data storage format;

[0029] The data table construction module is used to perform the following: For the source database table, the data acquisition module parses the fields in the source database table and constructs data tuples based on the field attributes. The data tuples are then sent to the message queue and sent to the data entry module through the message queue. After receiving the data tuples, the data entry module creates the database table in the target database based on the field attributes in the data tuples.

[0030] The data transmission module performs the following actions: For source database tables, after creating tables in the target database using the data table construction method, the data acquisition module collects the synchronization data to be synchronized from the source database tables, parses the synchronization data, and fills the synchronization data into the corresponding data blocks based on the data structure type. If the storage capacity of the current data block is less than the size of the synchronization data, a new data block is created to store the synchronization data across blocks and arrange them according to the block number. The data block corresponding to the synchronization data is sent to the data entry module through a message queue. The data entry module parses the data block and stores the parsed synchronization data into the corresponding table in the target database.

[0031] Preferably, the content of the data block includes verification information, block number, header information, Lower, Upper, data row index, and data row content;

[0032] The verification information is located at the beginning of the data block and includes a verification identifier and a verification code. When the verification identifier is 0, it means that the data block does not need to be verified and the verification code is empty. When the verification identifier is 1, it means that verification is required and the verification code is added after the verification identifier.

[0033] The block number is located at the back end of the verification information;

[0034] The header information is located after the block number, and includes the data format and data length of the data added to the block for synchronization.

[0035] Lower is located at the end of the header information and points to the beginning of the index of the next data row;

[0036] Upper is located after Lower and points to the end of the data in the next data row;

[0037] The data row index is located at the back end of the Upper. For each row of synchronized data added to the data block, the data row index is added from the end of the Upper in a front-to-back manner.

[0038] The data row content is located at the back end of the data row index. For each row of synchronized data added to the data block, the data content is added from the end of the data block in a back-to-front manner.

[0039] As a preferred option, the structure types of synchronized data include metadata, full data, incremental data, large data blocks, and full data end marker. The type identifier corresponding to metadata is A, the type identifier corresponding to full data is B, the type identifier corresponding to incremental data is C, the type identifier corresponding to large data blocks is D, and the type identifier corresponding to full data end marker is E.

[0040] The data storage format includes value type and data length. After the data acquisition module collects the synchronization data to be synchronized from the source database tables, it fills the synchronization data into the data block with the specified value type based on the pre-configured value type and data length. The data entry module parses the data block and writes it into the target database tables according to the format of the synchronization data in the source database tables.

[0041] As a preferred option, for metadata, the header information includes type identifier, table unique identifier, schema name and table name, and the data tuple includes field name, whether it is a primary key, whether it is null, default value and remarks;

[0042] For the full dataset, the header information includes a type identifier and a table unique identifier, and the data tuples include the data values ​​of each column in the source database tables.

[0043] For incremental data, the header information includes a type identifier, a table unique identifier, a timestamp, and a transaction number, and the data tuple includes the data value of each column and the After value of each column;

[0044] For the full end marker, the header information includes a type identifier, and the data tuple includes the schema name and table name;

[0045] For large data blocks, the data tuple includes the length of the data within the block, the remaining data length, and the location of the remaining large data block.

[0046] Preferably, after receiving a data block from the message queue, the data entry module verifies whether the received synchronized data is missing or duplicated based on the data block's verification information and block number.

[0047] The data transmission method and system of the present invention for data synchronization have the following advantages:

[0048] 1. Synchronous data is transmitted to the destination in the form of data blocks and arranged sequentially, reducing the dependence on the transport layer;

[0049] 2. It does not bind the programming language of the data collection and database entry end, but can be implemented in a familiar language, which improves the flexibility of implementation;

[0050] 3. Reduce the complexity of data entry and memory usage;

[0051] 3. Using standard data formats facilitates reuse and functional expansion;

[0052] 4. Changes made only at the transport layer can reuse existing data synchronization functions. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] The invention will be further described below with reference to the accompanying drawings.

[0055] Figure 1 This is a flowchart of a data transmission method applied to data synchronization, as described in Example 1.

[0056] Figure 2 This is a schematic diagram of the data block format in a data transmission method applied to data synchronization, as described in Example 1. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0058] This invention provides a data transmission method and system for data synchronization, which addresses the technical problems of ensuring the universality of data transmission formats, reducing dependence on data transmission layers, and reducing binding to programming languages.

[0059] Example 1:

[0060] This invention discloses a data transmission method for data synchronization, applied between a data acquisition module and a data storage module, comprising four steps: data block design, data value design, data table construction, and data transmission.

[0061] Step S100 Data Block Design: For synchronized data collected from the source database tables through the data acquisition module, a fixed-size data block is configured for each type of synchronized data. Each data block is configured with verification information and a unique block number.

[0062] The data block contains verification information, block number, header information, Lower, Upper, data row index, and data row content.

[0063] The verification information is located at the beginning of the data block and includes a verification identifier and a verification code. When the verification identifier is 0, it means that the data block does not need to be verified and the verification code is empty. When the verification identifier is 1, it means that verification is required and the verification code is added after the verification identifier.

[0064] The block number is located after the verification information; the header information is located after the block number, and includes the identifier of the type of synchronization data added to the block and the data length; the Lower is located after the header information, pointing to the beginning of the index of the next data row; the Upper is located after the Lower, pointing to the end of the data in the next data row; the data row index is located after the Upper, and for each row of synchronization data added to the data block, the data row index is added from the end of the Upper in a forward-to-back manner; the data row content is located after the data row index, and for each row of synchronization data added to the data block, the data content is added from the end of the data block in a backward-to-forward manner.

[0065] like Figure 2 The checksum shown is the checksum of the data following it in this block; 0 indicates no checksum. The sequence number is an 8-bit number starting from 1, incrementing by one for each block. It's used by the destination to determine if data has been retransmitted or lost. The header information contains information related to parsing the data in this block. The first byte specifies the data format, and subsequent data is parsed according to this format. `Lower` points to the beginning of the index of the next data line. `Upper` indicates the end position of the next data line; 1 is the index of data line tuple1 (offset within the block), and 2 is the index of data line tuple2 (offset within the block). Tuple1 contains the content of data line 1, and Tuple2 contains the content of data line 2.

[0066] The indexes of data rows are added from front to back, while the actual content of the data rows is added from back to front. Spaces in between are left empty. Data is added gradually until there is insufficient space in the middle to store a single data item.

[0067] In this embodiment, the overall structure of different types of data blocks remains consistent, and the difference lies in the header information and the specific data row content.

[0068] The structure types of synchronized data include metadata, full data, incremental data, large data blocks, and full data end marker. The type identifier corresponding to metadata is A, the type identifier corresponding to full data is B, the type identifier corresponding to incremental data is C, the type identifier corresponding to large data blocks is D, and the type identifier corresponding to full data end marker is E.

[0069] For metadata, the header information includes type identifier, table unique identifier, schema name and table name, and the data tuple includes field name, whether it is a primary key, whether it is nullable, default value and remarks.

[0070] When table metadata is stored in this format, each data row contains the attribute information for each column, such as the field name, whether it is a primary key, whether null values ​​are allowed, default value, and remarks. The header information includes the table name, schema name, and unique identifiers for the table metadata. Because the table may have its metadata changed due to adding or deleting fields, the unique identifiers for the table metadata will also change. Subsequent data must use the new unique identifiers for the table metadata; otherwise, data parsing errors will occur.

[0071] Table 1: Header information of metadata

[0072] Type identifier Table unique identifier mode name Table name

[0073] Table 2: Data Tuples of Metadata

[0074] field name Is it a primary key? Is it null? default value Remark

[0075] For the full dataset, the header information includes a type identifier and a table unique identifier, and the data tuples include the data values ​​of each column in the source database tables.

[0076] When the full data of a table is stored in this format, a data row is information about each data value in a row, and the header information contains a unique identifier of the table's metadata.

[0077] Table 3: Header information for the full dataset

[0078] Type identifier Table unique identifier

[0079] Table 4: Data Tuples of the Full Data Set

[0080] Column 1 data values Column 2 data values 。。。 Column N data values

[0081] For incremental data, the header information includes a type identifier, a table unique identifier, a timestamp, and a transaction number. The data tuple includes the data value of each column and the After value for each column.

[0082] When incremental data for a table is stored in this format, the data rows include information about adding, deleting, and modifying data for a particular data row, and the header information includes the table's unique identifier and transaction information, etc.

[0083] Table 5: Header information for incremental data

[0084] Type identifier Table unique identifier Timestamp Transaction number

[0085] Table 6: Data Tuples for Incremental Data

[0086]

[0087] For the full end marker, the header information includes a type identifier, and the data tuple includes the schema name and table name.

[0088] Table 7: Header information for the end-of-full-load flag

[0089] Type identifier

[0090] Table 8: Data tuples for the end-of-full-process identifier

[0091] mode name Table name

[0092] For large data blocks, the data tuple includes the length of the data within the block, the remaining data length, and the location of the remaining large data block.

[0093] Big data blocks are used to record values ​​that are too long to be recorded in data tuples. The data tuple format is shown in Table 9.

[0094] Table 9: Data Tuples of Big Data Blocks

[0095] Data length within a block Remaining data length Remaining data big data block location

[0096] The above data formats can meet general needs. Other data formats can be extended as needed. For example, if a table creation statement needs to be transmitted, a new type identifier can be added, and the data tuple can record the table schema name, table name, and the table creation SQL statement. Incremental data can be synchronized using SQL, and a data type can be added in a similar way.

[0097] Step S200 Data Value Design: For synchronized data collected from the source database tables through the data acquisition module, configure the data storage format for each data format of synchronized data, and fill the data blocks with the synchronized data through the pre-configured data storage format.

[0098] In this embodiment, the data storage format includes value type and data length. After the data acquisition module collects the synchronization data to be synchronized from the source database table, the synchronization data is filled into the data block with the specified value type based on the pre-configured value type and data length. The data block is parsed by the data entry module and written into the target database table according to the format of the synchronization data in the source database table.

[0099] It supports common data types, but the data type and attributes of values ​​may differ. For example, if the field is `int`, the value may be a string. Values ​​are divided into fixed-length and non-fixed-length values. Fixed-length types such as `int`, `long`, and `char` do not require a length in the value. For side-length values ​​such as `varchar`, the length is recorded first in the value. The recording position of values ​​falls into two categories: one is the value recorded within a tuple, which has a relatively short value length; the other is the value with a very long data length, which records the position of other blocks of data within the data tuple. The data value format is shown in Table 10.

[0100] Table 10: Data value format:

[0101] Value types Is it null? Data Length Is it a specific value? Specific value | Big data block location

[0102] Table 11: Block location format is as follows:

[0103] Block number Index number

[0104] For large datasets (e.g., exceeding 2KB or a certain threshold), large data blocks are used for storage. Multiple fragments of a large dataset are linked together by their block locations.

[0105] Step S300: Data Table Construction: For the source database tables, the data acquisition module parses the fields in the source database tables and constructs data tuples based on the field attributes. The data tuples are then sent to the message queue, which in turn sends them to the data entry module. After receiving the data tuples, the data entry module creates the tables in the target database based on the field attributes in the data tuples.

[0106] Step S400 Data Transmission: For the source database tables, after creating the tables in the target database through data table construction, the data acquisition module collects the synchronization data to be synchronized from the source database tables, parses the synchronization data, and fills the synchronization data into the corresponding data blocks based on the structure type of the synchronization data. If the storage capacity of the current data block is less than the size of the synchronization data, a new data block is created to store the synchronization data across blocks and arrange them according to the block number. The data block corresponding to the synchronization data is sent to the data entry module through a message queue. The data entry module parses the data blocks and stores the parsed synchronization data into the corresponding tables in the target database.

[0107] In this embodiment, the data blocks are incremented sequentially and a check is set to verify at the destination whether there is missing or duplicate data.

[0108] The same batch of data, such as multiple blocks of the full data, starts with the full data block, followed by any potentially large data blocks.

[0109] Data at the destination is arranged in files or memory according to block number. It is recommended to use memory file mapping to avoid excessive memory usage caused by inconsistent data reception and consumption progress.

[0110] During parsing, the starting sequence number of the parsing and the expected block sequence number are recorded. The index may have used data from a subsequent block, but it has not yet been received.

[0111] Data received and processed without errors at the destination can be periodically cleared.

[0112] Example 2:

[0113] This invention discloses a data transmission system for data synchronization, which is used between a data acquisition module and a data storage module. The system includes a data block design module, a data value design module, a data table construction module, and a data transmission module.

[0114] The data block design module is used to perform the following: For synchronized data collected from the source database tables through the data acquisition module, a fixed-size data block is configured for each type of synchronized data. Each data block is configured with verification information and a unique block number.

[0115] The data block contains verification information, block number, header information, Lower, Upper, data row index, and data row content.

[0116] The verification information is located at the beginning of the data block and includes a verification identifier and a verification code. When the verification identifier is 0, it means that the data block does not need to be verified and the verification code is empty. When the verification identifier is 1, it means that verification is required and the verification code is added after the verification identifier.

[0117] The block number is located after the verification information; the header information is located after the block number, and includes the identifier of the type of synchronization data added to the block and the data length; the Lower is located after the header information, pointing to the beginning of the index of the next data row; the Upper is located after the Lower, pointing to the end of the data in the next data row; the data row index is located after the Upper, and for each row of synchronization data added to the data block, the data row index is added from the end of the Upper in a forward-to-back manner; the data row content is located after the data row index, and for each row of synchronization data added to the data block, the data content is added from the end of the data block in a backward-to-forward manner.

[0118] In this embodiment, the overall structure of different types of data blocks remains consistent, and the difference lies in the header information and the specific data row content.

[0119] The structure types of synchronized data include metadata, full data, incremental data, large data blocks, and full data end marker. The type identifier corresponding to metadata is A, the type identifier corresponding to full data is B, the type identifier corresponding to incremental data is C, the type identifier corresponding to large data blocks is D, and the type identifier corresponding to full data end marker is E.

[0120] For metadata, the header information includes type identifier, table unique identifier, schema name and table name, and the data tuple includes field name, whether it is a primary key, whether it is nullable, default value and remarks.

[0121] For the full dataset, the header information includes a type identifier and a table unique identifier, and the data tuples include the data values ​​of each column in the source database tables.

[0122] For incremental data, the header information includes a type identifier, a table unique identifier, a timestamp, and a transaction number. The data tuple includes the data value of each column and the After value for each column.

[0123] For the full end marker, the header information includes a type identifier, and the data tuple includes the schema name and table name.

[0124] For large data blocks, the data tuple includes the length of the data within the block, the remaining data length, and the location of the remaining large data block.

[0125] The data value design module is used to perform the following: For synchronized data collected from the source database tables through the data acquisition module, a data storage format is configured for each data format of synchronized data, and the synchronized data is filled into the data block through the pre-configured data storage format.

[0126] In this embodiment, the data storage format includes value type and data length. After the data acquisition module collects the synchronization data to be synchronized from the source database table, the synchronization data is filled into the data block with the specified value type based on the pre-configured value type and data length. The data block is parsed by the data entry module and written into the target database table according to the format of the synchronization data in the source database table.

[0127] It supports common data types, but the data type and attributes of values ​​may differ. For example, if the field is `int`, the value may be a string. Values ​​are divided into fixed-length and non-fixed-length values. Fixed-length types such as `int`, `long`, and `char` do not require a length in the value itself. For side-length values ​​such as `varchar`, the length is recorded first. The recording position of values ​​falls into two categories: one is values ​​recorded within tuples with relatively short lengths, and the other is values ​​with very large lengths, where the position of other blocks of data is recorded within the data tuple.

[0128] The data table construction module performs the following: For the source database tables, the data acquisition module parses the fields in the source database tables and constructs data tuples based on the field attributes. The data tuples are then sent to the message queue, which in turn sends them to the data entry module. After receiving the data tuples, the data entry module creates the database tables in the target database based on the field attributes in the data tuples.

[0129] The data transmission module performs the following actions: For source database tables, after creating tables in the target database using the data table construction method, the data acquisition module collects the synchronization data to be synchronized from the source database tables, parses the synchronization data, and fills the synchronization data into the corresponding data blocks based on the data structure type. If the storage capacity of the current data block is less than the size of the synchronization data, a new data block is created to store the synchronization data across blocks and arrange them according to the block number. The data block corresponding to the synchronization data is sent to the data entry module through a message queue. The data entry module parses the data block and stores the parsed synchronization data into the corresponding table in the target database.

[0130] The system in this embodiment can execute the method disclosed in Embodiment 1 to achieve synchronous data transmission. During transmission, data blocks are sequentially incremented, and checksums are set to verify at the destination whether there is missing or duplicate data. For the same batch of data, such as multiple blocks of full data, the full data block is generated first, followed by any potentially large data blocks. The data at the destination is arranged in a file or memory according to the block sequence number. It is recommended to use a memory-file mapping method to avoid excessive memory consumption due to inconsistencies in data reception and consumption progress. During parsing, the starting sequence number of the parsing and the expected block sequence number are recorded. The data may contain indexes that use data from a later block that has not yet been received. Data received and processed without errors at the destination can be periodically cleared.

[0131] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above multiple embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the means in the different embodiments described above, and these embodiments are also within the protection scope of the present invention.

Claims

1. A data transmission method for data synchronization, characterized in that, This method is applied between the data acquisition module and the data storage module, and includes the following steps: Data block design: For synchronized data collected from the source database tables through the data acquisition module, a fixed-size data block is configured for each type of synchronized data structure. Each data block is configured with verification information and a unique block number. Data value design: For synchronized data collected from the source database tables through the data acquisition module, a data storage format is configured for each data format of synchronized data, and the synchronized data is filled into the data block according to the pre-configured data storage format; Data table construction: For the source database tables, the data acquisition module parses the fields in the source database tables and constructs data tuples based on the field attributes. The data tuples are then sent to the message queue, which in turn sends them to the data entry module. After receiving the data tuples, the data entry module creates the database tables in the target database based on the field attributes in the data tuples. Data transmission: For source database tables, after creating tables in the target database through data table construction, the data acquisition module collects the synchronization data to be synchronized from the source database tables, parses the synchronization data, and fills the synchronization data into the corresponding data blocks based on the structure type of the synchronization data. If the storage capacity of the current data block is less than the size of the synchronization data, a new data block is created to store the synchronization data across blocks and arrange them according to the block number. The data block corresponding to the synchronization data is sent to the data entry module through a message queue. The data entry module parses the data block and stores the parsed synchronization data into the corresponding table in the target database. The data block contains verification information, block number, header information, Lower, Upper, data row index, and data row content. The verification information is located at the beginning of the data block and includes a verification identifier and a verification code. When the verification identifier is 0, it means that the data block does not need to be verified and the verification code is empty. When the verification identifier is 1, it means that verification is required and the verification code is added after the verification identifier. The block number is located at the back end of the verification information; The header information is located after the block number and includes the identifier of the synchronization data type added to the block and the data length. Lower is located at the end of the header information and points to the beginning of the index of the next data row; Upper is located after Lower and points to the end of the data in the next data row; The data row index is located at the back end of the Upper. For each row of synchronized data added to the data block, the data row index is added from the end of the Upper in a front-to-back manner. The data row content is located at the back of the data row index. For each row of synchronized data added to the data block, the data content is added from the end of the data block in a back-to-front manner. The structure types of synchronized data include metadata, full data, incremental data, large data blocks, and full data end marker. The type identifier corresponding to metadata is A, the type identifier corresponding to full data is B, the type identifier corresponding to incremental data is C, the type identifier corresponding to large data blocks is D, and the type identifier corresponding to full data end marker is E. The data storage format includes value type and data length. After the data acquisition module collects the synchronization data to be synchronized from the source database tables, it fills the synchronization data into the data block with the specified value type based on the pre-configured value type and data length. The data entry module parses the data block and writes it into the target database tables according to the format of the synchronization data in the source database tables. For metadata, the header information includes type identifier, table unique identifier, schema name and table name, and the data tuple includes field name, whether it is a primary key, whether it is null, default value and remarks. For the full dataset, the header information includes a type identifier and a table unique identifier, and the data tuples include the data values ​​of each column in the source database tables. For incremental data, the header information includes a type identifier, a table unique identifier, a timestamp, and a transaction number, and the data tuple includes the data value of each column and the After value of each column; For the full end marker, the header information includes a type identifier, and the data tuple includes the schema name and table name; For large data blocks, the data tuple includes the length of the data within the block, the remaining data length, and the location of the remaining large data block. The data entry module receives a data block from the message queue and then verifies whether the received synchronized data is missing or duplicated based on the data block's verification information and block number.

2. A data transmission system for data synchronization, characterized in that, It is applied between the data acquisition module and the data storage module, including the data block design module, data value design module, data table construction module and data transmission module; The data block design module is used to perform the following: For synchronized data collected from the source database tables through the data acquisition module, a fixed-size data block is configured for each type of synchronized data structure. Each data block is configured with verification information and a unique block number. The data value design module is used to perform the following: For synchronized data collected from the source database tables through the data acquisition module, configure the data storage format for synchronized data of each data format, and fill the data blocks with the synchronized data through the pre-configured data storage format; The data table construction module is used to perform the following: For the source database table, the data acquisition module parses the fields in the source database table and constructs data tuples based on the field attributes. The data tuples are then sent to the message queue and sent to the data entry module through the message queue. After receiving the data tuples, the data entry module creates the database table in the target database based on the field attributes in the data tuples. The data transmission module performs the following: For the source database tables, after creating the tables in the target database through data table construction, the data acquisition module collects the synchronization data to be synchronized from the source database tables, parses the synchronization data, and fills the synchronization data into the corresponding data blocks based on the structure type of the synchronization data. If the storage capacity of the current data block is less than the size of the synchronization data, a new data block is created to store the synchronization data across blocks and arrange them according to the block number. The data block corresponding to the synchronization data is sent to the data entry module through a message queue. The data entry module parses the data block and stores the parsed synchronization data into the corresponding table in the target database. The data block contains verification information, block number, header information, Lower, Upper, data row index, and data row content. The verification information is located at the beginning of the data block and includes a verification identifier and a verification code. When the verification identifier is 0, it means that the data block does not need to be verified and the verification code is empty. When the verification identifier is 1, it means that verification is required and the verification code is added after the verification identifier. The block number is located at the back end of the verification information; The header information is located after the block number and includes the identifier of the synchronization data type added to the block and the data length. Lower is located at the end of the header information and points to the beginning of the index of the next data row; Upper is located after Lower and points to the end of the data in the next data row; The data row index is located at the back end of the Upper. For each row of synchronized data added to the data block, the data row index is added from the end of the Upper in a front-to-back manner. The data row content is located at the back of the data row index. For each row of synchronized data added to the data block, the data content is added from the end of the data block in a back-to-front manner. The structure types of synchronized data include metadata, full data, incremental data, large data blocks, and full data end marker. The type identifier corresponding to metadata is A, the type identifier corresponding to full data is B, the type identifier corresponding to incremental data is C, the type identifier corresponding to large data blocks is D, and the type identifier corresponding to full data end marker is E. The data storage format includes value type and data length. After the data acquisition module collects the synchronization data to be synchronized from the source database tables, it fills the synchronization data into the data block with the specified value type based on the pre-configured value type and data length. The data entry module parses the data block and writes it into the target database tables according to the format of the synchronization data in the source database tables. For metadata, the header information includes type identifier, table unique identifier, schema name and table name, and the data tuple includes field name, whether it is a primary key, whether it is null, default value and remarks. For the full dataset, the header information includes a type identifier and a table unique identifier, and the data tuples include the data values ​​of each column in the source database tables. For incremental data, the header information includes a type identifier, a table unique identifier, a timestamp, and a transaction number, and the data tuple includes the data value of each column and the After value of each column; For the full end marker, the header information includes a type identifier, and the data tuple includes the schema name and table name; For large data blocks, the data tuple includes the length of the data within the block, the remaining data length, and the location of the remaining large data block. After receiving a data block from the message queue, the data entry module verifies whether the received synchronized data is missing or duplicated based on the data block's verification information and block number.

Images