An irregular table database synchronization system based on query-based data crawling technology

Abstract

The present invention is applicable to the field of database synchronization technology and provides an irregular table database synchronization system based on query-based data capture technology, including: intelligent data capture, which is used to capture table structure information, unit information, and unit data of irregular tables from a source database; intelligent data management, which is used to filter and process data temporarily stored in the running memory to form a data carrier, and formulate a binary CI standard protocol as the core carrier for data transmission and processing; intelligent data transmission, which is used to transmit the data carrier CI file from the source database server to the destination database server, and the transmission process follows the TCP / IP protocol; intelligent data analysis, which is used to parse the data carrier CI file into an SQL statement that conforms to the specifications of the destination database; and intelligent data writing, which is used to write the SQL statement into the destination database. The present invention effectively solves many problems existing in existing database synchronization technologies and demonstrates significant advantages in resource utilization.

Description

Technical Field

[0001] The present invention belongs to the technical field of database synchronization, and in particular relates to an irregular table database synchronization system based on query-type data crawling technology. Background Art

[0002] Database synchronization technology is a key means of ensuring data consistency between the source and destination databases. It consists of five core components: 1) Data capture technology, responsible for capturing data from the source database; 2) Data processing technology, responsible for filtering and processing the captured data to form a data carrier; 3) Data transmission technology, responsible for transferring the data carrier from the source database server to the destination database server; 4) Data parsing technology, responsible for parsing the data in the data carrier into a programming language or database language; and 5) Data loading technology, responsible for loading the data into the destination database. Database synchronization technology is widely used in scenarios such as off-site database disaster recovery, non-stop database migration, real-time data storage in warehouses / lakes, read-write separation, and asynchronous business decoupling.

[0003] While there are a variety of database synchronization solutions on the market, they can be primarily categorized into three common types: RDB-based, XML-based, and transaction log analysis-based. With technological advancements, the number of databases supporting irregular tables or flexible table structures has increased in recent years, such as MongoDB, Cassandra, SequoiaDB, and ArangoDB. However, none of these three existing database synchronization technologies are capable of synchronizing irregular tables. Market research and academic studies indicate that database synchronization platforms that support irregular table synchronization are relatively scarce, and large-scale, systematic research on the development and application of such database synchronization technologies has yet to be conducted. This, to a certain extent, limits the efficiency and effectiveness of irregular data synchronization, posing challenges to ensuring high data availability and business continuity for application organizations. Therefore, it is necessary to strengthen the research and development of database synchronization technologies for irregular tables and promote academic research in related fields to improve the convenience and security of database synchronization for irregular tables. In order to solve the complex problem of irregular table database synchronization, ensure data consistency and integrity, protect the business continuity of application organizations, and achieve efficient use of storage space, the present invention proposes an irregular table database synchronization system based on query-based data crawling technology. Summary of the Invention

[0004] The purpose of the present invention is to provide an irregular table database synchronization system based on query-based data crawling technology, aiming to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] An irregular table database synchronization system based on query-based data crawling technology, including intelligent data capture, intelligent data governance, intelligent data transmission, intelligent data analysis and intelligent data writing.

[0007] Furthermore, the intelligent adaptive data capture is used to capture the table structure information, unit information and unit data of irregular tables from the source database; the intelligent adaptive data treatment is used to filter and process the data temporarily stored in the running memory to form a data carrier; the intelligent adaptive data transmission is used to transfer the data carrier CI file from the source database server to the destination database server; the intelligent adaptive data analysis is used to parse the data carrier CI file into an SQL statement that complies with the destination database specifications; and the intelligent adaptive data writing is used to write the SQL statement into the destination database.

[0008] Furthermore, the workflow of the intelligent adaptive data capture is as follows:

[0009] First, Zhishi Data Capture obtains the connection information and data capture rules of the source database by reading the configuration file; then, with the behavior as the minimum capture unit, it captures the column name, type, unit value, and unit value length information of each unit in the row; finally, the captured information is temporarily stored in the running memory in the form of a row information structure LineDetail and a chain unit structure ChainElement.

[0010] Furthermore, in the intelligent adaptive digital governance, a binary CI standard protocol is formulated as the core carrier for data transmission and processing.

[0011] Furthermore, the workflow of the intelligent digital governance is as follows:

[0012] First, the CI standard protocol eliminates the need to transmit the value, into, and set keywords in DML statements. Second, the row data information is binary-encoded in a specific order and written into the CI file. Next, the remaining parts of the DML statement are processed as necessary. At the same time, buffer technology is used to temporarily store multiple data items in memory, and then write them all at once to the CI file once a certain number is reached.

[0013] Furthermore, the transmission process of the intelligent adaptive data transmission complies with the TCP / IP protocol.

[0014] Furthermore, the workflow of the intelligent adaptive data analysis is as follows:

[0015] First, Zhishi Data Analysis reads the CI files in order from small to large according to their serial numbers, and reads them bit by bit within a single CI file; then, it parses the CI file bit by bit to obtain the instance name, table name, database operation type, and unit information of the operation; finally, according to the database specifications and syntax requirements, the parsed column name, data type, length limit, and unit value information are spliced ​​into SQL statements according to a certain logic and order.

[0016] Furthermore, the workflow of the smart data writing is as follows:

[0017] First, Zhishishuxie establishes a data interaction connection with the target database; then, it reads in SQL statements in sequence and loads data into the target database in sequence; finally, based on the loading status of the target database, it promptly feeds back the loading results to the user in text form.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] The present invention effectively solves many problems existing in existing database synchronization technologies through innovations in supporting database synchronization of irregular tables, designing row information storage structure, and formulating binary CI standard protocols, and demonstrates significant advantages in resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 These are the line information structure LineDetail and the chain unit structure ChainElement of the intelligent adaptive data capture in the present invention.

[0021] Figure 2 This is the workflow diagram of the intelligent digital treatment in the present invention.

[0022] Figure 3 This is a workflow diagram of the intelligent adaptive data transmission in the present invention.

[0023] Figure 4 This is the workflow diagram of the intelligent adaptive data analysis in the present invention.

[0024] Figure 5 This is the workflow diagram of Zhishishu in the present invention.

[0025] Figure 6 It is the workflow diagram of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0027] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0028] One embodiment of the present invention provides an irregular table database synchronization system based on query-based data capture technology (also known as "intelligent adaptive synchronization"). Intelligent adaptive synchronization is a technology for capturing, transmitting, and processing data to maintain data consistency between the source database and the destination database, including:

[0029] (1) Intelligent Adaptive Data Capture, which is used to capture the table structure information, unit information and unit data of irregular tables from the source database, and innovatively designs the row information structure LineDetail and the chain unit structure ChainElement (such as Figure 1 As shown) is used to store the captured information;

[0030] During data synchronization, ensuring information integrity and accuracy is crucial. Data synchronization isn't limited to simply synchronizing table entries from the source database to the destination database; more importantly, it involves synchronizing the table dictionary information associated with that data. This dictionary information includes, but is not limited to, column names, data types, length limits, and annotations. Synchronizing this dictionary information is crucial for ensuring the integrity and accuracy of data after cross-database transmission. By thoroughly capturing and synchronizing this dictionary information, we ensure that the synchronized data is not only correctly stored and queried in the destination database but also supports calls from higher-level applications, providing stable and reliable data support for these applications.

[0031] The workflow of the intelligent adaptive data capture is as follows:

[0032] First, Zhishi Data Capture obtains the connection information and data capture rules of the source database by reading the configuration file; then, with the behavior as the minimum capture unit, it captures the column name, type, unit value, unit value length and other information of each unit in the row; finally, the captured information is temporarily stored in the running memory in the form of the row information structure LineDetail and the chain unit structure ChainElement.

[0033] Smart Data Capture changes the traditional table-first, data-second capture method, which focuses on analyzing the table structure first and then capturing each row of table data one by one by column. It provides more efficient and flexible data unit storage, management, and access for irregular table row data. It eliminates the traditional linear storage structure's restriction of a fixed number of columns per row, eliminating the need to waste the memory space between shorter rows and the longest row, thereby achieving more efficient storage space utilization.

[0034] (2) Intelligent Adaptive Data Governance, which is used to filter and process the data temporarily stored in the running memory to form a data carrier, formulate a binary CI (CHAIN ​​INFORMATION) standard protocol as the core carrier for data transmission and processing, and also supports a flexible number of units per row; Figure 2 As shown, the workflow of the intelligent adaptive digital treatment is as follows:

[0035] First, the CI standard protocol eliminates keywords such as "value," "into," and "set" from the DML statement, eliminating the need to transmit them. Second, the row of data is binary-encoded in a specific order and written to the CI file. Next, the remaining parts of the DML statement are processed as necessary. Furthermore, to improve the efficiency of writing to the CI file, the present invention employs buffer technology to temporarily store multiple data items in memory, then writes them all at once once a certain number has been reached, thereby reducing the number of disk I / O operations. This approach, which eliminates standard SQL statement keywords and encodes data by row, significantly saves storage space compared to methods that store complete SQL statements or define memory lengths in storage units, and is more suitable for flexible and irregular tables.

[0036] (3) Intelligent data transmission is used to transfer the data carrier CI file from the source database server to the destination database server. The transmission process follows the TCP / IP protocol. The workflow of intelligent data transmission is as follows: Figure 3 shown.

[0037] (4) Intelligent adaptive data analysis, used to parse the data carrier CI file into SQL statements that conform to the target database specifications; Figure 4 As shown, the workflow of the intelligent adaptive data analysis is as follows:

[0038] First, Zhishi Data Analysis reads the CI files in order from small to large according to their serial numbers, and reads them bit by bit within a single CI file; then, it parses the CI file bit by bit to obtain the instance name, table name, database operation type, and unit information of the operation; finally, according to the database specifications and syntax requirements, it splices the parsed column name, data type, length limit, unit value and other information into an SQL statement according to a certain logic and order.

[0039] (5) Intelligent data writing, used to write SQL statements into the target database; Figure 5 As shown, the workflow of the smart adaptive data writing is as follows:

[0040] First, Zhishishuxie establishes a data interaction connection with the target database; then, it reads in SQL statements in sequence and loads data into the target database in sequence; finally, based on the loading status of the target database, it promptly feeds back the loading results to the user in text form.

[0041] In summary, Adaptive Sync is a database synchronization technology that adapts to irregular tables based on query-based data capture technology. The workflow is as follows: Figure 6 The present invention effectively solves many problems existing in existing database synchronization technologies through innovations such as supporting database synchronization of irregular tables, designing row information storage structures, and formulating binary CI standard protocols, and demonstrates significant advantages in resource utilization.

[0042] The above are only preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, several variations and improvements can be made without departing from the concept of the present invention. These should also be regarded as the scope of protection of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. An irregular table database synchronization system based on query-based data crawling technology, characterized in that: Including intelligent data capture, intelligent data management, intelligent data transmission, intelligent data analysis and intelligent data writing; The smart data capture is used to capture the table structure information, unit information and unit data of the irregular table from the source database; the workflow of the smart data capture is as follows: First, Smart Data Capture reads the configuration file to obtain the source database's connection information and data capture rules. Next, using rows as the minimum capture unit, it captures the column name, type, cell value, and cell value length information for each cell in the row. Finally, the captured information is temporarily stored in the runtime memory in the form of a row information structure (LineDetail) and a chain element structure (ChainElement). The intelligent adaptive data management is used to filter and process the data temporarily stored in the running memory to form a data carrier. In the intelligent adaptive data management, a binary CI standard protocol is formulated as the core carrier for data transmission and processing. The workflow of the intelligent adaptive data management is as follows: First, remove the value, into, and set keywords from DML statements that are not required by the CI standard protocol. Second, binary encode the row data information according to the field order defined in the row information structure LineDetail and the chain element structure ChainElement, and write it to the CI file. Next, remove the value, into, and set keywords that are not required to be transmitted from DML statements, and convert the data type format to adapt to the binary protocol. At the same time, use buffer technology to temporarily store multiple data items in memory, and then write them all at once to the CI file after reaching a preset number. The smart data transmission is used to transmit the data carrier CI file from the source database server to the destination database server; the transmission process of the smart data transmission complies with the TCP / IP protocol; The intelligent adaptive data analysis is used to parse the data carrier CI file into SQL statements that comply with the target database specifications. The workflow of the intelligent adaptive data analysis is as follows: First, Zhi Sui Data Analysis reads the CI files in ascending order, reading each bit within a single CI file. Next, it parses the CI file bit by bit to obtain the instance name, table name, database operation type, and unit information being operated on. Finally, according to the database's specifications and syntax requirements, it assembles the parsed column name, data type, length limit, and unit value information into an SQL statement. The Smart Data Writer is used to write SQL statements into the target database. The workflow of the Smart Data Writer is as follows: First, Zhishishuxie establishes a data interaction connection with the target database; then, it reads in SQL statements in sequence and loads data into the target database in sequence; finally, based on the loading status of the target database, it promptly feeds back the loading results to the user in text form.

Images