Irregular table database synchronization system based on query type data capture technology

Through an irregular table database synchronization system based on query data crawling technology, the problem of the existing technology being unable to synchronize irregular tables is solved, efficient data synchronization and resource utilization are achieved, and data consistency and business continuity are ensured.

CN118939729AActive Publication Date: 2024-11-12RURAL CREDIT BANKS FUNDS CLEARING CENT CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410992088.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12
Estimated Expiration
2044-07-23

AI Technical Summary

Technical Problem

Existing database synchronization technology cannot effectively synchronize databases with irregular table structures, resulting in limited data synchronization efficiency and effectiveness, affecting data consistency and business continuity.

Method used

Adopt irregular table database synchronization system based on query data grabbing technology, including intelligent number capture, intelligent number control, intelligent number transmission, intelligent number analysis and intelligent number writing. By crawling table structure information, processing data, transmission and analysis To achieve data synchronization for SQL statements that comply with the target database specifications.

Benefits of technology

It realizes efficient synchronization of irregular tables, ensures data integrity and consistency, improves resource utilization efficiency, and improves the convenience and security of database synchronization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939729A_ABST
    Figure CN118939729A_ABST
Patent Text Reader

Abstract

The invention is applicable to the technical field of database synchronization, and provides an irregular table database synchronization system based on a query type data capture technology, which comprises an intelligent data capture unit used for capturing table structure information, unit information and unit data of an irregular table from a source database; the intelligent digital controller is used for filtering and processing the data temporarily stored in the running memory to form a data carrier and formulating a binary CI standard protocol as a core carrier for data transmission and processing; the intelligent data transmission is used for transmitting the data carrier CI file from the source database server to the target database server, and the transmission process follows a TCP / IP protocol; the intelligent digital analysis is used for analyzing the data carrier CI file into an SQL statement conforming to a target database specification; and the intelligent data writing is used for writing the SQL statements into the target database. According to the method, many problems in the existing database synchronization technology are effectively solved, and obvious advantages are shown in the aspect of resource utilization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of database synchronization, and in particular relates to an irregular table database synchronization system based on query-type data crawling technology. Background Art

[0002] Database synchronization technology is a key means to ensure data consistency between the source database and the destination database. It consists of five core parts: 1) Data capture technology, which is responsible for capturing data from the source database; 2) Data processing technology, which is responsible for filtering and processing the captured data to form a data carrier; 3) Data transmission technology, which is responsible for transmitting the data carrier from the source database server to the destination database server; 4) Data parsing technology, which is responsible for parsing the data in the data carrier into a program language or database language; 5) Data loading technology, which is responsible for loading the data into the destination database. Database synchronization technology is widely used in scenarios such as database off-site disaster recovery, non-stop database migration, real-time data storage / lake, read-write separation, and asynchronous business decoupling.

[0003] There are various database synchronization solutions on the market, but they can be mainly divided into three categories: RDB-based database synchronization technology, XML-based database synchronization technology, and transaction log analysis-based database synchronization technology. With the development of technology, the number of databases that support irregular tables or flexible table structures has increased in recent years, such as MongoDB, Cassandra, SequoiaDB, ArangoDB, etc., and the above three types of existing database synchronization technologies cannot synchronize irregular tables. Market research and academic research show that database synchronization platforms that support irregular table synchronization are still relatively scarce, and large-scale and systematic research has not yet been conducted on the research and development and application of such database synchronization technologies. This has limited the efficiency and effectiveness of irregular data synchronization to a certain extent, and has posed certain challenges to ensuring high data availability and business continuity of application organizations. Therefore, it is necessary to strengthen the research and development of database synchronization technology for irregular tables and promote academic research in related fields to improve the convenience and security of database synchronization for irregular tables. In order to solve the complex problem of irregular table database synchronization, ensure data consistency and integrity, guarantee business continuity of application organizations, and achieve efficient use of storage space, the present invention proposes an irregular table database synchronization system based on query-based data capture technology. Summary of the invention

[0004] The purpose of the present invention is to provide an irregular table database synchronization system based on query-based data capture technology, aiming to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] An irregular table database synchronization system based on query-based data crawling technology includes intelligent data capture, intelligent data governance, intelligent data transmission, intelligent data analysis and intelligent data writing.

[0007] Furthermore, the intelligent data capture is used to capture the table structure information, unit information and unit data of irregular tables from the source database; the intelligent data governance is used to filter and process the data temporarily stored in the running memory to form a data carrier; the intelligent data transmission is used to transfer the data carrier CI file from the source database server to the destination database server; the intelligent data analysis is used to parse the data carrier CI file into an SQL statement that conforms to the destination database specification; the intelligent data writing is used to write the SQL statement into the destination database.

[0008] Furthermore, the workflow of the intelligent adaptive data capture is as follows:

[0009] First, Zhishi Data Capture obtains the connection information and data capture rules of the source database by reading the configuration file; then, with the line as the smallest capture unit, it captures the column name, type, unit value, and unit value length information of each unit in the row; finally, the captured information is temporarily stored in the running memory in the form of a row information structure LineDetail and a chain unit structure ChainElement.

[0010] Furthermore, in the intelligent and adaptive digital governance, a binary CI standard protocol is formulated as the core carrier for data transmission and processing.

[0011] Furthermore, the workflow of the intelligent and adaptive digital governance is as follows:

[0012] First, the CI standard protocol eliminates the need to transmit the value, into, and set keywords in DML statements. Second, the row data information is binary-encoded and written into the CI file in a specific order. Then, the other parts of the DML statement are processed as necessary. At the same time, the buffer technology is used to temporarily store multiple data items in the memory, and then write them into the CI file at once when a certain number is reached.

[0013] Furthermore, the transmission process of the intelligent adaptive data transmission complies with the TCP / IP protocol.

[0014] Furthermore, the workflow of the intelligent adaptive data analysis is as follows:

[0015] First, Zhishi Data Analysis reads the CI files in order from small to large, and reads bit by bit in a single CI file; then, it parses the CI file bit by bit to obtain the instance name, table name, database operation type and unit information of the operation; finally, according to the database specifications and syntax requirements, the parsed column name, data type, length limit, and unit value information are spliced ​​into SQL statements according to a certain logic and order.

[0016] Furthermore, the workflow of the smart data writing is as follows:

[0017] First, Zhishishuxie establishes a data interaction connection with the target database; then, it reads in SQL statements in sequence and loads data into the target database in turn; finally, based on the loading status of the target database, it promptly feeds back the loading results to the user in text form.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] The present invention effectively solves many problems existing in the existing database synchronization technology by supporting database synchronization of irregular tables, designing row information storage structure and formulating binary CI standard protocol, and shows significant advantages in resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the line information structure LineDetail and the chain unit structure ChainElement of the intelligent data capture in the present invention.

[0021] Figure 2 This is the workflow diagram of the intelligent and appropriate digital treatment in the present invention.

[0022] Figure 3 This is a workflow diagram of the intelligent adaptive data transmission in the present invention.

[0023] Figure 4 This is the workflow diagram of the intelligent adaptive data analysis in the present invention.

[0024] Figure 5 This is a workflow diagram of Zhishishu in the present invention.

[0025] Figure 6 It is the work flow chart of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0027] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0028] An embodiment of the present invention provides an irregular table database synchronization system based on query-based data capture technology (also known as "intelligent adaptive synchronization"), which is a technology for data capture, transmission and processing to maintain data consistency between a source database and a destination database, including:

[0029] (I) Intelligent data capture, which is used to capture the table structure information, unit information and unit data of irregular tables from the source database, and innovatively designs the line information structure LineDetail and the chain unit structure ChainElement (such as Figure 1 As shown) is used to store the captured information;

[0030] In the process of data synchronization, it is crucial to ensure the integrity and accuracy of information. Data synchronization is not limited to simply synchronizing the data items in the table from the source database to the destination database. More importantly, it is necessary to synchronize the table dictionary information of these data. This dictionary information includes but is not limited to column names, data types, length limits, annotations, etc. The synchronization of this dictionary information is crucial to ensure the integrity and accuracy of data after cross-database transmission. By capturing and synchronizing this dictionary information in detail, it can be ensured that the synchronized data can not only be correctly stored and queried in the destination database, but also support the call of upper-level applications, providing stable and reliable data support for upper-level applications.

[0031] The workflow of the intelligent adaptive data capture is as follows:

[0032] First, Zhishi Data Capture obtains the connection information and data capture rules of the source database by reading the configuration file; then, with the line as the smallest capture unit, it captures the column name, type, unit value, unit value length and other information of each unit in the row; finally, the captured information is temporarily stored in the running memory in the form of a row information structure LineDetail and a chain unit structure ChainElement.

[0033] Intelligent Data Capture has changed the traditional method of first focusing on the analysis of the table structure and then grabbing each row of table data one by one according to the number of columns. It provides more efficient and flexible data unit storage, management and access for irregular table row data; it abandons the limitation of the traditional linear storage structure on the fixed number of columns per row, so that shorter rows do not need to waste the memory space between the shorter rows and the longest row, thus achieving more efficient storage space utilization.

[0034] (ii) Intelligent and adaptive data governance is used to filter and process the data temporarily stored in the running memory to form a data carrier, formulate a binary CI (CHAIN ​​INFORMATION) standard protocol as the core carrier for data transmission and processing, and also support a flexible number of units per row; Figure 2 As shown, the workflow of the intelligent adaptive digital governance is as follows:

[0035] First, the CI standard protocol eliminates the need to transmit keywords such as value, into, and set in this DML statement; second, the row of data information is binary-encoded in a specific order and written into the CI file; then, the other parts of the DML statement are processed as necessary; at the same time, in order to improve the efficiency of writing CI files, the present invention uses buffer technology to temporarily store multiple data items in memory, and write them into the CI file at once after reaching a certain number, thereby reducing the number of disk I / O operations. This method of eliminating SQL statement standard keywords and encoding by row data can effectively save storage space compared to the method of storing complete SQL statements or defining memory lengths in storage units, and is more suitable for flexible and changeable irregular tables.

[0036] (III) Intelligent data transmission is used to transfer the data carrier CI file from the source database server to the destination database server. The transmission process follows the TCP / IP protocol. The workflow of intelligent data transmission is as follows: Figure 3 shown.

[0037] (iv) Intelligent data analysis, used to parse the CI file of the data carrier into SQL statements that meet the specifications of the target database; Figure 4 As shown, the workflow of the intelligent adaptive data analysis is as follows:

[0038] First, Zhishi Data Analysis reads the CI files in order from small to large, and reads bit by bit in a single CI file; then, it parses the CI file bit by bit to obtain the instance name, table name, database operation type and unit information of the operation; finally, according to the database specifications and syntax requirements, the parsed column name, data type, length limit, unit value and other information are spliced ​​into SQL statements according to a certain logic and order.

[0039] (V) Intelligent data writing, used to write SQL statements into the target database; Figure 5 As shown, the workflow of the smart adaptive data writing is as follows:

[0040] First, Zhishishuxie establishes a data interaction connection with the target database; then, it reads in SQL statements in sequence and loads data into the target database in turn; finally, based on the loading status of the target database, it promptly feeds back the loading results to the user in text form.

[0041] In summary, Adaptive Sync is a database synchronization technology that adapts to irregular tables based on query-based data capture technology. The workflow is as follows: Figure 6 The present invention effectively solves many problems existing in the existing database synchronization technology by supporting database synchronization of irregular tables, designing row information storage structure and formulating binary CI standard protocol and other innovations, and shows significant advantages in resource utilization.

[0042] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. An irregular table database synchronization system based on query-based data crawling technology, characterized in that: It includes intelligent data capture, intelligent data governance, intelligent data transmission, intelligent data analysis and intelligent data writing.

2. The irregular table database synchronization system based on query-based data crawling technology according to claim 1 is characterized in that: The intelligent data capture is used to capture the table structure information, unit information and unit data of irregular tables from the source database; the intelligent data governance is used to filter and process the data temporarily stored in the running memory to form a data carrier; the intelligent data transmission is used to transfer the data carrier CI file from the source database server to the destination database server; the intelligent data analysis is used to parse the data carrier CI file into an SQL statement that meets the specifications of the destination database; the intelligent data writing is used to write the SQL statement into the destination database.

3. The irregular table database synchronization system based on query-based data crawling technology according to claim 1 is characterized in that: The workflow of the intelligent adaptive data capture is as follows: First, Zhishi Data Capture obtains the connection information and data capture rules of the source database by reading the configuration file; then, with the line as the smallest capture unit, it captures the column name, type, unit value, and unit value length information of each unit in the row; finally, the captured information is temporarily stored in the running memory in the form of a row information structure LineDetail and a chain unit structure ChainElement.

4. The irregular table database synchronization system based on query-based data crawling technology according to claim 1 is characterized in that: In the intelligent adaptive digital governance, a binary CI standard protocol is formulated as the core carrier for data transmission and processing.

5. The irregular table database synchronization system based on query-based data crawling technology according to claim 4 is characterized in that: The workflow of the intelligent and adaptive digital governance is as follows: First, the CI standard protocol eliminates the need to transmit the value, into, and set keywords in DML statements. Second, the row data information is binary-encoded and written into the CI file in a specific order. Then, the other parts of the DML statement are processed as necessary. At the same time, the buffer technology is used to temporarily store multiple data items in the memory, and then write them into the CI file at once when a certain number is reached.

6. The irregular table database synchronization system based on query-based data crawling technology according to claim 1 is characterized in that: The transmission process of the intelligent adaptive data transmission complies with the TCP / IP protocol.

7. The irregular table database synchronization system based on query-based data crawling technology according to claim 1 is characterized in that: The workflow of the intelligent adaptive analysis is as follows: First, Zhishi Data Analysis reads the CI files in order from small to large, and reads bit by bit in a single CI file; then, it parses the CI file bit by bit to obtain the instance name, table name, database operation type and unit information of the operation; finally, according to the database specifications and syntax requirements, the parsed column name, data type, length limit, and unit value information are spliced ​​into SQL statements according to a certain logic and order.

8. The irregular table database synchronization system based on query-based data crawling technology according to claim 1 is characterized in that: The workflow of the smart data writing is as follows: First, Zhishishuxie establishes a data interaction connection with the target database; then, it reads in SQL statements in sequence and loads data into the target database in turn; finally, based on the loading status of the target database, it promptly feeds back the loading results to the user in text form.

Citation Information

Patent Citations

  • Method and system for data synchronism between heterogeneous databases

    CN103685543A

  • Data synchronization method and system among heterogeneous databases

    CN105005618A

  • Method for synchronously replicating data to Hadoop platform from MySQL database based on log analysis technology

    CN108228755A

  • Data synchronization method and system and computer readable storage medium

    CN114741453A

  • Data synchronization and desensitization integration method and device, electronic equipment and storage medium

    CN116881276A