Data transmission system, method, device and medium based on canal framework

Abstract

Embodiments of the present application provide a kind of data transmission system, method, equipment and medium based on canal framework, involve data processing technical field, wherein, the system includes: first database end, including: key acquisition module, obtains public-private key pair, sends public key to second database end;Second database end, including: encryption module, encrypts the data to be transmitted using public key, and through canal server, the encrypted data to be transmitted is input into second queue;Data sending module, the encrypted data to be transmitted in second queue is sent to first database end;First database end, including: data receiving module, receives the encrypted data to be transmitted from second database end, and the encrypted data to be transmitted is input into first queue;Decryption module, reads the encrypted data to be transmitted from first queue by canal adapter, and the encrypted data to be transmitted is decrypted using private key, obtains data to be transmitted.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data transmission system, method, device and medium based on the Canal framework. Background Technology

[0002] Due to business development needs, it is often necessary to deploy data centers in two different locations. In some cases, the scope of these two locations can even extend to two countries, creating a need for cross-data center synchronization. Currently, the main approach to addressing this need is to obtain incremental changes based on business triggers.

[0003] Over the past decade, some businesses have gradually tried to obtain incremental changes by parsing database logs and then synchronizing the incremental changes. This has led to the emergence of a large number of database incremental subscription services and consumption businesses, namely Canal.

[0004] like Figure 1 As shown, the principle of data replication in MySQL master-slave databases is as follows:

[0005] The MySQL master database writes data changes to the binary log. The records in the binary log are called binary log events, which can be viewed using the `show binlog events` command.

[0006] A MySQL slave (standby database) copies the binary log events of the MySQL master to its own relay log.

[0007] MySQL slaves replay events in the relay log, reflecting data changes in their own data.

[0008] The working principle of Canal during data copying is as follows:

[0009] Canal simulates the interaction protocol of a MySQL slave, disguising itself as a MySQL slave and sending dump protocols to the MySQL master.

[0010] After receiving the dump request, the MySQL master begins pushing binary logs to the slave (i.e., canal).

[0011] Canal parses binary log objects (binary logs are originally byte streams).

[0012] However, during this data replication process, both the sending of the binary log on the MySQL master side and the receiving of the binary log as a relay log on the MySQL slave side are transmitted in plaintext. Therefore, there is a risk that data may be intercepted and leaked during the transmission of data between databases, which reduces the security of the entire communication process. Summary of the Invention

[0013] In view of this, embodiments of the present invention provide a data transmission system based on the Canal framework to solve the technical problem of low security in inter-database communication in the prior art. The system includes: a first database endpoint and a second database endpoint, wherein...

[0014] The first database terminal includes:

[0015] The key acquisition module is used to acquire a public key-private key pair and send the public key of the public key-private key pair to the second database terminal;

[0016] The second database terminal includes:

[0017] An encryption module is used to encrypt the data to be transmitted using the public key, and input the encrypted data to be transmitted into a second queue through a Canal server;

[0018] The data sending module is used to send the encrypted data to be transmitted in the second queue to the first database terminal;

[0019] The first database terminal also includes:

[0020] The data receiving module is used to receive the encrypted data to be transmitted from the second database end, and input the encrypted data to be transmitted into the first queue;

[0021] The decryption module is used to read the encrypted data to be transmitted from the first queue through the Canal adapter, and to decrypt the encrypted data to be transmitted using the private key of the public-private key pair to obtain the data to be transmitted.

[0022] This invention also provides a data transmission method based on the Canal framework. This method runs on the destination database to address the low security issue in existing database communication technologies. The method includes:

[0023] Obtain a public-private key pair and send the public key of the public-private key pair to the source database, wherein the source database is used to send the data to be transmitted, and the public key is used to encrypt the data to be transmitted;

[0024] Receive the encrypted data to be transmitted from the source database and input the encrypted data to be transmitted into the first queue;

[0025] The encrypted data to be transmitted is read from the first queue using the Canal adapter, and the encrypted data to be transmitted is decrypted using the private key of the public-private key pair to obtain the data to be transmitted.

[0026] This invention also provides a data transmission method based on the Canal framework. This method runs on the source database side to solve the technical problem of low security in inter-database communication in the prior art. The method includes:

[0027] Receive a public key from the destination database, wherein the destination database is used to receive the data to be transmitted;

[0028] The data to be transmitted is encrypted using the public key, and the encrypted data is then input into the second queue via the Canal server.

[0029] The encrypted data to be transmitted in the second queue is sent to the destination database.

[0030] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned data transmission methods based on the Canal framework, thereby solving the technical problem of low security in inter-database communication in the prior art.

[0031] This invention also provides a computer-readable storage medium storing a computer program that executes any of the above-described data transmission methods based on the Canal framework, in order to solve the technical problem of low security in inter-database communication in the prior art.

[0032] Compared with the prior art, the beneficial effects that the above-mentioned at least one technical solution adopted in the embodiments of this specification can achieve include at least the following: A first database terminal and a second database terminal are proposed to transmit data based on the Canal framework. The first database terminal obtains a public-private key pair and sends the public key of the public-private key pair to the second database terminal, so that the second database terminal uses the public key to encrypt the data to be transmitted. The encrypted data to be transmitted is then input into a second queue through the Canal server. The second database terminal then sends the encrypted data to be transmitted from the second queue to the first database terminal. Finally, the first database terminal receives the encrypted data to be transmitted from the second database terminal, inputs the encrypted data to be transmitted into a first queue, reads the encrypted data to be transmitted from the first queue through the Canal adapter, and decrypts the encrypted data to be transmitted using the private key of the public-private key pair to obtain the data to be transmitted. This system enables encrypted data transmission between the first and second databases using the Canal framework. Even if an attacker intercepts the data during transmission, they can only obtain unreadable information and cannot access the original data. This prevents data leakage during inter-database transmission and improves the security of inter-database communication. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of a data replication principle using existing technology;

[0035] Figure 2 This is a structural block diagram of a data transmission system based on the Canal framework provided in an embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram illustrating the data transmission principle of a data transmission system based on the Canal framework provided in an embodiment of the present invention.

[0037] Figure 4 This is a flowchart of a data transmission method based on the Canal framework provided in an embodiment of the present invention;

[0038] Figure 5 This is a flowchart of another data transmission method based on the Canal framework provided in an embodiment of the present invention;

[0039] Figure 6 This is a structural block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0040] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0041] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] This disclosed solution can be used in privacy computing application scenarios. Privacy computing refers to a set of technologies that enable data analysis and computation while protecting the data itself from being disclosed to the outside world, so as to achieve the goal of making the data "usable but not visible"; and realize the transformation and release of data value under the premise of fully protecting data and privacy security.

[0043] Data security is an unavoidable topic in today's "big data era". Our daily lives are filled with all kinds of complicated data. This huge amount of data is a treasure trove that has not yet been explored. It contains all kinds of valuable information, and the most important information reflected in individuals is their personal privacy information.

[0044] In today's world, if one's personal privacy information is leaked, it's like being completely naked. At best, one's daily life will be disrupted by various harassing messages and advertising calls; at worst, it may be exploited by criminals as a perfect cover for their illegal activities.

[0045] Due to the needs of daily life, we often need to provide our personal information, including some private information. Normally, this extremely important personal information is stored in a database. With the increasing sophistication of database technology, it is becoming increasingly difficult for malicious attackers to steal the data they want through direct database attacks. However, database information is not static; data only generates value when it flows. Our robust database is like a fortified city, while communication between databases is like a caravan between two cities. Attackers coveting this wealth would certainly prefer to intercept a weaker caravan rather than attack a large and fortified city. Therefore, while ensuring the stability of our city, we must also ensure the security of the caravan, that is, ensure the security of communication between databases. Therefore, this application proposes the aforementioned data transmission system based on the Canal framework to provide a guarantee for communication between databases, ensuring that even if an attacker intercepts the ciphertext of the communication, they cannot decipher the original text, thereby improving the security of the entire communication process and protecting the security of transmitted data, including private data.

[0046] In this embodiment of the invention, a data transmission system based on the Canal framework is provided, such as... Figure 2 As shown, the system includes a first database terminal 201 and a second database terminal 202, wherein,

[0047] The first database terminal 201 includes:

[0048] The key acquisition module 2011 is used to acquire a public key-private key pair and send the public key of the public key-private key pair to the second database terminal;

[0049] The second database terminal 202 includes:

[0050] The encryption module 2021 is used to encrypt the data to be transmitted using the public key, and input the encrypted data to be transmitted into the second queue through the Canal server;

[0051] Data sending module 2022 is used to send the encrypted data to be transmitted in the second queue to the first database terminal;

[0052] The first database terminal 201 also includes:

[0053] The data receiving module 2012 is used to receive the encrypted data to be transmitted from the second database end, and input the encrypted data to be transmitted into the first queue;

[0054] The decryption module 2013 is used to read the encrypted data to be transmitted from the first queue through the canal adapter, and decrypt the encrypted data to be transmitted using the private key of the public-private key pair to obtain the data to be transmitted.

[0055] Depend on Figure 2 As shown, in this embodiment of the invention, data is encrypted during data transmission between the first and second database ends based on the Canal framework. This ensures that even if the data is intercepted by an attacker during transmission, the attacker can only obtain unreadable information and cannot obtain the original data. This avoids data leakage during inter-database transmission and improves the security of inter-database communication.

[0056] In practice, during the communication between databases, the first database end is the destination database for receiving data, and the second database end is the source database for sending data. In the case of communication between different databases, the first database end can also act as the source database for sending data, and the second database end can also act as the destination database for receiving data.

[0057] In practice, the aforementioned key acquisition module can generate public-private key pairs or obtain them from other devices.

[0058] In practical implementation, in order to further improve the communication security between databases, the private key can be changed periodically according to different security requirements. In this embodiment, in order to meet higher security requirements, a one-time public-private key pair is generated for each transmission task. For example, the key acquisition module is also used to generate a set of public-private key pairs for each data transmission task. That is, a one-time public-private key pair can be generated for each data transmission task, so as to realize that the public-private key pair is changed once for each data transmission task.

[0059] In specific implementation, during the above data transmission process, the first queue and the second queue can adopt Kafka message queues. For example, the encryption module is also used to input the encrypted data to be transmitted into the second Kafka message queue through the Canal server.

[0060] The data sending module is also used to send the encrypted data to be transmitted in the second Kafka message queue to the first database terminal;

[0061] The data receiving module is also used to receive the encrypted data to be transmitted from the second database end, and input the encrypted data to be transmitted into the first Kafka message queue;

[0062] The decryption module is further configured to read the encrypted data to be transmitted from the first Kafka message queue through the Canal adapter, and decrypt the encrypted data to be transmitted using the private key of the public-private key pair to obtain the data to be transmitted.

[0063] In practice, the data transmission system based on the Canal framework can be applied to various database communication scenarios, such as real-time database backup, database mirroring, and index updates (i.e., index building and real-time maintenance).

[0064] In practical implementation, for real-time database backup applications, when data in database A changes, the same changes need to be made in database B to ensure data consistency. For data timeliness, it's desirable for changes to database B to be made as quickly as possible, while also ensuring that this process doesn't introduce security risks. In this case, the data transmission system based on the Canal framework can be used to implement the data changes. Database B is the first database endpoint, and database A is the second database endpoint. The data to be changed in database A is the data to be transmitted. First, a public-private key pair is generated on database B, or a locally stored public-private key pair is retrieved. Then, the public key is sent to database A. Next, the changes in database A are recorded in the form of an operation log. This operation log is then encrypted using the public key, and the encrypted operation log is sent to database B via a Kafka message queue. Database B receives the encrypted operation log and stores it as a relay log. Database B then decrypts the relay log using the private key to obtain the original text of the operation log from database A. Finally, the changes made on database A can be replicated to database B based on the operation log.

[0065] Specifically, for the application scenario of real-time database backup, the encryption module is also used to generate an operation log from the changed data, use the operation log as the data to be transmitted, encrypt the operation log with the public key, and input the encrypted operation log into the second queue through the Canal server;

[0066] The data sending module is also used to send the encrypted operation log in the second queue to the first database terminal;

[0067] The data receiving module is also used to receive the encrypted operation log from the second database end, store the encrypted operation log as a relay log, and input the relay log into the first queue;

[0068] The decryption module is further configured to read the relay log from the first queue through the canal adapter, and decrypt the relay log using the private key of the public-private key pair to obtain the modified data.

[0069] In practical implementation, for database mirroring application scenarios, such as when it's necessary to set up a new data center in a certain location for deployment, or when it's necessary to completely record all information within the database at a specific point in time to reduce the risk of accidental deletion or corruption of the database, the aforementioned data transmission system based on the Canal framework can be used to implement database mirroring. The new empty database is the first database end mentioned above, and the mirrored database...

[0070] The database endpoint is the second database endpoint mentioned above, and all the data content of the mirrored database endpoint is the data to be transmitted as described in step 5. First, a new empty database is created, and a public-private key pair is generated on the empty database endpoint. Then, the public key is sent to the mirrored database endpoint. Next, all the content of the mirrored database endpoint is recorded as a binary file. This binary file is then encrypted using the public key, and the encrypted binary file is sent to the empty database endpoint via a Kafka message queue. The empty database endpoint receives the encrypted binary file.

[0071] After obtaining the encrypted binary file, the empty database end decrypts the binary file using its private key, i.e., 0, to obtain the original binary file containing all the contents of the mirrored database end. Then, it can copy all the contents of the mirrored database to the empty database based on this binary file.

[0072] Specifically, for database mirroring applications, the encryption module is further used to generate a binary file from the data to be mirrored (i.e., all data content of the database being mirrored), and to convert the binary file into a binary file.

[0073] The binary file is used as the data to be transmitted, and the public key is used to encrypt the binary file. The encrypted binary file is then input into the second queue through the Canal server.

[0074] The data sending module is also used to send the encrypted binary file in the second queue to the first database terminal;

[0075] The data receiving module is used to receive the encrypted binary file from the second database and input the encrypted binary file into the first queue;

[0076] The decryption module is used to read the encrypted binary file from the first queue through the canal adapter, and decrypt the encrypted binary file using the private key of the public-private key pair to obtain the data to be mirrored.

[0077] In practical implementation, for application scenarios involving index building and real-time maintenance (e.g., splitting heterogeneous indexes, inverted indexes, etc.), the data stored in the database often becomes increasingly complex and multifaceted over time. For a specific retrieval need, a dedicated index is typically built to facilitate that need. However, sometimes it's necessary to make structural adjustments to the index based on changes in retrieval requirements to better suit those needs. Similarly, it's necessary to synchronize the newly built index and these adjustments to other databases to address the same retrieval requirements. In this case, the data transmission system based on the Canal framework can be used to implement index building and real-time maintenance. The database to be synchronized is the first database, the database with newly created or adjusted indexes is the second database, and the newly built index and / or changes to the index are the data to be transmitted. The database that creates or modifies an index records the new index or the changes to the index in its operation log. The database to be synchronized retrieves the stored public-private key pair or generates a new one. The public key is then sent to the database that created or modified the index. Upon receiving the public key, the database encrypts the operation log and sends it to the database to be synchronized via a Kafka message queue. The database to be synchronized receives the encrypted operation log, stores it as a relay log, and decrypts it using its private key. After decryption, the database to be synchronized applies the newly created index or the changes to the index locally, thus completing the synchronization.

[0078] Specifically, for application scenarios involving index building and real-time maintenance, the encryption module is further configured to generate an operation log from the updated data of the index used for searching (i.e., the operation of building the new index and / or adjusting the index), use the operation log as the data to be transmitted, encrypt the operation log using the public key, and input the encrypted operation log into the second queue through the Canal server.

[0079] The data sending module is also used to send the encrypted operation log in the second queue to the first database terminal;

[0080] The data receiving module is also used to receive the encrypted operation log from the second database end, store the encrypted operation log as a relay log, and input the relay log into the first queue;

[0081] The decryption module is further configured to read the relay log from the first queue through the canal adapter, and decrypt the relay log using the private key of the public-private key pair to obtain the updated data.

[0082] In practice, the first database terminal and the second database terminal mentioned above can be two databases belonging to the same network, or they can be databases belonging to two different networks.

[0083] In specific implementation, taking the example of the first database terminal and the second database terminal belonging to two different networks, such as... Figure 3 As shown, the first database end is a MySQL slave end belonging to network 1, and the second database end is a MySQL master end belonging to network 2. The following describes the data transmission process between the first and second database ends based on the Canal framework:

[0084] 1. The MySQL slave server generates its own private key and a public key pair.

[0085] 2. The MySQL slave server stores the private key locally on the MySQL slave server, and can send the public key to the MySQL master server in plaintext;

[0086] 3. The MySQL master uses the received public key to encrypt the binary log to be sent (i.e., the data to be transmitted mentioned above);

[0087] 4. The MySQL master server inputs the encrypted binary log into the Kafka message queue of network 2 via the canal server (canal_server). Figure 3 In Kafka_Network 2 (i.e., the second queue mentioned above), the encrypted binary log is transmitted to the Kafka message queue of Network 1 via the Kafka message queue of Network 2. Figure 3 (in kafka_network 1, i.e., the first queue mentioned above);

[0088] 5. The MySQL slave server retrieves the encrypted binary log from the Kafka message queue of network 1 through the canal adapter (canal_adapter) and saves it as a relay log;

[0089] 6. The MySQL slave end decrypts the received relay log using its own private key to obtain the original text of the transmitted information and complete the entire communication process.

[0090] In specific implementation, this embodiment also provides a data transmission method based on the Canal framework. This method runs on the destination database (i.e., the aforementioned first database), such as... Figure 4 As shown, the method includes:

[0091] Step S401: Obtain a public key-private key pair and send the public key of the public key-private key pair to the source database terminal, wherein the source database terminal is used to send the data to be transmitted, and the public key is used to encrypt the data to be transmitted;

[0092] Step S402: Receive the encrypted data to be transmitted from the source database and input the encrypted data to be transmitted into the first queue;

[0093] Step S403: Read the encrypted data to be transmitted from the first queue through the Canal adapter, and decrypt the encrypted data to be transmitted using the private key of the public-private key pair to obtain the data to be transmitted.

[0094] In one embodiment, obtaining a public-private key pair includes: generating a set of the public-private key pairs for each data transmission task.

[0095] In one embodiment, receiving encrypted data to be transmitted from the source database and inputting the encrypted data to be transmitted into a first queue includes:

[0096] Receive encrypted operation logs from the source database, store the encrypted operation logs as relay logs, and input the relay logs into the first queue, wherein the operation logs are generated from the modified data of the source database.

[0097] The encrypted data to be transmitted is read from the first queue using the Canal adapter, and the encrypted data to be transmitted is decrypted using the private key of the public-private key pair to obtain the data to be transmitted, including:

[0098] The relay log is read from the first queue using the canal adapter, and the modified data is obtained by decrypting the relay log using the private key of the public-private key pair.

[0099] In one embodiment, receiving encrypted data to be transmitted from the source database and inputting the encrypted data to be transmitted into a first queue includes:

[0100] Receive the encrypted binary file from the source database and input the encrypted binary file into the first queue, wherein the binary file is generated from the data to be mirrored from the source database.

[0101] The encrypted data to be transmitted is read from the first queue using the Canal adapter, and the encrypted data to be transmitted is decrypted using the private key of the public-private key pair to obtain the data to be transmitted, including:

[0102] The encrypted binary file is read from the first queue using the canal adapter, and the encrypted binary file is decrypted using the private key of the public-private key pair to obtain the data to be mirrored.

[0103] In one embodiment, receiving encrypted data to be transmitted from the source database and inputting the encrypted data to be transmitted into a first queue includes:

[0104] The system receives the encrypted operation log from the source database and stores the encrypted operation log as a relay log. The relay log is then input into the first queue. The operation log is generated from the updated index data of the source database used for searching.

[0105] The encrypted data to be transmitted is read from the first queue using the Canal adapter, and the encrypted data to be transmitted is decrypted using the private key of the public-private key pair to obtain the data to be transmitted, including:

[0106] The relay log is read from the first queue using the canal adapter, and the updated data is obtained by decrypting the relay log using the private key of the public-private key pair.

[0107] In one embodiment, the first queue can be a Kafka message queue. That is, it receives the encrypted data to be transmitted from the source database and inputs the encrypted data into the first Kafka message queue; then, it reads data from the first Kafka message queue via a Canal adapter.

[0108] The encrypted data to be transmitted is decrypted using the private key of the public-private key pair to obtain the data to be transmitted.

[0109] In specific implementation, this embodiment also provides a data transmission method based on the Canal framework. This method runs on the source database (i.e., the aforementioned second database), such as... Figure 5 As shown, the method includes:

[0110] Step S501: Receive the public key from the destination database, wherein the destination database is used to receive the data to be transmitted;

[0111] Step S502: Encrypt the data to be transmitted using the public key, and input the encrypted data to be transmitted into the second queue through the Canal server;

[0112] Step S503: Send the encrypted data to be transmitted in the second queue to the destination database.

[0113] 5. In one embodiment, the public key is used to encrypt the data to be transmitted, and the encrypted data to be transmitted is input into a second queue through a Canal server, including:

[0114] The modified data is used to generate an operation log, which is then used as the data to be transmitted. The operation log is encrypted using the public key, and the encrypted operation log is input into the second queue through the Canal server.

[0115] Sending the encrypted data to be transmitted from the second queue to the destination database includes:

[0116] The encrypted operation log in the second queue is sent to the destination database.

[0117] In one embodiment, the data to be transmitted is encrypted using the public key, and the encrypted data is input into a second queue via a Canal server, including:

[0118] The data to be mirrored is generated into a binary file, which is then used as the data to be transmitted. The binary file is encrypted using the public key, and the encrypted binary file is input into the second queue through the Canal server.

[0119] Sending the encrypted data to be transmitted from the second queue to the destination database includes:

[0120] The encrypted binary file in the second queue is sent to the destination database. 5. In one embodiment, the data to be transmitted is encrypted using the public key, and the encrypted data to be transmitted is input into the second queue through a Canal server, including:

[0121] An operation log is generated from the update data of the index used for searching. The operation log is used as the data to be transmitted. The operation log is encrypted using the public key. The encrypted operation log is then input into the second queue through the canal server.

[0122] 0. Sending the encrypted data to be transmitted from the second queue to the destination database includes:

[0123] The encrypted operation log in the second queue is sent to the destination database.

[0124] In this embodiment, the second queue can be a Kafka message queue. That is, the data to be transmitted is encrypted using the public key, and the encrypted data is input into the second Kafka message queue via the Canal server; then, the encrypted data in the second Kafka message queue is sent to the destination database.

[0125] In this embodiment, a computer device is provided, such as... Figure 6 As shown, it includes a memory 601, a processor 602, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described data transmission methods based on the Canal framework.

[0126] Specifically, the computer device can be a computer terminal, a server, or a similar computing device.

[0127] In this embodiment, a computer-readable storage medium is provided, which stores a computer program that executes any of the above-described data transmission methods based on the Canal framework.

[0128] Specifically, computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.

[0129] The embodiments of the present invention achieve the following technical effects: they enable encrypted transmission of data during data transmission between the first and second database ends based on the Canal framework. This ensures that even if the data is intercepted by an attacker during transmission, the attacker can only obtain unreadable information and cannot obtain the original data, thereby preventing data leakage during inter-database transmission and improving the security of inter-database communication.

[0130] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

[0131] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data transmission system based on the Canal framework, characterized in that, include: The first database terminal and the second database terminal, among which... The first database terminal includes: The key acquisition module is used to acquire a public key-private key pair and send the public key of the public key-private key pair to the second database terminal; The second database terminal includes: An encryption module is used to encrypt the data to be transmitted using the public key, and input the encrypted data to be transmitted into a second queue through a Canal server; The data sending module is used to send the encrypted data to be transmitted in the second queue to the first database terminal; The first database terminal also includes: The data receiving module is used to receive the encrypted data to be transmitted from the second database end, and input the encrypted data to be transmitted into the first queue; The decryption module is used to read the encrypted data to be transmitted from the first queue through the Canal adapter, and decrypt the encrypted data to be transmitted using the private key of the public-private key pair to obtain the data to be transmitted; the data transmission system based on the Canal framework is applicable to database communication application scenarios including real-time database backup, database mirroring, and index updates. The key acquisition module is used to generate a one-time public-private key pair for each data transmission task; The encryption module is further configured to generate a binary file from the data to be mirrored, use the binary file as the data to be transmitted, encrypt the binary file using the public key, and input the encrypted binary file into the second queue through the canal server; The data sending module is also used to send the encrypted binary file in the second queue to the first database terminal; The data receiving module is used to receive the encrypted binary file from the second database and input the encrypted binary file into the first queue; The decryption module is used to read the encrypted binary file from the first queue through the canal adapter, and decrypt the encrypted binary file using the private key of the public-private key pair to obtain the data to be mirrored; The encryption module is further configured to generate an operation log from the update data of the index used for searching, use the operation log as the data to be transmitted, encrypt the operation log using the public key, and input the encrypted operation log into the second queue through the canal server; The data sending module is also used to send the encrypted operation log in the second queue to the first database terminal; The data receiving module is also used to receive the encrypted operation log from the second database end, store the encrypted operation log as a relay log, and input the relay log into the first queue; The decryption module is further configured to read the relay log from the first queue through the canal adapter, and decrypt the relay log using the private key of the public-private key pair to obtain the updated data.

2. The data transmission system based on the Canal framework as described in claim 1, characterized in that, The encryption module is further configured to generate an operation log from the changed data, use the operation log as the data to be transmitted, encrypt the operation log using the public key, and input the encrypted operation log into the second queue through the canal server; The data sending module is also used to send the encrypted operation log in the second queue to the first database terminal; The data receiving module is also used to receive the encrypted operation log from the second database end, store the encrypted operation log as a relay log, and input the relay log into the first queue; The decryption module is further configured to read the relay log from the first queue through the canal adapter, and decrypt the relay log using the private key of the public-private key pair to obtain the modified data.

3. The data transmission system based on the Canal framework as described in any one of claims 1 to 2, characterized in that, The encryption module is also used to input the encrypted data to be transmitted into the second Kafka message queue through the Canal server; The data sending module is also used to send the encrypted data to be transmitted in the second Kafka message queue to the first database terminal; The data receiving module is also used to receive the encrypted data to be transmitted from the second database end, and input the encrypted data to be transmitted into the first Kafka message queue; The decryption module is further configured to read the encrypted data to be transmitted from the first Kafka message queue through the Canal adapter, and decrypt the encrypted data to be transmitted using the private key of the public-private key pair to obtain the data to be transmitted.

4. A data transmission method based on the Canal framework, the method running on the destination database, characterized in that, include: Obtain a public-private key pair and send the public key of the public-private key pair to the source database, wherein the source database is used to send the data to be transmitted, and the public key is used to encrypt the data to be transmitted; Receive the encrypted data to be transmitted from the source database and input the encrypted data to be transmitted into the first queue; The encrypted data to be transmitted is read from the first queue through the Canal adapter, and the encrypted data to be transmitted is decrypted using the private key of the public-private key pair to obtain the data to be transmitted. The data transmission method based on the Canal framework is applicable to database communication scenarios including real-time database backup, database mirroring, and index updates. Obtain the public-private key pair, including: For each data transmission task, a one-time public-private key pair is generated; Receiving encrypted data to be transmitted from the source database and inputting the encrypted data to be transmitted into a first queue includes: Receive an encrypted binary file from the source database and input the encrypted binary file into the first queue, wherein the binary file is generated from the data to be mirrored from the source database. The encrypted data to be transmitted is read from the first queue using the Canal adapter, and the encrypted data to be transmitted is decrypted using the private key of the public-private key pair to obtain the data to be transmitted, including: The encrypted binary file is read from the first queue using the canal adapter, and the encrypted binary file is decrypted using the private key of the public-private key pair to obtain the data to be mirrored. Receiving encrypted data to be transmitted from the source database and inputting the encrypted data to be transmitted into a first queue includes: Receive encrypted operation logs from the source database, store the encrypted operation logs as relay logs, and input the relay logs into the first queue, wherein the operation logs are generated from the update data of the search index on the source database. The encrypted data to be transmitted is read from the first queue using the Canal adapter, and the encrypted data to be transmitted is decrypted using the private key of the public-private key pair to obtain the data to be transmitted, including: The relay log is read from the first queue using the canal adapter, and the updated data is obtained by decrypting the relay log using the private key of the public-private key pair.

5. A data transmission method based on the Canal framework, the method running on the source database side, characterized in that, include: Receive a public key from the destination database, wherein the destination database is used to receive the data to be transmitted, and the public key is the public key in a set of one-time public-private key pairs generated by the destination database for each data transmission task. The data to be transmitted is encrypted using the public key, and the encrypted data is then input into the second queue via the Canal server. The encrypted data to be transmitted in the second queue is sent to the destination database. The data transmission method based on the Canal framework is applicable to inter-database communication scenarios including real-time database backup, database mirroring, and index updates. The data to be transmitted is encrypted using the public key, and the encrypted data is then input into the second queue via the Canal server, including: The data to be mirrored is generated into a binary file, the binary file is used as the data to be transmitted, and the binary file is encrypted using the public key. The encrypted binary file is then input into the second queue through the Canal server. Sending the encrypted data to be transmitted from the second queue to the destination database includes: Send the encrypted binary file in the second queue to the destination database. The data to be transmitted is encrypted using the public key, and the encrypted data is then input into the second queue via the Canal server, including: An operation log is generated from the update data of the index used for searching. The operation log is used as the data to be transmitted. The operation log is encrypted using the public key. The encrypted operation log is then input into the second queue through the canal server. Sending the encrypted data to be transmitted from the second queue to the destination database includes: The encrypted operation log in the second queue is sent to the destination database.

6. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data transmission method based on the Canal framework as described in claim 4 or 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the data transmission method based on the Canal framework as described in claim 4 or 5.

Images