Synchronous transmission method and device for changed data, equipment and storage medium

Abstract

The invention discloses a synchronous transmission method and device for changed data, equipment and a storage medium. When the method provided by the embodiment of the invention is executed, firstly, an incremental log of a source database can be monitored and read through a database incremental log analysis tool, incremental data is generated based on the incremental log, and serialization processing and encryption processing are performed on the incremental data to obtain a to-be-transmitted log. And generating an MD5 verification information file based on the to-be-transmitted log, and binding a user certificate for the to-be-transmitted log. Acquiring user identity information uploaded by the client through the server port, and performing identity verification on the user identity information by using the user certificate to obtain a first verification result; and when the first verification result indicates that the identity verification is passed, sending the MD5 verification information file and the to-be-transmitted log to the client through the server port by adopting HTTPS, so that the client updates the target database based on the information. According to the invention, the security risk of data change is effectively reduced, and the synchronization real-time performance is improved.

Description

Technical Field

[0001] This application relates to the field of database synchronization technology, specifically to a method, apparatus, device, and storage medium for the synchronous transmission of changed data. Background Technology

[0002] In scenarios such as financial data sharing, government information exchange, and cross-regional business collaboration among enterprises, data service providers often need to synchronize changed data in the source database (such as transaction record updates, user information modifications, business form adjustments, etc.) to downstream systems (such as subsidiary databases, partner data platforms, data analysis terminals, etc.) to ensure that downstream businesses can operate based on the latest data.

[0003] Database synchronization, a common method for providing changed data, transmits modified data from the source to the target. However, traditional database synchronization technologies (such as Kettle) typically require both the source and target ends to have open database ports and communicate with each other over the public internet, thus transmitting data over the public network. This approach poses significant security risks. When data is transmitted over the public network, the open database ports are vulnerable to malicious scanning and attacks, leading to the theft and tampering of data and increasing the risk of core business data leakage.

[0004] In addition, traditional database synchronization technology relies on a scheduler to trigger synchronization tasks. Even if the scheduling period is set to the fastest minute level, it is impossible to avoid the time difference between the data change and the synchronization completion. This ultimately causes delays in the data obtained downstream, making it difficult to meet the business scenarios with high real-time requirements (such as real-time transaction monitoring, dynamic risk control, etc.).

[0005] Therefore, how to achieve high real-time performance and low latency in synchronizing changed data while ensuring data transmission security is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] To address the aforementioned issues, this application provides a method, apparatus, device, and storage medium for synchronizing and transmitting changed data, which can achieve efficient and real-time synchronization of database changed data while ensuring data transmission security.

[0007] The embodiments of this application disclose the following technical solutions:

[0008] A method for synchronously transmitting changed data, applied to a server, the method comprising:

[0009] The incremental log of the source database is monitored and read using a database incremental log parsing tool; the incremental log is used to record the change operations that occur in the source database, including table structure change statements or table data change statements.

[0010] Incremental data is generated based on the incremental log, and the incremental data is serialized and encrypted to obtain the log to be transmitted; the incremental data includes table structure information, data change status, and changed data;

[0011] An MD5 checksum file is generated based on the log to be transmitted, and user credentials are bound to the log to be transmitted.

[0012] Obtain the user identity information uploaded by the client through the server port opened by the server, and use the user credentials to verify the user identity information to obtain a first verification result;

[0013] When the first verification result indicates that the authentication is successful, the MD5 checksum file and the log to be transmitted are sent to the client via the server port using the Hypertext Transfer Protocol (HTTPS), so that the client can update the target database based on the MD5 checksum file and the log to be transmitted.

[0014] In one possible implementation, one incremental log entry records one change operation that occurs in the source database; one log entry to be transmitted corresponds to one change operation.

[0015] The step of sending the MD5 checksum file and the log to be transmitted to the client via the server port using HTTPS includes:

[0016] The number of logs to be transmitted is counted. When the number of logs to be transmitted equals the transmission threshold, all logs to be transmitted are packaged into a packaged log, and the MD5 checksum file and the packaged log are sent to the client via HTTPS; or...

[0017] Calculate the waiting time for the first log to be transmitted. When the waiting time is greater than or equal to the transmission time threshold, package all the logs to be transmitted into a packaged log and send the MD5 checksum file and the packaged log to the client via HTTPS.

[0018] In one possible implementation, the step of sending the MD5 checksum file and the log to be transmitted to the client via the server port using HTTPS further includes:

[0019] When the number of logs to be transmitted exceeds the transmission threshold, all logs to be transmitted are divided based on the transmission threshold. The portion of logs to be transmitted whose number is equal to the transmission threshold is packaged to obtain packaged logs. The MD5 checksum file and the packaged logs are then sent to the client via HTTPS.

[0020] In one possible implementation, the step of having the client update the target database based on the MD5 checksum file using the packaging log includes:

[0021] The client uses the MD5 checksum file to perform integrity verification on the packaged log to obtain a second verification result.

[0022] When the second verification result indicates that the integrity verification is successful, the client generates a synchronous Structured Query Language (SQL) statement based on the packaging log, and updates the target database using the synchronous SQL statement.

[0023] In one possible implementation, the step of using the MD5 checksum file by the client to perform integrity verification on the packaged log to obtain a second verification result includes:

[0024] The client calculates the MD5 value of each log to be transmitted in the packaged log to obtain the comparison object;

[0025] The client reads the MD5 value from the MD5 checksum file to obtain the comparison target.

[0026] The client compares the verification object and the comparison target to obtain the second verification result.

[0027] When the second verification result indicates that the integrity verification is successful, the client generates a synchronous SQL statement based on the packaging log, and updates the target database using the synchronous SQL statement, including:

[0028] When the second verification result indicates that the integrity verification is successful, the client decodes and deserializes the packaged log to obtain the decrypted log.

[0029] The client converts the decryption log into a structured data format to obtain the synchronous SQL statement, and then updates the target database using the synchronous SQL statement.

[0030] A device for synchronously transmitting changed data, the device comprising:

[0031] The listening and reading unit is used to listen to and read the incremental log of the source database through the database incremental log parsing tool; the incremental log is used to record the change operations that occur in the source database, including table structure change statements or table data change statements;

[0032] A generation and processing unit is used to generate incremental data based on the incremental log, and to perform serialization and encryption processing on the incremental data to obtain the log to be transmitted; the incremental data includes table structure information, data change status, and changed data;

[0033] A binding unit is used to generate an MD5 checksum file based on the log to be transmitted and bind user credentials to the log to be transmitted.

[0034] The verification unit is used to obtain user identity information uploaded by the client through the server port opened by the server, and use the user credentials to verify the user identity information to obtain a first verification result;

[0035] The log sending unit, when the first verification result indicates that the authentication is successful, is used to send the MD5 checksum file and the log to be transmitted to the client via the server port using HTTPS, so that the client can update the target database based on the MD5 checksum file and the log to be transmitted.

[0036] In one possible implementation, one incremental log entry records one change operation that occurs in the source database; one log entry to be transmitted corresponds to one change operation.

[0037] The log sending unit is specifically used for:

[0038] The number of logs to be transmitted is counted. When the number of logs to be transmitted equals the transmission threshold, all logs to be transmitted are packaged into a packaged log, and the MD5 checksum file and the packaged log are sent to the client via HTTPS; or...

[0039] Calculate the waiting time for the first log to be transmitted. When the waiting time is greater than or equal to the transmission time threshold, package all the logs to be transmitted into a packaged log and send the MD5 checksum file and the packaged log to the client via HTTPS.

[0040] In one possible implementation, the log sending unit is further configured to:

[0041] When the number of logs to be transmitted exceeds the transmission threshold, all logs to be transmitted are divided based on the transmission threshold. The portion of logs to be transmitted whose number is equal to the transmission threshold is packaged to obtain packaged logs. The MD5 checksum file and the packaged logs are then sent to the client via HTTPS.

[0042] A device for synchronously transmitting change data, characterized in that it comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the method for synchronously transmitting change data as described above.

[0043] A computer-readable storage medium is characterized in that the computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the synchronous transmission method for changed data as described above.

[0044] Compared with the prior art, this application has the following beneficial effects:

[0045] This application provides a method, apparatus, device, and storage medium for the synchronous transmission of changed data. Specifically, when executing the synchronous transmission method for changed data provided in this application embodiment, firstly, an incremental log recording the source database table structure and data change operations can be monitored and read using a database incremental log parsing tool. This step achieves real-time capture of database changes, improving the timeliness of data synchronization. Secondly, incremental data containing table structure information, data change status, and changed data is generated based on the incremental log, and serialized and encrypted to ensure data format uniformity and security. Then, an MD5 (Message-Digest Algorithm 5) verification information file is generated based on the log to be transmitted, and user credentials are bound to the log to verify data integrity and maintain identity security. Next, the server receives user identity information uploaded by the client through an open server port, performs authentication using user credentials, and effectively prevents unauthorized access. Finally, after successful authentication, the MD5 checksum file and the log to be transmitted are securely sent to the client through the server port using HyperText Transfer Protocol Secure (HTTPS). This allows the client to verify data integrity based on the checksum file and update the target database, thereby improving the security and reliability of data transmission and the accuracy of synchronization.

[0046] This application serializes and encrypts changed data before transmitting it over the network via HTTPS. It exposes only the server port, not the database port, reducing security risks associated with open database ports and mitigating the risk of malicious scanning and attacks. Furthermore, by monitoring the incremental logs of the source database using a database incremental log parsing tool, it achieves real-time capture of data changes, significantly reducing the latency from data change occurrence to detection, improving synchronization real-time performance, and meeting the needs of high real-time business scenarios. Additionally, it employs an authentication mechanism combining user credentials and client identity information, along with MD5 checksum files to ensure the integrity and correctness of transmitted data, further enhancing the overall security of the synchronization process. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A schematic diagram illustrating an exemplary application scenario provided in this application embodiment;

[0049] Figure 2 A flowchart illustrating a method for synchronously transmitting change data, as provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of a data synchronization transmission device provided in an embodiment of this application. Detailed Implementation

[0051] To facilitate understanding of the technical solutions provided in the embodiments of this application, the background technology involved in the embodiments of this application will be described below.

[0052] In scenarios such as cross-regional collaboration in finance, government affairs, and enterprises, data service providers need to synchronize changed data from the source database to downstream systems to ensure that business operations are based on the latest information. Although traditional database synchronization technology is a commonly used method for transmitting changed data, it usually relies on open database ports and transmission over the public network, which poses significant security risks. At the same time, due to scheduling cycle limitations, it is difficult to meet the requirements for high real-time performance, resulting in delays in data synchronization.

[0053] To address this issue, this application provides a method, apparatus, device, and storage medium for the synchronous transmission of changed data. First, a database incremental log parsing tool monitors and reads incremental logs recording changes to the source database table structure or data in real time. Based on these incremental logs, incremental data containing table structure information, data change status, and the changed data is generated and serialized and encrypted to obtain the log to be transmitted. Subsequently, a corresponding MD5 checksum file is generated, and user credentials are bound to the log to be transmitted. The client uploads identity information through an open server port. The server verifies the identity information using the user credentials. After successful verification, the MD5 checksum file and the log to be transmitted are securely transmitted to the client via the HTTPS protocol through the server port. This allows the client to verify data integrity based on the checksum file and update the target database using the log to be transmitted, achieving secure and efficient synchronization of changed data. This application serializes and encrypts the changed data, transmits it using the HTTPS protocol, and only opens the server port, avoiding direct exposure of the database port. This effectively reduces the security risks associated with open ports and decreases the possibility of malicious scanning and attacks. Meanwhile, by leveraging incremental database log parsing tools to monitor changes in the source database in real time, data can be captured rapidly, effectively shortening the time from when a change occurs to when it is detected, improving the real-time performance of synchronization, and meeting the high-real-time business requirements. Furthermore, authentication is performed by combining user credentials with client identity information, and the integrity and accuracy of data transmission are ensured through MD5 verification files, comprehensively enhancing the security protection capabilities of the synchronization process.

[0054] To facilitate understanding of the attack behavior detection method provided in the embodiments of this application, the following is combined with... Figure 1 The example scenario is shown below. See also... Figure 1 This figure is a schematic diagram of an exemplary application scenario provided in the embodiments of this application.

[0055] First, the source database generates new logs (binlog) during operation. The database incremental log parsing tool continuously monitors the changes in the source database's logs through a "listen for new logs" mechanism, and simultaneously performs a "read incremental log" operation to retrieve these new incremental logs from the source database. The incremental logs here can be understood as a collection of basic information recording various database change operations (such as adding, modifying, and deleting data), much like the original "record archive" of database operations.

[0056] Next, the database incremental log parsing tool will perform "serialization and encryption processing" on the acquired incremental logs, converting the originally scattered and diverse incremental logs into a unified, easy-to-transmit, and encrypted "log to be transmitted". This step is to make the logs more stable and secure in subsequent transmission, and to avoid information leakage or transmission errors.

[0057] Then, the server port sends the "MD5 checksum file" and the processed "login to be transmitted" to the client. Before the actual data transmission, an "authentication" process is performed to verify the client's legitimate identity, ensure data transmission security, and prevent unauthorized clients from obtaining sensitive database log information. After receiving the relevant files and logs, the client performs "integrity verification" based on the "MD5 checksum file." A specific algorithm compares the characteristics of the login to be transmitted with the content of the MD5 checksum file to determine whether the login to be transmitted is complete and has not been tampered with during transmission. This results in "login to be transmitted that has passed integrity verification," which can be seen as a judgment of log integrity and includes key information such as whether the log can be used for subsequent update operations.

[0058] Finally, the "target database" is updated using the "log to be transferred" that has passed integrity verification. The database changes recorded in the log are applied to the target database, ensuring that the target database's state is consistent with the changes in the source database caused by the incremental logs. This achieves data synchronization from the source database to the target database. This process ensures the accuracy and security of data from changes in the source database to the final synchronization to the target database. Mechanisms such as MD5 checksums prevent erroneous updates to the target database due to corrupted or tampered logs. Furthermore, the entire data synchronization process is traceable and verifiable, improving its reliability.

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0060] See Figure 2 The figure is a flowchart of a method for synchronously transmitting changed data provided in an embodiment of this application. This method is applied to the server.

[0061] like Figure 2 As shown, the method for synchronously transmitting the changed data may include steps S201-S205:

[0062] S201: Listen to and read the incremental logs of the source database using the database incremental log parsing tool.

[0063] To achieve real-time capture and accurate recording of data changes in the source database, ensuring that downstream systems can promptly obtain the latest database change information, this step continuously monitors and reads the incremental logs of the source database using database incremental log parsing tools (such as Canal and Debezium). The core purpose of this design is to break away from the inefficient mode of relying on periodic queries or full scans in traditional data synchronization, and instead obtain first-hand change information directly from the underlying change records of the source database through "real-time monitoring + log parsing," thereby laying a low-latency and highly accurate foundation for subsequent data synchronization.

[0064] The incremental log, as a change record carrier inherent to the source database (such as MySQL's binlog, PostgreSQL's wal log, etc.), has the core function of completely and orderly recording all change operations that occur in the source database, ensuring that every data change can be traced and restored. These change operations can be divided into two main categories: one is table structure change statements, which modify the metadata of the database table. For example, executing the statement "CREATE TABLE order(id INT PRIMARY KEY, amount DECIMAL)" to create an order table directly changes the data storage structure. The incremental log records key information such as the statement content, execution time, and operator. The other category is table data change statements, which modify the specific records in the table. For example, executing the statement "INSERT INTO product VALUES(1001, 'Laptop', 5999)" to add a product record, executing the statement "UPDATE user SET age =25 WHERE id = 101" to modify a user's age, or executing the statement "DELETE FROM log WHERE create_time <'2024-01-01'" to delete an expired log. This type of operation changes the actual data content in the table, and the incremental log records detailed information such as the changed table name, conditions, and data values ​​before and after the change.

[0065] In this way, the database incremental log parsing tool can accurately capture every "action" of the source database in real time. Whether it's an adjustment to the table structure or the addition, deletion, or modification of data content, it can be read in a timely manner and transformed into basic information that can be processed by subsequent synchronization processes. This effectively avoids the inconsistency problem between upstream and downstream data caused by delayed acquisition of change information in traditional synchronization methods. For example, in the order system of an e-commerce platform, when a user places an order, the order data modification statement "INSERT INTO orders..." is written to the incremental log of the source database in real time. The parsing tool immediately listens for and reads this log, ensuring that the downstream inventory management system can quickly obtain new order information and trigger the inventory deduction process, thereby achieving efficient collaboration in the business chain.

[0066] S202: Generate incremental data based on the incremental log, and perform serialization and encryption processing on the incremental data to obtain the log to be transmitted.

[0067] To ensure the integrity and security of changed data during transmission, this application can generate structured incremental data based on incremental logs. This incremental data not only includes detailed table structure information, such as field names, types, and constraints, but also covers data change status information (e.g., addition, modification, or deletion), as well as the specific content of the changed data. By comprehensively encapsulating this information, every change operation that occurs in the source database can be fully reflected. Subsequently, this incremental data is serialized, converting the complex data structure into a unified and easily transmitted byte stream format, facilitating cross-network transmission and subsequent parsing.

[0068] To prevent data theft or tampering during transmission, this application further employs encryption technology to encrypt the serialized data, such as using the SM2 asymmetric encryption algorithm. SM2, as a domestically developed public-key cryptography algorithm, possesses high security and fast encryption / decryption speed, making it suitable for protecting such sensitive data. For example, when a record in the order table is updated, the update operation and the updated field value are recorded in the incremental data, serialized into a standard format, and then encrypted using the server's SM2 public key to generate a log to be transmitted. Even if this log is intercepted in a network environment, it cannot be illegally deciphered due to encryption, greatly improving the security level of data transmission. Through this series of processes, not only is accurate data transmission ensured, but the risks of data leakage and tampering are also effectively prevented, meeting the stringent requirements of business for data synchronization security.

[0069] S203: Generate an MD5 checksum file based on the log to be transmitted, and bind user credentials to the log to be transmitted.

[0070] To ensure the integrity of transmitted data and prevent data tampering during transmission, this application generates a corresponding MD5 checksum file based on the encrypted log to be transmitted. MD5 (Message Digest Algorithm Version 5) is a widely used hash function that can convert data input of arbitrary length into a fixed-length 128-bit digest value. This digest value is unique and irreversible, effectively reflecting the integrity status of the original data. After the log to be transmitted is generated, the server performs MD5 calculation on its content, generating a unique checksum, and stores this checksum in the checksum file. After receiving the data, the client can also recalculate the MD5 value of the downloaded log file and compare it with the value in the checksum file to verify whether the data has been tampered with or corrupted during transmission.

[0071] In addition, to ensure the legitimacy and security of data access, the server also binds user credentials to the logs to be transmitted. User credentials typically include the user's username and password, serving as crucial evidence for verifying the client's identity. For example, when the server generates the logs to be transmitted and their corresponding MD5 checksum files, it attaches the currently authenticated user credentials for matching and verification when the client submits identity information later. If the client's credentials match the bound information, the request is considered legitimate, and data reception is permitted; otherwise, access is denied to prevent unauthorized users from obtaining sensitive data.

[0072] S204: Obtain the user identity information uploaded by the client through the server port opened by the server, and use the user credentials to verify the user identity information to obtain a first verification result.

[0073] To ensure security during data transmission, this application designs an authentication mechanism based on user credentials. Specifically, the client uploads its user identity information to the server through a server port opened by the server. This server port is typically a web server port that supports secure communication protocols such as HTTPS to prevent eavesdropping or tampering of the identity information during transmission. After receiving the identity information uploaded by the client, the server rigorously verifies the identity information using user credentials pre-bound to the transmission log.

[0074] The authentication process typically involves comparing the user's identity (such as username) uploaded by the client with the information stored in the user credentials (including username and password), and verifying the validity and scope of permissions of the credentials. For example, when a client uploads the username "User123" and the corresponding password, the server first checks if "User123" exists in the server's authorized list, then compares the uploaded password with the password bound to the server, and finally checks if the user credentials are valid and have the permission to access synchronized data. Through this series of verification steps, the server can accurately determine whether the client's request is legitimate, thus obtaining the first verification result, i.e., whether the authentication is successful. Only when authentication is successful will the server allow subsequent operations, such as sending change data logs to the client, to ensure that sensitive data is only provided to legitimate users.

[0075] For example, suppose a client uploads user identity information including the username "UserABC" and the corresponding password. The server uses stored user credentials to verify the username and password. If the password is correct and the user credentials are valid, the server returns "passed" as the first verification result; otherwise, the request is rejected and an authentication failure message is displayed. This mechanism effectively prevents identity forgery attacks and unauthorized access, improving the overall security protection capabilities of the server.

[0076] S205: When the first verification result indicates that the authentication is successful, the MD5 verification information file and the log to be transmitted are sent to the client via the server port using HTTPS, so that the client can update the target database based on the MD5 verification information file and the log to be transmitted.

[0077] When the server verifies the user's identity information uploaded by the client and the first verification result indicates successful authentication, the server confirms that the client has legitimate access rights and allows it to receive modified data. At this point, the server securely sends the log to be transmitted and the corresponding MD5 checksum file to the client via an open server port using the HTTPS protocol. The HTTPS protocol utilizes Transport Layer Security (TLS) or Secure Sockets Layer (SSL) encryption technology to provide confidentiality, integrity, and authentication guarantees for both communicating parties during data transmission, effectively preventing data eavesdropping, tampering, or man-in-the-middle attacks, and ensuring the secure transmission of synchronized data.

[0078] The log to be transmitted contains the latest change information of the source database, while the MD5 checksum file is used by the client to verify the integrity of the received data. After receiving the data, the client first compares the hash value of the log to be transmitted with the MD5 checksum file to verify whether the data has been corrupted or tampered with during transmission, ensuring the accuracy and integrity of the data. Only after successful verification will the client further use the log to be transmitted to update the target database, achieving data consistency and real-time synchronization between the source and target databases.

[0079] Based on the content of S201-S205, the database incremental log parsing tool first listens for and reads the incremental logs recording table structure or table data change operations in the source database, and generates incremental data containing table structure information, data change status, and the changed data. Then, the incremental data is serialized and encrypted to obtain the log to be transmitted, and a corresponding MD5 checksum file is generated and user credentials are bound to the log to be transmitted. The server receives user identity information uploaded by the client through its open server port to complete authentication using user credentials. After successful authentication, the MD5 checksum file and the log to be transmitted are also sent to the client via the same server port using the HTTPS protocol, allowing the client to update the target database based on the checksum file and the log to be transmitted. This application avoids exposing the database port by serializing and encrypting the changed data and using HTTPS transmission with only the server port open, effectively reducing security risks. Simultaneously, incremental logs are used to capture data changes in real time, improving synchronization real-time performance, and authentication and MD5 checksums ensure data integrity and transmission security.

[0080] In one possible implementation, each incremental log entry records a specific change operation in the source database. Correspondingly, each log entry to be transmitted also corresponds to a change operation. By parsing and transforming the incremental logs one by one to generate the logs to be transmitted, fine-grained tracking and synchronization of database changes can be achieved. This one-to-one correspondence ensures the accuracy and integrity of data transmission, enabling the target database to apply changes gradually and sequentially, improving the reliability of synchronization and the ease of maintenance.

[0081] In one possible implementation, the step S205, which involves sending the MD5 verification information file and the log to be transmitted to the client via HTTPS through the server port, is not a single trigger for transmission. Instead, it employs a dynamic triggering mechanism that combines data volume and time dimensions to balance transmission efficiency, data real-time performance, and database access pressure. Specifically, it includes two core triggering logics:

[0082] The first method is "quantity threshold triggering." The server continuously counts the accumulated number of logs to be transmitted. When the count equals a preset transmission threshold (e.g., 100 or 500 logs, which can be adjusted based on the data generation frequency and transmission capacity in the business scenario), all currently accumulated logs to be transmitted are packaged into a single unified packaged log. The key advantage of this packaging design is that when the server needs to provide the same incremental data to multiple clients, it does not need to generate and transmit scattered logs separately for each client. Instead, it can directly reuse the pre-packaged logs, significantly reducing repeated access to the source database and data processing operations. This effectively alleviates the database access pressure and avoids database performance overload caused by concurrent requests from multiple clients. For example, an e-commerce platform's server needs to synchronize product inventory change logs to inventory management clients in three regions. If the transmission threshold is set to 100 logs, when the accumulated logs reach 100, the server packages and generates "Inventory Change_Packaged Log_20240827_100.log". When the three clients request data subsequently, the server can directly reuse this packaged log and transmit it via HTTPS protocol, without having to repeatedly extract and process logs from the database, significantly reducing the database load.

[0083] The second method is "time threshold triggering." The server records the time when the first log to be transmitted is generated and calculates the waiting time from its generation to the present in real time. When the waiting time is greater than or equal to a preset transmission time threshold (such as 1 second or 5 seconds, adapting to business scenarios with high real-time requirements), all generated logs to be transmitted will be packaged into a packaged log, regardless of whether the cumulative number of logs to be transmitted has reached the transmission threshold. This mechanism is mainly to avoid the problem of data remaining on the server for a long time and failing to be synchronized to the client in a timely manner due to the number of logs to be transmitted not reaching the threshold for a long time, thus ensuring the timeliness of data synchronization. For example, in a financial transaction system, the transmission time threshold is set to 1 second. If only one "user transfer successful" log is generated at a certain moment, and no new log is generated within the next second, the waiting time of the first log has reached 1 second. The server will immediately package the log into "transfer record_packaged log_20240827_001.log" and send it to the client via HTTPS protocol to ensure that the downstream risk control system can obtain the transfer data in a timely manner and avoid the impact of data delay on risk control decisions.

[0084] Regardless of the triggering logic used, after generating the packaging log, the server will send the corresponding MD5 checksum file (which records the hash value of the packaging log for the client to verify its integrity) along with the packaging log to the client via an open server port using the HTTPS protocol. This ensures the security of the transmission link while also balancing database performance and data real-time performance through the packaging mechanism.

[0085] In one possible implementation, step S205, which involves sending the MD5 checksum file and the log to be transmitted to the client via HTTPS through the server port, further includes:

[0086] When the number of logs to be transmitted exceeds a preset transmission threshold, the server will initiate a log batch processing mechanism to avoid decreased transmission efficiency or excessive resource consumption caused by a large volume of data transmitted at once. Specifically, the server will evenly divide all logs to be transmitted into units equal to the transmission threshold. For example, if the transmission threshold is 100 logs and there are currently 350 logs to be transmitted, they will be divided into four parts (100 logs each in the first three parts, and 50 logs in the last part). For each part containing logs equal to the transmission threshold (such as the 100 logs in the first three parts), the server will package them separately, generating independent packaged logs. Subsequently, the server will send these batch-generated packaged logs and their corresponding MD5 checksum files sequentially to the client via the server port using the HTTPS protocol.

[0087] The advantages of this batch transmission mechanism are: on the one hand, by controlling the amount of data transmitted in a single batch, the risk of network congestion can be reduced and the transmission success rate can be improved; on the other hand, the client can receive and process data in batches without waiting for all logs to be transmitted before starting the processing flow, thereby reducing the overall synchronization time.

[0088] In one possible implementation, the process of having the client update the target database based on the MD5 verification information file using the packaged log follows a "verify first, execute later" security logic to ensure the accuracy of the data synchronized to the target database. This specifically includes two core steps:

[0089] First, the client uses the MD5 checksum file to perform integrity verification on the packaging log to obtain a second verification result. Then, when the second verification result indicates that the integrity verification passed, the client generates a synchronization SQL statement based on the packaging log and updates the target database using the synchronization SQL statement.

[0090] Specifically, the server first verifies the integrity of the received packaged logs using the MD5 checksum file from the client. This is achieved by calculating the MD5 hash value of each log entry to be transmitted within the packaged logs and comparing it to the MD5 value in the checksum file, thus obtaining a second verification result. This verification process ensures that the data has not been tampered with or corrupted during transmission, guaranteeing the accuracy of the synchronized data.

[0091] When the second verification result indicates that the integrity verification has passed, the client parses the change information contained in the verified packaged log and generates the corresponding synchronization SQL statements. These SQL statements accurately reflect the change operations in the source database. Subsequently, the client executes the synchronization SQL statements to update the target database, achieving data consistency and real-time synchronization.

[0092] This solution effectively ensures the security and accuracy of target database updates and improves the reliability of system data synchronization through integrity verification and incremental data-based synchronous execution.

[0093] In one possible implementation, the process of obtaining a second verification result by having the client perform integrity verification on the packaged logs using the MD5 checksum file employs a fine-grained verification logic of "log-by-log verification + overall matching" to ensure the integrity of each log to be transmitted. The specific steps are as follows:

[0094] First, the client calculates the MD5 value of each log entry to be transmitted in the packaged log to obtain the comparison objects. The client unpacks the received packaged log, extracts all log entries to be transmitted, and then calculates the hash value of each log entry using the MD5 hash algorithm, resulting in a set of independent MD5 values ​​(i.e., comparison objects). The purpose of this step is to break down the packaged log into the smallest verification unit, ensuring that even if a log entry in the packaged log is abnormal, it can be accurately identified. For example, if the packaged log contains three log entries to be transmitted (recording information changes for users A, B, and C respectively), the client will calculate the MD5 value of each of these three log entries, obtaining a set of comparison objects such as "a1b2c3...", "d4e5f6...", and "g7h8i9...".

[0095] Next, the client reads the MD5 values ​​from the MD5 checksum file to obtain the comparison targets. The client reads the MD5 checksum file sent by the server, which stores the MD5 values ​​(i.e., comparison targets) pre-calculated by the server when packaging the logs, each corresponding to a log entry to be transmitted in the packaged log. These comparison targets have a strict sequential correspondence with the log entries to be transmitted in the packaged log. For example, the comparison targets corresponding to the above three log entries might be "a1b2c3...", "d4e5f6...", and "g7h8i9...".

[0096] Finally, the client compares the verification object and the comparison target to obtain the second verification result: the client compares the comparison object and the comparison target one by one in sequence. If the MD5 value of all logs to be transmitted (the comparison object) is completely consistent with the corresponding value in the MD5 verification information file (the comparison target), then the second verification result of "integrity verification passed" is generated, indicating that each log to be transmitted in the packaged log has not been tampered with or damaged; if the MD5 value of any log to be transmitted does not match the comparison target, then the second verification result of "integrity verification failed" is generated. The client will determine that the packaged log is abnormal and refuse to use it for updating the target database. If necessary, it will request the server to retransmit the corresponding batch of logs.

[0097] Compared to performing a single MD5 check on the entire packaged log, this log-by-log verification method can more accurately locate abnormal logs, ensuring the overall integrity of the data and providing a clear basis for subsequent error investigation and retransmission. It is especially suitable for packaged transmission scenarios containing multiple important business logs (such as batch order data synchronization).

[0098] In one possible implementation, the process of generating a synchronization SQL statement based on the packaging log by the client when the second verification result indicates that the integrity verification has passed, and updating the target database by the synchronization SQL statement, ensures that changes to the source database can be accurately synchronized to the target database. The specific steps are as follows:

[0099] First, when the second verification result indicates that the integrity verification has passed, the client decodes and deserializes the packaged log to obtain a decrypted log. The client decodes and deserializes the packaged log that has passed integrity verification. Decoding involves using a decryption method corresponding to the server-side encryption algorithm (such as decrypting with the private key of SM2 asymmetric encryption) to restore the encrypted packaged log to plaintext. Deserialization converts the plaintext serialized data into a memory object or data structure that the client can recognize, ultimately obtaining a decrypted log containing the original change information. For example, a user information change log encrypted and serialized by the server, after decryption and deserialization, will be restored to a decrypted log containing key information such as "Table name: user, Operation type: update, Changed field: age, Old value: 25, New value: 26, Condition: id=1001".

[0100] Subsequently, the client converts the decryption log into a structured data format to obtain the synchronous SQL statement, and updates the target database using the synchronous SQL statement: the client converts the decryption log into a structured data format (i.e., synchronous SQL statement) that conforms to the syntax specifications of the target database. This conversion process automatically matches the corresponding SQL operation type based on the core information in the decryption log (table structure information, data change status, changed data, etc.): if the change status is "add", an INSERT statement is generated (e.g., "INSERT INTO user (id, name, age) VALUES(1002, 'Zhang San', 30)"); if it is "modify", an UPDATE statement is generated (e.g., "UPDATE user SET age =26 WHERE id = 1001"); if it is "delete", a DELETE statement is generated (e.g., "DELETE FROM user WHERE id= 1003"); if it involves table structure changes (e.g., adding a field), an ALTER statement is generated (e.g., "ALTER TABLE user ADDCOLUMN email VARCHAR (50)").

[0101] See Figure 3 , Figure 3 This is a schematic diagram of a data synchronization transmission device provided in an embodiment of this application. Figure 3 As shown, the synchronous transmission device for the changed data includes:

[0102] The listening and reading unit 301 is used to listen to and read the incremental log of the source database through the database incremental log parsing tool; the incremental log is used to record the change operations that occur in the source database, and the change operations include table structure change statements or table data change statements;

[0103] The generation and processing unit 302 is used to generate incremental data based on the incremental log, and to perform serialization and encryption processing on the incremental data to obtain the log to be transmitted; the incremental data includes table structure information, data change status, and changed data;

[0104] A binding unit 303 is used to generate an MD5 verification information file based on the log to be transmitted and bind user credentials to the log to be transmitted.

[0105] Verification unit 304 is used to obtain user identity information uploaded by the client through the server port opened by the server, and use the user credentials to verify the user identity information to obtain a first verification result;

[0106] The log sending unit 305, when the first verification result indicates that the authentication is successful, is used to send the MD5 verification information file and the log to be transmitted to the client via the server port using HTTPS, so that the client can update the target database based on the MD5 verification information file and the log to be transmitted.

[0107] In one possible implementation, one incremental log entry records one change operation that occurred in the source database; one log entry to be transmitted corresponds to one change operation.

[0108] In one possible implementation, the log sending unit 305 is specifically used for:

[0109] The number of logs to be transmitted is counted. When the number of logs to be transmitted equals the transmission threshold, all logs to be transmitted are packaged into a packaged log, and the MD5 checksum file and the packaged log are sent to the client via HTTPS; or...

[0110] Calculate the waiting time for the first log to be transmitted. When the waiting time is greater than or equal to the transmission time threshold, package all the logs to be transmitted into a packaged log and send the MD5 checksum file and the packaged log to the client via HTTPS.

[0111] In one possible implementation, the log sending unit 305 is further configured to:

[0112] When the number of logs to be transmitted exceeds the transmission threshold, all logs to be transmitted are divided based on the transmission threshold. The portion of logs to be transmitted whose number is equal to the transmission threshold is packaged to obtain packaged logs. The MD5 checksum file and the packaged logs are then sent to the client via HTTPS.

[0113] In one possible implementation, the step of having the client update the target database based on the MD5 checksum file using the packaging log includes:

[0114] The client uses the MD5 checksum file to perform integrity verification on the packaged log to obtain a second verification result.

[0115] When the second verification result indicates that the integrity verification is successful, the client generates a synchronous Structured Query Language (SQL) statement based on the packaging log, and updates the target database using the synchronous SQL statement.

[0116] In one possible implementation, the step of using the MD5 checksum file by the client to perform integrity verification on the packaged log to obtain a second verification result includes:

[0117] The client calculates the MD5 value of each log to be transmitted in the packaged log to obtain the comparison object;

[0118] The client reads the MD5 value from the MD5 checksum file to obtain the comparison target.

[0119] The client compares the verification object and the comparison target to obtain the second verification result.

[0120] In one possible implementation, when the second verification result indicates that the integrity verification has passed, the step of generating a synchronous SQL statement based on the packaging log by the client and updating the target database by the synchronous SQL statement includes:

[0121] When the second verification result indicates that the integrity verification is successful, the client decodes and deserializes the packaged log to obtain the decrypted log.

[0122] The client converts the decryption log into a structured data format to obtain the synchronous SQL statement, and then updates the target database using the synchronous SQL statement.

[0123] In addition, this application embodiment also provides a device for synchronous transmission of change data, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for synchronous transmission of change data as described above.

[0124] In addition, this application embodiment also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform the synchronous transmission method for changed data as described above.

[0125] This application achieves dual optimization of security and real-time performance during the synchronous transmission of changed data through multiple technical designs: Firstly, the changed data is first serialized and formatted, then encrypted to protect the data content, and finally transmitted over the network using the HTTPS protocol. Throughout the process, only the server port is exposed, abandoning the traditional approach of directly opening the database port. This cuts off the main path for malicious scanning and database attacks from the port exposure level, significantly reducing security risks such as data leakage and tampering caused by open database ports. Secondly, by using a database incremental log parsing tool to monitor the incremental logs of the source database in real time, changes in data can be captured instantly without relying on periodic triggering by the scheduler. This greatly shortens the time interval between data changes and their detection, effectively reducing synchronization latency and enabling near real-time data synchronization. This fully meets the stringent requirements for data timeliness in business scenarios such as real-time transaction monitoring and dynamic risk control. Simultaneously, by matching and verifying user credentials with the identity information uploaded by the client, it ensures that only authorized clients can access the data. Furthermore, the integrity of the transmitted data is verified using an MD5 checksum file, further strengthening security protection from both identity access and data integrity perspectives, and building a more comprehensive synchronous security system.

[0126] The foregoing has provided a detailed description of a method, apparatus, device, and storage medium for synchronous transmission of changed data provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0127] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0128] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for synchronously transmitting changed data, characterized in that, Applied to the server side, the method includes: The incremental log of the source database is monitored and read using a database incremental log parsing tool; the incremental log is used to record the change operations that occur in the source database, including table structure change statements or table data change statements. Incremental data is generated based on the incremental log, and the incremental data is serialized and encrypted to obtain the log to be transmitted; the incremental data includes table structure information, data change status, and changed data; An MD5 checksum file is generated based on the log to be transmitted, and user credentials are bound to the log to be transmitted. Obtain the user identity information uploaded by the client through the server port opened by the server, and use the user credentials to verify the user identity information to obtain a first verification result; When the first verification result indicates that the authentication is successful, the MD5 checksum file and the log to be transmitted are sent to the client via the server port using the Hypertext Transfer Protocol (HTTPS), so that the client can update the target database based on the MD5 checksum file and the log to be transmitted.

2. The method according to claim 1, characterized in that, One incremental log entry records one change operation that occurred in the source database; one log entry to be transmitted corresponds to one change operation. The step of sending the MD5 checksum file and the log to be transmitted to the client via the server port using HTTPS includes: The number of logs to be transmitted is counted. When the number of logs to be transmitted equals the transmission threshold, all logs to be transmitted are packaged into a packaged log. The MD5 checksum file and the packaged log are then sent to the client via HTTPS. or, Calculate the waiting time for the first log to be transmitted. When the waiting time is greater than or equal to the transmission time threshold, package all the logs to be transmitted into a packaged log and send the MD5 checksum file and the packaged log to the client via HTTPS.

3. The method according to claim 2, characterized in that, The step of sending the MD5 checksum file and the log to be transmitted to the client via the server port using HTTPS also includes: When the number of logs to be transmitted exceeds the transmission threshold, all logs to be transmitted are divided based on the transmission threshold. The portion of logs to be transmitted whose number is equal to the transmission threshold is packaged to obtain packaged logs. The MD5 checksum file and the packaged logs are then sent to the client via HTTPS.

4. The method according to claim 1, characterized in that, The step of enabling the client to update the target database based on the MD5 checksum file using the packaging log includes: The client uses the MD5 checksum file to perform integrity verification on the packaged log to obtain a second verification result. When the second verification result indicates that the integrity verification is successful, the client generates a synchronous Structured Query Language (SQL) statement based on the packaging log, and updates the target database using the synchronous SQL statement.

5. The method according to claim 4, characterized in that, The step of using the MD5 checksum file on the client to perform integrity verification on the packaged log to obtain a second verification result includes: The client calculates the MD5 value of each log to be transmitted in the packaged log to obtain the comparison object; The client reads the MD5 value from the MD5 checksum file to obtain the comparison target. The client compares the verification object and the comparison target to obtain the second verification result. When the second verification result indicates that the integrity verification is successful, the client generates a synchronous SQL statement based on the packaging log, and updates the target database using the synchronous SQL statement, including: When the second verification result indicates that the integrity verification is successful, the client decodes and deserializes the packaged log to obtain the decrypted log. The client converts the decryption log into a structured data format to obtain the synchronous SQL statement, and then updates the target database using the synchronous SQL statement.

6. A device for synchronously transmitting changed data, characterized in that, The device includes: The listening and reading unit is used to listen to and read the incremental log of the source database through the database incremental log parsing tool; the incremental log is used to record the change operations that occur in the source database, including table structure change statements or table data change statements; A generation and processing unit is used to generate incremental data based on the incremental log, and to perform serialization and encryption processing on the incremental data to obtain the log to be transmitted; the incremental data includes table structure information, data change status, and changed data; A binding unit is used to generate an MD5 checksum file based on the log to be transmitted and bind user credentials to the log to be transmitted. The verification unit is used to obtain user identity information uploaded by the client through the server port opened by the server, and to verify the user identity information using the user credentials to obtain a first verification result; The log sending unit, when the first verification result indicates that the authentication is successful, is used to send the MD5 checksum file and the log to be transmitted to the client via the server port using HTTPS, so that the client can update the target database based on the MD5 checksum file and the log to be transmitted.

7. The apparatus according to claim 6, characterized in that, One incremental log entry records one change operation that occurred in the source database; one log entry to be transmitted corresponds to one change operation. The log sending unit is specifically used for: The number of logs to be transmitted is counted. When the number of logs to be transmitted equals the transmission threshold, all logs to be transmitted are packaged into a packaged log. The MD5 checksum file and the packaged log are then sent to the client via HTTPS. or, Calculate the waiting time for the first log to be transmitted. When the waiting time is greater than or equal to the transmission time threshold, package all the logs to be transmitted into a packaged log and send the MD5 checksum file and the packaged log to the client via HTTPS.

8. The apparatus according to claim 7, characterized in that, The log sending unit is also used for: When the number of logs to be transmitted exceeds the transmission threshold, all logs to be transmitted are divided based on the transmission threshold. The portion of logs to be transmitted whose number is equal to the transmission threshold is packaged to obtain packaged logs. The MD5 checksum file and the packaged logs are then sent to the client via HTTPS.

9. A device for synchronously transmitting changed data, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the synchronous transmission method for changed data as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the synchronous transmission method for changed data as described in any one of claims 1-5.

Images