Database incremental synchronization method, electronic equipment, readable medium and program product

Abstract

The invention provides a database incremental synchronization method. The method comprises the following steps: selecting a to-be-synchronized source table; creating a temporary record table and a trigger for monitoring data change in a source end server for the selected source table; scanning the temporary record table by starting a timed scanning task, and detecting whether an increment change record needing to be processed exists or not; when it is detected that an incremental change record exists in the temporary record table, generating a user-defined synchronization protocol based on preset information; sending the user-defined synchronization protocol to a receiving end; and according to an analysis result of the user-defined synchronization protocol by the receiving end, determining transmission or termination of synchronization data. The disclosure also provides an electronic device, a computer readable medium, and a computer program product. According to the method disclosed by the invention, through flexible design of the trigger, the temporary record table and the user-defined synchronization protocol, efficient capture, secure transmission and accurate synchronization of source table data increment change are realized.

Description

Technical Field

[0001] The present disclosure relates to the field of database technology, and in particular to a database incremental synchronization method, an electronic device, a computer-readable medium, and a computer program product. Background Art

[0002] In the context of modern informatization, data has become the core asset of enterprise operations and decision-making. As the scale of enterprise business expands, the amount of data grows exponentially. In this case, data synchronization efficiency and security have become key issues that enterprises must face, especially in distributed systems and multi-data source environments. Efficiently and securely implementing incremental synchronization between databases has become a technical challenge.

[0003] Traditional data synchronization usually uses full synchronization, that is, synchronizing the entire database data each time. However, full synchronization will have an adverse impact on the normal operation of the business system, and data integrity cannot be guaranteed. Summary of the invention

[0004] The embodiments of the present disclosure provide a database incremental synchronization method, an electronic device, a computer-readable medium, and a computer program product.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for incremental synchronization of a database, comprising: selecting a source table to be synchronized; creating a temporary record table and a trigger for monitoring data changes in a source server for the selected source table; scanning the temporary record table to detect whether there are incremental change records that need to be processed; when an incremental change record is detected in the temporary record table, generating a custom synchronization protocol based on preset information; sending the custom synchronization protocol to a receiving end; and determining the sending or termination of synchronization data based on the parsing result of the custom synchronization protocol by the receiving end.

[0006] In some embodiments, a temporary record table and a trigger for monitoring data changes are created for the selected source table, including: creating a temporary record table for the selected source table to record incremental change information related to the source table; creating a trigger for monitoring data changes of the source table in the source server based on the primary key and field information of the selected source table, wherein the data changes include adding operations, modifying operations and deleting operations; when the data in the selected source table changes, the source table primary key information, change action and change time are written into the temporary record table through the trigger to form an incremental change record.

[0007] In some embodiments, the temporary record table is scanned to detect whether there are incremental change records that need to be processed, including: scanning the temporary record table by starting a scheduled scanning task; detecting whether there are incremental change records that need to be processed in the temporary record table; when there are incremental change records that need to be processed in the temporary record table, based on the source table primary key information stored in the temporary record table, extracting corresponding change data from the source table as synchronization data that needs to be synchronized to the receiving end.

[0008] In some embodiments, a custom synchronization protocol is generated based on preset information, including: setting preset information according to actual needs; generating a custom synchronization protocol based on the preset information; wherein the preset information includes the source table name, the source database name, the sum of the number of data in the scanned temporary record table, the current system time of the source database, the encoding format of the source data, and the custom version number, to generate a custom synchronization protocol.

[0009] In some embodiments, the custom synchronization protocol further includes an encryption algorithm field for indicating an encryption method of the synchronization data.

[0010] In some embodiments, the sending or termination of synchronization data is determined based on the parsing result of the custom synchronization protocol by the receiving end, including: receiving the parsing result of the custom synchronization protocol from the receiving end; when the parsing result indicates that the custom synchronization protocol cannot be parsed correctly, disconnecting the connection with the receiving end to terminate the synchronization operation; when the parsing result indicates that the custom synchronization protocol is parsed correctly, sending the synchronization data.

[0011] In some embodiments, when the parsing result indicates that the custom synchronization protocol is correctly parsed, the synchronization data is sent, including: when the parsing result indicates that the custom synchronization protocol is correctly parsed, encapsulating the synchronization data into a byte array; encrypting the byte array using an encryption algorithm; and sending the encrypted byte array to the receiving end.

[0012] In some embodiments, before sending the custom synchronization protocol to the receiving end, it includes: encapsulating the synchronization data into a byte array; and encrypting the byte array using an encryption algorithm.

[0013] In some embodiments, the sending or termination of synchronization data is determined based on the parsing result of the custom synchronization protocol by the receiving end, including: receiving the parsing result of the custom synchronization protocol from the receiving end; when the parsing result shows that the custom synchronization protocol cannot be parsed correctly, disconnecting the connection with the receiving end to terminate the synchronization operation; when the parsing result shows that the custom synchronization protocol is parsed correctly, sending the encrypted byte array to the receiving end.

[0014] In some embodiments, the database incremental synchronization method further includes: the receiving end decrypts the received encrypted byte array according to the encryption algorithm specified in the custom synchronization protocol; stores the decrypted data to the receiving end server; and after the data storage is completed, sends a storage success result to the source end.

[0015] In some embodiments, the database incremental synchronization method further includes: after the source end receives the storage success result, deleting the synchronized data in the temporary record table.

[0016] In a second aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a memory on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the database incremental synchronization method described in the first aspect of the embodiment of the present disclosure.

[0017] In a third aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, the database incremental synchronization method described in the first aspect of the embodiment of the present disclosure is implemented.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the database incremental synchronization method described in the first aspect of the embodiment of the present disclosure.

[0019] The present invention ensures the security and integrity of the data synchronization process by combining a custom protocol with encrypted transmission; by associating triggers and temporary record tables, it is easy to maintain and can achieve incremental synchronization of the database without any complex operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a database incremental synchronization method according to an embodiment of the present disclosure;

[0021] Figure 2 is an overall schematic timing diagram of a database incremental synchronization method according to an embodiment of the present disclosure;

[0022] Figure 3 A schematic flow chart of source data in a database incremental synchronization method according to an embodiment of the present disclosure is shown;

[0023] Figure 4 A schematic flow chart showing the receiving end data of the database incremental synchronization method according to an embodiment of the present disclosure;

[0024] Figure 5 is a schematic diagram of the composition of an electronic device according to an embodiment of the present disclosure;

[0025] Figure 6 It is a block diagram of a database incremental synchronization system according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the technical solution of the present disclosure is described in detail below with reference to the accompanying drawings.

[0027] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.

[0028] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0029] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0030] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a", "an" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded.

[0031] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0032] In this disclosure, unless otherwise specified, the following technical terms should be understood as follows:

[0033] A source table refers to a table in a source database that stores actual business data. It is the data source for the data synchronization or migration process. In data synchronization technology, a source table is used to provide data that needs to be copied, transferred, or synchronized to a target database.

[0034] The receiving table (sometimes called the target table or destination table) is a table in the receiving database that is responsible for receiving data synchronized from the source. In data synchronization, the receiving table is used to receive and store incremental data to ensure consistency with the source table data.

[0035] Incremental Synchronization is a data synchronization method that only transmits the data that has changed in the source table (such as new, modified, and deleted records).

[0036] Full Synchronization is a data synchronization method that transfers all data in the source table each time, regardless of whether the data has changed.

[0037] Join Query is a database query operation used to connect two or more tables based on their associated fields (such as primary keys or foreign keys) in order to obtain related data from these tables.

[0038] A trigger is an automatically executed program defined in the database that can trigger specific actions when table data changes (such as inserting, updating, or deleting).

[0039] AES encryption (Advanced Encryption Standard) is a symmetric encryption algorithm used to encrypt sensitive data.

[0040] Data persistence (Data Write) is the process of writing data from memory or temporary storage to persistent storage (such as a database or file system).

[0041] There are multiple problems with full synchronization: full synchronization requires a large amount of network bandwidth and storage resources; the synchronization process takes too long, affecting the real-time and normal operation of the business system; when processing large-scale data, full synchronization can easily lead to data inconsistency and affect data integrity.

[0042] To address the above problems, the present disclosure proposes an efficient and secure incremental synchronization method. Incremental synchronization only transmits the data that has changed in the database, which reduces resource and time consumption while improving data consistency and integrity.

[0043] The present disclosure proposes a database incremental synchronization method based on a temporary record table and a custom protocol, aiming to solve the synchronization efficiency and security issues in a distributed environment, optimize resource utilization of data synchronization and reduce system burden.

[0044] The present disclosure aims to overcome the deficiencies of the prior art in incremental database synchronization, and addresses the following technical issues: The present disclosure focuses on achieving efficient incremental synchronization of databases, especially solving the incremental synchronization problem of all databases that support triggers, so as to solve the efficiency problem of incremental data synchronization in a large data volume environment, and by using a trigger mechanism to capture and record data changes, improve the real-time nature and resource utilization of data synchronization, and avoid the high bandwidth and storage consumption of full synchronization; The present disclosure also pays attention to the security issues of database data synchronization.

[0045] Figure 1 It is a flowchart of a database incremental synchronization method according to an embodiment of the present disclosure.

[0046] First, refer to Figure 1 The present disclosure provides a method for incremental synchronization of a database, including:

[0047] S11. Select the source table to be synchronized;

[0048] S12. Create a temporary record table and a trigger for monitoring data changes for the selected source table in the source server;

[0049] S13, scanning the temporary record table to detect whether there is an incremental change record that needs to be processed;

[0050] S14, when it is detected that there is an incremental change record in the temporary record table, generating a custom synchronization protocol based on preset information;

[0051] S15, sending the custom synchronization protocol to a receiving end;

[0052] S16: Determine the sending or termination of synchronization data according to the parsing result of the custom synchronization protocol by the receiving end.

[0053] In the disclosed embodiment, the synchronization service in the source end initiates a request to "select a synchronization table (i.e., source table)" to the source end server, and specifies the source table to be synchronized as the object of data synchronization; after confirming the existence of the specified source table, the source end server returns a response of "processing successful" to the synchronization service in the source end; after the response is successful, the synchronization service in the source end prepares to further configure the monitoring mechanism according to the confirmed source table. The synchronization service in the receiving end initiates a request to "select a synchronization table (i.e., receiving table)" to the receiving end server, and specifies the receiving table for receiving synchronization data; after confirming the existence of the receiving table in the receiving end database, the receiving end server returns a response of "selection successful" to the receiving end synchronization service.

[0054] In the embodiment of the present disclosure, the source end is the sending end, the table to be monitored configured by the sending end is called the source table, and the source end server is the database server corresponding to the source table; the receiving end is the end that receives data, the table configured for receiving synchronized data is called the receiving table, and the receiving end server is the database server corresponding to the receiving table.

[0055] In some embodiments, creating a temporary record table and a trigger for monitoring data changes for the selected source table includes:

[0056] Creating a temporary record table for the selected source table to record incremental change information related to the source table;

[0057] According to the primary key and field information of the selected source table, a trigger for monitoring data changes of the source table is created in the source server, wherein the data changes include adding operations, modifying operations and deleting operations;

[0058] When the data in the selected source table is changed, the source table primary key information, change action and change time are written into the temporary record table through the trigger to form an incremental change record.

[0059] In the embodiment of the present disclosure, the synchronization service in the source end initiates a request to "create a temporary record table" to the source end server; the source end server creates a temporary record table based on the request to record data changes related to the source table; after the source end server completes the creation of the temporary record table, it returns a "successful creation" response to the synchronization service of the source end.

[0060] In the embodiment of the present disclosure, the temporary record table is created by the source server at the instruction of the source synchronization service; the temporary record table is closely associated with the trigger and is used to record data changes in the source table.

[0061] In the disclosed embodiment, the source end creates a trigger based on the selected source table using the primary key and field information. The function of the trigger is to monitor the data changes (addition, modification, deletion) of the source table. Three independent triggers can be used to monitor the addition, modification, and deletion operations respectively; or a comprehensive trigger can be used to monitor the three changes. When the data in the source table changes, the trigger writes the key information of the changed data into a temporary record table. After the trigger monitors the data changes in the source table, the source table primary key information, insert action, and insert time are written into the temporary record table.

[0062] In the disclosed embodiment, the temporary record table includes: source table primary key information, according to which the actual changed data is queried from the source table; insert action, indicating the action type of data change (such as adding, modifying, deleting); insert time, indicating the time when the current trigger writes information to the temporary record table, used to record the time point when the change occurs. The temporary record table is session-level, that is, it is automatically destroyed when the current source database session ends to avoid long-term occupation and redundancy of data.

[0063] In some embodiments, scanning the temporary record table to detect whether there is an incremental change record that needs to be processed includes:

[0064] Scan the temporary record table by starting a scheduled scanning task;

[0065] Detect whether there is an incremental change record that needs to be processed in the temporary record table;

[0066] When there are incremental change records that need to be processed in the temporary record table, corresponding change data is extracted from the source table based on the source table primary key information stored in the temporary record table as synchronization data that needs to be synchronized to the receiving end.

[0067] In the disclosed embodiment, the synchronization service at the source end starts a scheduled scanning task (supporting real-time scanning or periodic scanning) to monitor data changes in the temporary record table; if there are records in the temporary record table (that is, the trigger has detected that the data in the source table has changed and written it to the temporary record table), the scanning task will perform a table-join query between the temporary record table and the source table through the primary key field based on the primary key information in the temporary record table. The temporary record table records the "fact" that the source table data has changed, such as the primary key information captured by the trigger, the type of change (new, modified, deleted), the time of the change, etc. Through the table-join query, combined with the complete records in the source table, the specific change content (field data) is extracted to facilitate sending to the receiving end during subsequent synchronization. If there are no records in the temporary record table scanned, it means that there is no data change. At this time, the scanning task will enter a waiting state and wait for the next scan. The server at the receiving end opens the listening port and waits for the source server to establish a connection and send data.

[0068] In some embodiments, generating a custom synchronization protocol based on preset information includes:

[0069] Set preset information according to actual needs;

[0070] Generate a custom synchronization protocol based on preset information;

[0071] The preset information includes the source table name, the source database name, the number and data in the scanned temporary record table, the current system time of the source database, the encoding format of the source data, and the custom version number, and generates a custom synchronization protocol.

[0072] In some embodiments, the length of the custom synchronization protocol can be dynamically defined according to actual needs.

[0073] In some embodiments, the custom synchronization protocol further includes an encryption algorithm field for indicating an encryption method of the synchronization data.

[0074] In the disclosed embodiment, when the scanning task of the source detects that there is data in the temporary record table, the synchronization service of the source will actively establish a connection with the receiving end; after the connection is established, the source constructs a set of custom protocols according to the following information: the name of the source table, the name of the source database, the number and the data in the scanned temporary record table, the current system time of the source database, the encoding format of the source data, the custom version number, etc.; the length of the custom protocol is 64 bytes; if the information contained is less than 64 bytes, it is filled with 0 (to ensure that the protocol meets the minimum frame length requirement of the Ethernet protocol); the protocol can also be extended to include other flexible fields, such as: encryption algorithm negotiation information, client information, etc. This set of protocols is used to verify the legitimacy and consistency of the two connected parties before data transmission. Those skilled in the art should understand that the length and field information of the protocol can be freely defined according to the specific scenario. Any information field that can be used for verification can be included in the protocol, and the specific number of bytes occupied is determined according to the different types of fields. By flexibly designing the content and length of the fields, it can not only meet different application requirements, but also effectively enhance the security and flexibility of communication.

[0075] In the disclosed embodiment, the fields in the custom protocol header do not have to be completely derived from the source table, and can contain any information that is relevant and useful to the synchronization process. For example, "the number of data in the scanned temporary record table" and "custom version number" are auxiliary information of the incremental synchronization process. Although they have no direct correspondence with the fields in the source table, they are feasible for verification and parsing protocols.

[0076] In some embodiments, determining the sending or termination of synchronization data according to the parsing result of the custom synchronization protocol by the receiving end includes:

[0077] Receiving a parsing result of the custom synchronization protocol from the receiving end;

[0078] When the parsing result indicates that the custom synchronization protocol cannot be correctly parsed, disconnecting from the receiving end to terminate the synchronization operation;

[0079] When the parsing result indicates that the custom synchronization protocol is parsed correctly, the synchronization data is sent.

[0080] In some embodiments, when the parsing result indicates that the custom synchronization protocol is correctly parsed, sending the synchronization data includes:

[0081] When the parsing result indicates that the custom synchronization protocol is parsed correctly, encapsulating the synchronization data into a byte array;

[0082] Encrypt the byte array using an encryption algorithm;

[0083] The encrypted byte array is sent to the receiving end.

[0084] In some embodiments, before sending the custom synchronization protocol to the receiving end, the process includes:

[0085] Encapsulating the synchronization data into a byte array;

[0086] Encrypts a byte array using an encryption algorithm.

[0087] In some embodiments, determining the sending or termination of synchronization data according to the parsing result of the custom synchronization protocol by the receiving end includes:

[0088] Receiving a parsing result of the custom synchronization protocol from the receiving end;

[0089] When the parsing result indicates that the custom synchronization protocol cannot be correctly parsed, disconnecting from the receiving end to terminate the synchronization operation;

[0090] When the parsing result indicates that the custom synchronization protocol is parsed correctly, the encrypted byte array is sent to the receiving end.

[0091] In the disclosed embodiment, the fields to be synchronized (i.e., the changed data to be transmitted from the source to the receiving end) are encapsulated into a byte[] byte array for transmission; the synchronization data array is encrypted using a symmetric encryption algorithm (e.g., AES) to ensure the security of data transmission; according to the specific scenario, the communication between the source and the receiving end can support the following processing modes: first send a custom synchronization protocol, then encrypt the data; after establishing a connection, the source first builds and sends a custom synchronization protocol to ensure the uniqueness and security of the connection; after the receiving end parses and verifies the success of the custom synchronization protocol, the source then encapsulates the synchronization data into a byte array, encrypts the byte array using a symmetric encryption algorithm, and sends it to the receiving end through a socket stream. Encrypt the data first, then send the custom synchronization protocol; the source first encapsulates the synchronization data into a byte array, and encrypts it using a symmetric encryption algorithm; then, the source builds a custom synchronization protocol, and sends the protocol and the encrypted synchronization data to the receiving end through the same connection or a separate connection. The receiving end decides whether to process the encrypted synchronization data based on the parsing result of the received custom synchronization protocol.

[0092] Byte[] is a low-level data representation that can encapsulate any type of data (such as integers, strings, floating-point numbers, etc.), and is suitable for different platforms and languages. Byte[] can compactly represent data, reduce metadata and structural overhead, and is suitable for bandwidth-constrained or efficient transmission scenarios; Byte arrays allow fast byte-level operations, can efficiently parse and reorganize data, and are suitable for application scenarios with high performance requirements. Those skilled in the art should understand that other types can also be used as data carriers, such as JSON and other formats can be used for data carriers, but compared to byte[], its data volume is larger and the parsing speed is slower.

[0093] Those skilled in the art should understand that other encryption algorithms may also be selected, and other symmetric encryption algorithms include DES (Data Encryption Standard), 3DES (Triple Data Encryption Standard), etc., which can be flexibly adjusted according to the application scenario.

[0094] In some embodiments, the database incremental synchronization method further comprises:

[0095] The receiving end decrypts the received encrypted byte array according to the encryption algorithm specified in the custom synchronization protocol;

[0096] Store the decrypted data to the receiving server;

[0097] After the data storage is completed, a storage success result is sent to the source end.

[0098] In some embodiments, the database incremental synchronization method further comprises:

[0099] After receiving the storage success result, the source end deletes the synchronized data in the temporary record table.

[0100] In the disclosed embodiment, after receiving the connection, the receiving end server parses the 64-byte custom protocol in sequence. The fields in the custom protocol include the data encryption algorithm type, so that the receiving end can parse the subsequently received data according to the encryption algorithm specified by the protocol. If all the fields in the custom protocol are parsed correctly, the receiving end returns a "successful" result to the source end; after successfully parsing the custom protocol, the source end sends an encrypted synchronous data byte array; after receiving the data, the receiving end decrypts the data according to the encryption algorithm specified in the custom protocol, and stores the decrypted data in the receiving end server (ie "dropped to disk"); if the drop to disk is successful, the receiving end returns a "successful" result to the source end; after receiving the response of successful drop to disk, the source end deletes the corresponding record in the temporary record table to avoid long-term occupation and redundancy of data. If the field parsing in the custom protocol is wrong, the receiving end disconnects and does not receive subsequent data.

[0101] In the disclosed embodiment, after the source and the receiving end establish a socket connection, a 64-byte custom protocol and an encrypted data byte array are sent in sequence. When sending data, the protocol ensures that subsequent content is sent after data transmission is completed through a flush operation. The custom protocol header carries a field for the data encryption algorithm, so that the receiving end can dynamically parse the protocol and decrypt the data according to the encryption method. The step-by-step design of the protocol and data improves the security and flexibility of data transmission. The above steps realize an efficient and secure synchronous data transmission process through flexible protocol design, data encapsulation and encryption strategies, and avoid data redundancy and long-term occupancy problems by deleting temporary records.

[0102] Figure 2 It is an overall schematic timing diagram of a database incremental synchronization method according to an embodiment of the present disclosure.

[0103] like Figure 2 As shown, the source end sends a "select synchronization table" request to the source end server, specifying the source table to be synchronized; after receiving the request, the source end server confirms the source table in the request and returns a "selection successful" response to the source end;

[0104] The source end sends a request to "create a temporary record table" to the source end server. After receiving the request, the source end server creates a temporary record table in the source end database for recording data changes. After the temporary record table is successfully created, the source end server returns a "successful creation" response to the source end.

[0105] The source end sends a "create trigger" request to the source end server. After receiving the request, the source end server creates a corresponding incremental synchronization trigger for the source table. The trigger is used to monitor data changes (addition, modification, deletion) in the source table and write the change information to the temporary record table. After the trigger is successfully created, the source end server returns a "successful creation" response to the source end.

[0106] The source end periodically queries the temporary record table of the source end server to periodically check the data changes in the temporary record table to discover and record the data changes in the source end server; when it detects that there is data in the temporary record table, the source end generates a custom protocol to ensure the uniqueness and security of data transmission; the source end sends the custom protocol to the receiving end; the protocol can be sent separately or in the same connection with subsequent synchronization data;

[0107] After receiving the custom protocol, the receiver parses the fields in the protocol. If the parsing is successful, the receiver returns a "protocol parsing success" response to the source. If the parsing fails, the receiver immediately disconnects and refuses to receive any data.

[0108] After receiving the "protocol parsing success" response from the receiving end, the source end encapsulates the fields to be synchronized into a byte[] byte array; encrypts the byte array using a symmetric encryption algorithm (such as AES); and sends the encrypted synchronization data to the receiving end through a socket stream;

[0109] After receiving the encrypted data, the receiver decrypts the data according to the encryption algorithm specified in the custom protocol. After the decryption is successful, the receiver stores the data in the receiver database (i.e. "download to disk"). If the data is successfully stored, the receiver returns a response of "data successfully downloaded to disk" to the source.

[0110] After receiving the "data successfully stored on disk" response from the receiving end, the source end deletes the synchronized data in the temporary record table to avoid long-term occupation and redundancy; after the deletion is completed, the source server returns a "deletion successful" response to the source end.

[0111] Through the schematic timing steps of the above database incremental synchronization method, the incremental synchronization process of the database at the source end and the receiving end is realized, ensuring efficient and secure data transmission.

[0112] Figure 3 A schematic flow chart of source data in a database incremental synchronization method according to an embodiment of the present disclosure is shown.

[0113] like Figure 3 As shown, the specific process is as follows:

[0114] The source side builds a custom synchronization protocol based on the source table name, source database name, the amount of data in the scanned temporary record table, the current system time of the source database, the data encoding format, the custom version number, and other information. The custom synchronization protocol is usually 64 bytes long. If the protocol is less than 64 bytes, it is padded with 0.

[0115] Encapsulate the data fields that need to be synchronized into a byte[] byte array, and encrypt the byte array using XOR encryption or other symmetric encryption algorithms (such as AES);

[0116] The source sends the constructed custom synchronization protocol to the receiver, so that the receiver can verify the protocol fields.

[0117] After the receiving end successfully parses and verifies the synchronization protocol, the source end sends the encrypted synchronization data to the receiving end;

[0118] The data transmission process from the source end to the receiving end is completed.

[0119] This process ensures the uniqueness of the data synchronization process through the construction of a custom synchronization protocol and a field verification mechanism; it improves the security of data transmission through encryption mechanisms (such as XOR encryption or symmetric encryption algorithms such as AES).

[0120] Figure 4 A schematic flow chart of receiving-end data of the database incremental synchronization method according to an embodiment of the present disclosure is shown.

[0121] like Figure 4 As shown, the specific process is as follows:

[0122] The receiving end listens to the port and establishes a communication connection when it detects a connection request sent by the source end;

[0123] The receiver receives the custom protocol from the source and starts parsing it one by one according to the order of fields defined in the protocol. The custom protocol header contains information such as the source table name, the source database name, the number of data in the scanned temporary record table, the current system time of the source database, the data encoding format, and the custom version number. The parsing order and field values ​​must strictly comply with the agreement.

[0124] If the field order or field value of the protocol is found to be incorrect during the parsing process (such as missing fields, disordered order, or abnormal data), the receiving end will immediately disconnect and terminate subsequent operations to ensure safety;

[0125] If the protocol sequence and field value parsing are correct, the receiving end decrypts the data (supports encryption algorithms such as AES and XOR), and stores the decrypted data in the receiving end database;

[0126] After the data is successfully stored, the receiving end returns a success result, completing the data receiving process.

[0127] This process ensures that data is verified before being received and only verified data is further processed and stored.

[0128] Figure 5 It is a schematic diagram of the composition of an electronic device according to an embodiment of the present disclosure.

[0129] Second, refer to Figure 5 , an embodiment of the present disclosure provides an electronic device, comprising:

[0130] One or more processors 501;

[0131] A memory 502 on which one or more programs are stored. When the one or more programs are executed by one or more processors, the one or more processors implement the database incremental synchronization method described in the first aspect of the embodiment of the present disclosure;

[0132] One or more I / O interfaces 503 are connected between the processor and the memory and are configured to implement information exchange between the processor and the memory.

[0133] Among them, the processor 501 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 502 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 503 is connected between the processor 501 and the memory 502, and can realize information interaction between the processor 501 and the memory 502, including but not limited to a data bus (Bus), etc.

[0134] In some embodiments, the processor 501 , the memory 502 , and the I / O interface 503 are connected to each other via a bus 504 , and further connected to other components of the computing device.

[0135] In a third aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the database incremental synchronization method described in the first aspect of the embodiment of the present disclosure is implemented.

[0136] In a fourth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the database incremental synchronization method described in the first aspect of the embodiment of the present disclosure.

[0137] Figure 6It is a block diagram of a database incremental synchronization system according to an embodiment of the present disclosure.

[0138] Fifth, refer to Figure 6 The embodiment of the present disclosure provides a database incremental synchronization system, including:

[0139] A source table selection module 601 is configured to select a source table to be synchronized;

[0140] A creation module 602 is configured to create a temporary record table and a trigger for monitoring data changes for the selected source table;

[0141] A detection module 603 is configured to scan the temporary record table to detect whether there is an incremental change record that needs to be processed;

[0142] The protocol generation module 604 is configured to generate a custom synchronization protocol based on preset information when an incremental change record is detected in the temporary record table;

[0143] The protocol sending module 605 is configured to send the custom synchronization protocol to the receiving end;

[0144] The determination module 606 is configured to determine the sending or termination of synchronization data according to the parsing result of the custom synchronization protocol by the receiving end.

[0145] In some embodiments, the creation module 602 includes:

[0146] A record table creation module, configured to create a temporary record table for the selected source table to record incremental change information related to the source table;

[0147] A trigger creation module, configured to create a trigger for monitoring data changes of the source table in the source server according to the primary key and field information of the selected source table, wherein the data changes include adding operations, modifying operations and deleting operations;

[0148] The change trigger module is configured to write the source table primary key information, change action and change time into the temporary record table through the trigger when the data in the selected source table changes, so as to form an incremental change record.

[0149] In some embodiments, the detection module 603 includes:

[0150] A scanning module, configured to scan the temporary record table by starting a scheduled scanning task;

[0151] A judgment module, configured to detect whether there is an incremental change record that needs to be processed in the temporary record table;

[0152] The data extraction module is configured to extract corresponding change data from the source table as synchronization data to be synchronized to the receiving end based on the source table primary key information stored in the temporary record table when there is an incremental change record to be processed in the temporary record table.

[0153] In some embodiments, the protocol generation module 604 includes:

[0154] A setting module, configured to set preset information according to actual needs;

[0155] A generation module configured to generate a custom synchronization protocol based on preset information;

[0156] The preset information includes the source table name, the source database name, the number and data in the scanned temporary record table, the current system time of the source database, the encoding format of the source data, and the custom version number, and generates a custom synchronization protocol.

[0157] In some embodiments, the custom synchronization protocol further includes an encryption algorithm field for indicating an encryption method of the synchronization data.

[0158] In some embodiments, the determination module 606 includes:

[0159] A result receiving module, configured to receive a parsing result of the custom synchronization protocol from the receiving end;

[0160] A disconnection module, configured to disconnect the connection with the receiving end to terminate the synchronization operation when the parsing result indicates that the custom synchronization protocol cannot be correctly parsed;

[0161] The data sending module is configured to send the synchronization data when the parsing result indicates that the custom synchronization protocol is correctly parsed.

[0162] In some embodiments, the sending module includes:

[0163] An encapsulation module, configured to encapsulate the synchronization data into a byte array when the parsing result indicates that the custom synchronization protocol is correctly parsed;

[0164] an encryption module configured to encrypt the byte array using an encryption algorithm;

[0165] The sending module is configured to send the encrypted byte array to the receiving end.

[0166] In some embodiments, the database incremental synchronization system further comprises:

[0167] a decryption module configured to decrypt the received encrypted byte array by the receiving end according to the encryption algorithm specified in the custom synchronization protocol;

[0168] A storage module, configured to store the decrypted data to a receiving end server;

[0169] The success result sending module is configured to send a storage success result to the source end after the data storage is completed.

[0170] In some embodiments, the database incremental synchronization system further comprises:

[0171] The deletion module is configured to delete the synchronized data in the temporary record table after the source end receives the storage success result.

[0172] In order to enable those skilled in the art to more clearly understand the technical solution provided by the embodiment of the present disclosure, the database incremental synchronization method provided by the embodiment of the present disclosure is described in detail below through specific embodiments:

[0173] Embodiment 1

[0174] The present disclosure can be applied to perform real-time or periodic data synchronization from a source database to a target database.

[0175] According to the database incremental synchronization method disclosed in the present invention, the specific implementation steps are as follows:

[0176] 1) The source database synchronization service A obtains source table information from the source database B and determines the table that needs to be synchronized;

[0177] 2) Based on the acquired source table information, source database synchronization service A creates a temporary record table in source database B to store the incremental change data of the source table; at the same time, it creates corresponding triggers (addition, modification, and deletion) for the source table to automatically record the data changes in the source table into the temporary record table;

[0178] 3) The source database synchronization service A creates a scheduled task to scan the temporary record table and detect incremental change data; after scanning the data, assemble a 64-byte header protocol according to the format of the custom protocol. The protocol can contain the following fields: source table name, source database name, etc., the number of data in the scanned temporary record table, the current system time of the source database, the encoding format, the custom version number, etc. (the source of the fields in the header protocol is not limited to the information of the source table itself, but can also include fields related to the temporary record table, for example, "the number of data in the scanned temporary record table" means the number of incremental data); then, the source encapsulates the data fields of incremental changes in the table to be synchronized into a byte array (byte[]), and encrypts the byte array using a symmetric encryption algorithm (such as AES); first send the 64-byte custom protocol header to the destination database synchronization service C in sequence, and then send the encrypted synchronization data;

[0179] 4) After receiving the data, the destination database synchronization service C parses the field contents in the protocol in sequence according to the format of the custom protocol; if the protocol parsing is successful, service C receives the data and decrypts the encrypted data; if the protocol parsing fails (such as the field order or value is wrong), service C immediately disconnects and terminates the operation to ensure the security of data transmission;

[0180] 5) The destination database synchronization service C synchronizes the decrypted incremental data to the specified table of the destination database D; at this time, the operation of synchronizing column fields only involves the columns specified by the user. For example, the user can synchronize only certain fields in the source table (such as field B) instead of all fields in the table; the table structure and field type of the destination database need to be consistent with the structure of the source table to ensure data compatibility; after the synchronization is completed, the destination database synchronization service C sends a "successful synchronization" result feedback to the source database synchronization service A; after receiving the successful synchronization result, the source database synchronization service A deletes the synchronized data in the temporary record table corresponding to the source table to avoid long-term data occupation and redundancy.

[0181] This process ensures that data synchronization from the source to the destination is secure and efficient while maintaining data consistency and integrity.

[0182] This disclosure focuses on the scenario of cross-network file synchronization, and the key technical points involved are as follows:

[0183] 1) Before sending the database incremental data, the present invention designs and transmits a custom 64-byte protocol, which does not contain the actual incremental data, but is the content freely combined by the user according to the source table information, user information, etc.; and only when the receiving end verifies the protocol, the actual incremental data is sent. This mechanism effectively ensures the security of data transmission and avoids unauthorized data leakage.

[0184] 2) The present disclosure records data changes in source tables through a combination of triggers and temporary record tables. The temporary record tables are created in the source database and are session-level, which means that they are automatically deleted after the data transmission session ends or according to policy settings. This method reduces the impact of temporary record tables on the source database and is applicable to all database systems that support triggers. It does not require additional configuration or third-party tool intervention and has the feature of "ready-to-use". In addition, this flexible design can effectively record data information and reduce the pressure on database backup, ensuring that the data synchronization process is lightweight and efficient.

[0185] The database incremental synchronization method disclosed in the present invention can bring the following beneficial effects:

[0186] 1) Through the synchronization mechanism disclosed in the present invention, the overall efficiency of data synchronization and backup from the source database to the destination database can be significantly improved, and real-time or periodic efficient data transmission can be achieved;

[0187] 2) By combining custom protocols and AES encryption transmission technology, the data transmission security and integrity are ensured during the data synchronization process to prevent the data from being intercepted or tampered with during transmission;

[0188] 3) The design of triggers and temporary record tables makes the data change records and synchronization process clear and controllable, easy to maintain and manage, and reduces the complexity of operation and maintenance;

[0189] 4) The present disclosure realizes synchronization of incremental data of a database without complicated operations, and users can easily realize incremental synchronization without complicated configuration during operation, thereby improving the convenience of use.

[0190] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0191] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly noted, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A database incremental synchronization method, comprising: Select the source table to be synchronized; Create a temporary record table and a trigger for monitoring data changes for the selected source table on the source server; Scan the temporary record table by starting a scheduled scanning task to detect whether there is an incremental change record that needs to be processed; When it is detected that there is an incremental change record in the temporary record table, a custom synchronization protocol is generated based on preset information; Sending the custom synchronization protocol to a receiving end; The sending or termination of the synchronization data is determined according to the parsing result of the custom synchronization protocol by the receiving end.

2. The database incremental synchronization method according to claim 1, wherein: Create a temporary record table and triggers for monitoring data changes for the selected source table, including: Creating a temporary record table for the selected source table to record incremental change information related to the source table; According to the primary key and field information of the selected source table, a trigger for monitoring data changes of the source table is created in the source server, wherein the data changes include adding operations, modifying operations and deleting operations; When the data in the selected source table is changed, the source table primary key information, change action and change time are written into the temporary record table through the trigger to form an incremental change record.

3. The database incremental synchronization method according to claim 2, wherein: Scan the temporary record table to detect whether there are incremental change records that need to be processed, including: Scan the temporary record table by starting a scheduled scanning task; Detect whether there is an incremental change record that needs to be processed in the temporary record table; When there are incremental change records that need to be processed in the temporary record table, corresponding change data is extracted from the source table based on the source table primary key information stored in the temporary record table as synchronization data that needs to be synchronized to the receiving end.

4. The database incremental synchronization method according to claim 3, wherein: Generate a custom synchronization protocol based on preset information, including: Set preset information according to actual needs; Generate a custom synchronization protocol based on preset information; The preset information includes the source table name, the source database name, the number and data in the scanned temporary record table, the current system time of the source database, the encoding format of the source data, and the custom version number, and generates a custom synchronization protocol.

5. The database incremental synchronization method according to any one of claims 1 to 4, wherein: The custom synchronization protocol also includes an encryption algorithm field for indicating an encryption method of the synchronization data.

6. The database incremental synchronization method according to claim 5, wherein: Determining the sending or termination of synchronization data according to the parsing result of the custom synchronization protocol by the receiving end includes: Receiving a parsing result of the custom synchronization protocol from the receiving end; When the parsing result indicates that the custom synchronization protocol cannot be correctly parsed, disconnecting from the receiving end to terminate the synchronization operation; When the parsing result indicates that the custom synchronization protocol is parsed correctly, the synchronization data is sent.

7. The database incremental synchronization method according to claim 6, wherein: When the parsing result indicates that the custom synchronization protocol is correctly parsed, sending the synchronization data includes: When the parsing result indicates that the custom synchronization protocol is parsed correctly, encapsulating the synchronization data into a byte array; Encrypt the byte array using an encryption algorithm; The encrypted byte array is sent to the receiving end.

8. The database incremental synchronization method according to claim 5, wherein: Before sending the custom synchronization protocol to the receiving end, it includes: Encapsulating the synchronization data into a byte array; Encrypts a byte array using an encryption algorithm.

9. The database incremental synchronization method according to claim 8, wherein: Determining the sending or termination of synchronization data according to the parsing result of the custom synchronization protocol by the receiving end includes: Receiving a parsing result of the custom synchronization protocol from the receiving end; When the parsing result indicates that the custom synchronization protocol cannot be correctly parsed, disconnecting from the receiving end to terminate the synchronization operation; When the parsing result indicates that the custom synchronization protocol is parsed correctly, the encrypted byte array is sent to the receiving end.

10. The database incremental synchronization method according to claim 7 or 9, wherein: The database incremental synchronization method further comprises: The receiving end decrypts the received encrypted byte array according to the encryption algorithm specified in the custom synchronization protocol; Store the decrypted data to the receiving server; After the data storage is completed, a storage success result is sent to the source end.

11. The database incremental synchronization method according to claim 10, wherein: The database incremental synchronization method further comprises: After receiving the storage success result, the source end deletes the synchronized data in the temporary record table.

12. An electronic device comprising: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the database incremental synchronization method according to any one of claims 1 to 11.

13. A computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for incremental synchronization of a database according to any one of claims 1 to 11 is implemented.

14. A computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the method for incremental synchronization of a database according to any one of claims 1 to 11 is implemented.

Images