Data synchronization method and apparatus, computer device, and storage medium

Abstract

The application relates to the technical field of data migration, and discloses a data synchronization method and device, computer equipment and a storage medium. The method comprises the following steps: determining a reference transaction sequence number of a target transaction in a source database; scanning all archive logs of the target transaction in the source database, and obtaining all archive files with a transaction sequence number greater than the reference transaction sequence number in the archive logs; grouping each archive file according to a corresponding physical node, and sorting each physical node according to a preset node arrangement strategy to obtain a node queue; calculating the density of each archive file in the node queue, and performing a fragmentation processing on each archive file according to the density calculation result; and sequentially migrating each archive file after the fragmentation processing to a target database according to the arrangement order of each physical node in the node queue. The application can effectively improve the efficiency and accuracy of archive log synchronization.

Description

Technical Field

[0001] This invention relates to the field of data migration technology, and more specifically to a data synchronization method, apparatus, computer equipment, and computer-readable storage medium. Background Technology

[0002] Currently, data synchronization is a critical task in modern distributed database systems, especially in scenarios where maintaining data consistency across multiple database instances is crucial. Traditional data synchronization methods primarily rely on full data synchronization or timestamp-based incremental synchronization. While full synchronization is simple, it is inefficient, especially when dealing with large-scale data. Timestamp-based incremental synchronization, while improving efficiency, carries the risk of data loss and inconsistency when handling high-concurrency transactions and distributed environments.

[0003] In recent years, with the development of distributed computing and storage technologies, log-based incremental synchronization methods have emerged. These methods achieve data synchronization by capturing archived logs from the database, effectively reducing data transfer volume and improving synchronization efficiency. However, existing methods still have some problems when processing archived logs in distributed databases, such as poor data synchronization efficiency, consistency, and reliability.

[0004] Therefore, how to provide a data synchronization method, apparatus, computer equipment, and computer-readable storage medium that can effectively improve the efficiency and accuracy of archived log synchronization is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide a data synchronization method, apparatus, computer device and computer-readable storage medium, which aims to solve the problem of how to effectively improve the efficiency and accuracy of archived log synchronization.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a data synchronization method, comprising:

[0008] Determine the base transaction sequence number of the target transaction in the source database; wherein, the source database includes multiple physical nodes;

[0009] Scan all archived logs of the target transaction in the source database to obtain all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number;

[0010] Each archived file is grouped according to its corresponding physical node, and the physical nodes are sorted according to a preset node arrangement strategy to obtain a node queue.

[0011] Based on a preset file density model, the density of each archived file in the node queue is calculated, and the archived files are fragmented according to the density calculation results.

[0012] According to the order of the physical nodes in the node queue, the archived files after sharding are migrated to the target database in sequence.

[0013] In a second aspect, the present invention provides a data synchronization device, comprising:

[0014] The determination module is used to determine the base transaction sequence number of the target transaction in the source database; wherein, the source database includes multiple physical nodes;

[0015] The scanning module is used to scan all archived logs of the target transaction in the source database to obtain all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number;

[0016] The grouping module is used to group the archived files according to their corresponding physical nodes, and sort the physical nodes according to a preset node arrangement strategy to obtain a node queue.

[0017] The calculation module is used to perform density calculation on each archive file in the node queue based on a preset file density model, and to perform fragmentation processing on each archive file according to the density calculation results.

[0018] The migration module is used to migrate the sharded archive files to the target database in sequence according to the arrangement order of the physical nodes in the node queue.

[0019] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data synchronization method described above.

[0020] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the data synchronization method described above.

[0021] Compared to existing technologies, this invention provides a data synchronization method, apparatus, computer device, and computer-readable storage medium. The method involves determining a baseline transaction sequence number for a target transaction in a source database, where the source database includes multiple physical nodes. All archived logs of the target transaction in the source database are scanned to obtain all archived files whose transaction sequence numbers are greater than the baseline transaction sequence number. Each archived file is grouped according to its corresponding physical node, and the physical nodes are sorted according to a preset node arrangement strategy to obtain a node queue. Based on a preset file density model, the density of each archived file in the node queue is calculated, and the archived files are fragmented according to the density calculation results. The fragmented archived files are then sequentially migrated to the target database according to the arrangement order of the physical nodes in the node queue. This invention effectively improves the efficiency and accuracy of archived log synchronization. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram illustrating the application environment of a data synchronization method provided in an embodiment of the present invention.

[0024] Figure 2 This is a flowchart illustrating a data synchronization method according to an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram of the program modules of a data synchronization device provided in an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention.

[0027] Figure 5 This is another structural schematic diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0030] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0031] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0032] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0034] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0035] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0036] An embodiment of the present invention provides a data synchronization method that can be applied to, for example... Figure 1 In the application environment shown, the client and server communicate via a network. The client includes, but is not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud computing devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0037] Please see Figure 2 An embodiment of the present invention provides a data synchronization method, wherein the method includes the following steps:

[0038] S100. Determine the base transaction sequence number of the target transaction in the source database; wherein, the source database includes multiple physical nodes;

[0039] S200: Scan all archived logs of the target transaction in the source database to obtain all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number;

[0040] S300. Group each of the archived files according to its corresponding physical node, and sort each of the physical nodes according to a preset node arrangement strategy to obtain a node queue.

[0041] S400. Based on a preset file density model, perform density calculation on each archived file in the node queue, and perform fragmentation processing on each archived file according to the density calculation results.

[0042] S500. According to the arrangement order of each physical node in the node queue, each of the fragmented archive files is migrated to the target database in sequence.

[0043] In practical implementation, the data synchronization method in this embodiment significantly improves the efficiency and accuracy of archived log synchronization through a series of innovative steps. A detailed analysis follows:

[0044] 1. Determine the baseline transaction sequence number (S100)

[0045] By determining the baseline transaction sequence number of the target transaction in the source database, this method provides a clear starting point for the data synchronization task. This step ensures that the synchronization process begins from the correct transaction position, avoiding the duplication of already synchronized data, thereby improving the accuracy and efficiency of synchronization.

[0046] The transaction sequence number is a sequence number used to identify the order of transactions. It is usually a monotonically increasing number used to record the commit order of transactions and is applicable to all database systems that require transaction management, including relational databases and NoSQL databases.

[0047] Specifically, it can be used to describe the transaction sequence recording mechanism in different database systems, clearly expressing its function of recording the transaction order. Examples include Oracle's SCN (System Change Number), MySQL's transaction sequence number (implemented through an auto-incrementing field), and PostgreSQL's XID (Transaction ID).

[0048] 2. Scan archived logs (S200)

[0049] The process involves scanning all archived logs of the target transaction in the source database to identify all archived files with transaction sequence numbers greater than the baseline transaction sequence number. This step precisely filters the archived files that need to be synchronized, reducing unnecessary data processing and further improving synchronization efficiency. Simultaneously, filtering by transaction sequence number ensures data integrity and consistency.

[0050] 3. Archive file grouping and node sorting (S300)

[0051] Archived files are grouped according to their corresponding physical nodes, and these physical nodes are then sorted according to a preset node arrangement strategy to obtain a node queue. This step optimizes the data processing order through reasonable grouping and sorting, ensuring the high efficiency of parallel processing. Especially in a distributed environment, node sorting allows for priority processing of nodes with lower load and lower network latency, thereby improving overall synchronization efficiency.

[0052] 4. File density-based fragmentation processing (S400)

[0053] Based on a preset file density model, the archived files are density-calculated, and then fragmented according to the calculation results. This step dynamically adjusts the fragmentation strategy to generate fragments of appropriate size based on the transaction density of the files. High-density files are fragmented with fine granularity, while low-density files are fragmented with coarse granularity, thereby achieving load balancing and further improving resource utilization efficiency and synchronization speed.

[0054] 5. Migrate to the target database (S500) in sequence.

[0055] Based on the order of physical nodes in the node queue, the sharded archive files are migrated sequentially to the target database. This step ensures data order and consistency through ordered migration operations. Furthermore, sharding allows each shard to be processed independently in parallel, further improving migration efficiency. In addition, a dynamic optimization mechanism enables the system to automatically adjust migration strategies based on real-time data distribution and system load, ensuring efficient operation in various scenarios.

[0056] Through the synergistic effect of the above steps, the data synchronization method of the present invention can significantly improve the efficiency and accuracy of archived log synchronization. Specifically, this is reflected in:

[0057] High throughput: Through dynamic fragmentation and fragmented parallel processing, the throughput of data synchronization is significantly improved, with a throughput increase of 3-5 times compared to traditional serial processing methods;

[0058] Low latency: By optimizing node sorting and sharding strategies, the processing time of individual tasks is reduced, significantly lowering the latency of data synchronization;

[0059] High accuracy: Through precise filtering and orderly migration of transaction sequence numbers, data integrity and consistency are ensured;

[0060] Dynamic optimization: By dynamically adjusting real-time data distribution and system load, resource utilization is maximized, improving the overall performance and reliability of the system.

[0061] Therefore, this invention breaks through the bottleneck of traditional data synchronization methods through a series of innovative technologies, significantly improving the efficiency and accuracy of archived log synchronization, and is suitable for large-scale, high-concurrency distributed database environments.

[0062] Furthermore, in one embodiment, the data synchronization method, wherein determining the base transaction sequence number of the target transaction in the source database specifically includes the following steps:

[0063] Receive user input commands;

[0064] If the input instruction is a sequence number input value, then the sequence number input value is used as the base transaction sequence number of the target transaction in the source database;

[0065] If the input instruction is automatic detection, then the minimum uncommitted transaction sequence number of the target transaction in the source database is obtained and used as the baseline transaction sequence number.

[0066] In practice, the specific implementation process of the steps in this embodiment is roughly as follows:

[0067] Step 1: Receive user input commands

[0068] The system (the software system corresponding to the method of this invention) receives instructions input by the user. These instructions can be transaction sequence numbers manually entered by the user or automatic detection commands selected by the user.

[0069] Step 2: Determine the type of input command

[0070] The system determines the type of command input by the user. If the input command is a specific sequence number, that value is directly used as the base transaction sequence number; if the input command is for automatic detection, the automatic detection process is initiated.

[0071] Step 3: Automatically detect the smallest uncommitted transaction sequence number

[0072] If the user selects automatic detection, the system queries the source database to obtain the smallest uncommitted transaction sequence number in the target transaction and uses it as the base transaction sequence number.

[0073] Step 4: Set the baseline transaction sequence number

[0074] Based on the user-inputted sequence number or the result of automatic detection, the system sets the baseline transaction sequence number for the target transaction and records this value for use in subsequent steps.

[0075] Step 5: Verify the baseline transaction sequence number

[0076] The system verifies whether the set baseline transaction sequence number is valid. If invalid, the user is prompted to re-enter it or to perform an automatic check again; if valid, the system continues to execute subsequent synchronization tasks.

[0077] Through the above process, the system can flexibly determine the base transaction sequence number of the target transaction in the source database based on user input or automatic detection results, ensuring the accuracy and reliability of the data synchronization task.

[0078] Further, in one embodiment, the data synchronization method, wherein scanning all archived logs of the target transaction in the source database to obtain all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number specifically includes the following steps:

[0079] Traverse all archived logs of the target transaction in the source database and read the metadata information of each archived log;

[0080] Based on the metadata information, all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number are selected.

[0081] In practice, the specific implementation process of the steps in this embodiment is roughly as follows:

[0082] Step 1: Initialize the scanning environment

[0083] Configure scan parameters: The system loads the configuration information of the source database, including the base transaction sequence number of the target transaction, the storage path of the archived logs, etc.

[0084] Load the archived log index: If an archived log index exists, the system loads the index to speed up the process of locating archived logs. The index contains basic information for each archived log, such as file path, size, and transaction sequence number range.

[0085] Step 2: Traverse the archived logs and read metadata

[0086] Traversing Archived Logs: The system traverses all archived logs of the target transaction in the source database. For each archived log, the system reads its metadata information, including file size, transaction sequence number range (starting SCN and ending SCN), etc.

[0087] Record metadata: The system records the metadata of each archived log in a temporary storage area for subsequent processing.

[0088] Step 3: Filter archived files

[0089] Compare transaction sequence numbers: The system compares the start SCN and end SCN of each archived log with the baseline transaction sequence number based on the transaction sequence number range in the metadata.

[0090] Filter files that meet the criteria: If the starting SCN of the archived log is greater than the base transaction sequence number, or the ending SCN is greater than the base transaction sequence number, then mark the archived log as an archived file that needs to be synchronized.

[0091] Record the filtering results: The system records the paths and related information of the filtered archive files into a list for subsequent fragmentation and migration.

[0092] Step 4: Verify the screening results

[0093] Integrity check: The system performs an integrity check on the selected archive files to ensure that the files are not corrupted and are readable.

[0094] Log error messages: If a file is found to be corrupted or unreadable, the system logs the error message and decides whether to skip the file based on the configuration.

[0095] Through the above process, this embodiment enables the system to efficiently scan the archived logs in the source database and filter out all archived files with transaction sequence numbers greater than the baseline transaction sequence number. This process not only ensures the accuracy and integrity of data synchronization but also improves synchronization efficiency by optimizing the filtering results.

[0096] Further, in one embodiment, the data synchronization method, wherein grouping each archived file according to its corresponding physical node, and sorting each physical node according to a preset node arrangement strategy to obtain a node queue, specifically includes the following steps:

[0097] Configure node arrangement strategy;

[0098] Based on the metadata information, the physical node corresponding to each archive file is determined, and the archive files are grouped according to their corresponding physical nodes;

[0099] According to the node arrangement strategy, the physical nodes are arranged in order to generate a node queue.

[0100] In practice, the specific implementation process of the steps in this embodiment is roughly as follows:

[0101] Step 1: Configure node arrangement strategy

[0102] Define strategy parameters: The system defines the parameters of the node arrangement strategy. These parameters may include node load, network latency, transaction processing capability, etc.

[0103] Set priority rules: Configure node priority rules according to actual needs. For example, prioritize nodes with lower load and lower network latency, or prioritize nodes with stronger transaction processing capabilities.

[0104] Storage strategy configuration: Store the configured node arrangement strategy in the system configuration file or database for use in subsequent steps.

[0105] Step 2: Read the metadata of the archived file

[0106] Traversing Archive Files: The system traverses all selected archive files, reading the metadata information of each archive file. Metadata typically includes file path, size, transaction sequence number range, and the physical node to which it belongs.

[0107] Record metadata: Record the metadata of each archived file in a temporary storage area for later processing.

[0108] Step 3: Group and archive files by physical node

[0109] Determine the physical node: Based on the metadata of the archived files, determine the physical node to which each archived file belongs.

[0110] Grouping Archive Files: Archive files are grouped according to their corresponding physical nodes. Each physical node corresponds to a group list, and archive files belonging to that node are added to the corresponding list.

[0111] Step 4: Sort physical nodes by node arrangement strategy

[0112] Loading strategy configuration: The system loads a pre-configured node arrangement strategy.

[0113] Node priority assessment: Based on the node allocation strategy, assess the priority of each physical node. For example, calculate metrics such as load, network latency, and transaction processing capacity for each node.

[0114] Node queue generation: Based on the evaluation results, physical nodes are arranged in priority order to generate a node queue. The order in the node queue will determine the order of subsequent processing.

[0115] Step 5: Verify and optimize the node queue

[0116] Verify the node queue: The system verifies the generated node queue to ensure that the archive files of each node are correctly grouped and that the order of the nodes conforms to the preset node arrangement strategy.

[0117] Dynamic optimization: The order of nodes in the queue is dynamically adjusted based on real-time system load and resource usage. For example, if the load on a node suddenly increases, the system can automatically lower its priority and advance other nodes in the queue.

[0118] Record queue information: Record the final generated node queue information to the system log or configuration file for use in subsequent steps.

[0119] Through the above process, this embodiment enables the system to group archived files according to their corresponding physical nodes and generate an ordered node queue based on a preset node arrangement strategy. This process not only ensures the efficiency and reliability of data synchronization tasks but also improves the system's adaptability and flexibility through a dynamic optimization mechanism.

[0120] Furthermore, in one embodiment, the data synchronization method, wherein the step of calculating the density of each archived file in the node queue based on a preset file density model, and fragmenting each archived file according to the density calculation result, specifically includes the following steps:

[0121] Configuration file density model and file fragmentation strategy;

[0122] Using the file density model, the density of each archived file in the node queue is calculated, and the density calculation result is generated.

[0123] Based on the density calculation results and the file fragmentation strategy, each archived file is fragmented.

[0124] In practice, the specific implementation process of the steps in this embodiment is roughly as follows:

[0125] Step 1: Configure the file density model and file fragmentation strategy

[0126] Define the file density model: The system defines the parameters of the file density model, including file size, SCN interval size, etc. The file density model is used to evaluate the distribution density of transactions in archived files.

[0127] Configure file splitting strategy: Set the file splitting strategy according to actual needs, including split size, number of splits, etc. The file splitting strategy can be dynamically adjusted according to file density to optimize processing efficiency.

[0128] Store configuration information: Store the configured file density model and file sharding strategy in the system configuration file or database for use in subsequent steps.

[0129] Step 2: Load the file density model and file fragmentation strategy

[0130] Loading the model and strategy: The system loads the pre-configured file density model and file fragmentation strategy. It ensures that the model and strategy parameters are correctly loaded into memory for subsequent calculations and processing.

[0131] Initialize the computing environment: Initialize the environment used for density calculation and piecewise processing, including the necessary data structures and variables.

[0132] Step 3: Calculate the density of the archived files.

[0133] Traversing archive files: The system traverses each archive file in the node queue, reading the file's metadata information, including file size, transaction sequence number range (starting SCN and ending SCN), etc.

[0134] Calculate file density: The density of each archived file is calculated using a file density model. File density can be calculated using the following formula:

[0135] AFD = Normalized Density Function(File Size / SCN Interval Size), where AFD (Archive File Density) represents the density of archived files.

[0136] Record density calculation results: Record the density calculation results of each archive file to a temporary storage area for subsequent fragmentation processing.

[0137] Step 4: Perform segmentation based on density calculation results

[0138] Evaluate file fragmentation strategy: Based on the density calculation results and the preset file fragmentation strategy, evaluate the fragmentation method for each archive file. For example:

[0139] If AFD > threshold 0.7, fine-grained fragmentation is triggered, dividing the file into more smaller fragments.

[0140] If AFD ≤ threshold 0.3, coarse-grained fragmentation is triggered, dividing the file into fewer large fragments.

[0141] If 0.3 < AFD ≤ 0.7, keep the default sharding parameters unchanged.

[0142] Perform sharding: Based on the evaluation results, shard each archive file. Sharding may involve dividing the file into multiple sub-files and recording metadata information for each sub-file.

[0143] Storage sharding results: The sharded archive files and their metadata information are stored in a temporary storage area for subsequent migration operations.

[0144] Step 5: Verify and optimize the sharding results

[0145] Verify the sharding results: The system verifies the archived files after sharding to ensure that the size and transaction distribution of each shard conform to the preset file sharding strategy.

[0146] Dynamic optimization: The file sharding strategy is dynamically adjusted based on real-time system load and resource usage. For example, if the system detects that memory resources are scarce, smaller shards will be generated to reduce memory usage; if CPU resources are sufficient, larger shards will be generated to improve processing speed.

[0147] Record sharding information: Record the final generated sharding information to the system log or configuration file for use in subsequent steps.

[0148] Through the above process, the system can calculate the density of archived files based on a preset file density model and perform fragmentation based on the density calculation results. This process not only ensures the efficiency and reliability of data synchronization tasks but also improves the system's adaptability and flexibility through a dynamic optimization mechanism.

[0149] Furthermore, in one embodiment, the data synchronization method, wherein the step of sequentially migrating the fragmented archive files to the target database according to the arrangement order of the physical nodes in the node queue, specifically includes the following steps:

[0150] According to the order of the physical nodes in the node queue, the fragmented archive files are sent to the message middleware in sequence.

[0151] Read the archived file in the message middleware and convert it into the format required by the target database;

[0152] The archived file, after format conversion, is replayed and written to the target database.

[0153] In practice, the specific implementation process of the steps in this embodiment is roughly as follows:

[0154] Step 1: Initialize the message middleware

[0155] Configure message middleware: The system loads the configuration information of the message middleware (such as Kafka middleware), including connection parameters, queue name, topic, etc.

[0156] Initialize connection: The system establishes a connection with the message middleware to ensure that the connection is stable and available.

[0157] Prepare queues or topics: Create or prepare queues or topics in the message middleware for storing archived files.

[0158] Step 2: Send archived files to the message middleware in sequence.

[0159] Traversing the node queue: The system processes each physical node sequentially according to the order in which the physical nodes are arranged in the node queue.

[0160] Sending archive files: For each physical node, the system sends the fragmented archive files to the message middleware in sequence.

[0161] Record sending status: The system records the sending status of each archived file, including sending time, success or failure flags, etc., for subsequent monitoring and retry mechanisms.

[0162] Step 3: Read and convert archive file format

[0163] Read archived files from the message middleware: The system reads archived files from the message middleware, ensuring the order and integrity of the read operations.

[0164] Format conversion: The system converts the read archive files into the format required by the target database. The conversion process may include data cleaning, field mapping, encoding conversion, and other operations.

[0165] Record conversion status: The system records the format conversion status of each archived file, including conversion time, success or failure flags, etc., for subsequent monitoring and retry mechanisms.

[0166] Step 4: Replay and write to the target database

[0167] Connect to the target database: The system establishes a connection with the target database, ensuring that the connection is stable and available.

[0168] Playback Write: The system replays the converted archive files sequentially and writes them to the target database. During playback, the system ensures the order and consistency of the data.

[0169] Record write status: The system records the write status of each archived file, including write time, success or failure flags, etc., for subsequent monitoring and retry mechanisms.

[0170] Step 5: Verify and confirm the synchronization results

[0171] Verify data integrity: The system verifies the data in the target database to ensure that all archived files have been written correctly and that the integrity and consistency of the data are guaranteed.

[0172] Confirm Synchronization Completion: The system confirms the completion of the synchronization task and records its final status. If any problems are detected, the system can trigger a retry mechanism or an alarm mechanism.

[0173] Clean up temporary data: The system cleans up temporary data in the message middleware, freeing up resources and ensuring efficient system operation.

[0174] Through the above process, the system can sequentially migrate the fragmented archive files to the target database according to the order of the physical nodes in the node queue. This process not only ensures the efficiency and reliability of data synchronization, but also improves the system's fault tolerance and scalability through the buffering mechanism of the message middleware.

[0175] Furthermore, in one embodiment, the data synchronization method wherein the source database and the target database are heterogeneous databases of different database types.

[0176] In practice, after the fragmented archive file is sent to the message middleware, the method in this embodiment continues to handle cases where the source database and the target database are heterogeneous databases by determining data mapping rules, reading and converting the archive file, handling data type differences, replaying and writing to the target database, and verifying and confirming the synchronization results. This ensures the efficiency and reliability of data synchronization. These steps together ensure the correct conversion and synchronization of data between different database types.

[0177] As can be seen from the above method embodiments, the data synchronization method provided by the present invention includes: determining the baseline transaction sequence number of the target transaction in the source database; wherein the source database includes multiple physical nodes; scanning all archived logs of the target transaction in the source database to obtain all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number; grouping each archived file according to its corresponding physical node, and sorting each physical node according to a preset node arrangement strategy to obtain a node queue; calculating the density of each archived file in the node queue based on a preset file density model, and performing fragmentation processing on each archived file according to the density calculation result; and sequentially migrating each fragmented archived file to the target database according to the arrangement order of each physical node in the node queue. Thus, the method of the present invention can effectively improve the efficiency and accuracy of archived log synchronization.

[0178] It should be understood that although this application provides the method operation steps as described in the embodiments or flowcharts, conventional or non-inventive labor may include more or fewer operation steps, and these operation steps are not necessarily executed sequentially according to the order of the embodiments or flowcharts. The order of steps listed in the embodiments or flowcharts is merely one way of executing many steps and does not represent the only execution order. It should be noted that there is no necessary sequential order between the above steps. Those skilled in the art can understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in exchange, etc. Moreover, at least some steps in the embodiments or flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn, alternately, or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0179] Based on the above method embodiments, please refer to Figure 3 Another embodiment of the present invention also provides a data synchronization device, wherein the device includes:

[0180] The determination module 11 is used to determine the base transaction sequence number of the target transaction in the source database; wherein, the source database includes multiple physical nodes;

[0181] Scanning module 12 is used to scan all archived logs of the target transaction in the source database to obtain all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number;

[0182] Grouping module 13 is used to group each of the archived files according to their corresponding physical nodes, and sort each of the physical nodes according to a preset node arrangement strategy to obtain a node queue.

[0183] The calculation module 14 is used to perform density calculation on each archive file in the node queue based on a preset file density model, and to perform fragmentation processing on each archive file according to the density calculation results.

[0184] The migration module 15 is used to migrate each of the sharded archive files to the target database in sequence according to the arrangement order of each of the physical nodes in the node queue.

[0185] Furthermore, in one embodiment, the data synchronization device, wherein determining the base transaction sequence number of the target transaction in the source database specifically includes:

[0186] Receive user input commands;

[0187] If the input instruction is a sequence number input value, then the sequence number input value is used as the base transaction sequence number of the target transaction in the source database;

[0188] If the input instruction is automatic detection, then the minimum uncommitted transaction sequence number of the target transaction in the source database is obtained and used as the baseline transaction sequence number.

[0189] Further, in one embodiment, the data synchronization device, wherein scanning all archived logs of the target transaction in the source database to obtain all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number specifically includes:

[0190] Traverse all archived logs of the target transaction in the source database and read the metadata information of each archived log;

[0191] Based on the metadata information, all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number are selected.

[0192] Further, in one embodiment, the data synchronization device, wherein grouping each archived file according to its corresponding physical node, and sorting each physical node according to a preset node arrangement strategy to obtain a node queue, specifically includes:

[0193] Configure node arrangement strategy;

[0194] Based on the metadata information, the physical node corresponding to each archive file is determined, and the archive files are grouped according to their corresponding physical nodes;

[0195] According to the node arrangement strategy, the physical nodes are arranged in order to generate a node queue.

[0196] Furthermore, in one embodiment, the data synchronization device, wherein the step of calculating the density of each archived file in the node queue based on a preset file density model, and performing fragmentation processing on each archived file according to the density calculation result, specifically includes:

[0197] Configuration file density model and file fragmentation strategy;

[0198] Using the file density model, the density of each archived file in the node queue is calculated, and the density calculation result is generated.

[0199] Based on the density calculation results and the file fragmentation strategy, each archived file is fragmented.

[0200] Furthermore, in one embodiment, the data synchronization device, wherein the step of sequentially migrating the fragmented archive files to the target database according to the arrangement order of the physical nodes in the node queue specifically includes:

[0201] According to the order of the physical nodes in the node queue, the fragmented archive files are sent to the message middleware in sequence.

[0202] Read the archived file in the message middleware and convert it into the format required by the target database;

[0203] The archived file, after format conversion, is replayed and written to the target database.

[0204] Furthermore, in one embodiment, the data synchronization device wherein the source database and the target database are heterogeneous databases of different database types.

[0205] It should be noted that, in the device embodiments of the present invention, the information interaction and execution process between the above modules are based on the same concept as in the method embodiments of the present invention. For details on their specific functions and the resulting technical effects, please refer to the aforementioned method embodiments section, which will not be repeated here.

[0206] Based on the above method embodiments, another embodiment of the present invention also provides a computer device, which can be a server, and its internal structure diagram can be as follows. Figure 4 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of the data synchronization method on the server side as described in any of the above method embodiments.

[0207] Based on the above method embodiments, another embodiment of the present invention also provides a computer device, which can be a client, and its internal structure diagram can be as follows. Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements the client-side functions or steps of the data synchronization method as described in any of the above method embodiments.

[0208] Those skilled in the art will understand that Figure 4 and Figure 5 The structural schematic diagram shown is only a schematic diagram of a part of the structure related to the present invention and does not constitute a limitation on the computer device on which the present invention is applied. The specific computer device may include more components than shown in the figure, or combine certain components, or have different component arrangements.

[0209] The processor referred to herein can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0210] The memory includes readable storage media, internal memory, etc., where internal memory can be the RAM of a computer device. Internal memory provides an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of the computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal storage units and external storage devices of the computer device. The memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. The memory can also be used to temporarily store data that has been output or will be output.

[0211] Based on the above method embodiments, another embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the data synchronization method as described in any of the above method embodiments. The computer-readable storage medium may be non-volatile or volatile.

[0212] It should be noted that the functions or steps that can be achieved by the computer-readable storage medium or computer device, and the technical effects brought about by the functions / steps, can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0213] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc. The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0214] Those skilled in the art will understand that, for the sake of convenience and brevity, the embodiments of the device of the present invention are only illustrated by the division of the above-mentioned functional units and modules. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the present invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0215] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0216] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0217] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0218] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data synchronization method, characterized in that, include: Determine the base transaction sequence number of the target transaction in the source database; wherein, the source database includes multiple physical nodes; Scan all archived logs of the target transaction in the source database to obtain all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number; Each archived file is grouped according to its corresponding physical node, and the physical nodes are sorted according to a preset node arrangement strategy to obtain a node queue. Based on a preset file density model, the density of each archived file in the node queue is calculated, and the archived files are fragmented according to the density calculation results. According to the order of the physical nodes in the node queue, the archived files after sharding are migrated to the target database in sequence. The process of calculating the density of each archived file in the node queue based on a preset file density model, and then fragmenting each archived file according to the density calculation results, includes: Configuration file density model and file fragmentation strategy; Using the file density model, the density of each archived file in the node queue is calculated to generate a density calculation result; wherein, the file density is calculated by the following formula: AFD = Normalized Density Function (File Size / SCN Interval Size), where AFD represents the density of the archived file; Based on the density calculation results and the file fragmentation strategy, each archived file is fragmented; wherein, if AFD > 0.7, fine-grained fragmentation is triggered; if AFD ≤ 0.3, coarse-grained fragmentation is triggered; if 0.3 < AFD ≤ 0.7, the default fragmentation parameters remain unchanged.

2. The data synchronization method according to claim 1, characterized in that, Determining the base transaction sequence number of the target transaction in the source database includes: Receive user input commands; If the input instruction is a sequence number input value, then the sequence number input value is used as the base transaction sequence number of the target transaction in the source database; If the input instruction is automatic detection, then the minimum uncommitted transaction sequence number of the target transaction in the source database is obtained and used as the baseline transaction sequence number.

3. The data synchronization method according to claim 1, characterized in that, The step of scanning all archived logs of the target transaction in the source database to obtain all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number includes: Traverse all archived logs of the target transaction in the source database and read the metadata information of each archived log; Based on the metadata information, all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number are selected.

4. The data synchronization method according to claim 3, characterized in that, The step of grouping the archived files according to their corresponding physical nodes and sorting the physical nodes according to a preset node arrangement strategy to obtain a node queue includes: Configure node arrangement strategy; Based on the metadata information, the physical node corresponding to each archive file is determined, and the archive files are grouped according to their corresponding physical nodes; According to the node arrangement strategy, the physical nodes are arranged in order to generate a node queue.

5. The data synchronization method according to claim 1, characterized in that, The step of sequentially migrating the fragmented archive files to the target database according to the order of the physical nodes in the node queue includes: According to the order of the physical nodes in the node queue, the fragmented archive files are sent to the message middleware in sequence. Read the archived file in the message middleware and convert it into the format required by the target database; The archived file, after format conversion, is replayed and written to the target database.

6. The data synchronization method according to any one of claims 1-5, characterized in that, The source database and the target database are heterogeneous databases of different database types.

7. A data synchronization device, characterized in that, include: The determination module is used to determine the base transaction sequence number of the target transaction in the source database; wherein, the source database includes multiple physical nodes; The scanning module is used to scan all archived logs of the target transaction in the source database to obtain all archived files in the archived logs whose transaction sequence number is greater than the baseline transaction sequence number; The grouping module is used to group the archived files according to their corresponding physical nodes, and sort the physical nodes according to a preset node arrangement strategy to obtain a node queue. The calculation module is used to perform density calculation on each archive file in the node queue based on a preset file density model, and to perform fragmentation processing on each archive file according to the density calculation results. The migration module is used to migrate the fragmented archive files to the target database in sequence according to the arrangement order of the physical nodes in the node queue. The process of calculating the density of each archived file in the node queue based on a preset file density model, and then fragmenting each archived file according to the density calculation results, includes: Configuration file density model and file fragmentation strategy; Using the file density model, the density of each archived file in the node queue is calculated to generate a density calculation result; wherein, the file density is calculated by the following formula: AFD = Normalized Density Function (File Size / SCN Interval Size), where AFD represents the density of the archived file; Based on the density calculation results and the file fragmentation strategy, each archived file is fragmented; wherein, if AFD > 0.7, fine-grained fragmentation is triggered; if AFD ≤ 0.3, coarse-grained fragmentation is triggered; if 0.3 < AFD ≤ 0.7, the default fragmentation parameters remain unchanged.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the data synchronization method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the data synchronization method as described in any one of claims 1-6.

Images