Data synchronization method, device, equipment, medium and program product

Abstract

The invention provides a data synchronization method and device, equipment, a medium and a program product, and relates to the field of big data. Receiving remote log transmission requests of a plurality of synchronization services through a log agent, requesting a log file from the source end database through the log agent based on the log analysis breakpoint, receiving a target log file from the source end database, and storing the target log file in a local log storage unit, the plurality of synchronization services share and analyze; according to the method, the log requests of the multiple synchronization services are received in a centralized mode through the log agent, the target log files are uniformly obtained from the source end database based on respective analysis breakpoints and locally cached, multi-service efficient sharing and collaborative analysis are achieved, repeated pulling is avoided, the database load is reduced, and the synchronization efficiency is improved.

Description

Data synchronization methods, devices, equipment, media and program products Technical Field

[0001] This application relates to the field of big data, and in particular to a data synchronization method, apparatus, device, medium, and program product. Background Technology

[0002] In scenarios where incremental data synchronization is achieved using archived logs and online logs of a database, in order to reduce the resource consumption of the server where the database resides, there is often a need for multi-target synchronization in actual business: that is, the data of multiple databases in the source database instance needs to be synchronized to different target data storage.

[0003] For multi-database synchronization scenarios, the existing log broker solution works as follows: since a single log broker can only communicate with one synchronization service, a log broker needs to be deployed for each synchronization service. Each log broker collects log files from the database side and then provides them to its respective associated synchronization service for parsing, ultimately achieving parallel incremental synchronization of multiple databases.

[0004] However, the existing technology of multiple log agent clients collecting log files from the database server in parallel will cause the CPU, IO, bandwidth and other resources of the source database to increase exponentially, which will seriously affect the normal operation of the source database. Summary of the Invention

[0005] This application provides a data synchronization method, apparatus, device, medium, and program product to solve the technical problem that in the prior art, the parallel collection of log files from a database server by multiple log agent clients causes a significant increase in the CPU, IO, and bandwidth resources of the source database, thereby seriously affecting the normal operation of the source database.

[0006] In a first aspect, this application provides a data synchronization method, the method comprising:

[0007] The log agent receives remote log transmission requests from multiple synchronization services; wherein, the log agent is a proxy program that supports communication with multiple synchronization services simultaneously, the log agent is connected to the source database, and the request includes log parsing breakpoints for each of the synchronization services, the log parsing breakpoints being used to indicate the starting position of the logs that the synchronization service currently needs to parse;

[0008] Based on the log parsing breakpoint, the log agent requests the log file from the source database.

[0009] Receive the target log file from the source database;

[0010] The target log file is stored in a local log storage unit for sharing and parsing by the multiple synchronization services.

[0011] In one possible implementation, the step of requesting log files from the source database through the log broker based on the log parsing breakpoint includes:

[0012] Based on the range of the parsing breakpoints of the multiple synchronization services, determine the range of log files to be requested;

[0013] A log file request is sent to the source database, the log file request containing the range of log files to be requested.

[0014] In one possible implementation, the method further includes:

[0015] New registration activity for the synchronization service has been detected.

[0016] Obtain the log parsing breakpoint for the new synchronization service;

[0017] Update the log file range in the log file request based on the parsing breakpoint of the new synchronization service.

[0018] In one possible implementation, storing the target log file in a local log storage unit includes:

[0019] Based on the log parsing breakpoints of the multiple synchronization services, filter out the log files required by the multiple synchronization services;

[0020] The selected log files are stored in the local log storage unit.

[0021] In one possible implementation, the step of filtering out the log files required by the plurality of synchronization services based on the log parsing breakpoints of the plurality of synchronization services includes:

[0022] Record the log parsing breakpoints of each of the aforementioned synchronization services in the global breakpoint mapping table;

[0023] The filtering criteria are determined based on the minimum breakpoint value in the global breakpoint mapping table.

[0024] The log files required for the multiple synchronization services are selected based on the filtering criteria.

[0025] In one possible implementation, the method further includes:

[0026] Perform hash verification on the log files stored in the local log storage unit to generate a verification identifier;

[0027] When any of the synchronization services requests a log file stored in the local log storage unit, the verification identifier is provided to the synchronization service.

[0028] In one possible implementation, the method further includes:

[0029] Based on the historical analysis of breakpoint trends, the future log file requirements of the multiple synchronization services are predicted to obtain the preloaded log files;

[0030] The preloaded log file is stored in the local log storage unit.

[0031] Secondly, this application provides a data synchronization device, comprising:

[0032] A receiving module is used to receive remote log transmission requests from multiple synchronization services through a log proxy; wherein, the log proxy is a proxy program that supports communication with multiple synchronization services simultaneously, the log proxy is connected to the source database, and the request includes log parsing breakpoints for each of the synchronization services, the log parsing breakpoints being used to indicate the starting position of the logs that the synchronization service currently needs to parse;

[0033] The request module is used to request log files from the source database through the log broker based on the log parsing breakpoint.

[0034] The receiving module is also used to receive target log files from the source database;

[0035] The sharing module is used to store the target log file in a local log storage unit for sharing and parsing by the multiple synchronization services.

[0036] In one possible implementation, the data synchronization device further includes: a determining module and a sending module;

[0037] The determination module is used to determine the range of log files to be requested based on the range of the parsing breakpoints of the multiple synchronization services;

[0038] The sending module is used to send a log file request to the source database, wherein the log file request includes the range of log files to be requested.

[0039] In one possible implementation, the data synchronization device further includes: a detection module, an acquisition module, and an update module;

[0040] The detection module is used to detect the registration behavior of new synchronization services;

[0041] The acquisition module is also used to acquire the log parsing breakpoint of the new synchronization service;

[0042] The update module is used to update the log file range in the log file request based on the parsing breakpoint of the new synchronization service.

[0043] In one possible implementation, the data synchronization device further includes: a filtering module and a storage module;

[0044] The filtering module is used to filter out the log files required by the multiple synchronization services based on the log parsing breakpoints of the multiple synchronization services.

[0045] The storage module is used to store the selected log files in the local log storage unit.

[0046] In one possible implementation, the data synchronization device further includes: a recording module;

[0047] The recording module is used to record the log parsing breakpoint of each of the synchronization services in the global breakpoint mapping table;

[0048] The determining module is used to determine the filtering conditions based on the minimum breakpoint value of the global breakpoint mapping table.

[0049] The filtering module is also used to filter out the log files required by the multiple synchronization services according to the filtering conditions.

[0050] In one possible implementation, the data synchronization device further includes: a verification module and a providing module;

[0051] The verification module is used to perform hash verification on the log files stored in the local log storage unit and generate a verification identifier.

[0052] The providing module is configured to provide the verification identifier to any of the synchronization services when any of the synchronization services requests a log file stored in the local log storage unit.

[0053] In one possible implementation, the data synchronization device further includes: a prediction module;

[0054] The prediction module is used to predict the future log file requirements of the multiple synchronization services based on the historical parsing breakpoint trends, and to obtain preloaded log files.

[0055] The storage module is also used to store the preloaded log file in the local log storage unit.

[0056] Thirdly, embodiments of this application provide a data synchronization device, including: a memory and a processor;

[0057] The memory stores computer-executed instructions;

[0058] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0059] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0060] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0061] The data synchronization method provided in this application effectively solves the resource redundancy and performance bottleneck problems caused by multiple synchronization services independently requesting logs from the source database by introducing a log broker as an intermediate coordination layer. The log broker can simultaneously receive requests from multiple synchronization services carrying their log parsing breakpoints, and based on these breakpoints, uniformly pull the required log files from the source database. The acquired logs are then centrally cached in local storage for shared parsing by all services. This reduces the number of repeated accesses to the source database, not only lowering database I / O pressure and network load but also improving overall synchronization efficiency and system scalability. Simultaneously, the breakpoint mechanism ensures that each synchronization service can continue parsing from the correct position, guaranteeing the accuracy and consistency of data synchronization. Attached Figure Description

[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0063] Figure 1 is a flowchart illustrating the data synchronization method provided in this application.

[0064] Figure 2 is a flowchart illustrating the data synchronization method provided in this application.

[0065] Figure 3 is a flowchart illustrating the data synchronization method provided in this application.

[0066] Figure 4 is a schematic diagram of the data synchronization device provided in this application;

[0067] Figure 5 is a schematic diagram of the data synchronization device provided in this application.

[0068] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0069] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0070] It should be noted that the data synchronization method, apparatus, equipment, medium and program products provided in this application can be used in the field of big data, or in any field other than big data. This application does not limit the application field of the data synchronization method, apparatus, equipment, medium and program products.

[0071] In incremental data synchronization based on database archive logs and online logs, to minimize the performance impact on the source database server, it is often necessary to simultaneously synchronize data from multiple source databases to different target storage systems. This multi-target synchronization requirement necessitates that the system can efficiently and with low interference process log data from multiple databases within the same database instance.

[0072] In existing technologies, log broker solutions typically adopt a "one-to-one" architecture: each synchronization task needs to be equipped with an independent log broker, which is responsible for pulling logs from the database server and passing them to the corresponding synchronization service for parsing and writing.

[0073] However, while this approach achieves parallel synchronization across multiple databases, its architecture essentially involves repeatedly deploying multiple data collection clients. Since each log broker independently accesses the database server to read log files, the concurrent operation of multiple brokers leads to a significant increase in the consumption of critical resources such as CPU, disk I / O, and network bandwidth. This resource contention not only reduces synchronization efficiency but also interferes with the normal business processing of the source database.

[0074] To address the aforementioned issues, the data synchronization method provided in this application introduces a multi-connection-supporting "log broker" as an intermediate coordination layer to uniformly receive remote log transmission requests from multiple synchronization services. Each request carries its current log parsing breakpoint (i.e., the position already processed). Based on this, the log broker requests the corresponding log file from the source database and caches the obtained target log file in the local log storage unit. Each synchronization service can then read the required logs from this shared storage for parsing. Through this mechanism of centralized request, unified retrieval, local caching, and multi-service sharing, the redundant retrieval and resource contention caused by multiple synchronization services independently and frequently accessing the source database are effectively avoided, reducing the load pressure on the source database and improving the overall efficiency and coordination of data synchronization.

[0075] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0076] Figure 1 is a flowchart illustrating the data synchronization method provided in this application. The execution entity in this embodiment is, for example, a data synchronization system. As shown in Figure 1, the method includes:

[0077] S101: Receive remote log transmission requests from multiple synchronization services through the log broker; wherein, the log broker is a proxy program that supports communication with multiple synchronization services simultaneously. The log broker communicates with the source database and the request includes the log parsing breakpoints of each synchronization service. The log parsing breakpoints are used to indicate the starting position of the logs that the synchronization service needs to parse.

[0078] The log broker acts as a communication bridge between the synchronization service and the source database, undertaking middleware functions such as request forwarding, log reception and distribution, which can avoid connection congestion caused by multiple synchronization services interacting directly with the source database.

[0079] Synchronization services refer to business service programs responsible for log parsing and data synchronization. Different synchronization services may correspond to different business scenarios (such as data backup, real-time monitoring, and data analysis), and require obtaining operation data from the source database through log parsing.

[0080] The remote log transmission request is initiated by the synchronization service and sent to the log broker as a request instruction to retrieve log data, which includes key information such as the current log parsing breakpoint of the synchronization service.

[0081] The source database refers to the original database that generates log data. It can be a relational database (such as a database) or a non-relational database. Its log files record various database operations (such as adding, deleting, modifying, querying, and committing transactions), and it is the core data source for the synchronization service to obtain data.

[0082] Log parsing breakpoints are used to mark the end point of the last log parsing in the synchronization service. They are usually in the form of log timestamps, log sequence numbers, or offsets. They ensure that the synchronization service can continue parsing from the breakpoint and avoid duplicate or missed log parsing.

[0083] Specifically, firstly, the log broker needs to be built on a high-concurrency communication framework, configuring a multi-threaded listening mechanism and connection pool management strategy to ensure it can simultaneously receive TCP / UDP connection requests from multiple synchronization services. The connection pool can be dynamically scaled to handle fluctuations in the number of synchronization services. Secondly, the log broker needs to establish a persistent communication connection with the source database, which can be achieved through JDBC connection pools, database client SDKs, etc. Database connection parameters (such as address, port, username, password, and timeout) should be configured in advance to ensure stable interaction with the source database.

[0084] When the synchronization service initiates a remote log transmission request, it must encapsulate the request message according to a preset protocol (such as HTTP or a custom TCP protocol). In addition to containing log parsing breakpoints, the message must also carry auxiliary information such as the synchronization service identifier and the time range of the requested logs. After receiving the request, the log broker first verifies the validity of the request message (e.g., verifying the identity of the synchronization service and the validity of the breakpoint format). If the verification passes, the request is temporarily stored in the request queue, awaiting subsequent log requests to the source database. If the verification fails, an error response is returned, informing the synchronization service to correct the error and re-initiate the request.

[0085] S102: Based on log parsing breakpoints, request log files from the source database through the log broker.

[0086] Log files refer to files generated by the source database that record the operation trajectory. Different database types have different log file formats and contain key information such as operation time, operation object, operation content, and log offset.

[0087] Specifically, the log broker extracts request information for the synchronization service from the request queue, parses the logs, and identifies the type and value of the breakpoints. If the breakpoint is a timestamp, it is converted to a time format recognizable by the source database; if the breakpoint is a log offset or sequence number, the corresponding value is extracted directly. Subsequently, based on the type of the source database, a request instruction conforming to its log query specifications is constructed.

[0088] To improve request efficiency and stability, the log broker can employ a batch request strategy, aggregating log requests from multiple synchronization services based on factors such as the source database partition and log file type. This avoids the performance overhead caused by frequently sending single requests to the database. Simultaneously, a request timeout retry mechanism should be configured. If the initial request fails due to network fluctuations, database overload, or other reasons, it can be retried within a set time (e.g., 3 seconds). The number of retries can be configured as needed (e.g., 3 times). If the number of retries is exceeded, an error log is recorded, and a request failure response is returned to the corresponding synchronization service. Furthermore, the log broker needs to maintain a request session with the source database to ensure continuous log data acquisition.

[0089] S103: Receive the target log file from the source database.

[0090] The target log file refers to the set of log data that meets the log parsing breakpoint conditions, which is selected by the source database based on the request of the log broker. It may be a single complete log file fragment or a combination of multiple log files, and its content precisely corresponds to the log range that the synchronization service needs to continue parsing.

[0091] Specifically, after receiving a request from the log broker, the source database filters the target log data according to the request parameters. After filtering, the target log file is transmitted to the log broker via a preset transmission protocol (such as TCP). The log broker needs to deploy a dedicated log receiving module. This module uses an asynchronous receiving mechanism to avoid request blocking caused by synchronous receiving. At the same time, it is configured with a buffer to temporarily store the received log data to prevent data loss due to fluctuations in network transmission rate.

[0092] During the reception process, the log broker needs to perform integrity verification on the target log file. Verification methods can include file verification and data length verification, which involves comparing the verification information sent by the source database with the verification information calculated locally after reception. If they match, the reception is considered complete; if they do not match, a retransmission request is sent to the source database, specifying the missing or erroneous data segments. After successful verification, the log broker performs preliminary formatting on the target log file, standardizing the log encoding format, field separators, etc., to facilitate subsequent storage and synchronization service parsing. Simultaneously, it records the log reception log, including the reception time, the corresponding synchronization service identifier, the log file size, and breakpoint information.

[0093] S104: Store the target log file in the local log storage unit for multiple synchronization services to share and parse.

[0094] The local log storage unit refers to the storage component deployed on the server or cluster where the log agent is located. It can be a local file system, distributed file storage, cache database, etc., used to persistently store the received target log files and provide high-concurrency read access capabilities.

[0095] Shared parsing refers to the ability of multiple synchronization services to access the same batch of log files in the local log storage unit simultaneously. Each service performs parsing operations based on its own log parsing breakpoint, eliminating the need for the log broker to repeatedly request logs from the source database, thus improving parsing efficiency and resource utilization.

[0096] Specifically, firstly, if there are many synchronization services and a large volume of log data, distributed file storage can be used, with data sharding and replication mechanisms to ensure storage reliability and high availability. Secondly, if the data volume is small and the access frequency is low, a local file system can be used, organized by a directory structure of "source database identifier-date-log type" for easy quick location of log files. During storage, metadata information needs to be added to each log file, including the corresponding log parsing breakpoint range, reception time, and a list of associated synchronization services, for use in synchronization service queries and access control.

[0097] After the log agent completes the storage of log files, it sends a log-ready notification to the corresponding synchronization service. This notification includes the access path of the log files in the local storage unit, metadata information, etc. Upon receiving the notification, the synchronization service accesses the local log storage unit through the access interface provided by the log agent and reads and parses the corresponding log segments based on its own log parsing breakpoints. Simultaneously, the local log storage unit needs to be configured with an access control mechanism to allow only authorized synchronization services to access the corresponding log files, preventing data leakage; a log expiration cleanup policy should also be configured to periodically delete log files that have been parsed and no longer need to be retained, freeing up storage resources. Cleanup rules can be set based on dimensions such as log storage duration and file size.

[0098] The data synchronization method provided in this embodiment receives remote log transmission requests from multiple synchronization services through a log proxy. The log proxy is a proxy program that supports simultaneous communication with multiple synchronization services. The log proxy communicates with the source database and requests log parsing breakpoints for each synchronization service. These breakpoints indicate the starting position of the logs that the synchronization service needs to parse. Based on these breakpoints, the log proxy requests log files from the source database, receives the target log file from the source database, and stores the target log file in a local log storage unit for shared parsing by multiple synchronization services. This method centrally receives log requests from multiple synchronization services through a log proxy, uniformly retrieves and locally caches the target log file from the source database based on its respective parsing breakpoint, achieving efficient sharing and collaborative parsing among multiple services, avoiding redundant fetching, reducing database load, and improving synchronization efficiency.

[0099] Figure 2 is a flowchart illustrating the data synchronization method provided in this application. As shown in Figure 2, this embodiment describes the data synchronization method in detail based on the embodiment in Figure 1. The method includes:

[0100] S201: Receive remote log transmission requests from multiple synchronization services through the log broker; wherein, the log broker is a proxy program that supports communication with multiple synchronization services simultaneously. The log broker communicates with the source database and the request includes the log parsing breakpoints of each synchronization service. The log parsing breakpoints are used to indicate the starting position of the logs that the synchronization service needs to parse.

[0101] Step S201 is similar to step S101, and will not be described again here.

[0102] S202: Determine the range of log files to be requested based on the range of parsing breakpoints of multiple synchronization services.

[0103] The range of log files to be requested is the interval corresponding to the set of log files that need to be obtained from the source database, which is determined based on the range of parsing breakpoints. This range includes the minimum set of logs required by all synchronization services to avoid duplicate requests or omissions.

[0104] Specifically, the log broker client extracts the System Change Number (SCN) of all registered synchronization service resolution breakpoints from the multi-service resolution progress management module, and filters out the minimum SCN (SCN-min) and maximum SCN (SCN-max) using a numerical sorting algorithm. Then, the log broker client queries the mapping relationship between the source database log files and SCN numbers recorded in its local cache (this mapping relationship is preset by historical requests or configuration files and includes the SCN range corresponding to each log file), and compares SCN-min and SCN-max with the SCN range of the existing log files locally.

[0105] If the SCN range of the existing local log files cannot completely cover [SCN-min, SCN-max], then the missing SCN sub-intervals are calculated and mapped to the corresponding range of log files to be requested. If the local logs already completely cover this interval, then there is no need to determine a new range to be requested. At the same time, the determined range of log files to be requested is stored in the request queue, waiting to be requested to the source Proxy Server (PS).

[0106] S203: Send a log file request to the source database. The log file request includes the range of log files to be requested.

[0107] Specifically, the log proxy client extracts the range of log files to be requested from the request queue and encapsulates the log file request message according to a preset communication protocol (a private protocol or standard protocol agreed upon with the proxy server). The message must include parameters such as the log proxy client identifier, the file name to be requested, the SCN start value, the SCN end value, and the request timeout period. Subsequently, the log proxy client sends the request through a persistent connection established with the source proxy server to ensure communication stability.

[0108] After sending, the log proxy client starts a timeout listening mechanism. If no response is received within the preset timeout period (e.g., 30 seconds), a retry mechanism is triggered. The preset number of retries is 3, with a 5-second interval between each retrieval. If the retry fails, an error log is recorded and the request is marked as failed, and it will be re-initiated periodically thereafter. If a response is received from the proxy server, the response content is first parsed to confirm whether the request is accepted (e.g., whether the proxy server can obtain the corresponding log file). If accepted, the client waits to receive the log; if rejected, the request parameters are corrected based on the error information in the response, and the request is resent.

[0109] S204: A new registration for a synchronization service has been detected.

[0110] The registration process involves a new synchronization service sending a request to the log broker client containing its own identifier, log parsing breakpoints, and other information to complete identity verification and information entry, and then adding itself to the list of synchronization services managed by the log broker client.

[0111] Specifically, the log broker client continuously listens for incoming requests on the RPC (Remote Procedure Call) communication port. It uniquely identifies the service by using the synchronization service identifier in the request message: if this identifier does not exist in the synchronization service list of the multi-service parsing progress management module, it is determined to be a new synchronization service registration. Subsequently, the registration process is triggered. First, the new synchronization service is authenticated (using a preset key or whitelist verification). If authentication is successful, its identifier, log parsing breakpoints, and other information are entered into the synchronization service list, and a unique registration ID (Identifier) ​​is assigned. If authentication fails, registration is rejected, and an authentication failure response is returned.

[0112] Simultaneously, the log broker client records registration logs, including registration time, new synchronization service identifier, registration ID, and parsing breakpoint information, and returns a registration success response to the new synchronization service, informing it that subsequent log query and parsing requests can be initiated normally. Furthermore, the registration action triggers the log broker client's pending log file range re-verification mechanism, preparing for subsequent log file update requests.

[0113] S205: Obtain the log parsing breakpoint for the new synchronization service.

[0114] Specifically, after a new synchronization service successfully registers, the log proxy client extracts the log parsing breakpoint SCN number from its registration request message. If the message does not carry the SCN number or the SCN number format is invalid, a supplementary request is sent to the new synchronization service, requesting it to resend a valid SCN number. After obtaining a valid SCN number, the log proxy client verifies its validity, checking whether the SCN number is within the valid SCN range of the source database (by initiating an SCN validity query to the proxy server) and whether it is less than the latest SCN number in the current database (to avoid requesting logs that have not yet been generated).

[0115] Upon successful verification, the identifier of the new synchronization service and its corresponding SCN number are associated and stored in the cache of the multi-service parsing progress management module, updating the "synchronization service-parsing breakpoint" mapping relationship. If verification fails, an invalid SCN number response is returned, informing the new synchronization service to correct and resubmit. Simultaneously, the obtained SCN number is marked as a key parameter for participating in the log file range calculation.

[0116] S206: Update the log file range in the log file request based on the new synchronization service's parsing breakpoint.

[0117] Specifically, the log broker client extracts the SCN numbers of all synchronization service breakpoints after the update, and recalculates SCN-min and SCN-max (the new SCN-min is the smaller of the original SCN-min and the new synchronization service SCN number, and the new SCN-max is the larger of the original SCN-max and the new synchronization service SCN number). Then, it compares the new [SCN-min, SCN-max] with the SCN range corresponding to the original pending log file range to determine if range expansion exists.

[0118] If an expansion exists (e.g., the new SCN-min is less than the original SCN-min, or the new SCN-max is greater than the original SCN-max), the mapping relationship between the local log file and the SCN is queried again to determine the expanded range of log files to be requested. A new log file request message is generated to replace any unsent requests or to send a supplementary request to the origin proxy server. If the range is not expanded, there is no need to update the log file request. After the update is complete, the new range of log files to be requested is stored in the request queue, overwriting the original range record.

[0119] S207: Receive the target log file from the source database.

[0120] Specifically, the source proxy server, based on the request from the log proxy client, reads the corresponding range of log files from the database's file system and sends the target log files in segments to the log proxy client using a preset transport protocol (such as TCP). The log proxy client starts an asynchronous receiving module, opening an independent receiving buffer to avoid blocking other business processes during the receiving process. During the receiving process, each segment of log data is verified, including data integrity verification (through MD5 checksum comparison) and SCN range consistency verification (ensuring that the received log SCN is within the requested range).

[0121] If the verification passes, the segmented data is temporarily stored in the buffer; if the verification fails, a retransmission request is sent to the Proxy Server, specifying the missing or incorrect segments. After all segments have been received, the Log Proxy Client merges the segmented data into a complete target log file and marks the reception status as "complete." If a connection interruption occurs during reception, the connection is re-established and reception resumes from the point of interruption. After reception is complete, a reception log is recorded, including information such as reception time, target log file name, SCN range, and file size.

[0122] S208: Based on the log parsing breakpoints of multiple synchronization services, filter out the log files required by multiple synchronization services.

[0123] Specifically, the log broker client extracts the SCN number of the log parsing breakpoint for each synchronization service from the multi-service parsing progress management module, traverses the received target log files, and parses the SCN number corresponding to each log record in the log files. For each synchronization service, it selects log segments whose SCN numbers are greater than their parsing breakpoint SCN numbers, and generates a unique log segment index for that synchronization service. The index contains information such as the start and end positions of the log segment in the target log file and the SCN range.

[0124] If the dedicated log segments corresponding to the parsing breakpoints of multiple synchronization services overlap (e.g., the breakpoints of multiple synchronization services are all less than the minimum SCN number of the target log file), only one copy of the target log file is retained, and the dedicated segments of different synchronization services are pointed to by indexes, avoiding redundant storage of log data. After filtering, the dedicated log segment index of each synchronization service is stored in the cache for subsequent log file storage and synchronization service queries.

[0125] Optionally, based on log parsing breakpoints of multiple synchronization services, the required log files for multiple synchronization services can be filtered out. Specific implementation methods include:

[0126] Record the log parsing breakpoints for each synchronization service in the global breakpoint mapping table;

[0127] The filtering criteria are determined based on the minimum breakpoint value in the global breakpoint mapping table.

[0128] Select the log files required for multiple synchronization services based on the filtering criteria.

[0129] Specifically, firstly, the log broker client maintains a global breakpoint mapping table. When each synchronization service initiates a log request, its identifier (e.g., Sync-001) and the corresponding log parsing breakpoint SCN number (e.g., 123456) are recorded in the table, and the breakpoint data in the table is updated in real time. Then, the SCN numbers of all synchronization services are extracted from the global breakpoint mapping table, and the minimum breakpoint value is obtained by numerical sorting (e.g., when multiple service breakpoints are 123450, 123456, and 123458, the minimum breakpoint is 123450), which is used as a filtering condition (i.e., filtering logs with an SCN number ≥ the minimum breakpoint value). Finally, the log broker client traverses the locally received log files, parses the SCN range of each log file, filters out log files that cover the minimum breakpoint value and the subsequent range, and marks the unique log fragments for each synchronization service (e.g., in the log files corresponding to the minimum breakpoint 123450, Sync-001 only needs fragments with an SCN ≥ 123456), thus obtaining the set of log files required by all synchronization services.

[0130] S209: Store the selected log files in the local log storage unit.

[0131] Specifically, the local log storage unit adopts a storage structure of "directory classification + metadata association". Directories are created hierarchically according to "source database identifier - date - log type", and the filtered target log files are stored in the corresponding directories. Simultaneously, metadata is created for each log file, including the filename, SCN range, storage path, reception time, a list of associated synchronization services, and a dedicated fragment index for each service. This metadata is stored in the storage unit's metadata database for easy retrieval and management.

[0132] During storage, a data backup mechanism is initiated to create copies of the stored log files (such as local multi-disk backups or distributed storage copies) to prevent log file loss. After storage is complete, the file index of the local log storage unit is updated, the log file status is marked as "accessible," and the synchronization service notification mechanism is triggered to inform the relevant synchronization services that the log files are now parseable.

[0133] S210: For multiple synchronization services to share resolution.

[0134] Specifically, the log broker client sends a log-ready notification to all associated synchronization services. This notification includes information such as the dedicated log fragment index for each synchronization service and the access interface for its local log storage unit (e.g., RPC read interface, file system access path). Upon receiving the notification, the synchronization service initiates a log read request to the log broker client through a pre-defined access interface, carrying its own identifier and dedicated fragment index information in the request.

[0135] After receiving a read request, the Log Broker Client extracts the corresponding log fragment from the local log storage unit based on the synchronization service identifier and index information, and transmits the fragment data to the synchronization service via the RPC protocol. If the synchronization service requires batch parsing, it can support reading log fragments in batches. Simultaneously, the Log Broker Client records the reading progress of each synchronization service and updates the cache of the multi-service parsing progress management module.

[0136] S211: Based on the historical breakpoint trends, predict the future log file requirements of multiple synchronization services to obtain preloaded log files.

[0137] Among them, preloaded log files refer to log files that multiple synchronization services may need to parse in the future based on the historical parsing breakpoint trend prediction. These log files are obtained and stored in advance from the source database to improve the parsing response speed of subsequent synchronization services.

[0138] Specifically, the log broker client extracts historical parsing breakpoint data for each synchronization service from the multi-service parsing progress management module. The time span can be preset (e.g., the last 24 hours). It analyzes historical parsing breakpoint trends using data statistical algorithms: calculating the average and maximum parsing rates for each synchronization service, and identifying peak parsing periods (e.g., 10:00-12:00 daily). Combining the latest parsing breakpoint SCN number for each synchronization service with the log generation rate of the source database, it predicts the required log SCN range for each synchronization service within a preset time period (e.g., 1 hour).

[0139] The predicted SCN ranges of each synchronization service are merged to obtain the overall future log demand range. This range is used as the preload request parameter to initiate a preload log file request to the source proxy server. After receiving the preload log file, a preload-specific fragment index for each synchronization service is generated according to the filtering rules in step S208, completing the initial processing of the preload log file, which is then ready for subsequent storage.

[0140] S212: Store the preloaded log file in the local log storage unit.

[0141] Specifically, create independent directories (distinguished from received non-preloaded log files) according to the hierarchy of "source database identifier - prediction date - preloading - log type", and store the preloaded log files in the corresponding directories. Create metadata for the preloaded log files. In addition to regular information, the metadata includes additional information such as the "preload" attribute and prediction validity period (e.g., 1 hour), and store it in the metadata database.

[0142] Simultaneously, the preloaded dedicated fragment index and metadata of each synchronization service are associated and stored, the file index of the local log storage unit is updated, and the status of the preloaded log file is marked as "preload ready". If the preloaded log file is not accessed by the synchronization service within the predicted validity period, or exceeds the validity period, a cleanup mechanism is triggered to delete the preloaded file and release storage resources; if it is accessed, its status is updated to "used" and it is included in the regular log file management.

[0143] The data synchronization method provided in this embodiment deploys a log broker that supports concurrent communication with multiple synchronization services. This broker uniformly receives remote log transmission requests from each synchronization service, each request containing its current log parsing breakpoint to identify the starting position of the required logs. Based on the parsing breakpoint ranges of all synchronization services, the log broker aggregates and determines the minimum range of log files to be retrieved from the source database and initiates corresponding log file requests to the source database. When a new synchronization service registers, the log broker dynamically obtains its parsing breakpoint and expands or adjusts the existing log file request range accordingly. After receiving the target log file returned by the source database, the log broker filters out the required log content based on the parsing breakpoints of each synchronization service and stores these log files uniformly in a local log storage unit for shared use by multiple synchronization services. Furthermore, this method predicts the future log needs of each synchronization service based on the changing trends of historical parsing breakpoints, pre-loading potentially needed log files and storing them in local storage, thereby improving log synchronization efficiency and system response performance.

[0144] Figure 3 is a flowchart illustrating the data synchronization method provided in this application. As shown in Figure 3, this embodiment, based on the embodiment in Figure 2, provides a detailed description of verifying the integrity of the relevant dump files recorded in the immutable data table. The method includes:

[0145] S301: Perform hash verification on the log files stored in the local log storage unit and generate a verification identifier.

[0146] Hash verification is an integrity verification method based on hash algorithms (such as MD5 and SHA-256). It calculates a unique hash value by analyzing the binary data of the log file, which is used to verify whether the log file has been tampered with, damaged, or lost during transmission.

[0147] The verification identifier is a unique hash value generated after hash verification. It is a "digital fingerprint" of log file integrity. Different log files with different content correspond to different verification identifiers, while log files with the same content have a unique and fixed verification identifier.

[0148] Specifically, the log broker client starts an independent hash verification thread, employing a dual verification mechanism of timed triggering and event triggering. Timed triggering can traverse all log files (including regular log files and preloaded log files) in the local log storage unit at a preset period (e.g., every 5 minutes). Event triggering is triggered immediately after a log file is stored or a file change is detected (e.g., rewriting or supplementing data). During verification, the MD5 hash algorithm is used first (balancing efficiency and uniqueness). If the log file is large (e.g., exceeding 1GB), a segmented hash verification method is used. First, the hash value is calculated for each segment of the file, and then the hash value of each segment is calculated again to obtain the final verification identifier.

[0149] After verification, the log proxy client stores the mapping relationship of "log file path-file name-verification identifier-verification time-hash algorithm type" in the local verification information cache, and updates the log file's metadata, writing the verification identifier into the metadata field. If log file corruption is detected during verification (such as inconsistent segment hash values), the file is marked as "invalid," and a re-acquisition process is triggered (requesting the corresponding log file from the source proxy server); if verification passes, the file is marked as "complete and valid," ensuring that the synchronization service can be accessed securely.

[0150] S302: When any synchronization service requests a log file stored in the local log storage unit, provide a verification identifier to the synchronization service.

[0151] Specifically, when any synchronization service initiates a log file request, the log broker client first parses the request message, extracting the synchronization service identifier and relevant information about the required log file (such as filename and SCN range). Using this information, it queries the local verification information cache for the corresponding verification identifier. If the verification identifier is found, the log broker client constructs a response message containing two core parts: first, the log file data required by the synchronization service (or the log file's access path and dedicated fragment index); and second, the corresponding verification identifier and hash algorithm type.

[0152] If no verification identifier is found (e.g., the log file has not been fully verified or has been marked as "invalid"), the synchronization service will first return a "log file temporarily unavailable" response. Once the log file has been fully verified and marked as "complete and valid," the service will respond again with the verification identifier. Upon receiving the response, the synchronization service can verify the obtained log file using the same hash algorithm, comparing its own calculated hash value with the verification identifier provided by the log broker client to verify the log file's integrity.

[0153] Figure 4 is a schematic diagram of the data synchronization device provided in this application. As shown in Figure 4, the data synchronization device 400 provided in this embodiment includes:

[0154] The receiving module 401 is used to receive remote log transmission requests from multiple synchronization services through a log broker; wherein, the log broker is a proxy program that supports communication with multiple synchronization services at the same time. The log broker is connected to the source database and the request includes the log parsing breakpoints of each synchronization service. The log parsing breakpoints are used to indicate the starting position of the log that the synchronization service needs to parse at the moment.

[0155] Request module 402 is used to request log files from the source database through the log broker based on log parsing breakpoints;

[0156] The receiving module 401 is also used to receive the target log file from the source database;

[0157] The shared module 403 is used to store the target log file in the local log storage unit for multiple synchronization services to share and parse.

[0158] In one possible implementation, the data synchronization device further includes: a determining module 404 and a sending module 405;

[0159] The 404 determination module is used to determine the range of log files to be requested based on the range of parsing breakpoints of multiple synchronization services.

[0160] The sending module 405 is used to send a log file request to the source database. The log file request contains the range of log files to be requested.

[0161] In one possible implementation, the data synchronization device further includes: a detection module 406, an acquisition module 407, and an update module 408;

[0162] Detection module 406 is used to detect the registration behavior of a new synchronization service;

[0163] The 407 module is also used to obtain the log parsing breakpoint of the new synchronization service;

[0164] Update module 408 is used to update the log file range in the log file request based on the new synchronization service's parsing breakpoint.

[0165] In one possible implementation, the data synchronization device further includes: a filtering module 409 and a storage module 410;

[0166] The filtering module 409 is used to filter out the log files required by multiple synchronization services based on the log parsing breakpoints of multiple synchronization services.

[0167] Storage module 410 is used to store the filtered log files in the local log storage unit.

[0168] In one possible implementation, the data synchronization device further includes: a recording module 411;

[0169] Recording module 411 is used to record the log parsing breakpoints of each synchronization service in the global breakpoint mapping table;

[0170] Module 404 is used to determine the filtering criteria based on the minimum breakpoint value in the global breakpoint mapping table.

[0171] The filtering module 409 is also used to filter out the log files required by multiple synchronization services based on the filtering criteria.

[0172] In one possible implementation, the data synchronization device further includes: a verification module 412 and a providing module 413;

[0173] The verification module 412 is used to perform hash verification on the log files stored in the local log storage unit and generate a verification identifier;

[0174] Module 413 is provided to provide a verification identifier to any synchronization service when any synchronization service requests a log file stored in the local log storage unit.

[0175] In one possible implementation, the data synchronization device further includes: a prediction module 414;

[0176] Prediction module 414 is used to predict the future log file requirements of multiple synchronization services based on historical parsing breakpoint trends, and to obtain preloaded log files;

[0177] Storage module 410 is also used to store preloaded log files in a local log storage unit.

[0178] The data synchronization device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0179] Figure 5 is a schematic diagram of the data synchronization device provided in this application. As shown in Figure 5, the electronic device of this embodiment may include: at least one processor 501; and a memory 502 communicatively connected to the at least one processor; wherein the memory 502 stores instructions that can be executed by the at least one processor 501, and the instructions are executed by the at least one processor 501 to cause the electronic device to perform the method as described in any of the above embodiments.

[0180] Optionally, the memory 502 can be either standalone or integrated with the processor 501. When the memory 502 is set up independently, the device also includes a bus for connecting the memory 502 and the processor 501.

[0181] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the foregoing embodiments, and will not be repeated here.

[0182] This application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the methods provided in any of the foregoing embodiments can be implemented.

[0183] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in any of the foregoing embodiments.

[0184] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0185] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0186] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0187] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0188] Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc.

[0189] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0190] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0191] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0192] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data synchronization method, characterized in that, include: The system receives remote log transmission requests from multiple synchronization services via a log proxy. The log proxy is a proxy program that supports simultaneous communication with multiple synchronization services. The log proxy is connected to the source database. Each request includes a log parsing breakpoint for each synchronization service, indicating the starting position of the log that the synchronization service needs to parse. Based on the log parsing breakpoint, the system requests a log file from the source database through the log proxy and receives the target log file from the source database. The target log file is stored in a local log storage unit for sharing and parsing by the multiple synchronization services.

2. The method according to claim 1, characterized in that, The step of requesting log files from the source database through the log broker based on the log parsing breakpoints includes: determining the range of log files to be requested based on the range of parsing breakpoints of the multiple synchronization services; and sending a log file request to the source database, wherein the log file request includes the range of log files to be requested.

3. The method according to claim 2, characterized in that, The method further includes: detecting the registration behavior of a new synchronization service; obtaining the log parsing breakpoint of the new synchronization service; and updating the log file range in the log file request based on the parsing breakpoint of the new synchronization service.

4. The method according to claim 3, characterized in that, The step of storing the target log file in the local log storage unit includes: filtering out the log files required by the multiple synchronization services based on the log parsing breakpoints of the multiple synchronization services; and storing the filtered log files in the local log storage unit.

5. The method according to claim 4, characterized in that, The step of filtering the log files required by the multiple synchronization services based on the log parsing breakpoints of the multiple synchronization services includes: recording the log parsing breakpoint of each synchronization service in a global breakpoint mapping table; determining the filtering criteria based on the minimum breakpoint value of the global breakpoint mapping table; and filtering the log files required by the multiple synchronization services based on the filtering criteria.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: performing hash verification on the log files stored in the local log storage unit to generate a verification identifier; and providing the verification identifier to any of the synchronization services when the service requests a log file stored in the local log storage unit.

7. The method according to any one of claims 1-4, characterized in that, The method further includes: predicting the future log file requirements of the multiple synchronization services based on historical parsing breakpoint trends to obtain preloaded log files; and storing the preloaded log files in the local log storage unit.

8. A data synchronization device, characterized in that, include: A receiving module is configured to receive remote log transmission requests from multiple synchronization services via a log proxy; wherein the log proxy is a proxy program that supports simultaneous communication with multiple synchronization services, the log proxy is connected to the source database, and the request includes log parsing breakpoints for each of the synchronization services, the log parsing breakpoints indicating the starting position of the logs that the synchronization service currently needs to parse; a request module is configured to request a log file from the source database through the log proxy based on the log parsing breakpoints; the receiving module is further configured to receive a target log file from the source database; and a storage module is configured to store the target log file in a local log storage unit for shared parsing by the multiple synchronization services.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images