Distributed data synchronization system, method and program product

Abstract

The invention relates to the technical field of data synchronization, and discloses a distributed data synchronization system and method and a program product, which can process log information from a plurality of distributed sources through a log acquisition module. Furthermore, the collected distributed log information can be accurately analyzed through a log analysis module and a log collection module, and data deviation caused by analysis errors is avoided. Furthermore, a synchronization strategy can be formulated based on the analyzed accurate target log data through a synchronization strategy formulating module, the consistency of the data in the transmission process is further guaranteed, and the data received by the target system is complete and reliable. Therefore, by implementing the method, the systematicness and the collaboration of data synchronization are greatly improved, and compared with a traditional scattered and integrated-lacking synchronization mode, the problems of disconnection of all links, repeated labor, error accumulation and the like are effectively avoided, and it is ensured that the whole synchronization process is efficient and stable.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a distributed data synchronization system, method and program product. Background Art

[0002] In today's digital age, distributed systems are widely used in various fields, such as cross-border transaction processing in the financial field, massive data storage and distribution in the Internet industry, order and inventory management in the e-commerce field, etc. With the continuous expansion and complexity of business, the demand for distributed data synchronization is becoming more and more urgent. However, traditional distributed data synchronization systems have many shortcomings in terms of timeliness, transmission consistency, scalability, resource consumption, and flexibility. Summary of the invention

[0003] In view of this, the present invention provides a distributed data synchronization system, method and program product to solve the problems existing in the traditional distributed data synchronization system in terms of timeliness, transmission consistency, scalability, resource consumption and flexibility.

[0004] In a first aspect, the present invention provides a distributed data synchronization system, the system comprising: a metadata management module, a log collection module, a log parsing module, a synchronization strategy formulation module and a data transmission module;

[0005] A metadata management module is used to obtain multiple log metadata and send the multiple log metadata to a log parsing module; a log parsing module is used to determine log parsing rules based on multiple log metadata and send the log parsing rules to a log collection module; a log collection module is used to collect multiple distributed log information and parse the multiple distributed log information using log parsing rules to obtain multiple target log data, and send the multiple target log data to a synchronization strategy formulation module; a synchronization strategy formulation module is used to formulate a target synchronization strategy based on multiple target log data and send the target synchronization strategy to a data transmission module; a data transmission module is used to send multiple target log data to a target system based on the target synchronization strategy.

[0006] The distributed data synchronization system provided by the present invention obtains multiple log metadata through the metadata management module, and establishes a unified specification and standard for subsequent data processing. Further, the log collection module can process log information from multiple distributed sources, whether it is a change log generated by different types of databases, or an operation log generated by various applications, or a system-level operation log, all of which can be included in the collection category. Further, the log parsing module formulates parsing rules based on accurate metadata and parses the collected multiple distributed log information in the log collection module, which can ensure that the collected distributed log information can be accurately parsed, avoiding data deviations caused by parsing errors. Further, the synchronization strategy formulation module can formulate synchronization strategies based on the accurate target log data after parsing, further ensuring the consistency of data during transmission. Whether it is the structure, content or update order of the data, it can be kept unified in a distributed environment in strict accordance with the predetermined strategy, effectively solving the data inconsistency problem that often occurs in traditional technologies during multi-node synchronization, so that the data received by the target system is complete and reliable. Therefore, by implementing the present invention, by organically integrating metadata management, log collection, log parsing, synchronization strategy formulation and data transmission modules, the systematicness and synergy of data synchronization are greatly improved. Compared with the traditional fragmented and unintegrated synchronization method, it effectively avoids problems such as disconnection between links, duplication of work and accumulation of errors, ensuring that the entire synchronization process is efficient and stable.

[0007] In an optional implementation, the metadata management module includes: a metadata definition submodule and a metadata version management submodule;

[0008] The metadata definition submodule is used to define the data model and data dictionary of the log and form multiple log metadata; the metadata version management submodule is used to manage the target file, and the target file is used to record the change information of multiple log metadata.

[0009] The distributed data synchronization system provided by the present invention, the metadata definition submodule forms log metadata by constructing the data model and data dictionary of the log, and can flexibly cope with diverse log sources and formats. Whether it is a complex database change log or an application log with different formats, it can be standardized based on the precisely defined metadata, so that subsequent parsing and synchronization operations can be accurately implemented, avoiding the processing difficulties caused by differences in log formats, and greatly enhancing the system's compatibility with log data in different business scenarios compared to the traditional fixed format processing method. Further, the metadata version management submodule is responsible for managing the target file that records metadata change information. On the one hand, when data synchronization anomalies occur, the metadata version can be traced back to check what kind of changes caused the problem; on the other hand, for newly connected business modules or upgraded log formats, it can be orderly transitioned according to version management to prevent system crashes caused by metadata confusion, ensuring stable and continuous operation of the system, and making up for the shortcomings of traditional technology that lacks effective metadata management.

[0010] In an optional implementation, the log collection module includes: a plurality of log collection components, a control submodule and a parsing submodule;

[0011] The control submodule is used to load multiple log collection components through the service discovery mechanism and register multiple collection nodes through the distributed coordination service framework; the log collection component is used to monitor and collect multiple distributed log information based on multiple collection nodes, and send multiple distributed log information to the parsing submodule; the parsing submodule is used to parse multiple distributed log information using log parsing rules to obtain multiple target log data.

[0012] The distributed data synchronization system provided by the present invention controls the submodule to load multiple log collection components using a service discovery mechanism, so that the system has extremely strong scalability. With the development of business, if it is necessary to add a collection function for a special data source or a new log format, it is only necessary to develop a collection component that complies with the corresponding interface specification, and the system can automatically discover and load it without large-scale changes to the core architecture. Compared with the traditional model that relies on fixed component configuration and is difficult to expand, it can easily adapt to the ever-changing business environment. Further, registering multiple collection nodes with the help of a distributed coordination service framework can realize functions such as master-slave switching, ensuring the high availability of log collection. In the face of emergencies such as network failures and node hardware damage, the system can quickly switch to the backup node to continue the collection task, maintain the continuous inflow of data, and completely solve the risk of data interruption and loss that is prone to occur in traditional single-point collection. Finally, multiple distributed log information is parsed by the parsing submodule and corresponding multiple target log data are obtained.

[0013] In an optional implementation, the synchronization strategy formulation module includes: a first determination submodule, a second determination submodule, a third determination submodule and a strategy formulation submodule;

[0014] The first determination submodule is used to determine the synchronization target according to the preset data requirements and the data type of multiple target log data, and send the synchronization target to the policy formulation submodule; the second determination submodule is used to determine the synchronization method according to the multiple target log data, and send the synchronization method to the policy formulation submodule; the third determination submodule is used to determine the synchronization frequency according to the preset business requirements and the change frequency of multiple target log data, and send the synchronization frequency to the policy formulation submodule; the policy formulation submodule is used to determine the target synchronization strategy according to the synchronization target, synchronization method and synchronization frequency.

[0015] In the distributed data synchronization system provided by the present invention, the first determination submodule determines the synchronization target based on the preset data requirements and the data type of the target log data, and can tailor the synchronization strategy according to the differentiated demands of different businesses for data consistency and real-time performance. Furthermore, the second determination submodule determines the synchronization method based on the target log data, and can be flexibly adjusted according to the actual data working conditions, significantly improving the data transmission efficiency and system adaptability. Furthermore, the third determination submodule determines the synchronization frequency in combination with the preset business requirements and the change frequency of the target log data, avoiding the problems of insufficient peak resources and idle low-peak resources caused by the fixed-frequency synchronization of traditional technologies, and realizing the reasonable allocation and efficient utilization of system resources.

[0016] In an optional implementation, the synchronization strategy formulation module further includes:

[0017] The conflict handling submodule is used to determine the priority and resolution rules when multiple target log data conflict, and use the priority and resolution rules to handle the conflict.

[0018] The distributed data synchronization system provided by the present invention can determine effective operations by determining priorities and resolution rules in a conflict handling submodule when conflicts occur in multiple target log data, whether the conflicts are caused by concurrent updates of the same data by different nodes due to network delays or conflicts caused by changes in business rules. This ensures global consistency of data and prevents conflicting data from entering the target system and causing business confusion.

[0019] In an optional embodiment, the system further includes:

[0020] The first processing module is used to control the log collection module to collect multiple distributed log information using a preset retry mechanism when multiple target log data are not all sent to the target system; the second processing module is used to control the metadata management module to obtain new multiple log metadata when the log format is updated to the original format.

[0021] In the distributed data synchronization system provided by the present invention, when part of the target log data is not fully sent to the target system, the first processing module uses a preset retry mechanism to control the log collection module to collect distributed log information. In this way, the data is ensured to be successfully transmitted in the end, and the probability of data loss is reduced. Compared with the traditional data synchronization that lacks effective retry means, the reliability and integrity of the transmission are greatly enhanced, and a strong guarantee is provided for common transmission failure scenarios such as network jitter and temporary overload of the target system. Furthermore, when the log format is updated to the original format, the second processing module controls the metadata management module to obtain new log metadata, ensuring that the system can quickly adapt to changes in the log format, avoiding system stagnation caused by incompatible log formats, and maintaining the vitality and adaptability of the system.

[0022] In an optional embodiment, the system further includes:

[0023] The monitoring and alarm module is used to monitor the synchronization process of multiple target log data and use the preset alarm mechanism to alarm when synchronization failure or synchronization delay is found.

[0024] The distributed data synchronization system provided by the present invention monitors the synchronization process of multiple target log data through a monitoring and alarm module and uses a preset alarm mechanism to sound an alarm when synchronization failure or synchronization delay is found, which enables operation and maintenance personnel to quickly locate the fault point with the help of timely alarms. No matter whether it is a jam in the log collection link, a deviation in the execution of the synchronization strategy or a blockage in data transmission, it can intervene and handle it at the first time, thus ensuring the continuity and timeliness of data synchronization and changing the dilemma that traditional technology is difficult to detect synchronization problems in a timely manner.

[0025] In an optional embodiment, the system further includes: a testing module and an optimization module;

[0026] The test module is used to receive the target synchronization strategy sent by the synchronization strategy formulation module, test the target synchronization strategy, obtain the test results and send the test results to the optimization module; the optimization module is used to optimize the target synchronization strategy according to the test results.

[0027] The distributed data synchronization system provided by the present invention continuously tests and discovers problems with the synchronization strategy through the test module and uses the optimization module to perform targeted optimization, so that the system performance is continuously improved and always adapts to business development needs, breaking the deadlock that traditional technologies are difficult to optimize and improve once deployed.

[0028] In a second aspect, the present invention provides a distributed data synchronization method, which is used in the distributed data synchronization system of the first aspect or any corresponding embodiment thereof; the method comprises:

[0029] Acquire multiple log metadata and multiple distributed log information; determine log parsing rules based on multiple log metadata; parse multiple distributed log information using log parsing rules to obtain multiple target log data; formulate target synchronization strategies based on multiple target log data; and send multiple target log data to the target system based on the target synchronization strategies.

[0030] The distributed data synchronization method provided by the present invention synchronously transmits distributed data through a distributed data synchronization system, effectively avoiding problems such as disconnection of various links, duplication of work and accumulation of errors, and ensuring that the entire synchronization process is efficient and stable.

[0031] In a third aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the distributed data synchronization method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 is a structural block diagram of a distributed data synchronization system according to an embodiment of the present invention;

[0034] Figure 2 is a structural block diagram of a metadata management module according to an embodiment of the present invention;

[0035] Figure 3 is a structural block diagram of a log collection module according to an embodiment of the present invention;

[0036] Figure 4 is a structural block diagram of a synchronization strategy formulation module according to an embodiment of the present invention;

[0037] Figure 5 is a flow chart of a distributed data synchronization method according to an embodiment of the present invention;

[0038] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0040] Traditional log collection data synchronization usually has the following limitations:

[0041] 1. Delay issue:

[0042] Batch processing: Traditional methods often use scheduled batch processing, which leads to delays in data synchronization and cannot meet real-time requirements. Polling mechanism: Collecting logs through polling may cause data updates to lag, affecting the timeliness of business decisions.

[0043] 2. Data loss risk

[0044] Network failure: In the event of a network failure or system crash, logs may be lost, especially when processing high-concurrency data, due to the lack of an effective message confirmation mechanism. Single point of failure: When relying on a single node for data collection, if the node fails, data may be lost during the entire data synchronization process.

[0045] 3. Consistency Issues

[0046] Data inconsistency: In a multi-node environment, traditional methods may not be able to ensure that the data of each node is always consistent, especially in scenarios with high concurrent writes. Conflict handling: The lack of an effective conflict detection and resolution mechanism may lead to inconsistent results when multiple nodes update the same data at the same time.

[0047] 4. Poor scalability

[0048] Performance bottleneck: As the amount of data and access volume increases, traditional methods are often difficult to expand and easily form performance bottlenecks, affecting the overall efficiency of the system. Complex architecture: Requires complex configuration and management, increasing the system's operation and maintenance costs.

[0049] 5. High resource consumption

[0050] Computing and storage costs: Batch processing may lead to repeated processing and storage of large amounts of data, increasing computing and storage costs. Log processing capabilities: Traditional log collection tools may not be able to efficiently process massive amounts of data, resulting in uneven resource consumption.

[0051] 6. Lack of flexibility

[0052] Data source limitations: Traditional methods are often not flexible enough in supporting access to data sources and are difficult to adapt to diverse data sources and formats. Poor adaptability: When faced with new business needs or data types, traditional methods require significant changes and are not flexible enough.

[0053] Furthermore, in modern enterprise environments, with the wide variety and dispersion of data sources and service applications, including relational database change logs, system logs, application tracking logs, and other log data, the demand for data analysis and collection to support data analysis, business intelligence, and real-time processing is gradually increasing, and data migration and synchronization are becoming more important. An efficient, stable, flexible, and easy-to-maintain log data synchronization system and method is urgently needed to provide a solution.

[0054] Furthermore, the existing synchronization systems and methods have the following defects and deficiencies:

[0055] 1. Lack of flexibility: When data sources, source log formats or business requirements change, manually coded conversion rules often need to be rewritten or significantly modified, which is time-consuming and labor-intensive. In the face of new data sources or formats, traditional systems and methods require additional development and testing work and are difficult to integrate quickly.

[0056] 2. Difficult to maintain. Standardized log collection rules increase the complexity of the system, lack of centralized management, and increased maintenance costs.

[0057] 3. Insufficient performance and optimization: In many traditional log collection modes, it is difficult to fully utilize the performance of modern multi-core processors. Log parsing logic often lacks systematic optimization, which affects data processing speed and efficiency.

[0058] 4. Lack of real-time monitoring and analysis tools for the log collection process makes it difficult to detect performance bottlenecks and errors in a timely manner.

[0059] 5. Difficult to expand and unable to apply diversified logs, affecting the stability and maintainability of the system.

[0060] 6. Strong dependence. Traditional log collection often relies on multiple log parsing tools or rule engines, which increases the technical dependence of the system.

[0061] 7. Integration is difficult. Integration with other systems and tools often requires additional development and adaptation work, which increases complexity.

[0062] In this embodiment, a distributed data synchronization system is provided. Figure 1As shown, the distributed data synchronization system 1 includes: a metadata management module 11, a log collection module 12, a log parsing module 13, a synchronization strategy formulation module 14, a data transmission module 15, a first processing module 16, a second processing module 17, a monitoring and alarm module 18, a testing module 19 and an optimization module 20.

[0063] Optionally, the metadata management module 11 is used to obtain multiple log metadata and send the multiple log metadata to the log parsing module 13 .

[0064] Among them, Figure 2 As shown, the metadata management module 11 includes: a metadata definition submodule 111 and a metadata version management submodule 112 .

[0065] Optionally, the metadata definition submodule 111 is used to define a data model and a data dictionary of a log and form a plurality of log metadata.

[0066] Specifically, the metadata definition submodule 111 can define the data model and data dictionary of the log according to the pre-set rules. Further, for different types of log metadata, detailed definitions can be made according to business needs and log processing requirements. The specific process may include:

[0067] (1) Timestamp

[0068] Definition: Record the specific time when an event occurs, including date and time information.

[0069] Importance: Timestamps are a critical factor in log analysis, helping to track the sequence and duration of events.

[0070] (2) Log level

[0071] Definition: A classification indicating the importance or urgency of a log message, such as DEBUG, INFO, WARN, ERROR, FATAL, etc.

[0072] Importance: Log levels help users quickly filter and locate key information, facilitating troubleshooting and monitoring.

[0073] (3) Message content

[0074] Definition: The actual text information of the log, describing the specific details of the event.

[0075] Importance: Message content is the core part of the log, providing context and details of the event.

[0076] (4) Source system / application

[0077] Definition: Identification information of the system, service, or application that generated the log.

[0078] Importance: It helps to identify the source of the event and facilitates problem location and tracking in a multi-system environment.

[0079] (5) Request ID / Transaction ID

[0080] Definition: An ID that uniquely identifies a specific operation or transaction.

[0081] Importance: In a distributed system, using request IDs can help track the request flow across services, making it easier to analyze and debug.

[0082] (6) User Information

[0083] Definition: Identity information of the user associated with the event, such as user name, user ID, etc.

[0084] Importance: User information is very important when conducting security audits and user behavior analysis.

[0085] (7) Host information

[0086] Definition: Information about the server or device that generates the log, including host name, IP address, etc.

[0087] Importance: Helps identify the infrastructure of log sources and supports network security and operation management.

[0088] (8) Service version

[0089] Definition: Version information of the application or service that generates the log.

[0090] Importance: When analyzing a problem, knowing the version information can help determine whether it is related to a specific version of the code or configuration.

[0091] (9) Environmental information

[0092] Definition: A label indicating the environment where the logs were generated, such as development, test, production, etc.

[0093] Importance: Helps in troubleshooting and performance analysis in different environments.

[0094] (10) Operation type

[0095] Definition: Describes the specific operation type of the log record, such as create, update, delete, etc.

[0096] Importance: Helps analyze log content and identify the nature and impact of the operation.

[0097] (11) Stack trace information

[0098] Definition: When an error or exception occurs, the call stack information is recorded.

[0099] Importance: Stack traces are important clues for debugging and troubleshooting, and can help developers quickly locate the source of the problem.

[0100] (12) Custom fields

[0101] Definition: Additional fields defined by the user based on specific needs.

[0102] Importance: Log information can be flexibly expanded to enhance the context of logs according to specific business needs or scenarios.

[0103] Furthermore, metadata may be stored in formats such as JSON and XML to form an initial set of multiple log metadata to facilitate subsequent parsing and management operations.

[0104] In an optional implementation, metadata definition may be implemented by the code shown in Table 1 below.

[0105] Table 1: Metadata definitions

[0106]

[0107] Optionally, the metadata version management submodule 112 is used to manage the target file.

[0108] The target file is used to record the change information of multiple log metadata.

[0109] Specifically, when the business logic changes, such as adding a new business operation type, or the technical architecture is adjusted, such as introducing a new service version into the system, the log metadata needs to be changed accordingly. At this time, the metadata definition submodule 111 can modify, add or delete the affected metadata. For example, a new custom field is added to record data related to the new business operation.

[0110] Furthermore, when a metadata change operation occurs in the metadata definition submodule 111, the metadata version management submodule 112 can be triggered, and the corresponding change information can be recorded in the metadata version management submodule 112, which can be achieved by managing the target file (for example, using Git to manage the Schema file).

[0111] In an optional implementation, the metadata version management submodule 112 may create corresponding records in the database to store version information, for example, creating a table to record the currently used Schema version and change history.

[0112] Furthermore, each time the metadata definition submodule 111 performs metadata changes, the metadata version management submodule 112 may record information such as the content, time, and operation type (such as addition, modification, and deletion) of the changes in a target file or database table.

[0113] For example, when the metadata definition submodule 111 adds a custom field metadata, the metadata version management submodule 112 can record the definition of the custom field, the adding time and other information.

[0114] Furthermore, during the operation of the system, when a problem needs to be investigated, the operation and maintenance personnel may need to view the historical version of the metadata. At this time, the version record in the target file or database can be queried through the metadata version management submodule 112 to understand the change of the metadata.

[0115] For example, if an error is found in log parsing, by querying the metadata version record, you can check whether the problem is caused by a recent metadata change, thereby helping to quickly locate and solve the problem.

[0116] Therefore, through the cooperation of the metadata definition submodule 111 and the metadata version management submodule 112 , corresponding multiple log metadata can be acquired.

[0117] Furthermore, the obtained multiple log metadata may be transmitted to the log parsing module 13 .

[0118] In an optional implementation, the metadata definition submodule 111 may transmit the obtained multiple log metadata to the log parsing module 13 through a communication mechanism within the system. The communication mechanism may be based on a message queue, a function call or other inter-process communication methods.

[0119] Therefore, the metadata definition submodule forms log metadata by constructing the log data model and data dictionary, which can flexibly cope with diverse log sources and formats. Whether it is a complex database change log or an application log with different formats, it can be standardized based on the precisely defined metadata, so that subsequent parsing and synchronization operations can be accurately implemented, avoiding the processing difficulties caused by differences in log formats. Compared with the traditional fixed format processing method, the system's compatibility with log data in different business scenarios is greatly enhanced. Furthermore, the metadata version management submodule is responsible for managing the target file that records metadata change information. On the one hand, when data synchronization anomalies occur, the metadata version can be traced back to check what changes caused the problem; on the other hand, for newly connected business modules or upgraded log formats, they can be orderly transitioned according to version management to prevent system crashes caused by metadata confusion, ensure the stable and continuous operation of the system, and make up for the shortcomings of traditional technologies that lack effective metadata management.

[0120] Optionally, the log parsing module 13 is used to determine a log parsing rule according to a plurality of log metadata and send the log parsing rule to the log collection module 12 .

[0121] Among them, the log parsing rules may include JSON rules, text rules, etc.

[0122] Specifically, the corresponding log parsing rules can be determined according to the format specification of the log metadata. For example, when the log itself is in a standard JSON format, the JSON file defined by the metadata (the log metadata is stored in JSON format) can be directly converted into the corresponding java bean entity class.

[0123] In an optional implementation, the metadata definition JSON shown in Table 1 can be directly converted into a Java bean object recognized by a Java code parser:

[0124] "{

[0125] "timestamp":"2023-11-01T14:23:00Z",

[0126] "level":"INFO",

[0127] "service":"UserService",

[0128] "operation":"UserLogin",

[0129] "status":"SUCCESS",

[0130] "message":"User'john.doe'logged in successfully",

[0131] "additional_info":{"ip":"192.168.1.01","browser":"Chrome"}

[0132] }".

[0133] Furthermore, if the log itself is an irregular text with a field identifier defined by metadata, a regular expression can be generated through the field defined by metadata to intercept the text and obtain the log content.

[0134] For example:

[0135] When the log is an irregular JSON text, the JsonLogInterceptor parser will generate the corresponding regular interception expression based on the fields defined in the metadata:

[0136] timestamp->generates / "timestamp":"([^"]+)" / regular expression.

[0137] level->generates / "level":"([^"]+)" / regular expression.

[0138] Similarly, for each field defined in the metadata, content extraction is performed on the original log.

[0139] Further, after determining the above-mentioned log parsing rules (which may include conversion rules for JSON format and regular expression interception rules for irregular text), the log parsing module 13 may transmit the determined log parsing rules to the log collection module 12 .

[0140] Therefore, by formulating parsing rules based on precise metadata in the log parsing module and parsing the collected multiple distributed log information in the log collection module, it can ensure that the collected distributed log information can be accurately parsed, avoiding data deviation caused by parsing errors.

[0141] Optionally, the log collection module 12 is used to collect multiple distributed log information and parse the multiple distributed log information using log parsing rules to obtain multiple target log data, and send the multiple target log data to the synchronization strategy formulation module 14.

[0142] Among them, Figure 3 As shown, the log collection module 12 includes multiple log collection components 121 , a control submodule 122 and a parsing submodule 123 .

[0143] Optionally, the control submodule 122 is used to load multiple log collection components 121 through a service discovery mechanism, and register multiple collection nodes through a distributed coordination service framework.

[0144] Among them, the service discovery mechanism adopts the SPI (Service Provider Interface) mechanism.

[0145] Specifically, the control submodule 122 can use the SPI (Service Provider Interface) mechanism to dynamically load the required collection components. For example, when the system needs to collect logs from different sources, the SPI mechanism can discover and load different types of log collection components such as Apache Flume, Filebeat, Canal, etc. at runtime.

[0146] Among them, Apache Flume can be used to efficiently collect, aggregate and transmit large amounts of log data, and is particularly suitable for big data environments; Filebeat represents a lightweight log collector that focuses on reading logs from files and sending them to the target system; Canal can be used to capture database change data in real time, and supports databases such as MySQL.

[0147] Furthermore, this dynamic loading method can make the system highly flexible and scalable. When there are new log collection requirements or new collection tools, the system can automatically discover and load the new collection components according to the SPI specification without large-scale recoding of the system.

[0148] Furthermore, the control submodule 122 may also register multiple collection nodes through a distributed coordination service framework (such as Zookeeper). Furthermore, during the registration process, the control submodule 122 may register the collection node information corresponding to each log collection component 121 in Zookeeper.

[0149] For example, when there are multiple Flume agents as collection nodes, the control submodule 122 can register the relevant information of these Flume agents (such as node address, running status, etc.) to Zookeeper.

[0150] Furthermore, during the collection node registration process, each collection node can obtain a unique identifier in a distributed environment to facilitate management and coordination.

[0151] Furthermore, Zookeeper can implement a master-slave switching mechanism by maintaining the node information, ensuring that when a collection node fails (such as hardware failure, network problem, etc.), it can automatically switch to the backup node to continue log collection, ensuring the high availability of the collection process.

[0152] Specifically, Zookeeper can continuously monitor the status of the collection nodes. If a collection node fails, such as a network connection interruption or hardware failure, Zookeeper can detect it in time and switch the collection task to the backup node according to the active-standby switching mechanism.

[0153] Optionally, the log collection component 121 is used to monitor and collect multiple distributed log information based on multiple collection nodes.

[0154] Among them, different log collection components 121 have their specific functions and applicable scenarios:

[0155] (1)Apache Flume

[0156] Specifically, Flume can configure multiple data sources (such as multiple file directories or network ports) as input, and can achieve log collection, temporary storage and transmission by defining different Sources, Channels and Sinks. For example, it can read log data from log file directories on multiple servers in real time, aggregate the data and transmit it to the next processing link.

[0157] (2) Filebeat: As a lightweight log collector, Filebeat mainly reads logs from files. It can monitor specified files or directories. Once a new log line is written to the file, Filebeat will immediately read it and send it to the target system. For example, for an application's log file, Filebeat can continuously monitor and collect newly generated log content with very low resource consumption.

[0158] (3) Canal: It can obtain the binary log (binlog) of the database by simulating the MySQL slave library, and then parse the operation records such as insertion, update, and deletion in the database. For example, when order data is modified in the database of an e-commerce system, Canal can capture this change information in real time and collect it as log data.

[0159] Specifically, you can initialize the configuration of the selected log collection component. This includes setting the parameters of the collection component, for example, for Flume, configure the relevant parameters of Source (data source), Channel (data channel) and Sink (data sink) to determine where to collect logs, how to temporarily store data, and where to send data; for Filebeat, specify the file or directory path to be monitored, log format and other parameters; for Canal, configure the connection parameters with the target database, such as the database host address, port, user name, password, etc., as well as the database table and operation type to be monitored.

[0160] Further, after the initialization and node registration are completed, each log collection component 121 can start log monitoring on its corresponding collection node.

[0161] For example, Filebeat starts monitoring the specified file or directory, and can continuously check whether there are new log lines written to the file in a low-overhead manner. Canal starts listening to the binary log of the database, waiting for changes to occur in the database.

[0162] Further, when a new log is generated, the log collection component 121 starts collecting log information:

[0163] (1) For logs from files (such as logs collected by Filebeat): Once a new log line is detected in the file, the log content of that line is read immediately. For example, in a log file of a web server, whenever a new user access request is recorded in the log file, Filebeat can quickly read the new log line.

[0164] (2) For database change logs (such as logs collected by Canal): When an insert, update, or delete operation occurs in the database, Canal parses the relevant operation records from the database binary log. For example, in the database of an e-commerce system, when an order status is updated, Canal can capture this change information in real time and collect it as log data.

[0165] (3) For logs from other sources (such as logs collected by Flume): Flume can collect log data from multiple data sources (such as log file directories or network ports on multiple servers) according to its configuration. For example, in an enterprise data center, Flume can collect system logs, application logs, and other log information from servers in different departments.

[0166] Furthermore, through the above process, based on multiple collection nodes, the log collection component 121 can comprehensively, efficiently and reliably monitor and collect multiple distributed log information, and then send the obtained multiple distributed log information to the parsing submodule 123.

[0167] Optionally, the parsing submodule 123 is used to parse multiple distributed log information using log parsing rules to obtain multiple target log data.

[0168] Specifically, after receiving multiple distributed log information, the parsing submodule 123 can parse the multiple distributed log information using the log parsing rules determined above. For example, for JSON format log parsing, the data in JSON can be mapped to the corresponding data structure based on the definition of each field in the metadata, thereby extracting valuable information as the target log data.

[0169] Furthermore, for irregular text format log parsing, a regular expression can be generated through the fields defined by the metadata to intercept the text to obtain the log content.

[0170] Therefore, through this embodiment, the control submodule adopts the service discovery mechanism to load multiple log collection components, so that the system has strong scalability. With the development of business, if it is necessary to add a collection function for special data sources or new log formats, it is only necessary to develop a collection component that complies with the corresponding interface specification, and the system can automatically discover and load it without large-scale changes to the core architecture. Compared with the traditional model that relies on fixed component configuration and is difficult to expand, it can easily adapt to the ever-changing business environment. Further, registering multiple collection nodes with the help of a distributed coordination service framework can realize functions such as master-slave switching, ensuring the high availability of log collection. In the face of emergencies such as network failures and node hardware damage, the system can quickly switch to the backup node to continue the collection task and maintain the continuous inflow of data, which completely solves the risk of data interruption and loss that is prone to occur in traditional single-point collection. Finally, multiple distributed log information is parsed through the parsing submodule and corresponding multiple target log data are obtained.

[0171] Optionally, the synchronization strategy formulation module 14 is used to formulate a target synchronization strategy according to multiple target log data, and send the target synchronization strategy to the data transmission module 15 .

[0172] Among them, Figure 4 As shown, the synchronization strategy formulation module 14 includes a first determination submodule 141 , a second determination submodule 142 , a third determination submodule 143 and a strategy formulation submodule 144 .

[0173] Optionally, the first determination submodule 141 is used to determine the synchronization target according to preset data requirements and data types of multiple target log data, and send the synchronization target to the strategy formulation submodule 144.

[0174] Among them, the preset data requirements may include:

[0175] (1) Data consistency requirements: Ensure data consistency on all nodes, such as using a strong consistency strategy or an eventual consistency strategy.

[0176] Among them, the strong consistency strategy means that in log collection, you can confirm whether the log is the same according to the "timestamp", "level", "service", "operation", and "status" in the log metadata definition. If they are the same, then compare whether the business information is consistent. If they are inconsistent, both are retained. If they are consistent, only one is retained.

[0177] Furthermore, the eventual consistency strategy means that there is a unique identifier in the log, and idempotent storage is used to ensure data consistency during the final persistence.

[0178] (2) Real-time requirements: Determine the real-time requirements for data synchronization based on application requirements, whether real-time updates are required or delays are acceptable.

[0179] Furthermore, considering the type of data that needs to be synchronized (such as database changes, log data, files, etc.), different data types require different synchronization strategies.

[0180] Specifically, the first determining submodule 141 may determine the corresponding synchronization target by comprehensively considering the data consistency requirement, the real-time requirement, and the data types of the plurality of target log data.

[0181] Further, the determined synchronization target is sent to the strategy formulation submodule 144 .

[0182] Optionally, the second determination submodule 142 is used to determine the synchronization mode according to the multiple target log data, and send the synchronization mode to the strategy formulation submodule 144.

[0183] Specifically, the synchronization mode can include active synchronization and passive synchronization:

[0184] (1) Active synchronization: The data source actively pushes data to the target system (for example, through API calls or message queues). This method is suitable for scenarios where data needs to be updated to the target system in a timely manner. For example, in a financial transaction system, when a new transaction occurs, the data source immediately pushes the transaction data to the relevant analysis system.

[0185] (2) Passive synchronization: The target system pulls data periodically (for example, through scheduled tasks or polling). This method is suitable for data that does not require high real-time performance, such as some backend data analysis systems that pull data from the data source for analysis at regular intervals (such as every hour or every day).

[0186] Furthermore, you need to consider whether to perform incremental synchronization or full synchronization:

[0187] (1) Incremental synchronization: Only the changed data is synchronized. This is suitable for scenarios with large amounts of data and frequent changes. For example, in a large e-commerce system, a large amount of order data is generated every day, but only the newly generated or modified order data needs to be synchronized to the relevant logistics system and financial system.

[0188] (2) Full synchronization: When data changes are infrequent or the data volume is small, full synchronization can be considered. For example, in the internal management system of a small enterprise, full data synchronization is performed once a week to ensure data consistency in each system.

[0189] Furthermore, through the definition of the above-mentioned different synchronization modes, the corresponding synchronization mode can be determined according to the received multiple target log data.

[0190] Furthermore, the determined synchronization method may be sent to the policy formulation submodule 144 .

[0191] Optionally, the third determination submodule 143 is used to determine the synchronization frequency according to preset business requirements and the change frequency of multiple target log data, and send the synchronization frequency to the policy formulation submodule 144.

[0192] Specifically, the third determining submodule 143 may determine the synchronization frequency according to preset business requirements and the change frequency of multiple target log data.

[0193] For example, according to the frequency of data changes and business needs, set synchronization time intervals or trigger conditions (such as every minute, every hour, every day, etc.). If the business is an online transaction system with high real-time requirements, data synchronization may be required every minute or even shorter to ensure that the transaction data in each system is consistent in real time.

[0194] For some backend data statistics systems, business needs may only require data synchronization once a day to summarize and analyze the data generated on the previous day.

[0195] Further, after the synchronization frequency is determined in the above manner, the synchronization frequency may be sent to the strategy formulation submodule 144 .

[0196] Optionally, the strategy formulation submodule 144 is used to determine a target synchronization strategy according to a synchronization target, a synchronization mode and a synchronization frequency.

[0197] Specifically, after receiving the synchronization target from the first determination submodule 141 , the synchronization mode from the second determination submodule 142 , and the synchronization frequency from the third determination submodule 143 , the strategy formulation submodule 144 may determine the target synchronization strategy by integrating the information.

[0198] For example, if the synchronization goal is to ensure strong consistency and real-time performance of data, the synchronization mode is determined to be active synchronization and incremental synchronization, and the synchronization frequency is once a minute, then the strategy formulation submodule 144 can formulate a target synchronization strategy that performs active incremental synchronization once a minute to ensure strong consistency and real-time performance of data.

[0199] Furthermore, the strategy formulation submodule 144 may send the determined target synchronization strategy to the data transmission module 15 .

[0200] Therefore, through this embodiment, the first determination submodule determines the synchronization target based on the preset data requirements and the data type of the target log data, and can tailor the synchronization strategy according to the differentiated demands of different businesses for data consistency and real-time performance. Furthermore, the second determination submodule determines the synchronization method based on the target log data, and can flexibly adjust it according to the actual data conditions, significantly improving data transmission efficiency and system adaptability. Furthermore, the third determination submodule determines the synchronization frequency based on the preset business requirements and the change frequency of the target log data, avoiding the problems of insufficient peak resources and idle resources in the valley caused by the fixed-frequency synchronization of traditional technologies, and realizing the reasonable allocation and efficient use of system resources.

[0201] Optionally, the data transmission module 15 is used to send multiple target log data to the target system 2 according to the target synchronization strategy.

[0202] The target system 2 is connected to the distributed data synchronization system 1 .

[0203] Specifically, whether to adopt synchronous transmission or asynchronous transmission can be determined according to the target synchronization strategy.

[0204] If the target synchronization strategy requires high real-time requirements, such as in a financial transaction system, it is necessary to ensure that the transaction data is immediately confirmed to be received by the target system 2, then the data transmission module 15 will adopt a synchronous transmission mode. In synchronous transmission, the sender (data transmission module 15) will wait for the receiver (target system 2) to confirm after sending the data.

[0205] Furthermore, if the target synchronization strategy focuses on high-concurrency scenarios, such as in the transmission of user behavior logs in a large e-commerce system, where immediate confirmation of receipt is not required, the data transmission module 15 can select an asynchronous transmission mode. In asynchronous transmission, the sender does not need to wait for the receiver's confirmation after sending the data, which can usually be achieved with the help of a message queue.

[0206] Furthermore, you can select the corresponding transmission protocol according to the business scenario:

[0207] (1) HTTP / HTTPS: If the target system 2 is based on a client-server request-response model, such as a web application that performs data interaction via a RESTful API, the data transmission module 15 may select HTTP or HTTPS protocol.

[0208] Furthermore, this protocol is suitable for secure data transmission over the Internet, such as transmitting user login log data from a source system to a remote security audit system.

[0209] (2) WebSocket: When the target system 2 requires two-way real-time communication, such as real-time transmission of player operation logs and game status logs in an online game system, the data transmission module 15 can adopt the WebSocket protocol to support real-time data update and interaction.

[0210] (3) TCP / UDP: If the underlying transmission is considered, for scenarios where reliable data transmission is required, such as the transmission of critical business data within an enterprise, the data transmission module 15 can select the TCP protocol. For some scenarios that require high data transmission speed but can tolerate a certain amount of packet loss, such as video stream logs in a video surveillance system (partial packet loss does not affect the overall monitoring effect), the UDP protocol can be used.

[0211] (4) AMQP: In complex messaging scenarios, such as when log data interactions between multiple different types of subsystems are involved and reliable routing functions are required, the data transmission module 15 can select AMQP (Advanced Message Queuing Protocol). For example, in a distributed system of a large enterprise, business log data of different departments can be reliably routed and transmitted through the AMQP protocol.

[0212] Furthermore, if the asynchronous transmission mode is selected, the data transmission module 15 can use a message queue (such as Kafka, RabbitMQ) to implement data transmission.

[0213] Specifically, the data transmission module 15 can send multiple target log data as messages to the message queue. For example, in a big data processing platform, the data transmission module 15 can send the collected massive user behavior log data as messages to the Kafka queue.

[0214] Furthermore, the target system 2 can consume data from the message queue. For example, the data analysis system obtains user behavior log data from the Kafka queue for subsequent analysis and mining.

[0215] Optional, such as Figure 4 As shown, the synchronization strategy formulation module 14 also includes a conflict handling submodule 145, which is used to determine priorities and resolution rules when multiple target log data conflict, and use the priorities and resolution rules to handle the conflicts that occur.

[0216] Specifically, the conflict handling submodule 145 can continuously monitor the update of multiple target log data. In a distributed system, multiple nodes may operate on the same data at the same time, which may cause data conflicts. For example, in an e-commerce system, multiple warehouse nodes may modify the inventory data of the same product at the same time.

[0217] Further, when a data conflict is detected, a corresponding priority and a resolution rule are determined to resolve the conflict.

[0218] In an optional implementation, the priority may be determined by a timestamp.

[0219] Specifically, the conflict handling submodule 14 may check the timestamp of each conflicting data. The timestamp usually records the time when the data operation occurs.

[0220] For example, if node A modifies the inventory of a certain product at 2023-11-01 10:00:00, and node B modifies the inventory of the same product at 2023-11-01 10:05:00, then node A's operation time is earlier. Under the rule of determining priority based on timestamp, node A's operation may have a higher priority.

[0221] Furthermore, if the timestamps are the same (this situation may be rare, but it is possible, for example, when the clock synchronization is inaccurate), other resolution rules need to be considered.

[0222] Furthermore, the priority can be determined by the version number.

[0223] Specifically, each time data is modified, its version number will be updated accordingly. The conflict handling submodule 145 can compare the version numbers of conflicting data.

[0224] For example, the initial version number of the data is 1, and after node A modifies the data, the version number becomes 2. Node B also modifies the original data without obtaining the modified data of node A, and its version number also becomes 2. In this case, the order of operations or other related identifiers can be further checked to determine which version of the modification has a higher priority.

[0225] Furthermore, if the version numbers are different, the data modification operation with the higher version number generally has a higher priority because it represents a more recent operation.

[0226] Furthermore, in addition to timestamps and version numbers, there may be other resolution rules, such as based on node weights. In some distributed systems, different nodes have different weights, and data operations on nodes with higher weights may have higher priority.

[0227] Furthermore, the priority can be determined according to business rules. For example, in a financial transaction system, certain types of transactions (such as high-risk transactions) may have higher priority, and data operations related to these transactions will be given priority when data conflicts occur.

[0228] Further, after the priorities and resolution rules are determined, the conflict handling submodule 145 may resolve the data conflicts that occur according to the determined priorities and resolution rules.

[0229] For example, if the operation of node A is determined to have a higher priority based on the timestamp, then when handling conflicts, the operation result of node A will be used, and the operation of node B may be ignored, rolled back, or otherwise processed to ensure data consistency.

[0230] Furthermore, when priority is determined based on version numbers, data operation results with higher version numbers and that comply with the rules will be applied to the final data, while ensuring that other related data and operations in the system can correctly handle the impact of this conflict resolution.

[0231] Therefore, through the above implementation, when multiple target log data conflicts, whether it is due to concurrent updates of the same data by different nodes caused by network delays, or conflicts caused by changes in business rules, effective operations can be determined by determining priorities and resolution rules in the conflict handling submodule, thereby ensuring data consistency on a global scale and preventing conflicting data from entering the target system 2 and causing business chaos.

[0232] Optionally, the first processing module 16 is used to control the log collection module 12 to collect multiple distributed log information by using a preset retry mechanism when multiple target log data are not all sent to the target system 2.

[0233] Specifically, the first processing module 16 can monitor in real time the sending of multiple target log data to the target system 2. When it is found that multiple target log data are not all sent to the target system 2, for example, part of the data is lost or failed to be sent due to network failure, temporary unavailability of the target system 2, etc. during the data transmission process, a retry mechanism can be triggered.

[0234] Furthermore, the first processing module 16 can control the log collection module 12 according to a preset retry mechanism, wherein the retry mechanism is similar to that implemented by Kafka's ACK (confirmation mechanism).

[0235] Specifically, under normal circumstances, the scanning or collection operation of the log will be stopped only after the log collection is confirmed and successfully sent to the message queue. If the data is not successfully sent to the target system 2 (no ACK confirmation is obtained), the first processing module 16 can control the log collection module to re-collect multiple distributed log information and try to re-send the collected new multiple distributed log information to the target system 2.

[0236] Furthermore, through the above method, the data is ensured to be successfully transmitted in the end, and the probability of data loss is reduced. Compared with the traditional data synchronization that lacks effective retry means, the reliability and integrity of the transmission are greatly enhanced, and it provides a strong guarantee for common transmission failure scenarios such as network jitter and temporary overload of the target system 2.

[0237] Optionally, the second processing module 17 is used to control the metadata management module 11 to obtain multiple new log metadata when the log format is updated to the original format.

[0238] Specifically, the second processing module 17 can monitor the changes in the log format in real time. When the log format is updated to the original format, for example, after a system upgrade or configuration change, the log format may be restored to a previous state, and new log metadata needs to be obtained to adapt to this change.

[0239] Furthermore, the second processing module 17 may control the metadata management module 11 to obtain multiple new log metadata.

[0240] Among them, the metadata definition submodule 111 may need to redefine the format and meaning of metadata such as timestamp, log level, message content, etc. At the same time, the metadata version management submodule 112 can record the change information of these metadata to ensure that the system can perform subsequent log processing operations according to the new metadata, such as log parsing and synchronization strategy formulation.

[0241] Furthermore, in this way, when the log format changes back to the previous form, there is a corresponding metadata version to describe the correct format of the log for parsing, ensuring the continuity and accuracy of the log processing process when the format changes.

[0242] Furthermore, through the above method, it is ensured that the system can quickly adapt to changes in log formats, avoiding system stagnation caused by incompatible log formats, and maintaining the vitality and adaptability of the system.

[0243] Optionally, the monitoring and alarm module 18 is used to monitor the synchronization process of multiple target log data and to generate an alarm using a preset alarm mechanism when synchronization failure or synchronization delay is detected.

[0244] Specifically, the monitoring and alarm module 18 may determine key indicators for monitoring the synchronization process of multiple target log data.

[0245] Among them, key indicators may include:

[0246] (1) Data transfer volume: Monitor the actual amount of log data transferred during the synchronization process to ensure that the data volume meets expectations and that there is no data loss or excessive transfer.

[0247] (2) Transmission rate: measures the speed at which log data is transmitted from the source to the target system 2 to determine whether the transmission is carried out within a reasonable time frame.

[0248] (3) Synchronization frequency: Based on the synchronization frequency determined by the synchronization strategy formulation module, monitor whether the actual synchronization operation is performed according to the predetermined frequency, for example, whether the synchronization operation is performed on time every minute, every hour, or every day.

[0249] (4) Data consistency: Check the consistency of the target log data before and after synchronization to ensure that the data has not been tampered with or inconsistent during transmission.

[0250] Furthermore, the monitoring and alarm module 18 can continuously monitor these indicators during the synchronization process.

[0251] Furthermore, when the following situations are detected, synchronization failure can be determined:

[0252] (1) The data transmission volume is far lower than expected and does not return to normal within a certain period of time. This may mean that data is lost during transmission or the transmission link fails.

[0253] (2) The transmission rate is zero or extremely low for a long time, indicating that the data cannot be transmitted to the target system 2 normally. This may be caused by network congestion, target system 2 failure, etc.

[0254] (3) After the synchronization operation, it is found that the data consistency check fails, indicating that an error occurred during the data transmission process, resulting in the data received by the target system 2 being inconsistent with the source data.

[0255] Furthermore, if it is found that the actual synchronization operation is not performed according to the scheduled synchronization frequency, and the delay exceeds the preset threshold, it is judged as synchronization delay. For example, if the synchronization is scheduled to be performed once an hour, but the actual synchronization operation has not been performed for more than an hour, or although it is performed, it is much later than the scheduled time, this is considered synchronization delay.

[0256] Furthermore, when synchronization failure or synchronization delay occurs, a preset alarm mechanism can be used to alarm. The alarm method may include email notification, SMS notification, system internal message notification, etc. Furthermore, the alarm information may include:

[0257] (1) A detailed description of the problem, such as "Synchronization failed: Data transfer volume was zero in the past 30 minutes" or "Synchronization delayed: The synchronization operation that was supposed to be performed at 10:00 has not been performed as of 10:30."

[0258] (2) Identification of relevant data, such as the source, type, and time range of the target log data involved, to help operation and maintenance personnel quickly locate problems.

[0259] (3) Possible fault causes or troubleshooting directions, based on the data collected and analysis results of the monitoring module, provide certain references for solving the problem.

[0260] Furthermore, through the above method, operation and maintenance personnel can quickly locate the fault point with the help of timely alarms. Whether it is a jam in the log collection link, a deviation in the execution of the synchronization strategy or a blockage in data transmission, they can intervene and deal with it at the first time, ensuring the continuity and timeliness of data synchronization, and changing the dilemma that traditional technology is difficult to detect synchronization problems in a timely manner.

[0261] Optionally, the testing module 19 is used to receive the target synchronization strategy sent by the synchronization strategy formulation module 14 , test the target synchronization strategy, obtain the test result, and send the test result to the optimization module 20 .

[0262] Specifically, the testing module 19 may determine a corresponding testing scheme according to the received target synchronization strategy.

[0263] Among them, the test plan may include simulating different scenarios, determining test indicators, etc.

[0264] Further, the test operation can be performed according to the determined test scheme:

[0265] (1) Deploy the test environment: Build a test environment similar to the actual production environment, configure the corresponding data source, target system 2, and various modules involved in the middle (such as log collection module, data transmission module, etc.), and ensure that the test environment can truly reflect the actual operation situation.

[0266] (2) Start the synchronization test: Start the data synchronization process according to the target synchronization strategy, and observe and record the actual performance of various test indicators during the synchronization process in different simulated scenarios. For example, in a high-concurrency scenario, check whether there is congestion or loss in data transmission, and whether the synchronization can be completed on time; in an unstable network scenario, check the integrity of the data and whether the system can automatically handle synchronization anomalies caused by network problems.

[0267] Furthermore, the test module 19 can perform a comprehensive analysis on the collected test indicator data to generate test results.

[0268] The test results can show in detail how the target synchronization strategy performs in different test scenarios, indicating which aspects have met the expected requirements and which aspects have deficiencies. For example, the test results may show that in high-concurrency scenarios, although data consistency and integrity can be guaranteed, the timeliness of synchronization has declined and a certain degree of delay has occurred; or when the network is unstable and the packet loss rate is high, a small amount of data is lost.

[0269] Further, the testing module 19 may send the obtained test results to the optimization module 20 .

[0270] Optionally, the optimization module 20 is used to optimize the target synchronization strategy according to the test results.

[0271] Specifically, the optimization module 20 can perform a detailed analysis on the received test results and fully understand the advantages and problems of the target synchronization strategy in the actual test.

[0272] First, we can conduct in-depth analysis based on the test results to find out the key factors that affect the synchronization effect and then determine the optimization direction:

[0273] (1) Regarding synchronization timeliness: If the test results show that synchronization timeliness is insufficient, such as delays in high concurrency or unstable network scenarios, the optimization module 20 can consider adjusting the synchronization method or synchronization frequency. For example, if the passive synchronization method is used and the delay is obvious, you can try to change to the active synchronization method, let the data source actively push data, and speed up the synchronization speed; or appropriately increase the synchronization frequency and shorten the synchronization interval to reduce the delay caused by data accumulation.

[0274] (1) Regarding data integrity and consistency issues: When data loss or inconsistent data after synchronization occurs, it is possible to analyze whether the problem occurs in the log collection link, data transmission link, or other links. If it is in the collection link, the configuration of the log collection component may be optimized to improve the accuracy and reliability of the collection; if it is in the transmission link, consider replacing a more appropriate data transmission protocol, or enhancing the error checking and retransmission mechanism during the transmission process, etc., to ensure that the data is transmitted to the target system completely and consistently2.

[0275] (3) Regarding resource occupation issues: If it is found that the target synchronization strategy occupies too many system resources during execution, affecting the overall system performance, the optimization module 20 can streamline or optimize the target synchronization strategy. For example, for some large-scale full synchronization operations, it can be considered to change to incremental synchronization to reduce unnecessary data transmission, thereby reducing the occupation of network bandwidth, memory and other resources; or optimize the data processing algorithm during the synchronization process to improve resource utilization efficiency.

[0276] Furthermore, the optimization module 20 can make specific optimization adjustments to the target synchronization strategy according to the determined optimization direction. For example, the synchronization mode, synchronization frequency, data transmission protocol and other related parameters in the synchronization strategy can be modified, or the configuration and workflow of each module involved (such as log collection module, data transmission module, etc.) can be optimized and improved to form an optimized synchronization strategy.

[0277] Furthermore, the optimized target synchronization strategy can be fed back to the test module 19 for a new round of testing. Through continuous testing and optimization cycles, the effect of the synchronization strategy can be continuously improved to better adapt to actual business needs, ensuring efficient, accurate and reliable data synchronization in various complex actual operating situations.

[0278] Furthermore, through continuous testing with the test module, problems with the synchronization strategy are discovered, and targeted optimization is performed using the optimization module, so that system performance is continuously improved and always adapts to business development needs, breaking the deadlock that traditional technologies are difficult to optimize once deployed.

[0279] The distributed data synchronization system provided in this embodiment obtains multiple log metadata through the metadata management module, and establishes unified specifications and standards for subsequent data processing. Further, the log collection module can process log information from multiple distributed sources, whether it is the change log generated by different types of databases, or the operation log generated by various applications, or the operation log at the system level, can be included in the collection scope. Further, the log parsing module formulates parsing rules based on accurate metadata and parses the collected multiple distributed log information in the log collection module, which can ensure that the collected distributed log information can be accurately parsed, avoiding data deviation caused by parsing errors. Further, the synchronization strategy formulation module can formulate synchronization strategies based on the accurate target log data after parsing, further ensuring the consistency of data during transmission. Whether it is the structure, content or update order of the data, it can be kept unified in a distributed environment in strict accordance with the predetermined strategy, effectively solving the data inconsistency problem that often occurs in traditional technology during multi-node synchronization, so that the data received by the target system is complete and reliable. Therefore, by implementing the present invention, by organically integrating metadata management, log collection, log parsing, synchronization strategy formulation and data transmission modules, the systematicness and synergy of data synchronization are greatly improved. Compared with the traditional fragmented and unintegrated synchronization method, it effectively avoids problems such as disconnection between links, duplication of work and accumulation of errors, ensuring that the entire synchronization process is efficient and stable.

[0280] According to an embodiment of the present invention, an embodiment of a distributed data synchronization method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0281] In this embodiment, a distributed data synchronization method is provided, which can be used in the distributed data synchronization system 1 shown in the above embodiment of the present invention. Figure 5is a flow chart of a distributed data synchronization method according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0282] Step S501, obtaining multiple log metadata and multiple distributed log information.

[0283] For the specific process, reference may be made to the functional description of the metadata management module 11 and the log collection module 12 in the distributed data synchronization system 1 in the above embodiment, which will not be described in detail here.

[0284] Step S502: determining a log parsing rule according to a plurality of log metadata.

[0285] The specific process can refer to the functional description of the log parsing module 13 in the distributed data synchronization system 1 in the above embodiment, which will not be repeated here.

[0286] Step S503: parse multiple distributed log information using log parsing rules to obtain multiple target log data.

[0287] For the specific process, reference may be made to the functional description and interaction process description of the log parsing module 13 and the log collection module 12 in the distributed data synchronization system 1 in the above embodiment, which will not be repeated here.

[0288] Step S504: formulate a target synchronization strategy according to multiple target log data.

[0289] The specific process can refer to the functional description of the synchronization strategy formulation module 14 in the distributed data synchronization system 1 in the above embodiment, which will not be repeated here.

[0290] Step S505: sending a plurality of target log data to the target system according to the target synchronization strategy.

[0291] For the specific process, reference may be made to the functional description of the data transmission module 15 in the distributed data synchronization system 1 in the above embodiment, which will not be described in detail here.

[0292] The distributed data synchronization method provided in this embodiment performs synchronous transmission of distributed data through a distributed data synchronization system, effectively avoiding problems such as disconnection of various links, duplication of work, and accumulation of errors, and ensuring that the entire synchronization process is efficient and stable.

[0293] In one example, a distributed data synchronization method and application based on log collection is provided, aiming to achieve high availability, reliable data transmission and real-time synchronization to meet the requirements of modern distributed systems for data consistency and system stability. This method and application provides a log metadata definition, and ensures efficient and reliable synchronization of data between multiple distributed nodes by real-time collection and processing of database change logs. The specific steps include: log collection, data analysis, synchronization strategy formulation, and data transmission.

[0294] The metadata definition means defining the data model and data dictionary of the log, describing the structure and field information of the log, and storing the metadata in formats such as JSON and XML for easy parsing and management.

[0295] Furthermore, log collection means to fully capture data operation information (such as insert, update, delete and business operations) by real-time monitoring of logs from various sources, including database change logs, application logs and system logs. Use efficient log collection tools (such as Canal and Flume) to achieve real-time data capture with low overhead, ensuring timely acquisition of all change information under high concurrency conditions.

[0296] Furthermore, data analysis means parsing the collected log information and converting it into a unified standard format. This process covers the extraction and mapping of change types, target tables, and change data, ensuring the accuracy of subsequent processing and providing support for high-availability system design.

[0297] Furthermore, synchronization strategy formulation means formulating flexible synchronization strategies based on business needs and data characteristics, including highly available data routing, load balancing, and failover mechanisms. These strategies ensure that the system can automatically recover and continue to provide services in the event of node failure or network fluctuations.

[0298] Data transmission means using distributed message queues (such as Kafka) or streaming processing platforms to push the parsed data to the target system in real time. This process supports high-concurrency data transmission and has fault tolerance to ensure data integrity and consistency in the event of network failure or node anomaly.

[0299] Furthermore, the specific technical solutions include:

[0300] (1) Timestamp

[0301] Definition: Record the specific time when an event occurs, including date and time information.

[0302] Importance: Timestamps are a critical factor in log analysis, helping to track the sequence and duration of events.

[0303] (2) Log level

[0304] Definition: A classification indicating the importance or urgency of a log message, such as DEBUG, INFO, WARN, ERROR, FATAL, etc.

[0305] Importance: Log levels help users quickly filter and locate key information, facilitating troubleshooting and monitoring.

[0306] (3) Message content

[0307] Definition: The actual text information of the log, describing the specific details of the event.

[0308] Importance: Message content is the core part of the log, providing context and details of the event.

[0309] (4) Source system / application

[0310] Definition: Identification information of the system, service, or application that generated the log.

[0311] Importance: It helps to identify the source of the event and facilitates problem location and tracking in a multi-system environment.

[0312] (5) Request ID / Transaction ID

[0313] Definition: An ID that uniquely identifies a specific operation or transaction.

[0314] Importance: In a distributed system, using request IDs can help track the request flow across services, making it easier to analyze and debug.

[0315] (6) User Information

[0316] Definition: Identity information of the user associated with the event, such as user name, user ID, etc.

[0317] Importance: User information is very important when conducting security audits and user behavior analysis.

[0318] (7) Host information

[0319] Definition: Information about the server or device that generates the log, including host name, IP address, etc.

[0320] Importance: Helps identify the infrastructure of log sources and supports network security and operation management.

[0321] (8) Service version

[0322] Definition: Version information of the application or service that generates the log.

[0323] Importance: When analyzing a problem, knowing the version information can help determine whether it is related to a specific version of the code or configuration.

[0324] (9) Environmental information

[0325] Definition: A label indicating the environment where the logs were generated, such as development, test, production, etc.

[0326] Importance: Helps in troubleshooting and performance analysis in different environments.

[0327] (10) Operation type

[0328] Definition: Describes the specific operation type of the log record, such as create, update, delete, etc.

[0329] Importance: Helps analyze log content and identify the nature and impact of the operation.

[0330] (11) Stack trace information

[0331] Definition: When an error or exception occurs, the call stack information is recorded.

[0332] Importance: Stack traces are important clues for debugging and troubleshooting, and can help developers quickly locate the source of the problem.

[0333] (12) Custom fields

[0334] Definition: Additional fields defined by the user based on specific needs.

[0335] Importance: Log information can be flexibly expanded to enhance the context of logs according to specific business needs or scenarios.

[0336] 2. Log metadata version management

[0337] Specifically, metadata version management uses Git to manage Schema files, records version information in the database, creates a table to record the currently used Schema version and change history, and implements Schema version management in the metadata management module.

[0338] 3. Log Collection

[0339] Collection component integration:

[0340] Apache Flume: Used to efficiently collect, aggregate, and transmit large amounts of log data, especially suitable for big data environments.

[0341] Filebeat: A lightweight log collector that focuses on reading logs from files and sending them to target systems.

[0342] Canal: Used to capture database changes in real time, supporting databases such as MySQL.

[0343] Furthermore, the collection component is registered and started: the collection components that need to be started are dynamically loaded through the SPI mechanism and multiple collection nodes are registered through Zookeeper to achieve active-standby switching to ensure high availability of the collection components.

[0344] 4. Log analysis

[0345] The application loads log parsing rules into the corresponding log collection component through log metadata definition. The collection component parses the collected logs into standardized logs according to the metadata definition rules. The following is the parsing implementation of the corresponding log collector:

[0346] Metadata definition:

[0347]

[0348] Parser implementation:

[0349] Implement the log parsing logic through Java code and generate the parsing class com.example.JsonLogInterceptor$Builder.

[0350] Parser loading start:

[0351]

[0352]

[0353]

[0354] 5. Synchronous strategy formulation

[0355] 1. Define synchronization targets

[0356] Data consistency: Ensure data consistency on all nodes, such as using strong consistency or eventual consistency strategies.

[0357] Real-time requirements: Determine the real-time requirements for data synchronization based on application requirements, whether real-time updates are required or delays are acceptable.

[0358] Data type: Consider the type of data that needs to be synchronized (such as database changes, log data, files, etc.). Different data types may require different synchronization strategies.

[0359] 2. Select the synchronization method

[0360] Active Sync vs Passive Sync:

[0361] Active synchronization: The data source actively pushes data to the target system (for example, through API calls or message queues).

[0362] Passive synchronization: The target system pulls data periodically (for example, through a cron job or polling).

[0363] Incremental synchronization vs full synchronization:

[0364] Incremental synchronization: only synchronizes changed data. It is suitable for scenarios with large amounts of data and frequent changes.

[0365] Full synchronization: If data changes are infrequent or the data volume is small, you can consider regular full synchronization.

[0366] 3. Determine the synchronization frequency

[0367] According to the frequency of data changes and business needs, set synchronization time intervals or trigger conditions (such as every minute, every hour, every day, etc.).

[0368] 4. Deal with conflict

[0369] Develop conflict resolution strategies: When data conflicts occur (such as updating the same data at multiple nodes at the same time), determine priorities and resolution rules (such as timestamps, version numbers, etc.).

[0370] 5. Fault Tolerance

[0371] Consider how to handle network or system failures, and design a retry or rollback mechanism to ensure data reliability.

[0372] 6. Monitoring and alarm

[0373] Implement data synchronization monitoring to promptly detect synchronization failures or delays, and configure alarm mechanisms to quickly respond to problems.

[0374] 7. Testing and Optimization

[0375] After developing a strategy, conduct adequate testing to ensure that synchronization works as expected.

[0376] Continuously optimize the synchronization strategy based on actual operating conditions.

[0377] 6. Data Transfer

[0378] 1. Type of data transmission

[0379] Synchronous transmission: The sender waits for the receiver's confirmation after sending the data. This method is usually used in scenarios with high real-time requirements.

[0380] Asynchronous transmission: The sender does not need to wait for the receiver's confirmation after sending data, which is suitable for high-concurrency scenarios. Message queues are a commonly used asynchronous transmission method.

[0381] 2. Common data transmission protocols

[0382] HTTP / HTTPS: Commonly used in the request-response model between clients and servers, suitable for RESTful APIs.

[0383] WebSocket: Used for two-way real-time communication, suitable for applications that require real-time updates.

[0384] TCP / UDP: underlying transport protocol. TCP ensures reliable data transmission. UDP is suitable for scenarios that require fast transmission but can tolerate packet loss.

[0385] AMQP: Advanced Message Queuing Protocol, suitable for complex messaging scenarios, providing reliability and routing capabilities.

[0386] 3. Data transmission implementation method

[0387] 3.1 Use message queues (such as Kafka, RabbitMQ)

[0388] Producer: Sends data to a message queue.

[0389] Consumer: consumes data from the message queue.

[0390] The distributed data synchronization method and application based on log collection provided in this embodiment have the following advantages:

[0391] 1. High Availability

[0392] Fault-tolerant mechanism: A distributed architecture is used to ensure that the system can still operate normally in the event of a single point failure, thereby improving the overall availability of the system.

[0393] Load balancing: Avoid overloading a certain node through reasonable load distribution, thereby improving the responsiveness and stability of the service.

[0394] 2. Data consistency

[0395] Real-time synchronization: Utilize the log collection mechanism to achieve real-time data updates and ensure that the data in each system remains consistent.

[0396] Conflict resolution: The built-in conflict detection and resolution mechanism can effectively handle conflicts in data synchronization and maintain data integrity.

[0397] 3. Flexibility and scalability

[0398] Support for multiple data sources: supports log collection from multiple data sources (such as databases, application logs, etc.), suitable for different business scenarios.

[0399] Strong scalability: New data sources and target systems can be added as needed, and the system can quickly adapt to business changes.

[0400] 4. Simplified integration process

[0401] Standardized interface: Providing a unified interface simplifies the integration process with other systems and reduces development and maintenance costs.

[0402] Automatic configuration: Supports dynamic loading and automatic configuration, making system deployment and updates more convenient.

[0403] 5. Efficient data processing

[0404] Incremental synchronization: The incremental synchronization strategy is adopted to reduce the volume of data transmission and improve transmission efficiency.

[0405] Batch processing: Supports batch data processing, which can effectively improve the system throughput.

[0406] 6. Monitoring and early warning

[0407] Real-time monitoring: Built-in monitoring mechanism to track the status of data synchronization in real time, and promptly detect and handle abnormal situations.

[0408] Early warning mechanism: When system anomalies or data delays are detected, timely alarms can be issued to ensure the reliability of data synchronization.

[0409] The embodiment of the present invention also provides a computer device for executing the above Figure 5 The distributed data synchronization method shown.

[0410] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0411] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0412] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0413] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0414] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0415] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0416] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0417] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0418] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A distributed data synchronization system, characterized in that: The system includes: a metadata management module, a log collection module, a log analysis module, a synchronization strategy formulation module and a data transmission module; The metadata management module is used to obtain multiple log metadata and send the multiple log metadata to the log parsing module; The log parsing module is used to determine a log parsing rule according to the multiple log metadata and send the log parsing rule to the log collection module; The log collection module is used to collect multiple distributed log information and parse the multiple distributed log information using the log parsing rules to obtain multiple target log data, and send the multiple target log data to the synchronization strategy formulation module; The synchronization strategy formulation module is used to formulate a target synchronization strategy according to the multiple target log data, and send the target synchronization strategy to the data transmission module; The data transmission module is used to send the multiple target log data to the target system according to the target synchronization strategy.

2. The system according to claim 1, characterized in that The metadata management module includes: a metadata definition submodule and a metadata version management submodule; The metadata definition submodule is used to define the data model and data dictionary of the log and form the multiple log metadata; The metadata version management submodule is used to manage a target file, and the target file is used to record change information of the multiple log metadata.

3. The system according to claim 1, characterized in that The log collection module includes: multiple log collection components, a control submodule and a parsing submodule; The control submodule is used to load the multiple log collection components through a service discovery mechanism and register multiple collection nodes through a distributed coordination service framework; The log collection component is used to monitor and collect the multiple distributed log information based on the multiple collection nodes, and send the multiple distributed log information to the parsing submodule; The parsing submodule is used to parse the multiple distributed log information using the log parsing rules to obtain the multiple target log data.

4. The system according to claim 1, characterized in that The synchronization strategy formulation module includes: a first determination submodule, a second determination submodule, a third determination submodule and a strategy formulation submodule; The first determination submodule is used to determine the synchronization target according to the preset data requirements and the data types of the multiple target log data, and send the synchronization target to the strategy formulation submodule; The second determination submodule is used to determine a synchronization mode according to a plurality of target log data, and send the synchronization mode to the strategy formulation submodule; The third determination submodule is used to determine the synchronization frequency according to the preset business requirements and the change frequency of the multiple target log data, and send the synchronization frequency to the strategy formulation submodule; The strategy formulation submodule is used to determine the target synchronization strategy according to the synchronization target, the synchronization mode and the synchronization frequency.

5. The system according to claim 1, characterized in that The synchronization strategy formulation module also includes: The conflict handling submodule is used to determine priorities and resolution rules when conflicts occur among the plurality of target log data, and to handle the conflicts using the priorities and resolution rules.

6. The system according to claim 1, characterized in that The system further comprises: A first processing module, configured to control the log collection module to collect the plurality of distributed log information by using a preset retry mechanism when the plurality of target log data are not all sent to the target system; The second processing module is used to control the metadata management module to obtain the new plurality of log metadata when the log format is updated to the original format.

7. The system according to claim 1, characterized in that The system further comprises: The monitoring and alarm module is used to monitor the synchronization process of the plurality of target log data and to generate an alarm using a preset alarm mechanism when synchronization failure or synchronization delay is detected.

8. The system according to claim 1, characterized in that The system also includes: a testing module and an optimization module; The testing module is used to receive the target synchronization strategy sent by the synchronization strategy formulation module, test the target synchronization strategy, obtain the test result and send the test result to the optimization module; The optimization module is used to optimize the target synchronization strategy according to the test result.

9. A distributed data synchronization method, characterized in that: Used in the distributed data synchronization system according to any one of claims 1 to 8; the method comprising: Get multiple log metadata and multiple distributed log information; Determine a log parsing rule according to the multiple log metadata; Parsing the plurality of distributed log information using the log parsing rules to obtain a plurality of target log data; Formulate a target synchronization strategy according to the plurality of target log data; According to the target synchronization strategy, the plurality of target log data are sent to the target system.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the distributed data synchronization method as claimed in claim 9.

Images