Data synchronization method and device, electronic equipment and storage medium
By acquiring metadata information from the data source and adjusting the number of acquisition threads, and using parallel pipelines to transmit data, the load imbalance problem during data synchronization was solved, achieving load balancing and efficient data transmission.
Patent Information
- Application Number
- CN202511673565.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-02-06
AI Technical Summary
During data synchronization, existing technologies that directly adjust the data processing rate can easily lead to data congestion and load imbalance, failing to achieve load balancing and efficiency optimization.
By acquiring metadata information from the data source, we can determine the adjustment strategy for the number of acquisition threads, and use parallel pipelines to transmit data to achieve load balancing and fast synchronization.
This reduces the load on the data source and local system, ensures load balancing and data synchronization accuracy during data transmission, and improves data transmission efficiency.
Smart Images

Figure CN121478881A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a data synchronization method, apparatus, electronic device and storage medium. Background Technology
[0002] Driven by the wave of digital transformation, the data ecosystem is experiencing explosive growth, with data sources expanding from traditional structured databases to diverse and heterogeneous systems.
[0003] Currently, when data source pressure is high, automatic adjustment mechanisms are used to configure the data processing rate to prevent downstream data overload. However, directly adjusting the data processing rate can easily cause data congestion during the data synchronization process, leading to a load imbalance in the data synchronization process. Summary of the Invention
[0004] This invention provides a data synchronization method, apparatus, electronic device, and storage medium, which enables accurate adjustment of the number of acquisition threads, reduces the data transmission load pressure between the data source and the local device, and ensures load balance between the two ends during data transmission.
[0005] In a first aspect, embodiments of the present invention provide a data synchronization method, the method comprising:
[0006] Establish a connection with the data source, obtain and parse the metadata information of the data source, and obtain the characteristics of the data source.
[0007] The target metrics are determined based on the real-time operating metrics of the data source and the local real-time allowable performance metrics. The adjustment strategy for the number of collection threads is determined based on the target metrics, and the current number of collection threads is adjusted according to the adjustment strategy.
[0008] Based on the characteristics of the data source and the database logs, determine the content to be synchronized. Based on the content to be synchronized, determine the parallel pipeline and send the content to the parallel pipeline to synchronize the data to be synchronized.
[0009] The data synchronization method provided in this invention obtains data source characteristics that reflect the data attribute structure of each data source based on different data sources. This enables automatic identification and management of the characteristics of different data sources, enhancing the automation level and efficiency of metadata management, and providing a foundation for rapid data collection and synchronization. It acquires real-time operating metrics from the data source and corresponding real-time allowable performance metrics on the local machine, comparing the two to achieve a comprehensive understanding of the real-time load on both sides of the data interaction. Simultaneously, it can directly determine the adjustment strategy for the number of collection threads based on target metrics, solving the problem that current methods can only adjust the data processing rate under high load pressure. This allows for accurate adjustment of the number of collection threads, reducing the data transmission load pressure between the data source and the local machine while ensuring load balance between the two ends during data transmission. To cope with the simultaneous collection of large amounts of data from the data source, it uses parallel pipeline transmission of multiple synchronized contents, changing the "collection end" from single-threaded sequential reading to parallel streaming data transmission, achieving fast and accurate data transmission.
[0010] Secondly, embodiments of the present invention also provide a data synchronization device, the device comprising:
[0011] The acquisition module is used to establish a connection with the data source, acquire and parse the metadata information of the data source, and obtain the characteristics of the data source.
[0012] The adjustment module is used to determine the target metric based on the real-time operating metrics of the data source and the local real-time allowable performance metrics, determine the adjustment strategy for the number of acquisition threads based on the target metric, and adjust the current number of acquisition threads according to the adjustment strategy.
[0013] The synchronization module is used to determine the synchronization content of the data to be synchronized based on the characteristics of the data source and the database logs, determine the parallel pipeline based on the synchronization content, and send the synchronization content to the parallel pipeline to synchronize the data to be synchronized.
[0014] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising:
[0015] At least one processor; and
[0016] A memory that is communicatively connected to at least one processor; wherein,
[0017] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the data synchronization method of any embodiment of the present invention.
[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions that are used to cause a processor to execute and implement the data synchronization method of any embodiment of the present invention.
[0019] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the data synchronization method of any embodiment of the present invention.
[0020] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on a computer-readable storage medium. This computer-readable storage medium may be packaged together with the processor of the data synchronization device, or it may be packaged separately from the processor of the data synchronization device; this application does not impose any limitations on this.
[0021] The descriptions of the second, third, fourth, and fifth aspects in this application can be referenced to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second, third, fourth, and fifth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.
[0022] In this application, the name of the aforementioned data synchronization device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear under other names. As long as the function of each device or functional module is similar to that of this application, it falls within the scope of the claims of this application and its equivalents.
[0023] These or other aspects of this application will become more readily apparent in the following description. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1 A flowchart illustrating the data synchronization method provided in an embodiment of the present invention;
[0026] Figure 2 This is a schematic diagram of the structure of a data synchronization device provided in an embodiment of the present invention;
[0027] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0028] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.
[0029] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.
[0030] The terms “initial” and “target” in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.
[0031] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.
[0032] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc. Moreover, embodiments and features in the embodiments of the present invention can be combined with each other without conflict.
[0033] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0034] In the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0035] Figure 1This is a flowchart illustrating a data synchronization method provided in an embodiment of the present invention. This embodiment is applicable to situations where data is obtained from a data source and synchronized. The method can be executed by a data synchronization device, which can be implemented in software and / or hardware. In this embodiment, the data synchronization device is located in an electronic device, which can be a computer or a server. (Continue to refer to...) Figure 1 This embodiment specifically includes the following steps:
[0036] S101. Establish a connection with the data source, obtain and parse the metadata information of the data source, and obtain the characteristics of the data source.
[0037] The data source is where the data originates, that is, where the data is initially generated or persistently stored. Metadata information is the information used to describe the attributes of the data.
[0038] Specifically, before data collection can be performed, electronic devices need to establish connections with data sources, meaning they need to connect to different data sources before they can acquire data from these different sources. Furthermore, since different data sources have their own specific "syntactic structure," it is necessary to first read the metadata information of each data source and parse it to obtain the data source characteristics that reflect the data attribute structure within that data source.
[0039] For example, the metadata information of the data source is obtained and parsed to obtain the data source characteristics, including:
[0040] (i) Determine the acquisition method and metadata type based on the data source type, and acquire the metadata information corresponding to the metadata type based on the acquisition method.
[0041] Data source types include static data sources and dynamic data sources. Metadata types include quantity types, structure types, etc.
[0042] Specifically, different data source types require different acquisition methods, resulting in different metadata types. For example, if the data source is a distributed stream processing platform capable of high-throughput distributed message publishing, the acquisition method is through the data source's application programming interface (API); the metadata types include the number of partitions, the number of replicas, and the partitioning strategy. If the data source is a relational database, the acquisition method is a structured query statement; the metadata types include table structure and field types. Based on the acquisition method corresponding to the data source, the metadata information corresponding to the metadata type can be obtained.
[0043] (ii) Standardize and map the metadata information to obtain the characteristics of the data source.
[0044] Standardization and mapping are used to ensure that metadata from different data sources can be represented in a uniform way.
[0045] Specifically, metadata information can be standardized and mapped using a predefined data exchange format (JSON) to ultimately obtain data source characteristics.
[0046] Optionally, in this embodiment, metadata information can be obtained periodically from the data source and the characteristics of the data source can be obtained based on the above steps. In this way, when the characteristics of the data source change, the relevant information in the metadata registry can be automatically and quickly updated to ensure the real-time performance and accuracy of the metadata.
[0047] In this embodiment, the corresponding metadata information of different data sources is first obtained, and the metadata information is parsed, standardized and mapped to obtain the data source characteristics that can reflect the data attribute structure of each data source. This realizes the automatic identification and management of the characteristics of different data sources, enhances the automation level and efficiency of metadata management, and provides a foundation for rapid data collection and synchronization in the future.
[0048] S102. Determine the target metrics based on the real-time operating metrics of the data source and the local real-time allowable performance metrics, determine the adjustment strategy for the number of acquisition threads based on the target metrics, and adjust the current number of acquisition threads according to the adjustment strategy.
[0049] Among them, real-time performance metrics are used to determine the real-time load of the data source, and the real-time performance metrics differ for different data sources. Local refers to the data synchronization device, which is the electronic device in this embodiment. Real-time allowable performance metrics are used to determine the real-time load of the data synchronization device (electronic device). The number of acquisition threads refers to the number of worker threads concurrently executing data capture tasks during the data acquisition process. The current number of acquisition threads is the number of acquisition threads before adjustment according to the adjustment strategy.
[0050] Specifically, real-time operational metrics of the data source can be obtained, along with real-time local performance metrics. Then, metrics with poor load conditions are selected as target metrics. Based on these target metrics, an adjustment strategy for the current number of data collection threads is determined, and the number of data collection threads is adjusted accordingly.
[0051] In this embodiment, not only are real-time operating metrics from the data source acquired, but also local real-time allowable performance metrics corresponding to the real-time operating metrics are acquired and compared. This provides a comprehensive understanding of the real-time load of both parties involved in the data interaction. Furthermore, it allows for direct determination of the adjustment strategy for the number of acquisition threads based on target metrics. This solves the problem that current methods can only adjust the data processing rate when the load is high, enabling accurate adjustment of the number of acquisition threads. This reduces the data transmission load on both the data source and the local machine while ensuring load balance between the two ends during data transmission.
[0052] For example, the target metrics are determined based on real-time operational metrics from the data source and local real-time permissible performance metrics, including:
[0053] (i) Determine the real-time operating metrics of the data source based on the data source type of the data to be synchronized, and obtain the local real-time allowable performance metrics corresponding to the real-time operating metrics.
[0054] The data to be synchronized refers to the data that is prepared to be collected and synchronized from the data source. The data source type refers to the type of data source.
[0055] Specifically, different data sources have different real-time performance metrics. Therefore, the real-time performance metrics of a data source can be determined based on its type. For example, if the data source is a relational database, real-time performance metrics such as CPU utilization, I / O wait rate, number of connections, and query response time can be obtained. If the data source is a distributed stream processing platform, real-time performance metrics such as the number of partitions, message pair metering, and consumer latency can be obtained. Furthermore, for comparability, the specific real-time performance metrics required locally can be determined based on the obtained real-time performance metrics.
[0056] (ii) Determine the target indicators based on real-time operating indicators, real-time permissible performance indicators, and single indicator thresholds.
[0057] Among them, the single indicator threshold is the threshold corresponding to a single indicator.
[0058] Specifically, real-time operating metrics can be compared with real-time permissible performance metrics. The metric with the worse data can be selected as a candidate metric. Then, the candidate metric can be compared with its corresponding single-metric threshold. If it exceeds the single-metric threshold, it can be designated as the target metric; otherwise, it is not considered a target metric. For example, if the real-time operating metric is CPU utilization, the corresponding local real-time permissible performance metric can be obtained, and the one with the larger value can be selected as a candidate metric. This value can then be compared with the single-metric threshold for CPU utilization. If the candidate metric exceeds the single-metric threshold, it can be designated as the target metric. Other metrics can be determined using the same steps.
[0059] In this embodiment, the indicators with poor real-time load conditions are compared with the corresponding single indicator thresholds to further determine whether the indicators with poor load conditions have exceeded the critical value. The indicators that exceed the critical value are used as target indicators. This achieves a comprehensive comparison of the load conditions of both parties before comparing with the critical value, providing a basis for comprehensively and accurately adjusting the load conditions during data transmission.
[0060] For example, the strategy for adjusting the number of data collection threads based on the target metric includes:
[0061] (i) Determine the total indicator parameters based on the indicator parameters and indicator weights corresponding to each indicator in the target indicators.
[0062] In this embodiment, the target metric includes at least one metric. The metric parameter is the parameter value corresponding to the metric. The metric weight is used to characterize the importance of the metric compared to other metrics, or to characterize the degree of influence of the metric on the load situation.
[0063] Specifically, the parameter value of each indicator can be multiplied by its corresponding weight to obtain the weight value for each indicator. Then, the weight values of all indicators are added together to obtain the total indicator parameters.
[0064] (ii) Determine whether the total indicator parameter exceeds the total indicator threshold; if it exceeds, proceed to (iii); if it does not exceed, proceed to (iv).
[0065] The overall index threshold is the critical load value allowed by the system, which can be determined in practice based on different data sources and the load capacity of electronic devices; different data sources can ultimately correspond to different overall index thresholds.
[0066] Specifically, after calculating the total indicator parameters, the total indicator threshold is determined based on the data source, and the total indicator parameters are compared with the total indicator threshold. Based on the comparison result, it is determined whether to execute (III) or (IV).
[0067] (iii) If the total indicator parameter exceeds the total indicator threshold, the excess amount is determined based on the total indicator parameter and the total indicator threshold, and the number of additional collection threads is determined based on the excess amount.
[0068] Specifically, if the total indicator parameter exceeds the total indicator threshold, it indicates that the transmission pressure between the current data source and the local machine is already quite high. Therefore, in order to quickly alleviate the transmission pressure at both ends, the difference between the total indicator parameter and the total indicator threshold, i.e. the excess amount, can be directly calculated, and the number of additional collection threads that need to be added can be determined directly based on the excess amount.
[0069] (iv) If the total indicator parameters do not exceed the total indicator threshold, the real-time gap corresponding to each indicator is determined based on the indicator parameters corresponding to each indicator and the single indicator threshold corresponding to the indicator parameters, and the number of collection threads to be changed is determined based on the real-time gap corresponding to each indicator and the indicator weight corresponding to each indicator.
[0070] Among them, the single indicator threshold is the value corresponding to the indicator parameter of an indicator, which is used to limit the limit value of the indicator.
[0071] Specifically, if the total metric parameters do not exceed the total metric threshold, it can be determined that the data transmission pressure on the data source and the local system is not very high. Therefore, the number of data collection threads can be fine-tuned. That is, the difference between the metric parameter corresponding to each metric and the individual metric threshold corresponding to that parameter is calculated; this is the real-time difference. For example, by subtracting the corresponding individual metric threshold from the metric parameter of each metric, a real-time difference (positive or negative) can be obtained. Furthermore, based on the real-time difference for each metric and its corresponding metric weight, the number of data collection threads that need to be reduced can be determined.
[0072] For example, in one implementation, if the real-time difference is the indicator parameter minus the single indicator threshold, then when the real-time difference is positive, the indicator corresponding to the maximum positive value is selected as the target indicator, and the number of collection threads is increased based on the real-time difference corresponding to the target indicator; when the real-time difference is negative, the indicator corresponding to the maximum negative value and the indicator corresponding to the minimum negative value are selected as the target indicators, the number of collection threads to be increased is determined based on the indicator corresponding to the maximum negative value, the number of collection threads to be decreased is determined based on the indicator corresponding to the minimum negative value, and the final number of collection threads to be changed is determined based on the number of collection threads to be increased and the number of collection threads to be decreased.
[0073] Another implementation method determines the number of data collection threads to be changed based on the real-time difference corresponding to each indicator and the indicator weight corresponding to each indicator, including:
[0074] (1) Determine the impact parameters of each indicator based on the real-time gap and the indicator weight of each indicator.
[0075] Specifically, the impact parameter of each indicator can be obtained by multiplying the real-time difference corresponding to each indicator by its corresponding indicator weight.
[0076] (2) Sum the influence parameters of all indicators to obtain the total influence parameter, and determine the number of collection threads to be changed based on the total influence parameter and the fuzzy table.
[0077] The fuzzy table includes the change in the number of sampling threads when the total parameter is affected by different ranges.
[0078] Specifically, the total influencing parameters of all indicators are summed to obtain the total influencing parameters. At this point, the number of data collection threads that need to be changed can be determined based on the fuzzy table.
[0079] Optionally, the specific number of data acquisition threads to be increased or decreased can be determined based on a fuzzy table. For example, each entry in a fuzzy table represents the relationship between the range of differences between the indicator parameters of different indicators and their corresponding single indicator thresholds, as determined during the experiment or trial, and the relationship between the range of different total influencing parameters and the number of data acquisition threads to be changed; and the relationship between the range of different total influencing parameters and the number of data acquisition threads to be changed. Therefore, when the differences between the indicator parameters of different indicators and their corresponding single indicator thresholds or the total influencing parameters have been determined as described above, the final number of data acquisition threads to be increased or decreased can be determined based on the corresponding entries in the fuzzy table.
[0080] In this embodiment, when the total indicator parameter exceeds the total indicator threshold, it is determined that the data transmission pressure needs to be quickly reduced. Therefore, the excess amount between the total indicator parameter and the total indicator threshold is directly calculated, and the number of additional acquisition threads to be added is determined directly based on the excess amount. This enables the rapid and accurate determination of the number of acquisition threads to be added when there is an urgent need to reduce data transmission pressure, ensuring a rapid reduction in data transmission pressure and achieving stability and security of data transmission at both ends. Simultaneously, when the total indicator parameter does not exceed the total indicator threshold, to ensure system balance, the number of acquisition threads can be fine-tuned based on the specific circumstances of different indicators. That is, the real-time difference corresponding to each indicator is determined, and the number of acquisition threads to be changed is determined based on the real-time difference and indicator weight. This achieves fine-tuning of the number of acquisition threads based on actual data transmission pressure, reducing local resource consumption while ensuring stable and secure data transmission at both ends.
[0081] S103. Determine the synchronization content of the data to be synchronized based on the characteristics of the data source and the database logs, determine the parallel pipeline based on the synchronization content, and send the synchronization content to the parallel pipeline to synchronize the data to be synchronized.
[0082] The database log records all transactional and system events that occur in the database. The parallel pipeline, as configured in this embodiment, is a pipeline capable of processing events simultaneously.
[0083] Specifically, all data is fully loaded into the database the first time it is transmitted locally. Subsequent transmissions, when data is transmitted again, the content to be synchronized can be determined based on the data source characteristics and database logs. For example, by parsing the corresponding content in the database logs based on the data source characteristics and obtaining the changed data, this changed data becomes the content to be synchronized. Furthermore, since the data collected from the data source is usually not singular but rather a large volume, different pipelines can be assigned to different synchronization contents, and multiple synchronization contents can be distributed to their corresponding parallel pipelines for synchronization.
[0084] In this embodiment, in order to cope with the simultaneous collection of a large amount of data from the data source, multiple synchronous contents are transmitted based on parallel pipeline, which changes the "collection end" from single-threaded sequential reading to parallel streaming data transmission, thus realizing fast and accurate data transmission.
[0085] For example, the synchronization content of the data to be synchronized is determined based on the characteristics of the data source and the database logs, and a parallel pipeline is determined based on the synchronization content, including:
[0086] (i) Determine the structure of the data to be synchronized based on the characteristics of the data source, map the database logs to the user space memory through memory according to the structure, and perform parallel parsing on the information after memory mapping to obtain the parsed event body.
[0087] The user-mode memory space is a "private and protected workbench" allocated locally for each running process. The parsed event body is a well-structured memory object that is easy for the program to use.
[0088] Specifically, the structure of the data to be synchronized can be determined based on the characteristics of the data source. This structure can be used to decode log information in the database log. Therefore, the location of the required information in the database log can be determined based on the structure, and the required information can be directly mapped from the database log to the user-space memory space via memory mapping. In this way, electronic devices can directly access the log file through memory pointers without having to "copy" the required information. Finally, since the log information in the database is binary data, it can be parsed in parallel to obtain the parsed event body.
[0089] (ii) Determine the changed data based on the event type of the parsed event body, and identify the changed data as the synchronization content.
[0090] Specifically, by parsing the event type of the event body, consecutive changes to the same row of data can be merged by primary key to obtain the changed data. This changed data is then identified as the synchronized content.
[0091] (iii) Construct routing keys based on the synchronized content, calculate the summary data of the routing keys, and determine the parallel pipeline based on the summary data.
[0092] The routing key is used to identify different synchronization content, thereby determining the parallel pipeline corresponding to that synchronization content.
[0093] Specifically, the "primary key," "shard key," and "business logic key" of the synchronized content can be used as routing keys for the synchronized content. Then, a digest of the routing key is calculated, for example, by calculating a hash function of the routing key. Finally, the parallel pipeline can be determined based on the digest data. One implementation method is to pre-define the parallel pipeline index corresponding to each digest data. Another implementation method is to perform mathematical calculations on the digest data, such as modulo operations, and the resulting value is the parallel pipeline index. The parallel pipeline can then be determined based on the pipeline index.
[0094] In this embodiment, firstly, information from the database logs is directly mapped to user-space memory via memory mapping, achieving memory mapping and zero-copy. This enables the most efficient conversion of changes to the monitoring database into events, providing a foundation for subsequent data synchronization. Furthermore, based on parallel pipelines, data with the same business semantics can be allocated to the same processing pipeline. This not only ensures synchronous and real-time data processing but also allows unrelated data to run in parallel while establishing temporal relationships between related data, improving data throughput while maintaining data coherence.
[0095] For example, sending synchronization content to a parallel pipeline for synchronizing the data to be synchronized includes:
[0096] (i) If there are multiple synchronous contents from different data sources in the parallel pipeline, a logical timestamp is assigned to each synchronous content.
[0097] In this embodiment, multiple synchronized contents from different data sources are in the same pipeline and belong to the related data mentioned above, such as the data at the current moment and the updated data at the next moment.
[0098] Specifically, to resolve data transmission conflicts, when multiple related synchronized contents from different data sources exist in a parallel pipeline, a logical timestamp can be assigned to each synchronized content. For example, a vector can be used to timestamp each write operation, with the vector length being the number of nodes, and each component representing the latest write sequence number seen by the corresponding node.
[0099] (ii) Compare the logical timestamps of each pair of synchronized content to obtain the sorting order of the logical timestamps of all synchronized content.
[0100] Specifically, the logical timestamps of each synchronized content can be compared pairwise to determine the order of logical timestamps. For example, by comparing vector timestamps, assuming A and B are both synchronized data, if A is an ancestor of B, then B is the latest data; if B is an ancestor of A, then A is the latest data. Therefore, one implementation is to send the older data first, then the new data; another implementation is to send the latest data directly. Furthermore, there is a case where A and B are concurrent, in which case merging is required. This can be done through set rules (such as larger physical timestamps, priority based on business fields, or aggregation of numeric fields), client notifications, and immutable logs.
[0101] (iii) The synchronization content is sent to the parallel pipeline in the order of arrangement to synchronize the data to be synchronized.
[0102] Specifically, once the order of arrangement or the data to be distributed is determined, the synchronization content can be distributed to the corresponding parallel pipeline to synchronize the data to be synchronized.
[0103] In this embodiment, the comparison based on logical timestamps can ensure that data replicas on different nodes remain consistent and that the order of messages is consistent, thereby ensuring data consistency across data sources.
[0104] Optionally, in this embodiment, S102 can be a real-time step or a periodic adjustment step performed before S103. That is, S102 in this embodiment is mainly used to adjust the load pressure at both ends of the data synchronization process after the electronic device and the data source establish a connection.
[0105] Optionally, the data synchronization device in this embodiment can have an acquisition architecture consisting of a protocol adaptation layer, a stream processing layer, and elastic scheduling. The protocol adaptation layer, based on a non-blocking input / output model, provides an asynchronous and event-driven network application framework to build a high-throughput protocol gateway. It supports dynamic adjustment of multiple protocols and handles protocol parsing through a chain-of-responsibility pattern. The stream processing layer employs a serverless architecture programming model of stateful, elastic distributed applications to implement stateful computation, including a message middleware, and handles out-of-order events through a mechanism that balances computational integrity and latency. The elastic scheduling uses an elastic scaling controller that automatically adjusts the number of application replicas based on real-time load metrics to achieve dynamic scaling, while also incorporating custom metrics to drive resource allocation.
[0106] The data synchronization method provided in this invention obtains data source characteristics that reflect the data attribute structure of each data source based on different data sources. This enables automatic identification and management of the characteristics of different data sources, enhancing the automation level and efficiency of metadata management, and providing a foundation for rapid data collection and synchronization. It acquires real-time operating metrics from the data source and corresponding real-time allowable performance metrics on the local machine, comparing the two to achieve a comprehensive understanding of the real-time load on both sides of the data interaction. Simultaneously, it can directly determine the adjustment strategy for the number of collection threads based on target metrics, solving the problem that current methods can only adjust the data processing rate under high load pressure. This allows for accurate adjustment of the number of collection threads, reducing the data transmission load pressure between the data source and the local machine while ensuring load balance between the two ends during data transmission. To cope with the simultaneous collection of large amounts of data from the data source, it uses parallel pipeline transmission of multiple synchronized contents, changing the "collection end" from single-threaded sequential reading to parallel streaming data transmission, achieving fast and accurate data transmission.
[0107] Figure 2 This is a schematic diagram of the structure of the data synchronization device provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the device includes:
[0108] The acquisition module 201 is used to establish a connection with the data source, acquire and parse the metadata information of the data source, and obtain the characteristics of the data source.
[0109] The adjustment module 202 is used to determine the target indicator based on the real-time operating indicators of the data source and the local real-time allowable performance indicators, determine the adjustment strategy for the number of collection threads based on the target indicator, and adjust the current number of collection threads according to the adjustment strategy.
[0110] Synchronization module 203 is used to determine the synchronization content of the data to be synchronized based on the characteristics of the data source and the database logs, determine the parallel pipeline based on the synchronization content, and send the synchronization content to the parallel pipeline to synchronize the data to be synchronized.
[0111] Based on the above embodiments, the acquisition module 201 is specifically used for:
[0112] The acquisition method and metadata type are determined based on the data source type, and the metadata information corresponding to the metadata type is obtained based on the acquisition method; the metadata information is standardized and mapped to obtain the data source characteristics.
[0113] Based on the above embodiments, the adjustment module 202 is specifically used for:
[0114] Determine the real-time operating metrics of the data source based on the data source type of the data to be synchronized, and obtain the local real-time allowable performance metrics corresponding to the real-time operating metrics; determine the target metrics based on the real-time operating metrics, real-time allowable performance metrics, and single metric thresholds.
[0115] Based on the above embodiments, an adjustment strategy for the number of acquisition threads is determined according to the target indicators. The adjustment module 202 is specifically used for:
[0116] Based on the indicator parameters and weights corresponding to each indicator in the target indicators, determine the total indicator parameters; determine whether the total indicator parameters exceed the total indicator threshold; if the total indicator parameters exceed the total indicator threshold, determine the excess amount based on the total indicator parameters and the total indicator threshold, and determine the number of additional collection threads based on the excess amount; if the total indicator parameters do not exceed the total indicator threshold, determine the real-time difference corresponding to each indicator based on the indicator parameters and the single indicator threshold corresponding to each indicator parameter, and determine the number of changed collection threads based on the real-time difference corresponding to each indicator and the indicator weight corresponding to each indicator.
[0117] Based on the above embodiments, the number of data collection threads to be changed is determined according to the real-time gap corresponding to each indicator and the indicator weight corresponding to each indicator. The adjustment module 202 is specifically used for:
[0118] Based on the real-time gap and the weight of each indicator, the influence parameters of each indicator are determined; the influence parameters of all indicators are summed to obtain the total influence parameters, and the number of data collection threads to be changed is determined based on the total influence parameters and the fuzzy table.
[0119] Based on the above embodiments, the synchronization module 203 is specifically used for:
[0120] The structure of the data to be synchronized is determined based on the characteristics of the data source. Based on the structure, the database logs are mapped to the user-space memory space via memory mapping, and the information after memory mapping is parsed in parallel to obtain the parsed event body. The changed data is determined based on the event type of the parsed event body, and the changed data is determined as the synchronization content. The routing key is constructed based on the synchronization content, the summary data of the routing key is calculated, and the parallel pipeline is determined based on the summary data.
[0121] Based on the above embodiments, the synchronization module 203 is specifically used for:
[0122] If there are multiple synchronized contents from different data sources in the parallel pipeline, a logical timestamp is assigned to each synchronized content; the logical timestamps of each pair of synchronized contents are compared to obtain the sorting order of the logical timestamps of all synchronized contents; and the synchronized contents are sent to the parallel pipeline in the sorting order to synchronize the data to be synchronized.
[0123] The data synchronization device provided in the embodiments of the present invention can execute the data synchronization method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.
[0124] It is worth noting that in the above-described embodiments of the data synchronization device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.
[0125] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 3 A block diagram is shown of an exemplary electronic device 11 suitable for implementing embodiments of the present invention. Figure 3 The electronic device 11 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.
[0126] like Figure 3 As shown, the electronic device 11 is represented in the form of a general-purpose computing electronic device. The components of the electronic device 11 may include, but are not limited to: one or more processors or processing units 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processing unit 16).
[0127] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.
[0128] Electronic device 11 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by electronic device 11, including volatile and non-volatile media, removable and non-removable media.
[0129] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 11 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (… Figure 3 Not shown; usually referred to as a "hard drive"). Although Figure 3As not shown, disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disc drives for reading and writing to removable non-volatile optical discs (e.g., CD-ROMs, DVD-ROMs, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of the present invention.
[0130] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of the present invention.
[0131] Electronic device 11 can also communicate with one or more external devices 14 (e.g., keyboard, pointing device, display 24, etc.), and with one or more devices that enable a user to interact with electronic device 11, and / or with any device that enables electronic device 11 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, electronic device 11 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 20. Figure 3 As shown, network adapter 20 communicates with other modules of electronic device 11 via bus 18. It should be understood that, although... Figure 3 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 11, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0132] The processing unit 16 executes various functional applications and page displays by running programs stored in the system memory 28, such as implementing the data synchronization method provided in this embodiment. Of course, those skilled in the art will understand that the processor can also implement the technical solutions of the data synchronization method provided in any embodiment of this invention.
[0133] This invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements, for example, the data synchronization method provided in this invention. The computer storage medium of this invention can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0134] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.
[0135] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0136] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data synchronization method provided in any embodiment of this invention. The computer program code for performing the operations of this invention can be written in one or more programming languages or a combination thereof. Programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0137] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.
[0138] Furthermore, the acquisition, storage, use, and processing of data in the technical solution of this invention all comply with the relevant provisions of national laws and regulations.
[0139] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.
Claims
1. A data synchronization method, characterized in that, The method includes: Establish a connection with the data source, obtain and parse the metadata information of the data source, and obtain the characteristics of the data source; The target metric is determined based on the real-time operating metrics of the data source and the local real-time allowable performance metrics. The adjustment strategy for the number of acquisition threads is determined based on the target metric, and the current number of acquisition threads is adjusted according to the adjustment strategy. Based on the characteristics of the data source and the database logs, the synchronization content of the data to be synchronized is determined. Based on the synchronization content, a parallel pipeline is determined, and the synchronization content is sent to the parallel pipeline to synchronize the data to be synchronized.
2. The method according to claim 1, characterized in that, The process of acquiring and parsing the metadata information of the data source to obtain data source characteristics includes: The acquisition method and metadata type are determined based on the data source type, and the metadata information corresponding to the metadata type is acquired based on the acquisition method. The metadata information is standardized and mapped to obtain the characteristics of the data source.
3. The method according to claim 1, characterized in that, The process of determining the target metrics based on real-time operational metrics from the data source and local real-time permissible performance metrics includes: The real-time operating metrics of the data source are determined based on the data source type of the data to be synchronized, and the local real-time allowable performance metrics corresponding to the real-time operating metrics are obtained. The target indicator is determined based on the real-time operating indicators, the real-time allowed performance indicators, and the single indicator threshold.
4. The method according to claim 3, characterized in that, The adjustment strategy for determining the number of data collection threads based on the target metric includes: The total indicator parameters are determined based on the indicator parameters and indicator weights corresponding to each of the target indicators. Determine whether the total indicator parameter exceeds the total indicator threshold; If the total indicator parameter exceeds the total indicator threshold, the excess amount is determined based on the total indicator parameter and the total indicator threshold, and the number of additional collection threads is determined based on the excess amount. If the total indicator parameter does not exceed the total indicator threshold, then the real-time gap corresponding to each indicator is determined according to the indicator parameter corresponding to each indicator and the single indicator threshold corresponding to the indicator parameter, and the number of collection threads to be changed is determined according to the real-time gap corresponding to each indicator and the indicator weight corresponding to each indicator.
5. The method according to claim 4, characterized in that, The step of determining the number of data collection threads to be changed based on the real-time difference corresponding to each indicator and the indicator weight corresponding to each indicator includes: The impact parameters of each indicator are determined based on the real-time gap corresponding to each indicator and the indicator weight corresponding to each indicator. Sum the influence parameters of all indicators to obtain the total influence parameter, and determine the number of data collection threads to be changed based on the total influence parameter and the fuzzy table.
6. The method according to claim 1, characterized in that, The step of determining the synchronization content of the data to be synchronized based on the characteristics of the data source and the database logs, and determining the parallel pipeline based on the synchronization content, includes: The structure of the data to be synchronized is determined based on the characteristics of the data source. Based on the structure, the database logs are mapped to the user-space memory space via memory mapping. The information after memory mapping is parsed in parallel to obtain the parsed event body. The changed data is determined based on the event type of the parsed event body, and the changed data is identified as the synchronization content; A routing key is constructed based on the synchronization content, a summary data of the routing key is calculated, and the parallel pipeline is determined based on the summary data.
7. The method according to claim 1, characterized in that, Sending the synchronization content to the parallel pipeline to synchronize the data to be synchronized includes: If there are multiple synchronized contents from different data sources in the parallel pipeline, a logical timestamp is assigned to each synchronized content. By comparing the logical timestamps of each pair of synchronized content, the logical timestamps of all synchronized content are arranged in the correct order. The synchronization content is sent to the parallel pipeline in the order described to synchronize the data to be synchronized.
8. A data synchronization device, characterized in that, The device includes: The acquisition module is used to establish a connection with the data source, acquire and parse the metadata information of the data source, and obtain the characteristics of the data source; The adjustment module is used to determine the target indicator based on the real-time operating indicators of the data source and the local real-time allowable performance indicators, determine the adjustment strategy for the number of acquisition threads based on the target indicator, and adjust the current number of acquisition threads according to the adjustment strategy. The synchronization module is used to determine the synchronization content of the data to be synchronized based on the characteristics of the data source and the database logs, determine the parallel pipeline based on the synchronization content, and send the synchronization content to the parallel pipeline to synchronize the data to be synchronized.
9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the data synchronization method as described in any one of claims 1 to 7.
10. A readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data synchronization method as described in any one of claims 1 to 7.
Citation Information
Cited By
Table structure consistency checking and repairing method and system for GoldenDB database
CN122086673A
