Enterprise multi-source data real-time synchronization system based on data platform

Abstract

The application relates to a kind of enterprise multi-source data real-time synchronization systems based on data platform, the system adopts containerization deployment mode and layered architecture, including access layer, processing layer, distribution layer and control layer;Access layer communicates with data platform, obtains original data from data platform;Access layer is provided with data access adapter module, processing layer is provided with data format processing module, distribution layer is provided with data routing distribution module, control layer is provided with consistency guarantee module;Data access adapter module is used for the unified access of multi-source heterogeneous data;Data format processing module is used for data standardization processing to the multi-source heterogeneous data accessed;Data routing distribution module is used for intelligent data routing and distribution;Consistency guarantee module is used to ensure the consistency of data synchronization.The application reduces the delay of data synchronization, improves the system throughput, effectively guarantees the consistency of data, and is especially suitable for data synchronization and cross-system data integration scenarios of enterprise core business systems.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing and distributed systems, and particularly relates to a multi-source data real-time synchronization system based on a data middle platform. BACKGROUND

[0002] With the deepening of enterprise digital transformation, the number of internal systems of enterprises is increasing, and the data synchronization demand is becoming increasingly complex. The existing data synchronization scheme mainly has the following problems:

[0003] 1. Heterogeneous data source problem:

[0004] There are various types of data source systems in enterprises, and the data format and interface standard are not unified, the data synchronization adaptation development cost is high, and the system expansion and maintenance are difficult.

[0005] 2. Real-time guarantee problem:

[0006] The traditional batch synchronization method has large delay, the real-time synchronization mechanism is not perfect, the performance bottleneck problem is prominent, and it is difficult to meet the real-time requirements of businesses.

[0007] 3. Data consistency problem:

[0008] In a distributed environment, data consistency is difficult to guarantee, data is easy to be inconsistent under abnormal conditions, the data checking mechanism is not perfect, the repair cost is high and the efficiency is low.

[0009] 4. System coordination problem:

[0010] The integration with the data middle platform is not enough, there is a lack of unified management and control, the security mechanism is not perfect, and the operation and maintenance cost is high. SUMMARY

[0011] The purpose of the application is to provide a multi-source data real-time synchronization system based on a data middle platform to solve the real-time synchronization problem of multi-source heterogeneous data in enterprises.

[0012] The application provides a multi-source data real-time synchronization system based on a data middle platform, which adopts a containerized deployment method and a layered architecture, including a data access layer, a data processing layer, a data distribution layer and a data control layer; the data access layer communicates with the data middle platform to obtain raw data from the data middle platform;

[0013] The data access layer is provided with a data access adapter module, the data processing layer is provided with a data format processing module, the data distribution layer is provided with a data routing and distribution module, and the data control layer is provided with a consistency guarantee module; the layers communicate through standardized interfaces to ensure the scalability and maintainability of the system;

[0014] The data access adapter module is used for unified access of multi-source heterogeneous data, and includes a relational database adaptation unit, a distributed database adaptation unit, a data warehouse adaptation unit and a search engine adaptation unit, which are respectively used for accessing relational databases, distributed databases, data warehouses and search engines.

[0015] The data format processing module is used for data standardization processing of the accessed multi-source heterogeneous data, and sends the processed standardized data to the data routing and distribution module.

[0016] The data routing and distribution module is used for:

[0017] In the data routing process, the load state of each node is monitored in real time, and the ARIMA model is used for dynamic load prediction in combination with historical data.

[0018] Based on the load prediction result, an adaptive routing strategy is adopted to dynamically distribute data to the optimal node, and a flow control mechanism based on a message middleware is adopted to prevent system overload and ensure the stability of data processing; wherein the adaptive routing strategy considers node load, network condition and data affinity factors, and performs optimal routing selection based on an ant colony algorithm to ensure the balance and efficiency of data distribution.

[0019] The consistency guarantee module is used for:

[0020] Real-time change information of source data is captured, a multi-level consistency check method based on hash check is adopted, data of the source end and the target end is periodically checked for the established data consistency check point, if data anomaly is found, data repair is automatically performed to ensure the consistency of data synchronization, and a distributed transaction coordination mechanism is adopted to ensure the atomicity and consistency of data operation in a distributed environment.

[0021] Further, the data access adapter module adopts a plug-in structure, supports dynamic loading and management, and individual extension for specific data sources; the data access adapter module includes an adapter management unit for unified management of the life cycle of various adapters, and realizes dynamic deployment and upgrade of the adapters.

[0022] Further, the data format processing module includes:

[0023] A data format identification unit is used for identifying data format types.

[0024] A data cleaning unit is used for cleaning abnormal data.

[0025] A data conversion unit is used for data format conversion.

[0026] A data standardization unit is used for unifying data formats.

[0027] A quality control unit for data quality check.

[0028] Further, the step of performing optimal routing based on the ant colony algorithm is as follows:

[0029] Step 1: Initialization:

[0030] Initial pheromone concentration, set to a constant (τ ij (0) = τ0); Set to initial value (τ0); Initialize system state: Collect the current load (L j ) and path bandwidth (B ij ) of each node; Step 2: Calculate path weight:

[0031] Comprehensive weight formula:

[0032]

[0033] Where:

[0034] (L j ): Load of node (j), the lower the better;

[0035] (B ij ): Remaining bandwidth of path (P ij ), the higher the better;

[0036] (α, β): Weight factor, control the influence of load and bandwidth on decision;

[0037] Step 3: Path selection:

[0038] Selection probability formula:

[0039]

[0040] Where:

[0041] τ ij (t): Pheromone concentration of path (P ij );

[0042] (W ij ): Comprehensive weight of path (P ij );

[0043] (η, γ): Parameters, respectively control the importance of pheromone and path weight;

[0044] Step 4: Pheromone update:

[0045] Update formula:

[0046] τ ij (t+1) = (1-ρ)τ ij (t) + Δτij ;

[0047] Wherein:

[0048] (1-p): pheromone volatile factor, prevent infinite growth of pheromone;

[0049] (Δτ ij ): new pheromone, indicating path quality.

[0050] Further, the multi-level consistency check method based on hash check is as follows:

[0051] 1) generating a hash value for the transmission data;

[0052] 2) calculating the hash value again after receiving the data at the target end;

[0053] 3) comparing the hash values of the source end and the target end;

[0054] 4) if the data is inconsistent, trigger the repair process, accurately locate and repair abnormal data by backtracking historical change records.

[0055] Further, the synchronization system further comprises a performance optimization module, the performance optimization module comprising:

[0056] Increment identification unit, for identifying data increment change;

[0057] Parallel processing unit, for dispersing data processing tasks to multiple nodes for parallel execution based on the parallel processing mechanism of the distributed computing framework;

[0058] Data compression unit, for data transmission compression optimization;

[0059] Resource scheduling unit, realizing resource scheduling based on container technology;

[0060] Performance monitoring unit, for system performance monitoring.

[0061] Further, the increment identification unit is specifically used for:

[0062] Record the synchronized data through the Bloom filter, and only identify the newly added or changed data to reduce the data transmission amount, and the specific process is as follows:

[0063] 1) initializing the Bloom filter and loading the source data hash;

[0064] 2) calculating the hash value of the new data, and checking whether the Bloom filter already exists;

[0065] 3) adding the new data to the synchronization task.

[0066] Further, the data compression unit is specifically used for:

[0067] Based on the efficient LZ4 compression algorithm, an intelligent data compression strategy is implemented. The specific process is as follows:

[0068] 1) Process the input data stream in blocks and extract repeating patterns;

[0069] 2) Match data blocks using a sliding window and replace them with pointers or symbols;

[0070] 3) The compressed data is transmitted and then decompressed at the target end.

[0071] Furthermore, the resource scheduling unit is specifically used for:

[0072] Based on a containerized resource scheduling mechanism, the synchronization system can dynamically adjust the allocation of computing resources according to the load to ensure optimal processing performance.

[0073] Furthermore, the synchronization system also includes a security control module, which includes:

[0074] The identity authentication unit is used for unified identity authentication;

[0075] Access control unit, used to implement role-based access control;

[0076] The data encryption unit is used to implement secure encryption at the transport layer.

[0077] The data desensitization unit is used to process sensitive data;

[0078] The audit log unit is used to record data access logs.

[0079] By employing the above solution, an enterprise multi-source data real-time synchronization system based on a data middle platform achieves reduced data synchronization latency, increased system throughput, and effective assurance of data consistency. It is particularly suitable for data synchronization of core enterprise business systems and cross-system data integration scenarios. Specific technical effects are as follows:

[0080] 1. The system adopts a containerized deployment approach, supporting elastic scaling and ensuring reliable operation through a unified service governance framework. Communication between different layers is achieved through standardized interfaces, ensuring system scalability and maintainability. A layered architecture is employed: at the access layer, enterprise-grade data access adapters enable unified access to multi-source heterogeneous data; at the processing layer, standardized data format processing modules enable unified data processing; at the distribution layer, an intelligent routing mechanism based on a stream processing framework achieves efficient data distribution; and at the control layer, a distributed consistency guarantee mechanism ensures reliable real-time data synchronization.

[0081] 2、The system adopts a data access adaptation mechanism to realize a unified data access adaptation framework, supporting access to multiple heterogeneous data sources. The system provides standardized adapter interface specifications, supporting access to multiple types of data sources such as relational databases (e.g., MySQL), distributed databases (e.g., HBase), data warehouses (e.g., Hive), search engines (e.g., Elasticsearch), etc. The adapter uses a plug-in design, supporting dynamic loading and management. Each adapter implements standard functions such as data reading, format conversion, and state maintenance, and supports personalized extensions for specific data sources. The system manages the life cycle of various adapters through an adapter management unit, enabling dynamic deployment and upgrading of adapters.

[0082] 3、The system's intelligent data routing mechanism based on a stream processing framework realizes dynamic load prediction and adaptive routing strategies. In the data routing process, understanding the real-time load of each node helps to distribute tasks more efficiently, thereby avoiding overload. The system monitors the load status of each node in real time, combines historical data analysis, and realizes dynamic load prediction based on the ARIMA model. Based on the prediction results, the system uses adaptive routing strategies to dynamically distribute data streams to the optimal nodes. The routing strategy considers multiple factors such as node load, network conditions, and data affinity to ensure the balance and efficiency of data distribution. The system uses an optimal routing selection based on the ant colony algorithm to solve the problem of efficiency decline caused by node load and network bandwidth in data routing, improving the overall performance of the system. The system uses a traffic control mechanism based on message middleware to prevent system overload and ensure the stability of data processing.

[0083] 4、The system establishes a complete distributed consistency guarantee mechanism, using multi-level data verification based on database change capture (CDC) technology to capture source data changes in real time. The system uses multi-level consistency verification based on hash verification to quickly detect data anomalies, and when inconsistencies are detected, the system automatically triggers data repair processes. At the same time, the system uses a distributed transaction coordination mechanism to ensure the atomicity and consistency of data operations in a distributed environment.

[0084] 5、The system realizes a high-performance data processing mechanism, ensuring the real-time performance of the system. At the data collection level, the system realizes incremental data recognition based on a Bloom filter, synchronizing only data that has changed, significantly reducing data transmission volume and solving the problem of high cost of full data synchronization, significantly reducing computing and transmission costs. At the data processing level, the system uses a parallel processing mechanism based on a distributed computing framework to disperse data processing tasks to multiple nodes for parallel execution, providing high-performance data processing capabilities. The system uses an efficient compression algorithm based on LZ4 to realize an intelligent data compression strategy, reducing network bandwidth occupancy while ensuring performance. The system uses a containerized resource scheduling mechanism to dynamically adjust the allocation of computing resources according to load conditions, ensuring the optimization of processing performance.

[0085] 6、The system establishes a perfect security control mechanism to realize the security protection of data transmission and access. The system realizes fine-grained management of user permissions through single sign-on (SSO) and role-based access control (RBAC). At the data transmission level, the system uses the Transport Layer Security (TLS) protocol for encryption to ensure the security of data transmission. The system supports flexible data desensitization strategies, allowing different desensitization rules to be configured for different data fields to protect sensitive data. At the same time, the system implements a complete audit log mechanism to record all data access and operation behaviors, supporting subsequent security audits and traceability.

[0086] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, and to implement the content of the specification, the following describes the preferred embodiments of the present application in detail, with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0087] Figure 1 The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, and to implement the content of the specification, the following describes the preferred embodiments of the present application in detail, with reference to the accompanying drawings.

[0088] Figure 2 The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, and to implement the content of the specification, the following describes the preferred embodiments of the present application in detail, with reference to the accompanying drawings.

[0089] Figure 3 The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, and to implement the content of the specification, the following describes the preferred embodiments of the present application in detail, with reference to the accompanying drawings.

[0090] Figure 4 The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, and to implement the content of the specification, the following describes the preferred embodiments of the present application in detail, with reference to the accompanying drawings.

[0091] Figure 5 The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, and to implement the content of the specification, the following describes the preferred embodiments of the present application in detail, with reference to the accompanying drawings.

[0092] Figure 6 The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, and to implement the content of the specification, the following describes the preferred embodiments of the present application in detail, with reference to the accompanying drawings. DETAILED DESCRIPTION

[0093] The specific embodiments of the present application are described in further detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present application but are not intended to limit the scope of the present application.

[0094] The embodiment provides a multi-source data real-time synchronization system of an enterprise based on a data center, adopts a containerized deployment mode and a layered architecture, and comprises a data access layer, a data processing layer, a data distribution layer and a data control layer; the data access layer communicates with a data center, and obtains original data from the data center; the data access layer is provided with a data access adapter module, the data processing layer is provided with a data format processing module, the data distribution layer is provided with a data routing and distribution module, and the data control layer is provided with a consistency guarantee module; the layers communicate with each other through standardized interfaces, so as to ensure the scalability and maintainability of the system.

[0095] The data access adapter module is used for unified access of multi-source heterogeneous data, comprises a relational database adaptation unit, a distributed database adaptation unit, a data warehouse adaptation unit and a search engine adaptation unit, and is respectively used for accessing a relational database, a distributed database, a data warehouse and a search engine. The data access adapter module adopts a plug-in structure, supports dynamic loading and management, and is personalized and extended for specific data sources; the data access adapter module comprises an adapter management unit, which is used for unified management of the life cycle of various adapters, and realizes dynamic deployment and upgrading of the adapters.

[0096] The data format processing module is used for data standardization processing of the accessed multi-source heterogeneous data, and sends the processed standardized data to the data routing and distribution module. Specifically, the data format processing module comprises:

[0097] A data format identification unit is used for identifying a data format type;

[0098] A data cleaning unit is used for cleaning abnormal data;

[0099] A data conversion unit is used for data format conversion;

[0100] A data standardization unit is used for unified data format;

[0101] A quality control unit is used for data quality inspection.

[0102] The data routing and distribution module is used for:

[0103] In the data routing process, the load status of each node is monitored in real time, historical data is combined, and an ARIMA model is used for dynamic load prediction. The ARIMA model is an Auto-Regressive Integrated Moving Average model. The ARIMA model includes three parts: autoregression (AR), difference (I), and moving average (MA). Generally, the ARIMA model is mainly used for predicting time series data, and is suitable for data with the following characteristics:

[0104] 1. The data has a certain trend or periodicity.

[0105] 2. There is a time dependence between data.

[0106] 3. The data can become stationary after difference processing.

[0107] It is widely used in load prediction, economic indicator analysis and other fields. Its parameters can reflect historical data, CPU usage, memory utilization and other related data. The prediction model is as follows:

[0108] y t =c+φ1y t-1 +φ2y t-2 +…+φ p y t-p +∈ t ;

[0109] In the formula, yt is the predicted value at time t; c is the constant term, representing the long-term mean trend of the predicted data; φ1, φ2, …, φp are autoregressive coefficients, which measure the influence of the values at the past p time points on the current value; ∈t is a random disturbance term, or error. In this embodiment, several possible scenarios are included: sudden events in system operation (such as high concurrency requests caused by users submitting a large number of queries at the same time), random disturbances caused by external environment (such as network, hardware); p is the autoregressive order, representing the relationship between the current value yt and the data values yt-1, yt-2, …, yt-p at the previous p time points.

[0110] In load prediction, CPU usage, memory utilization and other indicators can be used as input data of time series (i.e. time series of yt). Through ARIMA modeling, the data trend, fluctuation rule, periodicity of these indicators can be captured, so as to realize the prediction of future time point load. Among them:

[0111] Historical data (yt-1, yt-2, …): including past CPU usage, memory utilization data, used to build time series model.

[0112] Autoregressive coefficients (φ1, φ2, …): Reflect the short-term trend of each metric. For example, the current value of CPU usage might be strongly influenced by the values at certain past times.

[0113] Random error (∈t): Represents the part of the load prediction process that cannot be explained by historical data, such as instantaneous irregular burst loads.

[0114] Based on the load prediction results, an adaptive routing strategy is adopted to dynamically distribute data flows to optimal nodes, and a message middleware-based flow control mechanism is used to prevent system overload and ensure data processing stability; wherein the adaptive routing strategy considers node load, network condition, data affinity factors, and performs optimal routing selection based on the ant colony algorithm to ensure the balance and efficiency of data distribution.

[0115] The ant colony algorithm simulates the foraging behavior of ants and uses pheromone concentration to guide path selection, with the following specific steps:

[0116] Step 1: Initialization:

[0117] Initial pheromone concentration, set to a constant (τ ij (0) = τ0). Set to the initial value (τ0).

[0118] Initialize system state: Collect the current load (L j ) and path bandwidth (B ij ) of each node.

[0119] Step 2: Calculate path weight:

[0120] Comprehensive weight formula:

[0121]

[0122] Where:

[0123] (L j ): Load of node (j) (lower is better).

[0124] (B ij ): Remaining bandwidth of path (P ij ) (higher is better).

[0125] (α, β): Weight factors, control the influence of load and bandwidth on decision-making.

[0126] Step 3: Path selection:

[0127] Selection probability formula:

[0128]

[0129] Where:

[0130] τ ij (t) : pheromone concentration of path (P ij ).

[0131] (W ij ) : integrated weight of path (P ij ).

[0132] (η, γ) : parameters, respectively control the importance of pheromone and path weight.

[0133] Step 4: pheromone update:

[0134] Update formula:

[0135] τ ij (t+1) = (1-ρ)τ ij (t) + Δτ ij ;

[0136] Where:

[0137] (1-ρ) : pheromone evaporation factor, prevent infinite growth of pheromone.

[0138] (Δτ ij ) : new pheromone, indicating path quality.

[0139] In a specific example, the data routing distribution module comprises:

[0140] A load prediction unit for dynamic load prediction based on a stream processing framework;

[0141] A routing strategy unit for implementing an adaptive routing strategy;

[0142] A traffic control unit for implementing traffic control based on a message middleware;

[0143] A task scheduling unit for synchronous task scheduling;

[0144] A resource management unit for system resource management.

[0145] The consistency guarantee module is used for:

[0146] Real-time capture of change information of source data, using a multi-level consistency check method based on hash check, periodically checking the data of the source end and the target end for the established data consistency check point, including data integrity check, format consistency check and business rule check in multiple dimensions, if data anomalies are found, automatically repairing the data to ensure the consistency of data synchronization, and through a distributed transaction coordination mechanism, ensuring the atomicity and consistency of data operation in a distributed environment.

[0147] In a distributed environment, data consistency is a core issue. The system uses multi-level consistency check based on hash check to quickly find data anomalies. The data is uniquely coded using a hash function. Hash values are calculated and compared at different checkpoints (field level, table level, global level). The steps are as follows:

[0148] 1) Generate a hash value for the transmitted data;

[0149] 2) Recalculate the hash value after receiving the data at the target end;

[0150] 3) Compare the hash values of the source and target ends;

[0151] 4) If the data is inconsistent, trigger the repair process, locate and repair abnormal data by tracing historical change records.

[0152] In a specific example, the specific process of each step is as follows:

[0153] Step 1: Generate a hash value for the transmitted data

[0154] (1.1) Generate data blocks:

[0155] Divide the data to be transmitted into multiple parts (such as field level, table level, global level), and generate a hash value for each part.

[0156] Let the data block be (D i ), where (i = 1, 2,..., n).

[0157] (1.2) Calculate the source end hash value:

[0158] Use the hash function (H) to generate the corresponding hash value (h i ) for the data block (D i ):

[0159] h i = H(D i );

[0160] Generate a comprehensive hash value (Hglobal) at the global level by combining the hash values of all data blocks:

[0161] Hglobal = H(h1||h2||...||h n );

[0162] Where (||) represents the string concatenation operation.

[0163] (1.3) Attach the hash value:

[0164] h i) and (Hglobal) are appended to the data packet and transmitted to the destination along with the data.

[0165] Step 2: Destination receives data and re-computes hash values

[0166] (2.1) Extract data blocks: After receiving data at the destination, extract the transmitted data (D i ) and the appended hash values (h i ) and (Hglobal).

[0167] (2.2) Compute destination hash values:

[0168] Recalculate hash values (h′ i ) using the same hash function (H) on the received data blocks (D i ):

[0169] h′ i = H(D′ i );

[0170] Compute the global hash value (Hglobal′) at the destination:

[0171] Hglobal′ = H(h′1||h′2||…||h′ n ).

[0172] Step 3: Compare hash values at source and destination

[0173] (3.1) Compare level by level:

[0174] Field-level check:

[0175] If h i ≠ h′ i , trigger field-level repair

[0176] Table-level check:

[0177] If Htable ≠ Htable′, trigger table-level repair.

[0178] Global-level check:

[0179] If Hglobal ≠ Hglobal′, trigger global-level repair.

[0180] (3.2) Check logic:

[0181] Compare layer by layer from field level to global level, stop further comparison after finding the smallest granularity difference.

[0182] Step 4: Trigger the repair process

[0183] (4.1) Trace back the history change records: Use the change record system to find the changes that occurred during transmission between the source and target:

[0184] Find the modification log in the transmission path. i ) in the transmission path.

[0185] Determine the difference location during data transmission.

[0186] (4.2) Accurately locate the anomaly:

[0187] According to the change records, determine the affected data block (D i ) and its location.

[0188] (4.3) Repair the data:

[0189] Re-send the correct data from the source to the target to cover the abnormal data block. Recalculate the hash value of the repaired data and perform the check again to ensure data consistency.

[0190] Related formulas:

[0191] Field-level verification:

[0192] hfield, i = H(Dfield, i);

[0193] Table-level verification:

[0194] htable = H(hfield, 1 || hfield, 2 ||... || hfield, n);

[0195] Global verification:

[0196] hglobal = H(htable, 1 || htable, 2 ||... || htable, m);

[0197] Parameter explanation:

[0198]

[0199] In a specific example, the consistency guarantee module includes:

[0200] A change capture unit for capturing data changes based on CDC technology;

[0201] A data verification unit for implementing multi-level data verification;

[0202] An anomaly detection unit for detecting data anomalies;

[0203] A data repair unit is configured to automatically repair abnormal data.

[0204] A transaction coordination unit is configured to coordinate distributed transactions.

[0205] The enterprise multi-source data real-time synchronization system further comprises a performance optimization module configured to manage and optimize system performance, and specifically comprising:

[0206] An incremental identification unit is configured to identify data incremental changes.

[0207] A parallel processing unit is configured to disperse data processing tasks to multiple nodes for parallel execution based on a parallel processing mechanism of a distributed computing framework.

[0208] A data compression unit is configured to optimize data transmission compression.

[0209] A resource scheduling unit is configured to implement resource scheduling based on container technology.

[0210] A performance monitoring unit is configured to monitor system performance.

[0211] In the embodiment, the incremental identification unit

[0212] The data that has been synchronized is recorded by a Bloom filter, and only the newly added or changed data is identified to reduce the data transmission amount. The Bloom filter is a kind of efficient probabilistic data structure used to detect whether an element exists in a set. The steps are as follows:

[0213] 1) Initialize the Bloom filter and load the source data hash;

[0214] 2) Calculate the hash value of the new data and check whether the Bloom filter already exists;

[0215] 3) Add the new data to the synchronization task.

[0216] In a specific example, the specific procedures of the above steps are as follows:

[0217] Step 1: Initialize the Bloom filter and load the source data hash

[0218] (1.1) Bloom filter initialization:

[0219] The Bloom filter is a bit array with a length of (m), and all bits are initially set to 0.

[0220] Define a set of (k) independent hash functions (H = H1, H2,..., H k Each function maps the input to a position in the Bloom filter.

[0221] (1.2) Load source data:

[0222] For each element (x) in the source data set, calculate its (k) hash values:

[0223] H i (x)(1≤i≤k), i is the current hash function number used, taking values 1, 2, … k; according to the hash value, the corresponding position is set to 1:

[0224] B[H i (x)]=1 for all i

[0225] Result: The hash values of all elements in the source data are marked as 1 in the Bloom filter.

[0226] Step 2: Calculate the hash value of the new data and check the Bloom filter

[0227] (2.1) New data detection:

[0228] For each new data element (y), calculate the hash value using the same (k) hash functions:

[0229] H i (y)(1≤i≤k)(2.2) Bloom filter check:

[0230] Check if the corresponding bit in the Bloom filter is all 1:

[0231]

[0232] Where:

[0233] (F(y)): indicates whether (y) exists in the Bloom filter.

[0234] If (F(y) = True), it is considered that (y) already exists.

[0235] If (F(y) = False), it is considered that (y) is new or changed data.

[0236] Step 3: Add new data to the synchronization task

[0237] (3.1) Data classification:

[0238] If (F(y) = True), mark (y) as new data and add it to the synchronization task queue.

[0239] If (F(y) = False), skip this data to avoid duplicate synchronization.

[0240] (3.2) Bloom filter update:

[0241] For the new data (y), set the Bloom filter bit corresponding to its hash value to 1:

[0242] [B[H i (y)]=1 for all i.

[0243] In this embodiment, the data compression unit

[0244] Based on the efficient compression algorithm of LZ4, an intelligent data compression strategy is realized. In a distributed environment, transmitting a large amount of data will increase the bandwidth occupation, and the transmission overhead is reduced by the compression algorithm. LZ4 algorithm is a fast compression algorithm, which matches repeated data through sliding window mechanism and reduces storage requirements. The steps are as follows:

[0245] 1) Block processing of input data stream, extracting repeated patterns;

[0246] 2) Match data blocks through sliding window, replace with pointer or symbol;

[0247] 3) Transmit compressed data, target decompression.

[0248] In a specific example, the specific process of each step is as follows:

[0249] Step 1: Block processing of input data stream, extracting repeated patterns;

[0250] (1.1) Divide the input data stream S into multiple data blocks B i , each data block size L(B i ).

[0251] (1.2) Total amount of original data:

[0252]

[0253] (1.3) In each data block, extract repeated patterns through sliding window mechanism, and mark the start position and length of repeated segment.

[0254] Step 2: Match data blocks through sliding window, replace with pointer or symbol;

[0255] (2.1) Use sliding window to match repeated segments in data blocks, replace matched segments with pointer (p, l), and keep unmatched data as literal.

[0256] (2.2) Calculate the size of each data block after compression:

[0257] C(B i )=L(B i )-R(B i );

[0258] Wherein:

[0259] R(B i ): The number of bytes reduced after replacing the matching data segment with a pointer.

[0260] Step 3: Transmit the compressed data, and the target end decompresses

[0261] (3.1) Transmit all compressed data blocks, and the total amount of compressed data:

[0262]

[0263] (3.2) After the target end receives the compressed data, the original data is reconstructed according to the pointer (p, l) and the literal amount, and the verification is as follows:

[0264] L(S) = C(S) + R(S):

[0265] In the formula, L(S) represents the original data amount, C(S) represents the compressed data amount, and R(S) is the compression rate.

[0266] In the embodiment, the resource scheduling unit is based on a containerized resource scheduling mechanism, so that the allocation of computing resources can be dynamically adjusted according to the load condition of the synchronization system, to ensure the optimization of processing performance.

[0267] The enterprise multi-source data real-time synchronization system further comprises a security control module, which is used for data access control and security protection, and comprises:

[0268] An identity authentication unit is configured to perform unified identity authentication.

[0269] An access control unit is configured to implement role-based access control.

[0270] A data encryption unit is configured to implement transport layer security encryption.

[0271] A data desensitization unit is configured to process sensitive data.

[0272] An audit log unit is configured to record data access logs.

[0273] Compared with the prior art, the present application has the following advantages:

[0274] 1. Access capability advantage: supporting unified access of multiple heterogeneous data sources; a standardized adapter framework reduces development costs; a plug-in design improves system scalability; unified management improves operation and maintenance efficiency; and a dynamic loading mechanism enhances system flexibility.

[0275] 2. Performance advantage: data synchronization delay is reduced; system throughput is improved; system availability is improved; data consistency is effectively guaranteed; and large-scale concurrent synchronization tasks are supported.

[0276] 3. Reliability advantage: multi-level data verification ensures data accuracy; automated repair improves system reliability; distributed transaction ensures data consistency; perfect exception handling mechanism; data operation traceability.

[0277] The overall architecture of the application and the structure and function of each module are further described below.

[0278] I. Specific implementation of the overall architecture of the system

[0279] As shown in Figure 1 , the enterprise multi-source data real-time synchronization system of the application adopts a layered architecture design, specifically including the following parts:

[0280] 1. Hardware environment configuration:

[0281] Server configuration:

[0282] CPU: Intel Xeon E5-2680 v4 and above

[0283] Memory: 256GB DDR4

[0284] Storage: NVMe SSD 2TB

[0285] Network card: Gigabit network card (10GbE).

[0286] 2. Cluster size:

[0287] Management node: 3 nodes

[0288] Computing node: 10-100 nodes

[0289] Storage node: elastic configuration according to data volume.

[0290] 3. Infrastructure layer:

[0291] Container orchestration platform: Kubernetes cluster

[0292] Distributed file system: HDFS cluster

[0293] Distributed cache: Redis cluster

[0294] Message middleware: Kafka cluster

[0295] Load balancing: high availability solution based on HAProxy

[0296] 4. Service component deployment:

[0297] Service registry center: unified service registration and discovery

[0298] Configuration center: unified configuration management

[0299] Monitoring Center: System Operation Status Monitoring

[0300] Log Center: Centralized Log Management

[0301] Alarm Center: Real-time Alarm for Abnormal Situations

[0302] II. Specific Implementation of Data Access Adaptation Module

[0303] As shown in Figure 2 , the data access adapter module realizes unified access of multi-source heterogeneous data, including an adapter framework 201, a data source connection pool 202, a data reading engine 203, a first state manager 204, and a configuration manager 205. The adapter framework 201 includes a standard interface and a plug-in management module. The data source connection pool 202 is used for connection management and resource reuse. The data reading engine 203 is used for data extraction and data reading. The first state manager 204 is used to display synchronization status and monitoring records. The configuration manager 205 is used for configuration information management and parameter management. The configuration manager 205 performs configuration management on the adapter framework 201, the data source connection pool 202, and the data reading engine 203. The first state manager 204 performs state monitoring on the data source connection pool 202 and the data reading engine 203. Specifically, the following is included:

[0304] 1. Adapter framework implementation:

[0305] Standard interface definition:

[0306] Data reading interface

[0307] State management interface

[0308] Configuration management interface

[0309] Life cycle management interface.

[0310] Plug-in implementation:

[0311] Dynamic class loading mechanism

[0312] Plug-in registration mechanism

[0313] Version management mechanism hot loading support.

[0314] 2. Data source access implementation:

[0315] Relational database access:

[0316] JDBC-based general adaptation based on log analysis incremental collection

[0317] Connection pool optimization

[0318] Concurrent control.

[0319] Distributed database access:

[0320] Special API interface

[0321] Sharding data processing

[0322] State synchronization mechanism

[0323] Abnormal recovery.

[0324] 3. Adapter management implementation:

[0325] Adapter life cycle:

[0326] Registration, initialization

[0327] Start, run

[0328] Stop, uninstall.

[0329] State monitoring:

[0330] Running state detection

[0331] Performance index collection

[0332] Abnormal situation alarm automatic failover.

[0333] Three, the specific implementation of the data routing and distribution module

[0334] As shown in Figure 3 , the data routing and distribution module implements an intelligent data distribution mechanism, which realizes data distribution through load monitoring 301, strategy calculation 302, data distribution 303, flow control 304, and state feedback 305. During the distribution process, the load monitoring 301 is updated and the strategy calculation 302 is optimized through the state feedback 305; the load monitoring 301 is monitored through the flow control 304. Specifically, the load monitoring 301 includes system load and resource usage monitoring; the strategy calculation 302 includes optimized selection of routing calculation machines; the data distribution 303 includes data routing and data forwarding; the flow control 304 includes rate limiting and flow balancing; the state feedback 305 includes result collection and state update. Specifically, it includes:

[0335] 1. Load prediction implementation:

[0336] Index collection:

[0337] CPU usage

[0338] Memory usage

[0339] Network bandwidth utilization

[0340] Disk I / O status

[0341] Prediction algorithm:

[0342] Sliding window analysis

[0343] Trend prediction

[0344] Threshold dynamic adjustment

[0345] Early warning mechanism

[0346] 2、Routing strategy implementation:

[0347] Strategy calculation:

[0348] Node load weight

[0349] Network delay factor

[0350] Data affinity

[0351] Resource utilization.

[0352] Dynamic adjustment:

[0353] Real-time load balancing

[0354] Automatic failover

[0355] Performance optimization adjustment

[0356] Configuration dynamic update.

[0357] Four, the specific implementation of the consistency guarantee module

[0358] As Figure 4 shown, the consistency guarantee module realizes data consistency guarantee in a distributed environment, including a CDC engine 401, a verifier 402, a repair processor 403, a transaction coordinator 404, and a second state manager 405. Among them, the CDC engine 401 is used for change capture and data analysis; the verifier 402 is used for data verification and consistency check; the repair processor 403 is used for difference analysis and data repair; the transaction coordinator 404 is used for transaction management and submission control; and the second state manager 405 is used for state tracking and state maintenance. The second state manager 405 monitors the state of the CDC engine 401, the verifier 402, the repair processor 403, and the transaction coordinator 404; and the transaction coordinator 404 controls the transaction of the verifier 402 and the repair processor 403. Specifically, it includes:

[0359] 1、Change capture implementation:

[0360] CDC engine configuration:

[0361] Data source connection configuration

[0362] Table-level filtering rules

[0363] Field-level filtering rules

[0364] Change event definition.

[0365] Data collection processing:

[0366] Log analysis processing

[0367] Transaction integrity assurance

[0368] Resume mechanism

[0369] Concurrency control management.

[0370] 2、Data verification implementation:

[0371] Checkpoint setting:

[0372] Time point verification

[0373] Transaction point verification

[0374] Data volume verification

[0375] Business rule verification.

[0376] Verification process:

[0377] Full data verification

[0378] Incremental data verification

[0379] Real-time consistency verification

[0380] Periodic consistency verification.

[0381] 3、Data repair implementation:

[0382] Repair strategy:

[0383] Automatic repair rules

[0384] Manual intervention mechanism

[0385] Priority management

[0386] Impact analysis.

[0387] Repair process:

[0388] Anomaly detection

[0389] Difference comparison

[0390] Data backfill

[0391] Result verification.

[0392] Five, the specific implementation of the performance optimization module

[0393] For example Figure 5As shown, the performance optimization module implements comprehensive optimization of system performance, including incremental processor 501, parallel scheduler 502, resource manager 503, performance monitor 504, and optimization decision maker 505. Among them, the incremental processor 501 is used for incremental identification and incremental synchronization; the parallel scheduler 502 is used for task allocation and concurrency control; the resource manager 503 is used for resource allocation and load balancing; the performance monitor 504 is used for performance collection and index analysis; and the optimization decision maker 505 is used for strategy generation and optimization execution. The optimization decision maker 505 provides processing strategies, scheduling strategies, and optimization strategies to the incremental processor 501, the parallel scheduler 502, and the resource manager 503, respectively; the incremental processor 501 and the parallel scheduler 502 provide performance data to the performance monitor 504; and the resource manager 503 provides resource state information to the performance monitor 504. Specifically, it includes:

[0394] 1. Incremental processing implementation:

[0395] Incremental identification:

[0396] Timestamp marking

[0397] Version number marking

[0398] Change flag incremental log analysis. Data processing:

[0399] Incremental data extraction

[0400] Data merging processing

[0401] Conflict resolution state maintenance.

[0402] 2. Parallel processing implementation: task decomposition:

[0403] Data sharding strategy

[0404] Task priority division dependency analysis resource evaluation.

[0405] Parallel scheduling:

[0406] Work thread pool management task dynamic allocation progress monitoring exception handling.

[0407] 3. Resource scheduling implementation: container management:

[0408] Container resource configuration

[0409] Elastic scaling policy

[0410] Resource limit policy

[0411] Scheduling optimization.

[0412] Performance monitoring:

[0413] Real-time performance data collection performance bottleneck analysis

[0414] Trend prediction

[0415] Alarm triggering

[0416] Six, specific implementation of the security control module

[0417] As Figure 6 shown, the security control module realizes comprehensive security protection, including authentication center 601, permission manager 602, encryption processor 603, desensitization processor 604, and audit log manager 605. Among them, the authentication center 601 is used for identity verification and credential management; the permission manager 602 is used for access control and policy execution; the encryption processor 603 is used for encryption operation and key management; the desensitization processor 604 is used for data desensitization and rule application; and the audit log manager 605 is used for log recording and log analysis. The audit log manager 605 audits the authentication center 601, the permission manager 602, the encryption processor 603, and the desensitization processor 604; and the permission manager 602 controls the encryption processor 603 and the desensitization processor 604. Specifically, it includes:

[0418] 1. Identity authentication implementation:

[0419] Authentication mechanism:

[0420] SSO integration configuration

[0421] Multi-factor authentication

[0422] Token management

[0423] Session control

[0424] Permission management:

[0425] Role definition

[0426] Permission assignment

[0427] Permission inheritance

[0428] Dynamic authorization

[0429] 2. Data security implementation:

[0430] Transmission encryption:

[0431] TLS configuration management

[0432] Certificate management

[0433] Key update

[0434] Encryption policy

[0435] Data desensitization:

[0436] Desensitization rule configuration

[0437] Field-level de-sensitization

[0438] Dynamic de-sensitization processing

[0439] De-sensitization effect verification.

[0440] 3. Audit log implementation:

[0441] Logging:

[0442] Operation log collection

[0443] Access log recording

[0444] System log management

[0445] Audit log storage.

[0446] Log analysis:

[0447] Real-time log analysis

[0448] Abnormal behavior detection

[0449] Audit report generation

[0450] Compliance check.

[0451] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. It should be noted that for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, and these improvements and modifications should be considered as the protection scope of the present application.

Claims

1. A data middle platform-based enterprise multi-source data real-time synchronization system, characterized in that, Adopt containerization deployment mode and layered architecture, including data access layer, data processing layer, data distribution layer and data control layer; the data access layer communicates with the data center, and obtains raw data from the data center; The data access layer is provided with a data access adapter module, the data processing layer is provided with a data format processing module, the data distribution layer is provided with a data routing and distribution module, and the data control layer is provided with a consistency guarantee module; the layers communicate through standardized interfaces to ensure the scalability and maintainability of the system; The data access adapter module is used for unified access of multi-source heterogeneous data, and includes a relational database adaptation unit, a distributed database adaptation unit, a data warehouse adaptation unit and a search engine adaptation unit, which are respectively used for accessing relational databases, distributed databases, data warehouses and search engines; the data access adapter module adopts a plug-in structure, supports dynamic loading and management, and individualized expansion of data sources; The data access adapter module includes an adapter management unit for unified management of the life cycle of various adapters, and realizes dynamic deployment and upgrading of the adapters; The data format processing module is used for data standardization processing of the accessed multi-source heterogeneous data, and sends the processed standardized data to the data routing and distribution module; The data routing and distribution module is used for: In the data routing process, the load state of each node is monitored in real time, historical data is combined, and dynamic load prediction is performed by using an ARIMA model; CPU usage and memory utilization are used as input data of time series, and the prediction model is as follows: y t = c + φ1y t-1 + φ2y t-2 +... + φ p y t-p + ε t ; where y t is the predicted value of the current time t; c is a constant term, representing the long-term mean trend of the predicted data; φ1, φ2, …, φp are autoregressive coefficients, which measure the influence of the values at the past p time points on the current value, reflecting the short-term trend of each indicator. The current value of CPU usage may be strongly affected by the values at some time points in the past; yt-1, yt-2, … yt-p are the historical data at the previous p time points, including past CPU usage and memory utilization data; ∈ t is a random error, representing the part of the load prediction process that cannot be explained by historical data, including instantaneous irregular burst loads; Based on the load prediction result, an adaptive routing strategy is adopted to dynamically distribute data flow to the optimal node, and a flow control mechanism based on message middleware is adopted to prevent system overload and ensure the stability of data processing; wherein, the adaptive routing strategy considers node load, network condition and data affinity factors, and performs optimal routing selection based on an ant colony algorithm to ensure the balance and efficiency of data distribution; The steps of performing optimal routing selection based on the ant colony algorithm are as follows: Step 1: initialization: The initial pheromone concentration is set to a constant value (τ ij (0) = τ0); set to an initial value (τ0); Initialize system state: collect the current load (L j ) and path bandwidth (Bij) of each node; Step 2: calculate path weight: Comprehensive weight formula: Wherein: (L j ) : load of node (j), lower is better; (Bij): the remaining bandwidth of the path (Pij), the higher the better; (α, β): weight factor, control the influence of load and bandwidth on decision-making; Step 3: path selection: Selection probability formula: Wherein: τij(t): pheromone concentration of path (Pij); (Wij): comprehensive weight of path (Pij); (η, γ): parameters, respectively control the importance of pheromone and path weight; Step 4: pheromone update: Update formula: τij(t + 1) = (1 - p)τij(t) + Δτ ij ; Wherein: (1-ρ): pheromone evaporation factor, prevent pheromone from growing indefinitely; (Δτij): new pheromone, indicating path quality; The consistency guarantee module is used for: Real-time capture of source data change information, using a multi-level consistency check method based on hash check, periodically checking the data of the source and target ends for the established data consistency checkpoint, if data anomalies are found, automatically repairing the data to ensure the consistency of data synchronization, and through the distributed transaction coordination mechanism, ensuring the atomicity and consistency of data operation in a distributed environment; The multi-level consistency check method based on hash check is as follows: Step 1): Generate hash value for transmission data Step 1.1, generate data block: Divide the data to be transmitted into multiple parts, including field level, table level, and global level, and generate a hash value for each part; Let the data block be (D i ), where (i = 1, 2,..., n). Step 1.2, calculate the source end hash value: A hash function (H) is used to generate a corresponding hash value (h i ) for a data block (D i ): h i = H(D i ): Generate a comprehensive hash value (Hglobal) at the global level by combining the hash values of all data blocks: Hglobal = H(h1||h2||...||h n ); Where (||) represents the string concatenation operation; Step 1.3, append hash value: The calculated hash value (h i ) and (Hglobal) are appended to the data packet and transmitted with the data to the destination. Step 2) target end receives data and recalculates hash value Step 2.1, extract data block: After the data is received at the destination, the transmitted data (Di') and the appended hash values (h i ) and (Hglobal) are extracted: Step 2.2, calculate target end hash value: The same hash function (H) is used to re-compute the hash value (h') for the received data block (Di'): i ): h′ i = H(D′ i ); Calculate the global hash value (Hglobal') of the target end: yHglobal′=H(h′1||h′2||...||h′ n ); Step 3) compare the hash values of the source end and the target end (3.1) compare level by level: Field level check: hfield, i = H(Dfield, i); wherein (h field，i ) denotes the hash value of field i; (D field，i ) denotes the original data of field i; Table level check: htable = H(hfield, 1||hfield, 2||...||hfield, n); wherein (h table ) denotes the hash value of the table; Global level check: hglobal = H(htable, 1||htable, 2||...||htable, m); wherein (h global ) represents a global hash value; Step 3.2, check logic: Compare layer by layer from the field level to the global level, and stop further comparison after finding the smallest granularity difference; Step 4) trigger repair process Step 4.1, backtrack history change record: Use the change record system to find the changes that occurred during transmission between the source end and the target end: Find (D i ) a modification log in the transport path; Determine the difference position during data transmission; Step 4.2, accurately locate the anomaly: According to the change record, determine the affected data block (D i ) and its location; Step 4.3, repair data: Send the correct data from the source end to the target end to overwrite the abnormal data block; Recalculate the hash value of the repaired data and perform the check again to ensure data consistency; The enterprise multi-source data real-time synchronization system also includes a performance optimization module, which includes: Increment identification unit, used to identify data increment changes, including: Record synchronized data through the Bloom filter, and only identify new or changed data to reduce data transmission volume, the specific process is as follows: Step 1) initialize Bloom filter and load source data hash Step 1.1, Bloom filter initialization: Bloom filter is a bit array with a length of (m), all bits are initially set to 0; Define (k) independent sets of hash functions (H = H1, H2, ..., Hk). k Each function maps the input to a location in the Bloom filter; Step 1.2, load source data: For each element (x) in the source data set, calculate its (k) hash values: H i (x) (1≤i≤k), i is the current used hash function number, the value range is 1, 2, … k; According to the hash value, set the corresponding bit to 1: B[H i (x)] = 1 for all i; Result: all element hash values in the source data are marked as 1 in the Bloom filter; Step 2) calculate hash value for new data and check Bloom filter Step 2.1, new data detection: For each new data element (y), use the same (k) hash functions to calculate the hash value: H i (y) (1≤i≤k) (2.2) Bloom filter check: Check if the corresponding bit in the Bloom filter is all 1: Where: F(y): indicates whether (y) already exists in the Bloom filter; If (F(y) = True), it is considered that (y) already exists; If (F(y) = False), it is considered that (y) is new or changed data; Step 3) Add new data to the synchronization task Step 3.1, data classification: If (F(y) = True), mark (y) as new data and add it to the synchronization task queue; If (F(y) = False), skip this data to avoid repeated synchronization; Step 3.2, Bloom filter update: For new data (y), set the Bloom filter bit corresponding to its hash value to 1: [B[H i (y)] = 1 for all i]; Parallel processing unit, for parallel processing mechanism based on distributed computing framework, disperses data processing tasks to multiple nodes for parallel execution; Data compression unit, for data transmission compression optimization, including: Efficient compression algorithm based on LZ4, implements intelligent data compression strategy, the specific process is as follows: Step 1) Block processing of input data stream, extract repeated patterns; Step 1.

1. Split the input data stream S into a plurality of data blocks B i , each of size L(B i ) Step 1.2, total amount of original data: Step 1.3, in each data block, extract repeated patterns through sliding window mechanism, and mark the starting position and length of the repeated segment; Step 2) Replace the data block with a pointer or symbol through sliding window matching; Step 2.1, use sliding window to match the repeated segment in the data block, replace the matched segment with a pointer (p, l), and keep the unmatched data as literal; Step 2.2, calculate the size of each data block after compression: C(B i ) = L(B i ) - R(B i ); Where: R(B i ) is the number of bytes reduced after replacing the matching data segment with a pointer; Step 3: Transmit compressed data, target end decompression Step 3.1, transmit all compressed data blocks, total data size after compression: Step 3.2, after receiving the compressed data at the target end, reconstruct the original data according to the pointer (p, l) and literal, and verify: L(S) = C(S) + R(S): In the formula, L(S) represents the original data size, C(S) represents the compressed data size, and R(S) is the compression rate; Resource scheduling unit, based on container technology to realize resource scheduling; Performance monitoring unit, for system performance monitoring. 2.The data-distribution-platform-based enterprise multi-source data real-time synchronization system according to claim 1, characterized in that, The data format processing module includes: Data format identification unit, for identifying data format type; Data cleaning unit, for cleaning abnormal data; Data conversion unit, for data format conversion; Data standardization unit, for unified data format; Quality control unit, for data quality inspection. 3.The data-distribution-platform-based enterprise multi-source data real-time synchronization system according to claim 1, characterized in that, The resource scheduling unit is specifically used for: Container-based resource scheduling mechanism, for the synchronization system to dynamically adjust the allocation of computing resources according to the load condition, to ensure the optimization of processing performance. 4.The data-distribution-platform-based enterprise multi-source data real-time synchronization system according to claim 1, characterized in that, It also includes a security control module, which includes: Identity authentication unit, for unified identity authentication; Access control unit, for implementing role-based access control; Data encryption unit, for implementing transport layer security encryption; Data desensitization unit, for processing sensitive data; Audit log unit, for recording data access logs.

Images