Cloud-based data synchronization system

Abstract

The present invention relates to the field of data synchronization technology, specifically to a cloud-based data synchronization system, the system includes a shard scheduling module, a data optimization module, an asynchronous push module and a version conflict resolution module, the shard scheduling module calculates the intrinsic structure of data partitions based on data types and sizes, obtains partition splitting points, and allocates data blocks to designated processing threads. The present invention significantly optimizes the speed and response time of data processing through a refined data partitioning strategy and dynamic allocation, reduces system resource pressure, implements multi-threaded parallel processing, allows the system to synchronously process multiple data blocks, greatly improves processing capabilities and system throughput, and pushes data through an asynchronous channel immediately after processing, improves the real-time nature of data synchronization, and reduces the risk in the transmission process through encryption and compression. The intelligent conflict resolution strategy ensures data consistency and accuracy through real-time monitoring and data version control, and adapts to the needs of multi-user operations.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a cloud-based data synchronization system. Background Art

[0002] Data synchronization focuses on technologies that maintain data consistency and real-time updates between different systems, devices, or locations, including various algorithms and protocols to ensure that at any given point in time, multiple data copies can reflect the latest status regardless of their respective physical storage locations. This technology is widely used in distributed systems, database management, mobile devices, and cloud computing environments to support cross-platform data access and backup solutions.

[0003] Among them, the data synchronization system refers to a collection of software and hardware that implements data synchronization functions. Its main purpose is to synchronize data between multiple computing devices or network nodes to ensure that data in all locations is up-to-date and consistent. The system monitors data changes and automatically updates all related copies when changes occur, thereby achieving data consistency and integrity. The system is particularly important in multi-user operating environments, remote data access, and disaster recovery plans. It can ensure the continuous availability and integrity of data and quickly restore business operations even in the event of a data center failure.

[0004] In the absence of optimized data processing procedures, existing technologies often have delay problems in synchronization tasks and are unable to meet the needs of real-time data processing. In addition, security measures are inflexible, resulting in inefficiency in heterogeneous environments. Existing conflict resolution strategies, such as timestamp comparison, cannot effectively handle data version conflicts in multi-user environments, which can easily lead to data inconsistency and affect the reliability of data synchronization. These problems are particularly prominent in application scenarios that require fast response and high data consistency, such as online trading platforms, leading to business interruptions and data loss. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings in the prior art and to propose a cloud-based data synchronization system.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a cloud-based data synchronization system, the system comprising:

[0007] The shard scheduling module calculates the internal structure of data partitions based on data type and size, obtains partition split points, allocates data blocks to designated processing threads, determines processing priorities, establishes parallel processing queues, and generates parallel task allocation tables;

[0008] The data optimization module performs outlier detection and data fusion on each data block through a multi-threaded processing mechanism based on the parallel task allocation table, and refines the data quality and generates a data quality optimization record based on the preliminary processing results of each data block;

[0009] The asynchronous push module extracts each data block from the data quality optimization record, establishes an asynchronous communication channel with the cloud, performs encryption and compression processing on each data block, asynchronously pushes the compressed data block to the cloud through Apache Kafka, and monitors the data push status to obtain data synchronization status information;

[0010] The version conflict resolution module monitors the consistency of cloud and local data versions based on the data synchronization status information, identifies data version differences, determines data selection based on priority rules and user authority rules for detected data conflicts, and generates data version consistency results.

[0011] The present invention is improved in that the step of obtaining the partition splitting point is specifically as follows:

[0012] Based on the data type and size, a density-based clustering algorithm is used to identify high-density areas in the data set and obtain a set of potential split points;

[0013] Based on the potential split point set, the data density score is calculated using the formula:

[0014]

[0015] According to the score, the point with the best data density is selected as the partition point, where: Indicate point The data density score, Indicate point The set of neighboring points of Indicate point arrive The Euclidean distance of is the attenuation coefficient, which is used to adjust the influence of distance.

[0016] The present invention is improved in that the step of obtaining the parallel task allocation table is specifically as follows:

[0017] Analyze the load capacity and data block size of each processing thread, and calculate the processing efficiency of each thread for its assigned data block using the formula:

[0018]

[0019] Estimate the load of processing threads and allocate data blocks to balance the workload of multiple threads. Indicates The estimated load of threads, Indicates the allocation to the thread No. The size of the data block, Indicates The processing complexity coefficient of a data block is is the total number of data blocks;

[0020] Based on the load of the processing thread, the allocation order and processing priority of the data blocks are optimized, and a parallel task allocation table is established.

[0021] The present invention is improved in that the steps of outlier detection and data fusion are specifically as follows:

[0022] Based on the parallel task allocation table, determine the data blocks to be processed by each thread to obtain an initial mapping relationship;

[0023] Based on the initial mapping relationship, outlier detection is performed using the formula:

[0024]

[0025] Determine whether the data point deviates from the normal range and obtain the data abnormality status ,in, represents the data point value, and are the mean and standard deviation of the data set, respectively;

[0026] Based on the data abnormality state, data fusion is performed to integrate data blocks from the same source, and data integrity is verified to obtain a fused data block set.

[0027] The present invention is improved in that the steps of obtaining the data quality optimization record are specifically as follows:

[0028] Collecting preliminary processing results of each data block, including information from outlier detection to data fusion, to obtain a processing detail record;

[0029] The processing detail records are analyzed using the formula:

[0030]

[0031] Assessing the degree of bias in a dataset , identify and correct data quality issues, and obtain data quality optimization records, among which, Representative data points, Yes all The average value of Yes all The standard deviation of is the total number of data points.

[0032] The present invention is improved in that the execution steps of the encryption and compression processing are specifically as follows:

[0033] Extracting each data block from the data quality optimization record, establishing an asynchronous communication channel with the cloud, and obtaining a preprocessed data block;

[0034] Based on the preprocessed data block, encrypt the data using AES-256, and verify the security of data transmission to obtain an encrypted data block;

[0035] Based on the encrypted data block, the Brotli compression algorithm is applied to reduce the data volume and optimize the storage and transmission efficiency, using the formula:

[0036]

[0037] Calculating the Compression Efficiency Ratio , get the compressed data block, where Indicates the size of the compressed data. Indicates the original data size.

[0038] The present invention is improved in that the step of acquiring the data synchronization status information is specifically as follows:

[0039] Use the Apache Kafka to asynchronously push data in batches to the cloud, optimize the traceability of the transmission process, and obtain an encrypted and compressed data block queue;

[0040] Based on the encrypted and compressed data block queue, the formula is adopted:

[0041]

[0042] Calculate the success rate of data transmission , get the data transmission efficiency log, where, is the number of data blocks received, is the number of data blocks lost during transmission, Indicates the total number of data blocks that are attempted to be sent. and Represent the weights of received and lost data blocks respectively;

[0043] Based on the data transmission efficiency log, the data block reception status is analyzed and data synchronization status information is established.

[0044] The present invention is improved in that the step of obtaining the data version consistency result is specifically as follows:

[0045] Based on the data synchronization status information, collect cloud and local data version information, and create a version record list in combination with the release time and version number;

[0046] Based on the version record list, the collected data versions are compared using the formula:

[0047]

[0048] Calculate the difference between data versions , identify data points with inconsistent versions and obtain version difference information, where, and Represents the local and cloud Data version number, is the total number of data points being compared;

[0049] According to the version difference information, priority rules and user authority rules are applied to perform data selection judgment, determine the optimal data version, and obtain a data version consistency result.

[0050] Compared with the prior art, the advantages and positive effects of the present invention are:

[0051] In the present invention, through the refined data partitioning strategy and dynamic allocation, the speed and response time of data processing are significantly optimized, the pressure on system resources is reduced, and multi-threaded parallel processing is implemented, allowing the system to synchronously process multiple data blocks, greatly improving the processing capacity and system throughput. The data is pushed through the asynchronous channel immediately after processing, which improves the real-time performance of data synchronization, and reduces the risk in the transmission process through encryption and compression. The intelligent conflict resolution strategy ensures data consistency and accuracy through real-time monitoring and data version control, and adapts to the needs of multi-user operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a system flow chart of the present invention;

[0053] Figure 2 This is a flow chart for obtaining partition points in the present invention;

[0054] Figure 3 A flowchart for obtaining a parallel task allocation table in the present invention;

[0055] Figure 4 is a flow chart of outlier detection and data fusion in the present invention;

[0056] Figure 5 A flowchart for obtaining data quality optimization records in the present invention;

[0057] Figure 6 is a flowchart of the execution of encryption and compression processing in the present invention;

[0058] Figure 7This is a flow chart for obtaining data synchronization status information in the present invention;

[0059] Figure 8 This is a flow chart for obtaining data version consistency results in the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0061] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating positions or positional relationships, are based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0062] Example

[0063] See also Figure 1 , the present invention provides a technical solution: a cloud-based data synchronization system includes:

[0064] The shard scheduling module calculates the internal structure of data partitions based on data type and size, obtains partition split points, allocates data blocks to designated processing threads, determines processing priorities, establishes parallel processing queues, and generates parallel task allocation tables;

[0065] The data optimization module performs outlier detection and data fusion on each data block through a multi-threaded processing mechanism based on the parallel task allocation table, and refines the data quality and generates data quality optimization records based on the preliminary processing results of each data block;

[0066] The asynchronous push module extracts each data block from the data quality optimization record, establishes an asynchronous communication channel with the cloud, encrypts and compresses each data block, asynchronously pushes the compressed data block to the cloud through Apache Kafka, monitors the data push status, and obtains data synchronization status information;

[0067] The version conflict resolution module monitors the consistency of cloud and local data versions based on data synchronization status information, identifies data version differences, and uses priority rules and user permission rules to determine data selection for detected data conflicts and generate data version consistency results.

[0068] Priority rules are used to determine which data updates have priority for synchronization when data version conflicts occur. Specifically:

[0069] Timestamp, using the latest data modification to resolve conflicts, that is, the (latest modification first) principle, which considers the most recent data update to be the most accurate;

[0070] Data importance: some data has a higher priority due to its business importance. In this case, the system will prioritize synchronization of critical data based on the preset importance level.

[0071] Operation type. Some operations have higher priority than others, such as deletion operations or structural changes taking precedence over ordinary data updates.

[0072] User permission rules determine the control that different users have over data. During data synchronization and conflict resolution, the user's permission level can affect the priority of the changes they make. Specifically:

[0073] Role-based, different user roles (such as administrator, ordinary user, guest) have different permission settings, and administrators' data changes usually have higher priority;

[0074] Department / organizational structure. In large organizations, certain departments or teams may have higher priority for data, depending on business processes and internal rules.

[0075] Access levels. The access rights of individual users also affect data synchronization priority. For example, changes made by users with full access rights take precedence over those made by users with only partial access rights.

[0076] The parallel task allocation table includes partition identification, thread correspondence, and priority list. The data quality optimization records include abnormal data indicators, fusion efficiency information, and quality improvement points. The data synchronization status information includes compression efficiency, push timestamp, and synchronization verification status. The data version consistency results include version comparison analysis results, conflict handling process, and verification selection results.

[0077] See also Figure 2 , the specific steps for obtaining the partition split point are:

[0078] Based on the data type and size, a density-based clustering algorithm is used to identify high-density areas in the data set and obtain a set of potential split points;

[0079] By refining the type and size of the data, we can get an overview of the data structure. With this overview, we can accurately identify high-density areas in the data set. The identification process involves determining which areas show higher data aggregation than other areas through the distribution of spatial coordinates, ensuring that which areas become key nodes for data processing and analysis can be accurately located. It not only relies on the analysis of the spatial distribution characteristics of the data, but also needs to consider the overall scale and uneven distribution of the data, thereby generating high-density area identification results containing a set of potential partition points.

[0080] Based on the set of potential split points, the data density score is calculated using the formula:

[0081]

[0082] According to the score, the point with the best data density is selected as the partition point, where: Indicate point The data density score, Indicate point The set of neighboring points of Indicate point arrive The Euclidean distance of is the attenuation coefficient, which is used to adjust the influence of distance. Represents the cardinality of the set, that is, the number of neighboring points;

[0083] If there is a point and its neighbor point set ,and Contains 100 points, cluster centers Point with a neighbor Distance is 2, the attenuation coefficient is 0.5, then the calculation process is as follows:

[0084]

[0085] The results show that The data density score is 0.1353, which is calculated by the distance of the actual data points and the attenuation factor, showing Potential effectiveness as a data clustering center.

[0086] See also Figure 3 , the specific steps for obtaining the parallel task allocation table are:

[0087] Analyze the load capacity and data block size of each processing thread, and calculate the processing efficiency of each thread for its assigned data block using the formula:

[0088]

[0089] Estimate the load of processing threads and allocate data blocks to balance the workload of multiple threads. Indicates The estimated load of threads, Indicates the allocation to the thread No. The size of the data block, Indicates The processing complexity coefficient of a data block is is the total number of data blocks;

[0090] Analyze the load capacity of each processing thread and the size of the data block. According to the processing efficiency of each thread and the characteristics of the data block, optimize the initial strategy for data block allocation. The strategy takes into account the performance parameters of each thread and the processing requirements of the data block to achieve balanced load distribution. The process involves analysis of thread performance and data requirements. By quantifying the processing capacity of each thread and the complexity of each data block, the appropriate amount of data is allocated to each thread to ensure that each thread can run at its optimal efficiency. This not only improves the processing efficiency of the system, but also optimizes resource utilization and balances the entire workload through quantified parameters.

[0091] Based on the load of the processing thread, the data block allocation order and processing priority are optimized, and a parallel task allocation table is established. The parallel task allocation table records the data blocks allocated to each thread and the corresponding processing priority;

[0092] Set Thread Three data blocks are allocated, the size of the data blocks The processing complexity of each data block is 3, 2, and 1 GB respectively. They are 0.5, 2 and 1.5 respectively, and are substituted into the formula to calculate:

[0093]

[0094] The calculation results show that the thread The estimated load is 7, which is based on the multiplication and accumulation of data block size and complexity, indicating that the thread The total load incurred when processing these three data blocks.

[0095] See also Figure 4 ,The specific steps of outlier detection and data fusion are:

[0096] Based on the parallel task allocation table, determine the data block to be processed by each thread and obtain the initial mapping relationship;

[0097] Through the parallel task allocation table, information about each data block and the corresponding processing thread is provided, and the task list of each thread is initialized to ensure that each data block is correctly identified and configured to the corresponding processing thread. It involves the identification and allocation of data blocks, as well as the configuration and preparation of processing threads to ensure that each thread can start the processing flow immediately after receiving the data. The mapping relationship is the basis of data processing and provides the necessary prerequisite for subsequent outlier detection and data fusion.

[0098] Based on the initial mapping relationship, outlier detection is performed using the formula:

[0099]

[0100] Determine whether the data point deviates from the normal range and obtain the data abnormality status ,in, represents the data point value, and are the mean and standard deviation of the data set, respectively;

[0101] Assume that the data block contains the value , calculate its average value and standard deviation , for the last data point 100, The value is calculated as:

[0102]

[0103] If | |>3 is an anomaly. The result shows that although 100 is not statistically abnormal, its value is significantly higher than other observations in this small sample situation and needs attention, which is crucial for subsequent data processing and quality control.

[0104] Based on the data anomaly status, data fusion is performed to integrate data blocks from the same source and verify data integrity to obtain a fused data block set;

[0105] After completing the outlier detection, the data blocks are fused, relying on the outlier detection records. Using the records as a reference, the data is screened and merged. The purpose is to integrate data blocks from the same data source. Through this integration, redundancy and inconsistency in the data set can be eliminated, and the overall quality and availability of the data can be improved. The fused data blocks will have higher information completeness and lower error rates, which is critical to ensuring the accuracy and reliability of data analysis, and output an optimized and quality-controlled data set.

[0106] See also Figure 5 , the specific steps for obtaining data quality optimization records are:

[0107] Collect the preliminary processing results of each data block, including information from outlier detection to data fusion, and obtain the processing details record;

[0108] The data blocks obtain preliminary processing results after outlier detection and data fusion. After collecting the data blocks, data quality assessment is performed, including assessment of data integrity, consistency and accuracy. The assessment is based on the data fusion results, and in-depth analysis is performed to determine specific optimization measures for data quality, including correcting minor errors and inconsistencies in the data, ensuring that all data blocks meet the predetermined quality standards, and generating optimized data blocks. The data blocks provide the basis for subsequent operations.

[0109] To analyze the processing detail records, use the formula:

[0110]

[0111] Assessing the degree of bias in a dataset , identify and correct data quality issues, and obtain data quality optimization records, among which, Representative data points, Yes all The average value of Yes all The standard deviation of is the total number of data points;

[0112] If there are data points , where 250 is a significant outlier, calculate the average and standard deviation , calculated using the formula value:

[0113]

[0114]

[0115] The calculation shows that the overall data quality is high, although there is one outlier. The average deviation is relatively low, but this value reminds us that further analysis or treatment of the anomaly is needed. The calculation of the value can quantitatively evaluate the quality status of the data set and identify outliers and quality issues in the data.

[0116] See also Figure 6 , the execution steps of encryption and compression processing are as follows:

[0117] Extract each data block from the data quality optimization record, establish an asynchronous communication channel with the cloud, and obtain the preprocessed data block;

[0118] Extract data blocks from data quality optimization records, screen and verify the integrity and consistency of data blocks to ensure that the extracted data blocks are the latest and most accurate, and then establish an asynchronous communication channel. The channel uses network protocols, including HTTPS, TLS, SSL, TCP / IP and MQTT, to ensure the security and reliability of data during transmission, obtain the processed data block set, and prepare for the next stage of security encryption.

[0119] Based on the preprocessed data block, AES-256 is used to encrypt the data, and the security of data transmission is verified to obtain the encrypted data block;

[0120] Based on the preprocessed data blocks, AES-256 encryption operations are performed to effectively prevent unauthorized access and tampering of data during transmission. During the encryption process, each data block is encrypted independently to ensure that even if part of the data is intercepted, the entire information cannot be interpreted. The data transmission security is verified, including a double check of data integrity and encryption quality to ensure that the status of each piece of data when it leaves the local server is consistent with the status when it arrives at the cloud, and an encrypted data block is generated to ensure the high security of the entire process and the availability of data.

[0121] Based on the encrypted data blocks, the Brotli compression algorithm is applied to reduce the data volume and optimize the storage and transmission efficiency. The formula is:

[0122]

[0123] Calculating the Compression Efficiency Ratio , get the compressed data block, where Indicates the size of the compressed data. Indicates the original data size;

[0124] Set the original data size , compressed size , calculated as follows:

[0125]

[0126] The results show that the compression process reduces the data volume by 30%, effectively reducing the resources required for storage and transmission and improving data processing efficiency.

[0127] See also Figure 7 , the specific steps for obtaining data synchronization status information are:

[0128] Use Apache Kafka to asynchronously push data in batches to the cloud, optimize the traceability of the transmission process, and obtain an encrypted and compressed data block queue;

[0129] In building the data transmission process, Apache Kafka is used to push batch asynchronous data to the cloud, which not only optimizes the traceability of the transmission process, but also improves the efficiency and security of data transmission through batch processing and compression technology. During the data transmission process, the built-in compression mechanism reduces the network load and increases the transmission speed. At the same time, Kafka's high throughput characteristics ensure that the needs of large-scale data processing are met. Through the application of technology, data blocks are effectively compressed and encrypted before transmission, ensuring the integrity and security of data before storage in the cloud, greatly reducing the risk of data theft or damage during transmission, and strengthening data protection measures.

[0130] Based on the encrypted and compressed data block queue, the formula is:

[0131]

[0132] Calculate the success rate of data transmission , get the data transmission efficiency log, where, is the number of data blocks received, is the number of data blocks lost during transmission, Indicates the total number of data blocks that are attempted to be sent. and Represent the weights of received and lost data blocks respectively;

[0133] In one transmission cycle, a total of 1000 data blocks are sent, of which 950 are successfully received and 50 are lost. Set the weight of the received data blocks. and the weight of the missing block , calculated according to the formula:

[0134]

[0135]

[0136] The result shows that the overall transmission efficiency is 92.5% considering the reception and loss of data blocks. The score reflects the high efficiency of data transmission and the impact of small data loss on the overall transmission quality.

[0137] Based on the data transmission efficiency log, the data block reception situation is analyzed and data synchronization status information is established;

[0138] Analyzing the reception of data blocks based on the transmission efficiency log involves establishing information on the data synchronization status, ensuring that all transmitted data blocks are correctly recorded and monitored, and extracting key performance indicators from the log, including the reception time and processing speed of each data block, as well as any delays or errors that occur. The data can be used to dynamically adjust transmission parameters and optimize the performance of the entire system. This not only improves the transparency of data processing, but also increases control over data flow, making data management more efficient and reliable, to ensure the integrity and punctuality of data throughout the entire transmission process.

[0139] See also Figure 8 , the specific steps for obtaining the data version consistency result are:

[0140] Based on the data synchronization status information, collect the cloud and local data version information, combine the release time and version number, and establish a version record list;

[0141] During the data synchronization process, cloud and local data version information is collected and recorded, including obtaining data version numbers and release times from various data sources and establishing a version record list. In this way, the history and changes of each data version can be seen. Each version is marked with an exact timestamp and version number, providing basic data for subsequent data version comparison and management.

[0142] Based on the version record list, the collected data versions are compared using the formula:

[0143]

[0144] Calculate the difference between data versions , identify data points with inconsistent versions and obtain version difference information, where, and Represents the local and cloud Data version number, is the total number of data points being compared;

[0145] The local and cloud version numbers with 5 data points are and , calculate the difference between each version:

[0146]

[0147]

[0148] The result shows that there are a total of 3 version differences, indicating that there are three data version conflicts that need further processing to resolve the conflicts and ensure data consistency.

[0149] According to the version difference information, the priority rules and user authority rules are applied to make data selection judgments, determine the optimal data version, and obtain the data version consistency result;

[0150] After analyzing the version differences, the data of different versions are prioritized and permission controlled by comprehensively applying priority rules and user permission rules. This not only involves the comparison of data versions, but also covers the differentiated permission processing between versions. For each data version, its applicability will be judged according to the set rules to determine which versions can overwrite existing data and which versions need to be retained to avoid data loss or erroneous overwriting. This rule-based judgment process ensures the correctness and security of data updates and generates data version consistency results.

[0151] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A cloud-based data synchronization system, characterized in that: The system comprises: The shard scheduling module calculates the internal structure of data partitions based on data type and size, obtains partition split points, allocates data blocks to designated processing threads, determines processing priorities, establishes parallel processing queues, and generates parallel task allocation tables; The data optimization module performs outlier detection and data fusion on each data block through a multi-threaded processing mechanism based on the parallel task allocation table, and refines the data quality and generates a data quality optimization record based on the preliminary processing results of each data block; The asynchronous push module extracts each data block from the data quality optimization record, establishes an asynchronous communication channel with the cloud, performs encryption and compression processing on each data block, asynchronously pushes the compressed data block to the cloud through Apache Kafka, and monitors the data push status to obtain data synchronization status information; The version conflict resolution module monitors the consistency of cloud and local data versions based on the data synchronization status information, identifies data version differences, determines data selection using priority rules and user authority rules for detected data conflicts, and generates data version consistency results; The execution steps of the encryption and compression process are specifically as follows: Extracting each data block from the data quality optimization record, establishing an asynchronous communication channel with the cloud, and obtaining a preprocessed data block; Based on the preprocessed data block, encrypt the data using AES-256, and verify the security of data transmission to obtain an encrypted data block; Based on the encrypted data block, the Brotli compression algorithm is applied to reduce the data volume and optimize the storage and transmission efficiency, using the formula: Calculating the Compression Efficiency Ratio , get the compressed data block, where Indicates the size of the compressed data. Indicates the original data size; The steps for obtaining the data synchronization status information are specifically as follows: Use the Apache Kafka to asynchronously push data in batches to the cloud, optimize the traceability of the transmission process, and obtain an encrypted and compressed data block queue; Based on the encrypted and compressed data block queue, the formula is adopted: Calculate the success rate of data transmission , get the data transmission efficiency log, where, is the number of data blocks received, is the number of data blocks lost during transmission, Indicates the total number of data blocks that are attempted to be sent. and Represent the weights of received and lost data blocks respectively; Based on the data transmission efficiency log, the data block reception status is analyzed and data synchronization status information is established.

2. The cloud-based data synchronization system according to claim 1, characterized in that: The steps for obtaining the partition splitting point are specifically as follows: Based on the data type and size, a density-based clustering algorithm is used to identify high-density areas in the data set and obtain a set of potential split points; Based on the set of potential split points, the data density score is calculated using the formula: According to the score, the point with the best data density is selected as the partition point, where: Indicate point The data density score, Indicate point The set of neighboring points of Indicate point arrive The Euclidean distance of is the attenuation coefficient, which is used to adjust the influence of distance.

3. The cloud-based data synchronization system according to claim 1, characterized in that: The steps for obtaining the parallel task allocation table are specifically as follows: Analyze the load capacity and data block size of each processing thread, and calculate the processing efficiency of each thread for its assigned data block using the formula: Estimate the load of processing threads and allocate data blocks to balance the workload of multiple threads. Indicates The estimated load of each thread, Indicates the allocation to the thread No. The size of the data block, Indicates The processing complexity coefficient of a data block is is the total number of data blocks; Based on the load of the processing thread, the allocation order and processing priority of the data blocks are optimized, and a parallel task allocation table is established.

4. The cloud-based data synchronization system according to claim 1, characterized in that: The steps of outlier detection and data fusion are specifically as follows: Based on the parallel task allocation table, determine the data blocks to be processed by each thread to obtain an initial mapping relationship; Based on the initial mapping relationship, outlier detection is performed using the formula: Determine whether the data point deviates from the normal range and obtain the data abnormality status ,in, represents the data point value, and are the mean and standard deviation of the data set, respectively; Based on the data abnormal state, data fusion is performed to integrate data blocks from the same source, and data integrity is verified to obtain a fused data block set.

5. The cloud-based data synchronization system according to claim 1, characterized in that: The steps for obtaining the data quality optimization record are specifically as follows: Collecting preliminary processing results of each data block, including information from outlier detection to data fusion, to obtain a processing detail record; The processing detail records are analyzed using the formula: Assessing the degree of bias in a dataset , identify and correct data quality issues, and obtain data quality optimization records, among which, Representative data points, Yes all The average value of Yes all The standard deviation of is the total number of data points.

6. The cloud-based data synchronization system according to claim 1, characterized in that: The steps for obtaining the data version consistency result are specifically as follows: Based on the data synchronization status information, collect cloud and local data version information, and create a version record list in combination with the release time and version number; Based on the version record list, the collected data versions are compared using the formula: Calculate the difference between data versions , identify data points with inconsistent versions and obtain version difference information, where, and Represents the local and cloud Data version number, is the total number of data points being compared; According to the version difference information, priority rules and user authority rules are applied to perform data selection judgment, determine the optimal data version, and obtain a data version consistency result.

Images