Real-time data synchronization method for cloud computing heterogeneous resources

Abstract

The invention belongs to the technical field of data synchronization, and particularly relates to a real-time data synchronization method for cloud computing heterogeneous resources. In order to solve the problem of data synchronization among heterogeneous resources, the method comprises the following steps: aggregating a plurality of files in batches to generate a packaged file; performing metadata compression processing on the generated packaged file, and eliminating repeated data through a deduplication algorithm; dynamically selecting a lightweight data transmission protocol; establishing a multi-dimensional index structure; only retaining the file of the latest version through a version control mechanism; dynamically adjusting the allocation of synchronization tasks based on the load condition of available resources in the cloud computing environment; and rescheduling the uncompleted synchronization task. Through a multi-dimensional index structure, version control, self-adaptive transmission scheduling and a dynamic resource allocation mechanism, efficient and stable file synchronization among heterogeneous resources is realized, repeated transmission and storage waste are reduced, and real-time performance, reliability and resource utilization efficiency of synchronization are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data synchronization, and in particular relates to a real-time data synchronization method for heterogeneous cloud computing resources. Background Art

[0002] With the rapid development of cloud computing technology, data synchronization between heterogeneous resources has become a key link in modern cloud computing architecture. Heterogeneous resources usually refer to different types of computing nodes, storage devices, and network devices. These resources run on different operating systems and platforms. Due to the differences in geographical location, hardware environment, and network conditions of these resources, how to ensure efficient and real-time data synchronization between heterogeneous resources is a core technical challenge in the cloud computing environment. File creation, modification, and deletion operations are frequent, especially when it comes to synchronization between multiple heterogeneous resources. Large amounts of data transmission will cause synchronization delays, resource waste, and other problems.

[0003] The existing technologies usually have the following difficulties in data synchronization: first, there is a lack of flexible transmission protocol selection mechanism for different resources and network environments, which easily leads to network congestion or transmission failure; second, the transmission and storage management of duplicate data in the file synchronization process is insufficient, and there is a lack of deduplication and version control mechanisms, resulting in waste of storage resources and low transmission efficiency; third, traditional data synchronization strategies cannot dynamically adapt to the load changes of heterogeneous resources, lack priority scheduling of synchronization tasks and real-time allocation of resources; finally, when the transmission fails or the network is interrupted, the existing technology cannot effectively trigger the automatic recovery mechanism, resulting in the need to restart the synchronization task, causing duplication of transmission and low efficiency. Therefore, there is an urgent need for an efficient solution that can provide real-time and dynamic adjustment for file synchronization between heterogeneous resources to improve the reliability of data transmission and resource utilization efficiency. Summary of the invention

[0004] In view of the above problems, the present invention provides a real-time data synchronization method for heterogeneous resources in cloud computing.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a real-time data synchronization method for heterogeneous cloud computing resources, comprising the following steps:

[0007] S1: Deploy a file monitoring client on each heterogeneous resource to monitor the creation, modification, and deletion of files in real time. When the file update operation meets the preset conditions, multiple files are aggregated in batches to generate a package file based on time correlation or spatial proximity.

[0008] S2: Perform metadata compression on the packaged files generated by S1, optimize the metadata information of the file path, size, and timestamp, and eliminate duplicate data through a deduplication algorithm to reduce the redundant metadata overhead generated during the synchronization process;

[0009] S3: Dynamically selects a lightweight data transmission protocol based on the size of the packaged file, bandwidth between resources, and network latency to ensure data security while improving transmission efficiency;

[0010] S4: Based on the hash value of the file and the timestamp information of the file update, a multi-dimensional index structure is established to identify the synchronized and unsynchronized files, and the B+ tree structure is used to achieve search and real-time differential synchronization to ensure that only the unsynchronized files or their updated contents are transmitted;

[0011] S5: Perform version control management on the packaged files received from various heterogeneous resources on the central synchronization server. Through the version control mechanism, only the latest version of the file is retained, and historical versions or duplicate files are excluded;

[0012] S6: Prioritize synchronization tasks according to the importance and urgency of the packaged files, and dynamically adjust the allocation of synchronization tasks based on the load of available resources in the cloud computing environment through a dynamic resource allocation mechanism;

[0013] S7: During the file transfer and synchronization process, the synchronization progress and error information are monitored in real time. When a transmission failure or error is detected, the automatic recovery mechanism will be triggered to reschedule the unfinished synchronization tasks to ensure the integrity and consistency of file synchronization.

[0014] Further, the S1 specifically includes:

[0015] S11: deploying a file monitoring client on each heterogeneous resource, wherein the client is connected to the file system through a system kernel interface or API, and is used to capture file operation events in the file system in real time, including file creation, modification, and deletion operations, and generate a unique identifier and timestamp for each operation;

[0016] S12: By analyzing the captured file operation events, the events are compared with preset conditions, where the preset conditions include the size, type, operation frequency, and time window of the file. Only when the update operation of the file meets the preset size threshold, operation frequency, and specified time window, the file is marked as a file that needs to be synchronized.

[0017] S13: Based on the file operations that meet the preset conditions, a time correlation algorithm is used to calculate the operation time intervals of multiple files, and files that are related or operated in the same batch within a specified time range are identified; and a spatial proximity algorithm is used to analyze and identify the closeness of the files in the physical storage structure or logical file path, and files with close paths are preferentially selected for aggregation;

[0018] S14: batch-aggregate the files analyzed by time correlation and spatial proximity, package the files into a single transmission file package according to the order of creation and modification time or spatial location of the files, and attach corresponding metadata.

[0019] Furthermore, the S13 specifically includes:

[0020] S131: Based on the timestamps of the file operations, a time correlation algorithm is used to calculate the time intervals of the operations of multiple files. The time correlation algorithm calculates the time difference Δt of the operations of adjacent files. i,j , when Δt i,j When the time difference is less than the preset time threshold τ, it is considered that file i and file j have time correlation, and the time difference Δt i,j The calculation formula is as follows:

[0021] Δt i,j =|T i -T j |

[0022] Among them, T i and T j Respectively represent the operation timestamps of the i-th file and the j-th file;

[0023] S132: Calculate the closeness of the files in the physical storage structure or logical file path by using a spatial proximity analysis algorithm;

[0024] For the physical storage structure, use the storage block number B i and B j Calculate the physical distance D between adjacent files i,j , when D i,j When the distance is less than the preset spatial distance threshold σ, the file is considered to have spatial proximity in physical storage, and the physical distance D i，j The calculation formula is as follows:

[0025] D i，j =|B i -B j |

[0026] Among them, B i and B j Respectively represent the storage block numbers of the i-th file and the j-th file;

[0027] For the logical file path, calculate the path level similarity. When the path level similarity S i，j Greater than the set threshold S th When , the files are considered to have spatial proximity on the logical path; the calculation formula is:

[0028]

[0029] Among them, L common Indicates the length of the common part of the file path, L total Indicates the total length of the entire file path;

[0030] S133: By means of the dual constraints of time relevance and space proximity, files that satisfy both time and space proximity conditions are preferentially selected to form a packaged file.

[0031] Further, the S2 specifically includes:

[0032] S21: Parse the metadata information contained in the packaged file generated in S1, and extract the attribute information of the file path, file size, and timestamp of the file operation;

[0033] S22: compressing the extracted file path information using a path compression algorithm, merging the common parts in the path, and reducing the storage of redundant paths in the metadata;

[0034] S23: Standardizing the file size and timestamp information, that is, performing timestamp differential storage, thereby reducing repeated storage of absolute timestamps;

[0035] S24: Use a hash algorithm to deduplicate metadata information, generate a unique hash value for each file, and delete redundant metadata information for files with the same hash value to ensure that the metadata is unique during the synchronization process;

[0036] S25: re-encapsulate the metadata after path compression, timestamp differential storage and hash deduplication processing to generate optimized packaged file metadata.

[0037] Further, the S3 specifically includes:

[0038] S31: Get the size F of the package file s and the bandwidth B between the current heterogeneous resources w and network delay L d ;

[0039] S32: Calculate the expected transmission time T of the packaged file under different bandwidth and network delay conditions t , the formula is:

[0040]

[0041] Among them, T t Indicates the expected transfer time of the file in seconds;

[0042] S33: According to the calculated expected transmission time T t , compare whether the time threshold T required for real-time synchronization is met th , if T t ≤T th When the traffic is transmitted, lightweight transport protocols are preferred;

[0043] S34: According to the size of the packaged file F s Select the transport protocol. The options are:

[0044] When F s <F th1 When , select UDP protocol;

[0045] When F th1 ≤F s ≤F th2 When the QUIC protocol is selected, it ensures the transmission stability and speed;

[0046] When F s >F th2 , then select TCP protocol to ensure the reliability of transmission;

[0047] Among them, F th1 The first preset threshold for file size, used to distinguish small files from medium-sized files; F th2 A second preset threshold for file size, used to distinguish between medium-sized files and large files;

[0048] S35: In the process of selecting the transmission protocol, an adaptive transmission scheduling algorithm is introduced to adjust the protocol priority and transmission parameters in real time according to the dynamic changes of actual bandwidth and network delay, so as to avoid transmission failure or reduced efficiency due to network fluctuations;

[0049] S36: Applying the finally selected transmission protocol to the data synchronization process of the packaged files.

[0050] Furthermore, the S35 specifically includes:

[0051] S351: Real-time monitoring of bandwidth between heterogeneous resources B w (t) and network delay L d (t); Adjust protocol priorities in real time based on changes in bandwidth and latency;

[0052] S352: Define protocol priority function P protocol(t), calculated based on the bandwidth and delay at the current time t; when P protocol When the (t) value is high, the efficient transmission protocol is preferred; when P protocol When the (t) value is low, a more stable protocol is selected;

[0053] S353: dynamically adjusting the parameters of the transmission protocol according to the current network conditions by calculating the transmission parameters, including the transmission window size W(t) and the transmission rate R(t);

[0054] The adjustment formula of the transmission window size W(t) is:

[0055]

[0056] Where W(t) represents the transmission window size at time t, γ is a control parameter, which represents the adjustment ratio of bandwidth and delay to the transmission window;

[0057] The transmission rate R(t) is determined according to the bandwidth and the size of the transmission window, and the formula is:

[0058]

[0059] Where R(t) represents the transmission rate at time t, T t is the expected transmission time;

[0060] S354: According to the calculated priority P protocol (t) and transmission parameters W(t), R(t), dynamically adjusting the protocol selection and specific transmission strategy.

[0061] Further, the S4 specifically includes:

[0062] S41: Based on the hash value H of each file i and the timestamp T of the file update i , generate a unique identifier ID for each file i , as the only index item of the file;

[0063] S42: construct a multidimensional index structure for distinguishing synchronized files from unsynchronized files. The multidimensional index structure consists of two dimensions, including a hash value dimension and a timestamp dimension. i and timestamp T i As a keyword, it is identified in the index; files with the same hash value and unchanged timestamp are identified as synchronized files; if the hash value does not match or the timestamp changes, T i >T j , it is marked as an unsynchronized file;

[0064] S43: Using B+ tree structure to index file item ID i To organize, each node in the B+ tree stores the hash value and timestamp combination of the file, and the nodes are arranged in the order of the hash value;

[0065] S44: When performing file synchronization, first match the hash values ​​and timestamps of all files in the current package file, and search for corresponding items in the B+ tree; if the hash values ​​match and the timestamps are the same, the file is considered to be synchronized and does not need to be transmitted again; if the hash values ​​do not match or the timestamps are different, it is marked as an unsynchronized file and its updated content is prepared for transmission.

[0066] Furthermore, the version control mechanism in S5 specifically includes:

[0067] S51: The central synchronization server receives the packaged files transmitted from the heterogeneous resources, and parses the metadata information of each file, including the hash value, file size and update timestamp of the file;

[0068] S52: Through the version control mechanism, the central synchronization server checks the hash value of each file for duplication. If a file with the same hash value already exists in the file library of the central synchronization server, its timestamp T is compared. i and the corresponding file time T stored in the central synchronization server server , if T i >T server When receiving a file, it is determined that the latest version of the file is received and the old version of the file in the central synchronization server is replaced by the file; otherwise, the file is ignored to avoid duplicate storage;

[0069] S53: For a file with multiple historical versions, only the latest version file is retained. Specifically, when multiple versions of the file have the same hash value H i But different timestamps T i When the central synchronization server sorts files by timestamp, it keeps the files with the largest timestamp and deletes or archives other historical version files to reduce storage space usage;

[0070] S54: To prevent repeated file transmission, the central synchronization server will mark the processed files with a synchronized flag F. sync , and set the synchronization flag F sync Bind to the file's hash value and timestamp to ensure that the same version of the file will not be repeatedly transmitted or stored during subsequent synchronization.

[0071] Further, the S6 specifically includes:

[0072] S61: Priority evaluation is performed on each packaged file based on the importance of the file. i and urgency level Ui , the evaluation formula is:

[0073] P i =α·I i +β·U i

[0074] Among them, P i is the priority score of the packaged file, α and β are weight parameters;

[0075] S62: Based on the priority score P i , assign a preliminary synchronization order to each package file;

[0076] S63: Monitor available computing resources in the cloud computing environment R t , including the current load of CPU, memory and bandwidth, and through the dynamic resource allocation mechanism, the load is analyzed in real time to calculate the resource utilization U at the current time t r (t), the formula is:

[0077]

[0078] Among them, R used (t) represents the amount of resources used at the current time t, R total (t) represents the total resource capacity;

[0079] S64: When the resource utilization U r (t) is lower than the preset load threshold U th When U r (t) exceeds the threshold U th When syncing, the synchronization tasks of low-priority files are postponed, and available resources are allocated to the synchronization of high-priority files first.

[0080] Further, the S7 specifically includes:

[0081] S71: During the file transfer and synchronization process, the central synchronization server monitors the transmission status in real time, detects transmission failures or errors, and each file F i The transmission task will generate a transmission state S i , if S i =1 indicates success, if S i =0 means failure;

[0082] S72: When a transmission failure or error is detected, a recovery mechanism is automatically triggered to record the failed file and its transmission status information through a log system, including the reason for the failure and the transmission progress at the time of the failure;

[0083] S73: The recovery mechanism first checks the transfer progress of the file, and determines the size of the fragments of the file that need to be retransmitted by calculating the transferred part and the unfinished part of the file;

[0084] S74: Optimize and adjust the transmission parameters according to the reason for the transmission failure. If the failure is caused by network interruption or high latency, the transmission protocol will be dynamically adjusted to select a protocol suitable for the current network environment. If the failure is caused by insufficient bandwidth or resource overload, the execution of low-priority tasks will be postponed.

[0085] S75: rescheduling the unfinished transmission tasks. The recovery mechanism dynamically arranges the retransmission tasks based on the current system resource status and task priority, giving priority to the transmission of high-priority files;

[0086] S76: After the recovery task is completed, update the file transfer status S i =1, and perform a consistency check to ensure that the file content is consistent with the original file; if the consistency check passes, the file will be marked as synchronized and the fault state will be released.

[0087] Compared with the prior art, the present invention has the following advantages:

[0088] The present invention effectively solves the core problem of data synchronization between heterogeneous resources in cloud computing by introducing a multidimensional index structure, a hash algorithm, version control, and an adaptive transmission scheduling algorithm. By managing the hash value and timestamp of the file, using the B+ tree structure to achieve real-time differential synchronization, only the unsynchronized files or their updated contents are transmitted, reducing the transmission of duplicate data and greatly improving the efficiency of data synchronization. At the same time, the version control mechanism ensures that the central synchronization server only retains the latest version of the file, automatically excluding historical versions or duplicate files, thereby effectively reducing the waste of storage space.

[0089] The present invention uses a dynamic resource allocation mechanism to dynamically adjust the priority and transmission protocol of synchronization tasks according to the importance and urgency of packaged files and the load of available resources in a cloud computing environment. The adaptive transmission scheduling algorithm can automatically select the optimal transmission protocol according to real-time network conditions and trigger an automatic recovery mechanism in the event of transmission failure or error, so that only the unfinished partial files are transmitted to ensure the integrity and consistency of file synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0091] Figure 1 It is a flow chart of the real-time data synchronization method of cloud computing heterogeneous resources of the present invention;

[0092] Figure 2 This is a flow chart of data compression and deduplication in S2 of the present invention. DETAILED DESCRIPTION

[0093] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0094] It should be noted that the references to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0095] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0096] like Figure 1 and 2 As shown, a real-time data synchronization method for heterogeneous cloud computing resources in this embodiment includes the following steps:

[0097] S1: Deploy a file monitoring client on each heterogeneous resource to monitor the creation, modification, and deletion of files in real time. When the file update operation meets the preset conditions, multiple files are aggregated in batches to generate a packaged file based on time correlation or spatial proximity.

[0098] Specifically: S11: deploy a file monitoring client on each heterogeneous resource, the client connects to the file system through a system kernel interface or API, and is used to capture file operation events in the file system in real time, including file creation, modification and deletion operations, and generate a unique identifier and timestamp for each operation.

[0099] S12: By analyzing the captured file operation events, the events are compared with preset conditions, which include the file size, type, operation frequency and time window in which the operation occurs. Only when the file update operation meets the preset size threshold, operation frequency and specified time window, it is marked as a file that needs to be synchronized.

[0100] S13: Based on file operations that meet preset conditions, a time correlation algorithm is used to calculate the operation time intervals of multiple files, and files that are correlated within a specified time range or operated in the same batch are identified; and a spatial proximity algorithm is used for analysis to identify the closeness of files in the physical storage structure or logical file path, and files with close paths are prioritized for aggregation.

[0101] Specifically: S131: Based on the timestamps of the file operations, a time correlation algorithm is used to calculate the time intervals of the operations of multiple files. The time correlation algorithm calculates the time difference Δt of the operations of adjacent files. i,j , when Δt i,j When the time difference is less than the preset time threshold τ, it is considered that file i and file j have a time correlation. Based on this correlation, all eligible files are classified into the same batch. i,j The calculation formula is as follows:

[0102] Δt i,j =|T i -T j |

[0103] Among them, T i and T j Respectively represent the operation timestamps of the i-th file and the j-th file;

[0104] S132: Calculate the closeness of the files in the physical storage structure or logical file path by using a spatial proximity analysis algorithm;

[0105] For the physical storage structure, use the storage block number B i and B j Calculate the physical distance D between adjacent files i,j , when D i,j When the distance is less than the preset spatial distance threshold σ, the file is considered to have spatial proximity in physical storage, and the physical distance D i,j The calculation formula is as follows:

[0106] D i,j =|B i -B j |

[0107] Among them, B i and Bj Represent the storage block numbers of the i-th file and the j-th file respectively;

[0108] For the logical file path, calculate the path level similarity. When the path level similarity S i,j Greater than the set threshold S th When , the files are considered to have spatial proximity on the logical path; the calculation formula is:

[0109]

[0110] Among them, L common Indicates the length of the common part of the file path, L total Indicates the total length of the entire file path;

[0111] S133: By applying the dual constraints of time correlation and spatial proximity, files that meet both time and space proximity conditions are selected first to form a packaged file. The above steps, through the combination of the time correlation algorithm and the spatial proximity analysis algorithm, can more intelligently identify files that are closely related to the storage structure within a specific time range and perform effective batch packaging processing, which not only optimizes the file packaging efficiency, but also reduces the transmission and storage resource overhead during the file synchronization process, further improving the real-time synchronization performance in a heterogeneous resource environment.

[0112] S14: Batch aggregate the files analyzed by time correlation and spatial proximity, and package the files into a single transmission file package according to the order of creation and modification time or spatial location of the files, and attach corresponding metadata for subsequent transmission and synchronization. Through the above steps, it is ensured that file operations can be captured efficiently and accurately on each heterogeneous resource and batch aggregate files based on time and space correlation, effectively reducing the number of transmitted files and metadata redundancy, thereby improving the overall performance and accuracy of real-time data synchronization.

[0113] S2: Perform metadata compression on the packaged files generated by S1, optimize the metadata information of the file path, size, and timestamp, and eliminate duplicate data through a deduplication algorithm to reduce the redundant metadata overhead generated during the synchronization process.

[0114] Specifically: S21: Parse the metadata information contained in the packaged file generated by S1, and extract the attribute information of the file path, file size and the timestamp of the file operation.

[0115] S22: Use a path compression algorithm to compress the extracted file path information, merge the common parts in the path, and set the file path as P i , multiple paths P in the file collection i and Pj The common path part P can be extracted by common To merge, the formula is:

[0116] P compressed =P common +P unique

[0117] Among them, P compressed represents the compressed path, P common is the public part of the file path, P unique Represents the unique portion of each file in the path. This algorithm reduces the storage of redundant paths in the metadata.

[0118] S23: Standardize the file size and timestamp information, that is, perform timestamp differential storage; use the timestamp differential storage method to store only the difference ΔT between adjacent file operation times. i，j , the formula is:

[0119] ΔT i，j =T i -T j

[0120] Where, ΔT i，j Represents the timestamp difference between the i-th file and the j-th file, T i Indicates the operation timestamp of the i-th file, T i Represents the operation timestamp of the jth file; through the differential storage method, the repeated storage of absolute timestamps is reduced.

[0121] S24: Use a hash algorithm to deduplicate metadata information and generate a unique hash value for each file. The formula is:

[0122] H i = hash(P i +S i +T i )

[0123] Among them, H i represents the hash value of the i-th file, P i Indicates the file path, S i Indicates the file size, T i Represents the file timestamp, by comparing the hash value H i and H j , determine whether the file is duplicated. For files with the same hash value, delete the redundant metadata information to ensure that the metadata is unique during the synchronization process.

[0124] S25: Re-encapsulate the metadata after path compression, timestamp differential storage and hash deduplication processing to generate optimized packaged file metadata to minimize storage space and data transmission overhead, ready for subsequent synchronization and transmission steps; the above steps significantly reduce the metadata redundancy of packaged files by compressing, differencing and deduplicating file paths, sizes and timestamps, optimize the data storage and transmission process, and improve the data synchronization efficiency between heterogeneous resources in cloud computing.

[0125] S3: Dynamically selects a lightweight data transmission protocol based on the size of the packaged file, bandwidth between resources, and network latency to ensure data security while improving transmission efficiency.

[0126] Specifically: S31: Get the size F of the package file s and the bandwidth B between the current heterogeneous resources w and network delay L d , F s Indicates the total size of the packaged file, B w Indicates the available transmission bandwidth between resources, L d Indicates the network delay during transmission.

[0127] S32: Calculate the expected transmission time T of the packaged file under different bandwidth and network delay conditions t , the formula is:

[0128]

[0129] Among them, T t Indicates the expected transfer time of the file in seconds.

[0130] S33: According to the calculated expected transmission time T t , compare whether the time threshold T required for real-time synchronization is met th , if T t ≤T th When the transmission delay is low and the bandwidth usage is small, lightweight transmission protocols are preferred.

[0131] S34: According to the size of the packaged file F s Select the transport protocol. The options are:

[0132] When F s <F th1 When , select UDP protocol;

[0133] When F th1 ≤F s ≤F th2 When the QUIC protocol is selected, it ensures the transmission stability and speed;

[0134] When F s >F th2 , then select TCP protocol to ensure the reliability of transmission;

[0135] Among them, F th1 The first preset threshold for file size, used to distinguish small files from medium-sized files; F th2 The second preset threshold for file size is used to distinguish between medium-sized files and large files.

[0136] S35: An adaptive transmission scheduling algorithm is introduced in the transmission protocol selection process to adjust the protocol priority and transmission parameters in real time according to the dynamic changes of actual bandwidth and network delay, so as to avoid transmission failure or reduced efficiency due to network fluctuations.

[0137] Specific: S351: Real-time monitoring of bandwidth between heterogeneous resources B w (t) and network delay L d (t), where B w (t) represents the available bandwidth at time t, Ld(t) represents the network delay at time t, and the bandwidth and delay data are collected periodically to record their dynamic changes; the priority of the protocol is adjusted in real time according to the changes in bandwidth and delay. The basic principle of priority adjustment is that when the bandwidth B w (t)Higher and delayed L d When (t) is small, high-efficiency and low-latency protocols (such as UDP) are preferred; when B w (t) Lower or L d When (t) is large, a more reliable protocol (such as TCP) is selected.

[0138] S352: Define protocol priority function P protocol (t), calculated based on the bandwidth and delay at the current time t, the formula is:

[0139]

[0140] Among them, P protocol (t) represents the priority of the transmission protocol at time t, B max is the maximum bandwidth in the system, L max is the maximum tolerable delay; α and β are weight parameters, which control the impact of bandwidth and delay on priority respectively;

[0141] When P protocol When the (t) value is high, efficient transport protocols (such as UDP or QUIC) are preferred; when P protocol When the (t) value is low, a more stable protocol (such as TCP) is chosen.

[0142] S353: dynamically adjusting the parameters of the transmission protocol according to the current network conditions by calculating the transmission parameters, including the transmission window size W(t) and the transmission rate R(t);

[0143] The adjustment formula of the transmission window size W(t) is:

[0144]

[0145] Where W(t) represents the transmission window size at time t, γ is a control parameter, which represents the adjustment ratio of bandwidth and delay to the transmission window;

[0146] The transmission rate R(t) is determined according to the bandwidth and the size of the transmission window, and the formula is:

[0147]

[0148] Where R(t) represents the transmission rate at time t, T t is the expected transfer time.

[0149] S354: According to the calculated priority P protocol (t) and transmission parameters W(t), R(t), dynamically adjust the protocol selection and specific transmission strategy, and ensure that the system can always transmit data with the best performance under different network conditions by adjusting the protocol and transmission parameters in real time; the above steps can adjust the protocol priority and transmission parameters in real time according to the dynamic changes of network bandwidth and delay through the adaptive transmission scheduling algorithm, ensuring that data transmission remains efficient and stable when the network status changes. The adaptive algorithm not only improves the flexibility of the transmission protocol, but also optimizes the data synchronization performance in heterogeneous resource environments.

[0150] S36: Apply the finally selected transmission protocol to the data synchronization process of the packaged files, ensure that both transmission efficiency and reliability are taken into account during the data transmission process, and ensure that the packaged files can be efficiently transmitted under different network conditions; through the above steps, the most suitable lightweight transmission protocol can be dynamically selected according to parameters such as the size of the packaged file, bandwidth and network delay, thereby improving data transmission efficiency and reducing delays, and adaptively adjusting the protocol according to actual network conditions to ensure the stability and reliability of data synchronization between heterogeneous resources.

[0151] S4: Based on the hash value of the file and the timestamp information of the file update, a multidimensional index structure is established to identify the synchronized and unsynchronized files, and the B+ tree structure is used to achieve search and real-time differential synchronization to ensure that only the unsynchronized files or their updated contents are transmitted.

[0152] Specifically: S41: Based on the hash value H of each file i and the timestamp T of the file updatei , generate a unique identifier ID for each file i , where H i represents the hash value of the i-th file, and the file content is mapped to a hash value of fixed length through the hash algorithm; T i Indicates the latest update timestamp of the i-th file; the generated identifier ID i =H i +T i , as the only index entry for the file.

[0153] S42: construct a multidimensional index structure for distinguishing synchronized files from unsynchronized files. The multidimensional index structure consists of two dimensions, including a hash value dimension and a timestamp dimension. i and timestamp T i As a keyword, it is identified in the index; files with the same hash value and unchanged timestamp are identified as synchronized files; if the hash value does not match or the timestamp changes, T i >T j , it is marked as an unsynchronized file.

[0154] S43: Using B+ tree structure to index file item ID i Each node in the B+ tree stores the hash value and timestamp combination of the file, and the nodes are arranged in the order of the hash value. The search and insertion operations in the B+ tree are completed with a time complexity of O(logn). The B+ tree structure can quickly locate the update status of the file.

[0155] S44: When executing file synchronization, first match the hash values ​​and timestamps of all files in the current package file, and search for corresponding items in the B+ tree; if the hash values ​​match and the timestamps are the same, the file is deemed to be synchronized and does not need to be transmitted again; if the hash values ​​do not match or the timestamps are different, it is marked as an unsynchronized file and its updated content is prepared to be transmitted; the above steps can effectively identify the synchronization status of files through the combination of multi-dimensional index structure and B+ tree, quickly locate synchronized and unsynchronized files, and ensure that only unsynchronized files or their updated content are transmitted, thereby greatly reducing the amount of data transmission and improving the real-time performance of file synchronization and the transmission efficiency of the system.

[0156] S5: Perform version control management on the packaged files received from various heterogeneous resources on the central synchronization server. Through the version control mechanism, only the latest version of the file is retained, and historical versions or duplicate files are excluded.

[0157] Specifically: S51: The central synchronization server receives the packaged files transmitted from various heterogeneous resources, and parses the metadata information of each file, including the hash value, file size and update timestamp of the file.

[0158] S52: Through the version control mechanism, the central synchronization server checks the hash value of each file for duplication. If a file with the same hash value already exists in the file library of the central synchronization server, its timestamp T is compared. i and the corresponding file time T stored in the central synchronization server server , if T i >T server When the latest version of the file is received, it is determined that the file is the latest version and the old version of the file in the central synchronization server is replaced by the file; otherwise, the file is ignored to avoid duplicate storage.

[0159] S53: For a file with multiple historical versions, only the latest version file is retained. Specifically, when multiple versions of the file have the same hash value H i But different timestamps T i When the central synchronization server sorts files by timestamp, it retains the files with the largest timestamps and deletes or archives other historical version files to reduce storage space usage.

[0160] S54: To prevent repeated file transmission, the central synchronization server will mark the processed files with a synchronized flag F. sync , and set the synchronization flag F sync Binding with the hash value and timestamp of the file ensures that the same version of the file will not be repeatedly transmitted or stored during subsequent synchronization. The above steps can ensure that the central synchronization server only retains the latest version of the file through the above version control mechanism, excluding historical versions and duplicate files, greatly reducing storage space usage, avoiding the transmission and storage operations of duplicate files, and improving the overall efficiency of file synchronization and the system's resource management capabilities.

[0161] S6: Prioritize synchronization tasks according to the importance and urgency of the packaged files, and dynamically adjust the allocation of synchronization tasks based on the load of available resources in the cloud computing environment through a dynamic resource allocation mechanism.

[0162] Specifically: S61: Priority evaluation is performed on each packaged file based on the importance of the file. i and urgency level U i , among which, the importance I i Assess the urgency of the file based on its business criticality and usage. i According to the last updated timestamp T of the file i The calculation is based on the next scheduled synchronization time. The evaluation formula is:

[0163] P i =α·I i +β·U i

[0164] Among them, P i is the priority score of the packaged file, α and β are weight parameters, which respectively control the impact of file importance and urgency on priority.

[0165] S62: Based on the priority score P i , assign a preliminary synchronization order to each packaged file. The files with higher priority will be given priority to enter the synchronization queue to ensure that core business data and urgent files are synchronized in time.

[0166] S63: Monitor available computing resources in the cloud computing environment R t , including the current load of CPU, memory and bandwidth, and through the dynamic resource allocation mechanism, the load is analyzed in real time to calculate the resource utilization U at the current time t r (t), the formula is:

[0167]

[0168] Among them, R used (t) represents the amount of resources used at the current time t, R total (t) represents the total resource capacity, and resource utilization is used to evaluate the availability of current resources.

[0169] S64: When the resource utilization U r (t) is lower than the preset load threshold U th When U r (t) exceeds the threshold U th When synchronizing files with low priority, the synchronization tasks of low priority files are postponed, and available resources are allocated to the synchronization of high priority files first to ensure the efficiency and stability of synchronization. The above steps can effectively distinguish the importance and urgency of files through priority scheduling and dynamic resource allocation mechanism, and reasonably allocate synchronization resources, ensuring that the task order can be dynamically adjusted according to the system load in the cloud computing environment, so as to achieve efficient synchronization task scheduling and resource utilization optimization.

[0170] S7: During the file transfer and synchronization process, the synchronization progress and error information are monitored in real time. When a transmission failure or error is detected, the automatic recovery mechanism will be triggered to reschedule the unfinished synchronization tasks to ensure the integrity and consistency of file synchronization.

[0171] Specifically: S71: During the file transfer and synchronization process, the central synchronization server monitors the transmission status in real time, detects transmission failures or errors, and each file F i The transmission task will generate a transmission state S i , if S i=1 indicates success, if S i =0 means failure.

[0172] S72: When a transmission failure or error is detected (i.e., S i =0), the recovery mechanism is automatically triggered to record the failed files and their transfer status information through the log system, including the reasons for the failure (such as network interruption, timeout, etc.) and the transfer progress P at the time of failure. i , where P i Indicates the portion of the file that was successfully transferred.

[0173] S73: The recovery mechanism first checks the file transfer progress P i , by calculating the transferred part and the incomplete part of the file, the size of the fragment that needs to be retransmitted is determined. The size of the incomplete fragment is:

[0174] ΔF i =F s -P i

[0175] Among them, F s For file F i The total size, ΔF i For the unfinished part, the recovery mechanism retransmits only the unfinished fragment, avoiding retransmitting the entire file.

[0176] S74: Optimize and adjust the transmission parameters according to the cause of the transmission failure. If the failure is caused by network interruption or high latency, the transmission protocol will be dynamically adjusted to select a protocol suitable for the current network environment (for example, switch from UDP to TCP to improve stability). If the failure is caused by insufficient bandwidth or resource overload, the execution of low-priority tasks will be postponed and resources will be allocated to recovery tasks first.

[0177] S75: Reschedule unfinished transmission tasks. The recovery mechanism dynamically arranges the retransmission tasks based on the current system resource status and task priority, giving priority to the transmission of high-priority files to ensure that the transmission tasks are completed as soon as possible if resources permit.

[0178] S76: After the recovery task is completed, update the file transfer status S i =1, and perform a consistency verification to ensure that the file content is consistent with the original file; if the consistency check passes, the file will be marked as synchronized and the fault state will be released; the above steps can automatically adjust the transmission strategy when the file transfer fails or errors occur through the above automatic recovery mechanism, and only transmit the incomplete file fragments, effectively reducing the overhead of repeated transmission. At the same time, through dynamic scheduling and protocol adjustment, the efficient execution of the recovery task is ensured, further ensuring the integrity and consistency of file synchronization.

[0179] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0180] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A real-time data synchronization method for heterogeneous cloud computing resources, characterized in that: The following steps are involved: S1: Deploy a file monitoring client on each heterogeneous resource to monitor the creation, modification, and deletion of files in real time. When the file update operation meets the preset conditions, multiple files are aggregated in batches to generate a package file based on time correlation or spatial proximity. S2: Perform metadata compression on the packaged files generated by S1, optimize the metadata information of the file path, size, and timestamp, and eliminate duplicate data through a deduplication algorithm to reduce the redundant metadata overhead generated during the synchronization process; S3: Dynamically selects a lightweight data transmission protocol based on the size of the packaged file, bandwidth between resources, and network latency to ensure data security while improving transmission efficiency; S4: Based on the hash value of the file and the timestamp information of the file update, a multi-dimensional index structure is established to identify the synchronized and unsynchronized files, and the B+ tree structure is used to achieve search and real-time differential synchronization to ensure that only the unsynchronized files or their updated contents are transmitted; S5: Perform version control management on the packaged files received from various heterogeneous resources on the central synchronization server. Through the version control mechanism, only the latest version of the file is retained, and historical versions or duplicate files are excluded; S6: Prioritize synchronization tasks according to the importance and urgency of the packaged files, and dynamically adjust the allocation of synchronization tasks based on the load of available resources in the cloud computing environment through a dynamic resource allocation mechanism; S7: During the file transfer and synchronization process, the synchronization progress and error information are monitored in real time. When a transmission failure or error is detected, the automatic recovery mechanism will be triggered to reschedule the unfinished synchronization tasks to ensure the integrity and consistency of file synchronization.

2. A real-time data synchronization method for heterogeneous cloud computing resources according to claim 1, characterized in that: The S1 specifically includes: S11: deploying a file monitoring client on each heterogeneous resource, wherein the client is connected to the file system through a system kernel interface or API, and is used to capture file operation events in the file system in real time, including file creation, modification, and deletion operations, and generate a unique identifier and timestamp for each operation; S12: By analyzing the captured file operation events, the events are compared with preset conditions, where the preset conditions include the size, type, operation frequency, and time window of the file. Only when the update operation of the file meets the preset size threshold, operation frequency, and specified time window, the file is marked as a file that needs to be synchronized. S13: Based on the file operations that meet the preset conditions, a time correlation algorithm is used to calculate the operation time intervals of multiple files, and files that are related or operated in the same batch within a specified time range are identified; and a spatial proximity algorithm is used to analyze and identify the closeness of the files in the physical storage structure or logical file path, and files with close paths are preferentially selected for aggregation; S14: batch-aggregate the files analyzed by time correlation and spatial proximity, package the files into a single transmission file package according to the order of creation and modification time or spatial location of the files, and attach corresponding metadata.

3. A real-time data synchronization method for heterogeneous cloud computing resources according to claim 2, characterized in that: The S13 specifically includes: S131: Based on the timestamps of the file operations, a time correlation algorithm is used to calculate the time intervals of the operations of multiple files. The time correlation algorithm calculates the time difference Δt of the operations of adjacent files. i,j , when Δt i,j When the time difference is less than the preset time threshold τ, it is considered that file i and file j have time correlation, and the time difference Δt i,j The calculation formula is as follows: Δt i,j =|T i -T j | Among them, T i and T j Respectively represent the operation timestamps of the i-th file and the j-th file; S132: Calculate the closeness of the files in the physical storage structure or logical file path by using a spatial proximity analysis algorithm; For the physical storage structure, use the storage block number B i and B j Calculate the physical distance D between adjacent files i,j , when D i,j When the distance is less than the preset spatial distance threshold σ, the file is considered to have spatial proximity in physical storage, and the physical distance D i,j The calculation formula is as follows: D i,j =|B i -B j | Among them, B i and B j Represent the storage block numbers of the i-th file and the j-th file respectively; For the logical file path, calculate the path level similarity. When the path level similarity S i,j Greater than the set threshold S th When , the files are considered to have spatial proximity on the logical path; the calculation formula is: Among them, l common Indicates the length of the common part of the file path, L total Indicates the total length of the entire file path; S133: By means of the dual constraints of time relevance and space proximity, files that satisfy both time and space proximity conditions are preferentially selected to form a packaged file.

4. A real-time data synchronization method for heterogeneous cloud computing resources according to claim 1, characterized in that: The S2 specifically includes: S21: Parse the metadata information contained in the packaged file generated in S1, and extract the attribute information of the file path, file size, and timestamp of the file operation; S22: compressing the extracted file path information using a path compression algorithm, and merging the common parts in the path; S23: Standardizing the file size and timestamp information, that is, performing timestamp differential storage; S24: Use a hash algorithm to deduplicate metadata information, generate a unique hash value for each file, and delete redundant metadata information for files with the same hash value to ensure that the metadata is unique during the synchronization process; S25: re-encapsulate the metadata after path compression, timestamp differential storage and hash deduplication processing to generate optimized packaged file metadata.

5. The real-time data synchronization method for heterogeneous cloud computing resources according to claim 1, characterized in that: The S3 specifically includes: S31: Get the size F of the package file s and the bandwidth B between the current heterogeneous resources w and network delay L d ; S32: Calculate the expected transmission time T of the packaged file under different bandwidth and network delay conditions t , the formula is: Among them, T t Indicates the expected transfer time of the file in seconds; S33: According to the calculated expected transmission time T t , compare whether the time threshold T required for real-time synchronization is met th , if T t ≤T th When the traffic is transmitted, lightweight transport protocols are preferred; S34: According to the size of the packaged file F s Select the transport protocol. The options are: When F s <F th1 When , select UDP protocol; When F th1 ≤F s ≤F th2 When the QUIC protocol is selected, it ensures the transmission stability and speed; When F s >F th2 , then select TCP protocol to ensure the reliability of transmission; Among them, F th1 The first preset threshold for file size, used to distinguish small files from medium-sized files; F th2 A second preset threshold for file size, used to distinguish between medium-sized files and large files; S35: In the process of selecting the transmission protocol, an adaptive transmission scheduling algorithm is introduced to adjust the protocol priority and transmission parameters in real time according to the dynamic changes of actual bandwidth and network delay, so as to avoid transmission failure or reduced efficiency due to network fluctuations; S36: Applying the finally selected transmission protocol to the data synchronization process of the packaged files.

6. A real-time data synchronization method for heterogeneous cloud computing resources according to claim 5, characterized in that: The S35 specifically includes: S351: Real-time monitoring of bandwidth between heterogeneous resources B w (t) and network delay L d (t); Adjust protocol priorities in real time based on changes in bandwidth and latency; S352: Define protocol priority function P protocol (t), calculated based on the bandwidth and delay at the current time t; when P protocol When the (t) value is high, the efficient transmission protocol is preferred; when P protocol When the (t) value is low, a more stable protocol is selected; S353: dynamically adjusting the parameters of the transmission protocol according to the current network conditions by calculating the transmission parameters, including the transmission window size W(t) and the transmission rate R(t); The adjustment formula of the transmission window size W(t) is: Where W(t) represents the transmission window size at time t, γ is a control parameter, which represents the adjustment ratio of bandwidth and delay to the transmission window; The transmission rate R(t) is determined according to the bandwidth and the size of the transmission window, and the formula is: Where R(t) represents the transmission rate at time t, T t is the expected transmission time; S354: According to the calculated priority P protocol (t) and transmission parameters W(t), R(t), dynamically adjusting the protocol selection and specific transmission strategy.

7. A real-time data synchronization method for heterogeneous cloud computing resources according to claim 1, characterized in that: The S4 specifically includes: S41: Based on the hash value H of each file i and the timestamp T of the file update i , generate a unique identifier ID for each file i , as the only index item of the file; S42: construct a multidimensional index structure for distinguishing synchronized files from unsynchronized files. The multidimensional index structure consists of two dimensions, including a hash value dimension and a timestamp dimension. i and timestamp T i As a keyword, it is identified in the index; files with the same hash value and no updated timestamp are identified as synchronized files; if the hash value does not match or the time Changes T i >T j , it is marked as an unsynchronized file; S43: Using B+ tree structure to index file item ID i To organize, each node in the B+ tree stores the hash value and timestamp combination of the file, and the nodes are arranged in the order of the hash value; S44: When performing file synchronization, first match the hash values ​​and timestamps of all files in the current package file, and search for corresponding items in the B+ tree; if the hash values ​​match and the timestamps are the same, the file is considered to be synchronized and does not need to be transmitted again; if the hash values ​​do not match or the timestamps are different, it is marked as an unsynchronized file and its updated content is prepared for transmission.

8. A real-time data synchronization method for heterogeneous cloud computing resources according to claim 1, characterized in that: The version control mechanism in S5 specifically includes: S51: The central synchronization server receives the packaged files transmitted from the heterogeneous resources, and parses the metadata information of each file, including the hash value, file size and update timestamp of the file; S52: Through the version control mechanism, the central synchronization server checks the hash value of each file for duplication. If a file with the same hash value already exists in the file library of the central synchronization server, its timestamp T is compared. i and the corresponding file time T stored in the central synchronization server server , if T i >T server When receiving a file, it is determined that the latest version of the file is received and the old version of the file in the central synchronization server is replaced by the file; otherwise, the file is ignored to avoid duplicate storage; S53: For a file with multiple historical versions, only the latest version file is retained. Specifically, when multiple versions of the file have the same hash value H i But different timestamps T i When the central synchronization server sorts files by timestamp, it keeps the files with the largest timestamp and deletes or archives other historical version files to reduce storage space usage; S54: To prevent repeated file transmission, the central synchronization server will mark the processed files with a synchronized flag F. sync , and set the synchronization flag F sync Bind to the file's hash value and timestamp to ensure that the same version of the file will not be repeatedly transmitted or stored during subsequent synchronization.

9. A real-time data synchronization method for heterogeneous cloud computing resources according to claim 1, characterized in that: The S6 specifically includes: S61: Priority evaluation is performed on each packaged file based on the importance of the file. i and urgency level U i , the evaluation formula is: P i =α·I i +β·U i Among them, P i is the priority score of the packaged file, α and β are weight parameters; S62: Based on the priority score P i , assign a preliminary synchronization order to each package file; S63: Monitor available computing resources in the cloud computing environment R t , including the current load of CPU, memory and bandwidth, and through the dynamic resource allocation mechanism, the load is analyzed in real time to calculate the resource utilization U at the current time t r (t), the formula is: Among them, R used (t) represents the amount of resources used at the current time t, R total (t) represents the total resource capacity; S64: When the resource utilization U r (t) is lower than the preset load threshold U th When U r (t) exceeds the threshold U th When syncing, the synchronization tasks of low-priority files are postponed, and available resources are allocated to the synchronization of high-priority files first.

10. The real-time data synchronization method for heterogeneous cloud computing resources according to claim 1, characterized in that: The S7 specifically includes: S71: During the file transfer and synchronization process, the central synchronization server monitors the transmission status in real time, detects transmission failures or errors, and each file F i The transmission task will generate a transmission state S i , if S i =1 indicates success, if S i =0 means failure; S72: When a transmission failure or error is detected, a recovery mechanism is automatically triggered to record the failed file and its transmission status information through a log system, including the reason for the failure and the transmission progress at the time of the failure; S73: The recovery mechanism first checks the transfer progress of the file, and determines the size of the fragments of the file that need to be retransmitted by calculating the transferred part and the unfinished part of the file; S74: Optimize and adjust the transmission parameters according to the reason for the transmission failure. If the failure is caused by network interruption or high latency, the transmission protocol will be dynamically adjusted to select a protocol suitable for the current network environment. If the failure is caused by insufficient bandwidth or resource overload, the execution of low-priority tasks will be postponed. S75: rescheduling the unfinished transmission tasks. The recovery mechanism dynamically arranges the retransmission tasks based on the current system resource status and task priority, giving priority to the transmission of high-priority files. S76: After the recovery task is completed, update the file transfer status S i =1, and perform a consistency check to ensure that the file content is consistent with the original file; if the consistency check passes, the file will be marked as synchronized and the fault state will be released.

Images