Secure storage synchronization method for cloud edge fusion

Abstract

The invention discloses a secure storage synchronization method for cloud edge fusion, and belongs to the field of computer storage, in the method, a synchronization participant initiates a synchronization request when a file needs to be synchronously modified, a cloud center node is responsible for processing the synchronization request and monitoring resources of each edge node, and the cloud center node is responsible for processing the synchronization request; the system performance is prevented from being affected by a large number of requests of malicious users through a token bucket algorithm, current limiting is achieved under coordination of a cloud center node, direct data communication of all edge nodes is coordinated, a safe communication channel between two synchronous parties is established, and the system safety is improved. By adopting the adaptive method provided by the invention, the synchronization mode is adaptively selected for efficient synchronization according to the network state during synchronization and the size of the file needing to be synchronized; by adopting the improved incremental synchronization method provided by the invention, the overhead of the incremental synchronization process can be obviously reduced; the problems of large storage synchronization calculation overhead, high time delay, limited edge device resources, high isomerism and poor reliability in the existing cloud edge fusion storage are solved.

Description

Technical Field

[0001] This invention belongs to the field of computer storage, and more specifically, relates to a secure storage synchronization method for cloud-edge convergence. Background Technology

[0002] With the development of big data technology and the popularization of the Internet of Things and edge computing, the global data volume is growing at an astonishing exponential rate. Due to this explosive growth, storing data in a central cloud data center would place a heavy burden on the central cloud; simultaneously, the sheer volume of data would inevitably cause core network congestion and a significant waste of communication resources. Therefore, edge storage is a more rational approach. Edge devices offer advantages such as low cost and low latency. A large number of edge devices exist on the internet, possessing considerable computing power and ample storage space. Storing and processing data at the edge, closer to users, can significantly improve service quality, fully utilize the capabilities of edge devices, and reduce the waste of communication resources on the backbone network.

[0003] Existing data storage synchronization technologies are mainly divided into two categories: full synchronization and incremental synchronization. Full synchronization is suitable for synchronizing small files, transmitting the entire file during synchronization with almost no computational overhead. However, this method wastes communication bandwidth when the file is large or modified too frequently. Incremental synchronization, on the other hand, does not transmit the entire file during synchronization. Instead, it divides the file into blocks and only transmits the modified blocks, thus significantly reducing network traffic and making it suitable for storing and synchronizing large files. Incremental synchronization is further divided into incremental synchronization based on fixed-size partitioning (FSC) and incremental synchronization based on content definition chunks (CDC) (i.e., block sizes are not fixed). Incremental synchronization based on fixed-size partitioning suffers from boundary offset problems, which are solved by incremental synchronization based on content definition chunks, but this introduces a larger computational overhead. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a secure storage synchronization method for cloud-edge convergence, which can significantly reduce the computational overhead and time latency of the synchronization process.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a secure storage synchronization method for cloud-edge convergence is provided, comprising: S1: The synchronization participants monitor the local synchronization folder in real time. When a file change is detected, a synchronization request is initiated. After receiving the synchronization request, the cloud center node uses the token bucket algorithm to determine whether the synchronization conditions are met. If so, a secure communication connection is established between the synchronization participants and the synchronization receiver, and the process proceeds to S2. Otherwise, the synchronization request is paused. The synchronization participants wait for a preset time and then re-initiate the synchronization request. S2, the synchronization participants will synchronize the current network bandwidth. Substitute into the threshold function to calculate the file size threshold To confirm the method of data synchronization with the synchronization receiver; wherein, when the size of the file to be synchronized is greater than When the data is synchronized, it is incremental synchronization; otherwise, it is full synchronization. , and All are preset coefficients; Wherein, the synchronization participant is a cloud center node or an edge node; when the synchronization participant is a cloud center node, the synchronization receiver is an edge node; when the synchronization participant is an edge node, the synchronization receiver is a cloud center node or an edge node.

[0006] According to a second aspect of the present invention, a cloud-edge converged secure storage synchronization system is provided, comprising: a cloud module and an edge module; The cloud module is deployed on a cloud center node and includes: The synchronization request processing module is used to receive synchronization requests initiated by synchronization participants and use the token bucket algorithm to determine whether they meet the synchronization conditions. The resource monitoring module is used to pause synchronization requests when synchronization conditions are not met. The cloud-edge communication coordination module is used to establish a secure communication connection between the synchronization participants and the synchronization receiver when the synchronization conditions are met. The Pod synchronization module includes: a change monitoring module, used to monitor the local sync folder in real time and initiate a synchronization request when a file change is detected; and a network assessment module, used to obtain the current network bandwidth. The file evaluation module is used to obtain the size of the file to be synchronized; the synchronization module is used to measure the current network bandwidth. Substitute into the threshold function to calculate the file size threshold To confirm the method of data synchronization with the synchronization receiver; wherein, when the size of the file to be synchronized is greater than When the data is synchronized, it is incremental synchronization; otherwise, it is full synchronization. , and All are preset coefficients; The edge module is deployed on the edge node, including the synchronization Pod.

[0007] According to a third aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to perform the method as described in the first aspect.

[0008] According to a fourth aspect of the invention, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the method described in the first aspect.

[0009] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: The cloud-edge converged secure storage synchronization method provided by this invention involves the participating parties initiating a synchronization request when they need to modify files. The cloud central node is responsible for processing the synchronization request, monitoring the resources of each edge node, and using a token bucket algorithm to prevent malicious users from making a large number of requests that could affect system performance. Rate limiting is implemented under the coordination of the cloud central node, and data communication between each edge node is coordinated to establish a secure communication channel between the synchronizing parties, thereby improving system security. The adaptive method proposed in this invention selects the synchronization mode adaptively based on the network status and the size of the file to be synchronized during synchronization, resulting in efficient synchronization and significantly reducing the computational overhead and time latency of the synchronization process.

[0010] As a further preferred embodiment, the present invention provides an improved incremental synchronization method based on CDC, which combines a guess-jump accelerated CDC technique to optimize the computation speed in the incremental synchronization process, thereby reducing resource consumption and lowering computational overhead during synchronization block division. This solves both the boundary offset problem of the traditional fixed block division method and the problem of high computational overhead of the general CDC method.

[0011] As a further preferred embodiment, the improved CDC-based incremental synchronization method provided by the present invention accumulates the rolling hash calculated after each window slide when calculating the rolling hash, and uses it as the weak hash value for subsequent blocks, thereby further reducing computational overhead. Attached Figure Description

[0012] Figure 1 An overview diagram of the secure storage synchronization method for cloud-edge convergence provided in this embodiment of the invention; Figure 2 A flowchart of the adaptive storage synchronization method provided in an embodiment of the present invention; Figure 3 A schematic diagram of an improved CDC incremental synchronization based on guesswork jumps provided for an embodiment of the present invention; Figure 4 This is a diagram of a secure storage synchronization system architecture for cloud-edge convergence provided in an embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0014] This invention provides a secure storage synchronization method for cloud-edge convergence, such as... Figure 1 As shown, it includes: S1: The synchronization participants monitor the local synchronization folder in real time. When a file change is detected, a synchronization request is initiated. After receiving the synchronization request, the cloud center node uses the token bucket algorithm to determine whether the synchronization conditions are met. If so, a secure communication connection is established between the synchronization participants and the synchronization receiver, and the process proceeds to S2. Otherwise, the synchronization request is paused. The synchronization participants wait for a preset time and then re-initiate the synchronization request. S2, the synchronization participants will synchronize the current network bandwidth. Substitute into the threshold function to calculate the file size threshold To confirm the method of data synchronization with the synchronization receiver; wherein, when the size of the file to be synchronized is greater than When the data is synchronized, it is incremental synchronization; otherwise, it is full synchronization. , and All are preset coefficients; Wherein, the synchronization participant is a cloud center node or an edge node; when the synchronization participant is a cloud center node, the synchronization receiver is an edge node; when the synchronization participant is an edge node, the synchronization receiver is a cloud center node or an edge node.

[0015] Specifically, the synchronization participants monitor the local synchronization folder in real time. When a file change is detected, a synchronization request is initiated (the synchronization participants are also called synchronization initiators). The cloud center node processes the synchronization request, uses the token bucket algorithm to limit the rate, coordinates the synchronization process, and confirms whether the synchronization conditions are met.

[0016] The token bucket algorithm is used on cloud nodes to control the synchronous request processing rate. The token bucket size and token generation rate are determined based on the scenario and the carrying capacity of each node in the system.

[0017] The token bucket algorithm can be implemented in C. A token bucket description structure is defined, containing fields such as cps, burst, pos, and token, representing the token generation rate, the maximum number of tokens in the token bucket, the token's index in the token bucket array, and the maximum number of tokens, respectively. An array is then used to represent the token bucket. During system runtime, the above parameters are set according to the system's capacity, and tokens are added to the token bucket at a certain rate. A certain number of tokens are consumed when processing each synchronization request. When the number of tokens in the token bucket is insufficient, the synchronization request is paused; otherwise, the two synchronizing nodes coordinate to establish a data communication link.

[0018] Meanwhile, the cloud center node is responsible for establishing secure channels for direct connections between edge nodes and monitoring the usage of resources such as CPU, memory, and disk on each edge node.

[0019] When the synchronization conditions are met, the cloud center establishes a secure communication connection between the synchronization participants and the synchronization receiver.

[0020] The cloud center negotiates and establishes a communication connection between two nodes using the EdgeMesh component. This component is the data plane component of the KubeEdge cluster, providing applications with simple service discovery and traffic proxying capabilities. When applications on two edge nodes located in different subnets need to communicate, the EdgeMesh-agent resolves the domain name and then attempts to establish a P2P connection through the EdgeMesh-server. The domain name here can be edgen-sync.default.svc.cluster.local, where 'n' represents the nth node in the cloud-edge system.

[0021] After a secure communication connection is established between the synchronization participants and the synchronization receiver, the synchronization participants use an adaptive synchronization method to select either a full synchronization mode or an incremental synchronization mode to synchronize files, i.e., they use methods such as... Figure 2 The adaptive synchronization method shown selects a synchronization mode for storage synchronization, including: Assess the current network bandwidth, calculate a file size threshold using a threshold function, and then compare the size of the file to be synchronized with this threshold. If the file size is greater than the threshold, incremental synchronization is used; otherwise, full synchronization is used. The formula for calculating the threshold function is as follows: ,in This is the current network bandwidth. It is a file size threshold, a preset coefficient. and It can be derived through experimental simulation or manually set based on user experience.

[0022] The incremental synchronization can be achieved using existing FSC-based incremental synchronization or CDC-based incremental synchronization.

[0023] To save computing resources and reduce the time consumed by block partitioning, preferably, the incremental synchronization is the improved CDC incremental synchronization based on guessing jump proposed in the embodiments of the present invention, including: Pre-set jump and truncation conditions. Calculate the rolling hash of the data within the block window during the sliding process. If the rolling hash meets the jump condition, further determine if it meets the truncation condition. If not, the block window jumps forward by a certain number of bytes. If it does, the end of the block window is used as the block cutting point. If the rolling hash does not meet the jump condition, the block window slides forward by 1 byte. The sliding window jump byte count calculation rule is as follows: three constants are preset: minimum block size, maximum block size, and average block size. During the block division process, if the remaining file length is less than the minimum block size, the split point is returned directly without calculating the jump byte count. If the remaining file length is less than the average block size, the remaining file length is used as normalSize to calculate the jump byte count. In other cases, the average block size is used as normalSize to calculate the jump byte count. The jump byte count is calculated as follows: take the logarithm of normalSize to the base 2 and subtract 1 as a, and b as a-1. Then, shift 1 left by b bits and multiply it with a to get the numerator. Then, shift 1 left by a bits and subtract it from b bits to get the denominator. Finally, divide the numerator by the denominator to get the jump byte count.

[0024] Specifically, the improved incremental synchronization based on CDC proposed in this embodiment of the invention uses content-defined segmentation technology to solve the boundary offset problem of fixed segmentation technology, and sets jump conditions and truncation conditions in the segmentation. During the sliding of the segmentation window, the rolling hash of the data in the window is calculated. When the rolling hash meets the jump condition, the window slides forward a certain number of bytes. When the rolling hash meets the truncation condition, the tail of the segmentation window is taken as the segmentation point. The mask corresponding to the jump condition is less than the mask corresponding to the truncation condition and the high bits are the same.

[0025] like Figure 3 As shown, the jump condition and truncation condition are set to two masks, maskJ and maskC, respectively. For example, they can be set to 0x590003560000 and 0x590003570000, respectively. The jump condition is smaller than the truncation condition and the high bits are the same. This way, if the current window's scrolling hash does not meet the jump condition, it is even less likely to meet the truncation condition. This reduces the number of judgments during block division, thereby saving computing resources and reducing the time spent on block division.

[0026] For the blocks in incremental synchronization, a structure can be defined in C language to represent them, which mainly includes the following key fields: (1) id field: used to uniquely identify the number of each block; (2) offset field: used to indicate the offset of the block in the entire file, which can be used for file integration later; (3) length field: used to indicate the length of a block. The position of the block in the file can be accurately determined by the offset field and the length field. (4) Weakhash field: A weak hash field with a length of 8 bytes, which can be represented by uint64_t in C language; (5) stronghash field: The strong hash field has a length of 16 bytes and can be represented by a character array in C language.

[0027] Incremental synchronization requires both parties to find the block to be synchronized by matching strong and weak hash tables. To further reduce computational overhead, the improved CDC incremental synchronization method based on guessing and jumping proposed in this embodiment of the invention accumulates the rolling hash calculated after each window slide when calculating the rolling hash, and uses it as the weak hash value of the subsequent block. The strong hash value is calculated using an existing algorithm (such as the MD5 algorithm).

[0028] Preferably, the rolling hash calculated after each slide of the block window is accumulated and used as the weak hash value of the next block.

[0029] Preferably, the data synchronization process between the synchronization participants and the synchronization receiving method uses an encryption algorithm (such as the RSA algorithm) to establish a session key, and uses an encryption algorithm (such as the AES algorithm) on an encrypted channel to encrypt the transmitted data, thereby ensuring security.

[0030] Specifically, each node stores its public key certificate in the cloud center. The synchronization initiator generates a random number and encrypts it using its private key. The synchronization receiver decrypts the number using the other party's public key certificate stored in the cloud center, establishes a session key, and then communicates data symmetrically using the AES algorithm.

[0031] The public key certificate can be obtained from a Certificate Authority (CA), and Let's Encrypt can be used to provide the certificate. This certificate can then be used to negotiate the session key and for authentication. The session key is then negotiated and established. The initiator generates a 16-bit random number, encrypts it using their private key, and sends it to the receiver. The receiver then decrypts the session key using the initiator's public key and encrypts the transmitted data using AES in CBC mode. This process can be implemented using the OpenSSL library in C.

[0032] Preferably, when the data synchronization method is full synchronization, the synchronization participants send the corresponding synchronization flag and the file to be synchronized to the synchronization receiver; after receiving the corresponding synchronization flag, the synchronization receiver replaces the original file data with the file to be synchronized. When the data synchronization method is incremental synchronization, the synchronization participants send the corresponding synchronization flag and file description information of the file to be synchronized to the synchronization receiver. After receiving the corresponding synchronization flag, the synchronization receiver divides the local file into blocks using incremental synchronization and calculates the strong and weak hashes of each block to obtain a block strong and weak hash list, which is then sent to the synchronization participants. The synchronization participants match this list with the block strong and weak hash list of the local file and send any mismatched data blocks to the synchronization receiver. The synchronization receiver then modifies the mismatched data blocks to their corresponding local files. The synchronization flag for full synchronization is different from the synchronization flag for incremental synchronization.

[0033] Specifically, taking the synchronization flag bits corresponding to full synchronization and incremental synchronization as 1 and 0 respectively as examples, when the synchronization initiator determines to use full synchronization based on the network status and threshold function, it sends the synchronization flag bit (synchronization flag bit is 0) and all file data to the synchronization receiver. When the synchronization initiator determines to use incremental synchronization based on the network status and threshold function, it sends the synchronization flag (synchronization flag is 1) and file description information to the synchronization receiver, waits for the synchronization receiver to return the block strength and weakness hash list of the corresponding file calculated using the improved incremental synchronization method, and then matches it with the block strength and weakness hash list of the local file, and sends the mismatched data blocks to the synchronization receiver. The receiver checks the synchronization flag. When the flag is 0, it replaces the original file data. When the flag is 1, it uses an improved incremental synchronization method to divide the local file into blocks, calculates the hash, sends the block strong and weak hash table to the synchronization initiator, and then accepts the modified blocks and modifies them to the local file.

[0034] This invention provides a cloud-edge converged secure storage synchronization system, including a cloud module and an edge module. The architecture is based on KubeEdge and can solve the problems of limited resources, strong heterogeneity and poor reliability of edge devices. It is easy to deploy, has low overhead and ensures security.

[0035] The cloud module is deployed on a cloud center node and includes: The synchronization request processing module is used to receive synchronization requests initiated by synchronization participants and use the token bucket algorithm to determine whether the synchronization conditions are met.

[0036] Specifically, the synchronization request processing module receives and processes synchronization requests from edge nodes, determines the nodes to participate in synchronization based on the rules set during service deployment, and runs synchronization Pods on the corresponding nodes. At the same time, this module also limits the rate of synchronization requests to ensure the stable operation of the entire system.

[0037] The resource monitoring module is used to pause synchronization requests when synchronization conditions are not met.

[0038] Specifically, the resource monitoring module monitors the resource usage of edge nodes and the resource usage of synchronization Pods, and reduces the resource usage of the synchronization program or suspends synchronization requests when resources are scarce.

[0039] The cloud-edge communication coordination module is used to establish a secure communication connection between the synchronization participants and the synchronization receiver when the synchronization conditions are met. Specifically, the cloud-edge communication coordination module needs to store and synchronize edge nodes that may not be on the same local area network and may not have public IP addresses. This module coordinates the establishment of direct and secure communication tunnels between edge nodes. This module can be implemented by EdgeMesh through domain name resolution and includes two components: EdgeMesh-agent and EdgeMesh-server.

[0040] The sync Pod is responsible for data synchronization, including: The change monitoring module is used to monitor the local synchronized folder in real time. When a file change is detected, a synchronization request is initiated.

[0041] Specifically, the change monitoring module periodically monitors local data changes and detects modified files in the current node that need to be synchronized. This module can be implemented using the fanotify library provided by the Linux kernel, which can monitor creation, deletion, and modification operations in the synchronization directory at the system level.

[0042] The network assessment module is used to obtain the current network bandwidth. .

[0043] Specifically, the network evaluation module assesses the network status of the current node, mainly bandwidth information, to determine the method of subsequent synchronization. It can read network interface traffic data from the / proc / net / dev file and then calculate the bandwidth value by measuring the traffic change per unit time.

[0044] The file evaluation module is used to obtain the size of the file to be synchronized in order to select the subsequent synchronization method.

[0045] The synchronization module is used to synchronize the current network bandwidth. Substitute into the threshold function to calculate the file size threshold To confirm the method of data synchronization with the synchronization receiver; wherein, when the size of the file to be synchronized is greater than When the data is synchronized, it is incremental synchronization; otherwise, it is full synchronization. , and All are preset coefficients.

[0046] Specifically, the file evaluation module synchronizes data with the target edge node, which is divided into incremental synchronization and full synchronization. The synchronization method is selected based on network status and file size to reduce computational overhead and latency. In the cloud, data synchronization between the cloud and the edge is achieved by deploying a synchronization Pod.

[0047] The edge module is deployed on the edge node, including the synchronization Pod.

[0048] At the edge, data synchronization is achieved by deploying synchronization Pods.

[0049] Furthermore, sync Pods also include: Network communication module: When performing storage synchronization across network segments, it is necessary to establish a direct communication tunnel with the cloud. When performing synchronization within the same local area network, it transmits data directly. Security module: It is divided into several sub-modules, including an identity authentication module, a DOS prevention module, and a man-in-the-middle (MITM) prevention module. In addition, in order to ensure the confidentiality of data transmission, the transmitted data needs to be encrypted. A session key is established through the RSA algorithm, and then the transmitted data is encrypted using the AES algorithm to ensure security.

[0050] In addition, the system includes a CA module to provide basic security for the system. Its main function is to provide public key certificates to each node, which can be used to establish session keys during subsequent synchronization and to mitigate man-in-the-middle attacks. Each node uses OpenSSL to generate a public and private key. The private key is stored locally, and then the public key and node identity information are packaged and sent to the CA. The CA verifies the validity of the certificate, and if successful, sends a signed public key certificate back to the node. Each node then backs up its public key certificate in the cloud.

[0051] The secure storage synchronization method and system for cloud-edge convergence provided in this invention can realize data synchronization between cloud and edge, and between edge and edge.

[0052] The synchronization process between the cloud and the edge is as follows, taking the synchronization between the cloud node and the edge node edge1 as an example, including: (1) After edge1 detects changes to local files through the local change monitoring module, it sends a synchronization request to the cloud synchronization request processing module. (2) The cloud-based synchronization request processing module determines whether to allow the synchronization request based on whether there are enough tokens in the current token bucket. If there are enough, step (4) is executed; otherwise, step (3) is executed. (3) edge1 pauses for a period of time and then re-initiates the synchronization request, returning to step (2); (4) The cloud node and edge node edge1 establish a session key through key negotiation to ensure the security of data during communication; (5) Node edge1 determines the synchronization method to use by assessing the current network bandwidth and file resource size; (6) Data synchronization between cloud nodes and edge nodes edge1.

[0053] The synchronization process between edge nodes is as follows, taking the synchronization between node edge1 and node edgen as an example, including: (1) After edge1 detects changes to local files through the local change monitoring module, it sends a synchronization request to the cloud synchronization request processing module. (2) The cloud synchronization request processing module determines whether to allow the synchronization request based on whether the number of tokens in the current token bucket is sufficient. If it is sufficient, step (4) is executed; otherwise, step (3) is executed. (3) edge1 pauses for a period of time and then re-initiates the synchronization request, returning to step (2); (4) The cloud-based synchronization request processing module determines the synchronization recipient edgen and its domain name; the cloud-edge communication coordination module in the cloud assists in establishing a direct communication channel through the EdgeMesh component. (5) Node edge1 and edgen establish a session key through key negotiation to ensure data security during communication; (6) Node edge1 determines the synchronization method to use by assessing the current network bandwidth and file resource size; (7) Synchronize data between edge1 and edgen.

[0054] This invention provides a computer-readable storage medium storing computer instructions that cause a processor to perform the method described in any of the above embodiments.

[0055] This invention provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the method described in any of the above embodiments.

[0056] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cloud-edge fusion secure storage synchronization method, characterized in that, Comprise: S1, the synchronization participant monitors the local synchronization folder in real time, and initiates a synchronization request when a file change is monitored; After the cloud center node receives the synchronization request, it uses the token bucket algorithm to determine whether the synchronization condition is met. If yes, it establishes a secure communication connection between the synchronization participant and the synchronization receiver, and enters S2; otherwise, the synchronization request is suspended; the synchronization participant waits for a preset time and reinitiates the synchronization request; S2, the synchronization participant will send the current network bandwidth to the synchronization receiver Substitute the threshold function to calculate the file size threshold In order to confirm the data synchronization with the synchronization receiver; wherein, when the size of the file to be synchronized is greater than The data synchronization mode is incremental synchronization, otherwise it is full synchronization; wherein, , And are preset coefficients; Wherein, the synchronization participant is a cloud center node or an edge node; when the synchronization participant is a cloud center node, the synchronization receiver is an edge node; when the synchronization participant is an edge node, the synchronization receiver is a cloud center node or an edge node.

2. The method of claim 1, wherein, The incremental synchronization is an improved CDC-based incremental synchronization, comprising: The jump condition and the truncation condition are set in advance, the rolling hash of the data in the block window is calculated during the sliding process of the block window, if the rolling hash meets the jump condition, it is further determined whether it meets the truncation condition, if not, the block window slides forward by a certain number of bytes, if yes, the tail end of the block window is taken as the block cutting point; if the rolling hash does not meet the jump condition, the block window slides forward by 1 byte; Wherein, the calculation method of the sliding window jump byte number is: if the remaining file length is less than the minimum block size, the block cutting point is directly returned without calculating the jump byte number; if the remaining file length is less than the average block size, the remaining file length is taken as normalSize to calculate the jump byte number; otherwise, the average block size is taken as normalSize to calculate the jump byte number; the jump byte number is to take the logarithm of normalSize with 2 as the base-1 as a, take a-1 as b, then left shift 1 by b bits and multiply by a as the numerator, then subtract 1 left shifted by a and b bits as the denominator, and finally divide the numerator by the denominator to get; Wherein, the mask corresponding to the jump condition is less than the mask corresponding to the truncation condition, and the high bits of the two are the same.

3. The method of claim 2, wherein, The rolling hash calculated after each sliding of the cumulative block window is taken as the weak hash value of the next block.

4. The method according to any one of claims 1 to 3, characterized in that, The data synchronization process of the synchronization participant and the synchronization receiver method uses a public key encryption algorithm to establish a session key, and uses a symmetric encryption algorithm to encrypt and transmit data on an encrypted channel.

5. The method according to any one of claims 1 to 3, wherein When the data synchronization mode is full synchronization, the synchronization participant sends the corresponding synchronization flag bit and the file to be synchronized to the synchronization receiver; after the synchronization receiver receives the corresponding synchronization flag bit, it replaces the original file data with the file to be synchronized; When the data synchronization mode is incremental synchronization, the synchronization participant sends the corresponding synchronization flag bit and the file description information of the file to be synchronized to the synchronization receiver; after the synchronization receiver receives the corresponding synchronization flag bit, it uses the incremental synchronization method to block and calculate the strong and weak hash of each block to get the block strong and weak hash list, and sends it to the synchronization participant; the synchronization participant matches it with the block strong and weak hash list of the local file, and sends the unmatched data block to the synchronization receiver; the synchronization receiver modifies the corresponding local file according to the unmatched data block. Wherein, the synchronization flag corresponding to the full synchronization is different from the synchronization flag corresponding to the incremental synchronization.

6. A cloud-edge converged secure storage synchronization system, comprising: Comprise: The cloud module and the edge module; The cloud module is deployed in a cloud center node, comprising: A synchronization request processing module, configured to receive a synchronization request initiated by a synchronization participant, and determine whether the synchronization condition is met by using a token bucket algorithm; A resource monitoring module, configured to suspend the synchronization request when the synchronization condition is not met; A cloud-edge communication coordination module, configured to establish a secure communication connection between the synchronization participant and a synchronization receiver when the synchronization condition is met; The synchronization pod comprises a change monitoring module, a network evaluation module, a file evaluation module and a synchronization module. The change monitoring module is used for monitoring a local synchronization folder in real time, and initiating a synchronization request when a file change is monitored. The network evaluation module is used for obtaining a current network bandwidth The file evaluation module is used for obtaining a size of a file to be synchronized. The synchronization module is used for substituting the file size threshold value into a threshold function to calculate a file size threshold value to confirm a data synchronization manner with the synchronization receiver. When the size of the file to be synchronized is greater than the file size threshold value, the data synchronization manner is incremental synchronization, otherwise, the data synchronization manner is full synchronization. and are preset coefficients. The edge module is deployed in an edge node, comprising the synchronization Pod.

7. A computer readable storage medium characterized by The computer readable storage medium stores computer instructions, and the computer instructions are used to make the processor execute the method in any one of claims 1-5.

8. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are executed by the processor to realize the method in any one of claims 1-5.

Images