Cross-cloud increment synchronization method and device, equipment, medium and program product

Abstract

The invention provides a cross-cloud increment synchronization method which can be applied to the field of cloud computing or big data. The method comprises the following steps: in response to an increment synchronization request of a source public cloud accessing a log analysis platform, collecting an access log on the source public cloud in real time; obtaining an access log newly generated by the source public cloud according to a preset time window through an incremental synchronization service to generate an ith source list file; checking the consistency of the ith source list file and each target list file stored in the target public cloud through a difference comparison algorithm; and in response to inconsistency between the ith source list file and each target list file, synchronizing the ith source list file to the target public cloud. The invention further provides a cross-cloud increment synchronization device and equipment, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of cloud computing or big data, and more specifically to a method, device, equipment, medium and program product for synchronizing increments across clouds. Background Art

[0002] In a cloud computing environment, in order to improve data security and reliability, some important data needs to be stored in the storage of multiple cloud service providers. However, when it comes to data synchronization between different clouds, challenges arise. Traditional data synchronization methods often only support full synchronization, that is, each synchronization requires the data on the source end to be completely copied to the target end, regardless of whether the data has changed. This method may be feasible in scenarios with small amounts of data or infrequent changes, but in scenarios with large-scale and high-frequency data changes, its efficiency and cost issues become prominent. Summary of the invention

[0003] In view of the above problems, embodiments of the present disclosure provide a method, apparatus, device, medium and program product for synchronizing increments across clouds.

[0004] According to a first aspect of the present disclosure, a cross-cloud incremental synchronization method is provided, the method comprising: in response to an incremental synchronization request of a source public cloud accessing a log analysis platform, collecting access logs on the source public cloud in real time; obtaining, through an incremental synchronization service, the access logs newly generated by the source public cloud according to a preset time window to generate an i-th source manifest file; checking the consistency of the i-th source manifest file with each target manifest file stored in the target public cloud through a difference comparison algorithm; and in response to inconsistency between the i-th source manifest file and each target manifest file, synchronizing the i-th source manifest file to the target public cloud.

[0005] According to an embodiment of the present disclosure, the method also includes: determining the synchronization priority of the access log on the source public cloud based on the basic attributes of the access log; and in response to multiple source public clouds initiating the incremental synchronization request at the same time, comparing the synchronization priorities so that the access log with a high synchronization priority is synchronized first.

[0006] According to an embodiment of the present disclosure, the method further includes: verifying the i-th source manifest file to determine the consistency between the i-th source manifest file and the i-1-th source manifest file in two adjacent time windows; and in response to the i-th source manifest file being consistent with the i-1-th source manifest file, skipping the synchronization operation on the i-th source manifest file.

[0007] According to an embodiment of the present disclosure, obtaining the newly generated access log of the source public cloud according to a preset time window through the incremental synchronization service to generate the i-th source inventory file includes: in response to the log analysis platform receiving a query request of the incremental synchronization service, querying the time stamp of the creation or change of the access log through the log analysis platform; in response to the time stamp of the creation or change being within the preset time window and the access log not being synchronized, determining that the access log is the newly generated access log; and packaging the newly generated access log to form the i-th source inventory file.

[0008] According to an embodiment of the present disclosure, checking the consistency of the i-th source manifest file with each target manifest file stored in the target public cloud through a difference comparison algorithm includes: adding a uniqueness tag to the i-th source manifest file and each target manifest file, and comparing whether the uniqueness tag of the i-th source manifest file and the uniqueness tag of the target manifest file are the same; and in response to the uniqueness tag of the i-th source manifest file and the uniqueness tag of each target manifest file being different, determining that the i-th source manifest file and the target manifest file are inconsistent.

[0009] According to an embodiment of the present disclosure, checking the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud through a difference comparison algorithm also includes: obtaining the size of the file block in the i-th source manifest file and the size of the file block in each of the target manifest files; comparing the size of the file block in the i-th source manifest file with the size of the file block in each of the target manifest files; and in response to the size of the file block in the i-th source manifest file being inconsistent with the size of the file block in each of the target manifest files, determining that the i-th source manifest file is inconsistent with the target manifest file.

[0010] According to an embodiment of the present disclosure, the method also includes: creating a central server, registering and configuring the multiple source public clouds and the target public clouds on the central server; and in response to creating the incremental synchronization request on the central server, managing and monitoring the synchronization operations of each of the source public clouds and each of the target public clouds through the central server; and adjusting the synchronization operations of each of the source public clouds and each of the target public clouds based on the management and monitoring results of the synchronization operations.

[0011] A second aspect of the present disclosure provides a cross-cloud incremental synchronization device, including: a log collection module, which is used to respond to an incremental synchronization request of a source public cloud that accesses a log analysis platform, and collect access logs on the source public cloud in real time; a file generation module, which is used to obtain the newly generated access logs of the source public cloud according to a preset time window through an incremental synchronization service to generate an i-th source manifest file; a file checking module, which is used to check the consistency of the i-th source manifest file with each target manifest file stored in the target public cloud through a difference comparison algorithm; and a file synchronization module, which is used to synchronize the i-th source manifest file to the target public cloud in response to the i-th source manifest file being inconsistent with each of the target manifest files.

[0012] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above-mentioned cross-cloud incremental synchronization method.

[0013] The fourth aspect of the present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned cross-cloud incremental synchronization method when executed by a processor.

[0014] The fifth aspect of the present disclosure also provides a computer program product, including a computer program, which implements the steps of the above-mentioned cross-cloud incremental synchronization method when executed by a processor.

[0015] In the embodiments of the present disclosure, through the incremental synchronization service, the capture of incremental data is achieved while ensuring data security, avoiding the overhead of full synchronization, and reducing the bandwidth and storage costs of data transmission; at the same time, a difference comparison algorithm is used to check the consistency of files to quickly determine which data is new and which data has been synchronized, avoiding unnecessary data transmission and improving the intelligence and efficiency of cross-cloud incremental synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0017] Figure 1 The application scenario diagram of the method, apparatus, device, medium and program product for synchronizing increments across clouds according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 2 A flowchart of a method for synchronizing increments across clouds according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 3A flowchart of a method for checking consistency of a file block according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 4 A structural block diagram of a device for synchronizing increments across clouds according to an embodiment of the present disclosure is schematically shown; and

[0021] Figure 5 A block diagram of an electronic device for a method for synchronizing increments across clouds according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0022] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0023] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0024] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0025] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0026] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] Figure 1 The application scenario diagram of the method, apparatus, device, medium and program product for synchronizing increments across clouds according to an embodiment of the present disclosure is schematically shown.

[0028] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0029] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0030] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0031] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0032] It should be noted that the cross-cloud incremental synchronization method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the cross-cloud incremental synchronization device provided in the embodiment of the present disclosure can generally be set in the server 105. The cross-cloud incremental synchronization method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the cross-cloud incremental synchronization device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0033] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0034] First, the technical terms described in this article are explained and described as follows.

[0035] Log analysis platform: used to analyze access logs stored on the source public cloud and obtain recently generated file information.

[0036] Incremental synchronization service: responsible for synchronizing newly generated files to the target public cloud storage according to the manifest file to avoid repeated synchronization.

[0037] Intelligent incremental synchronization strategy: Dynamically adjust the priority of incremental synchronization based on factors such as file access frequency, size, and importance.

[0038] Difference comparison technology: Use efficient algorithms to synchronize only the changed parts and reduce the amount of data transmission.

[0039] Multi-cloud collaborative synchronization: Considering the collaborative synchronization between multiple cloud service providers, cross-cloud incremental synchronization is more flexible and scalable.

[0040] Central coordinator: responsible for coordinating incremental synchronization tasks between different clouds, managing task status and error handling.

[0041] The following will be based on Figure 1 The scene described by Figure 2 and Figure 3 The cross-cloud incremental synchronization method of the embodiment of the present disclosure is described in detail.

[0042] Figure 2 The flowchart of the method for synchronizing increments across clouds according to an embodiment of the present disclosure is schematically shown.

[0043] like Figure 2As shown, the cross-cloud incremental synchronization method of this embodiment includes operations S210 to S240.

[0044] In operation S210, in response to an incremental synchronization request of a source public cloud accessing a log analysis platform, access logs on the source public cloud are collected in real time.

[0045] In an embodiment of the present disclosure, before collecting the access logs on the source public cloud, the user's consent or authorization is obtained. For example, before operation S210, a request to collect the access logs on the source public cloud is issued to the user. When the user agrees or authorizes that the access logs on the source public cloud can be collected, the operation S210 is performed.

[0046] In operation S220, the access log newly generated by the source public cloud is obtained according to a preset time window through an incremental synchronization service to generate an i-th source manifest file.

[0047] In operation S230 , the consistency of the i-th source manifest file and each target manifest file stored in the target public cloud is checked by using a difference comparison algorithm.

[0048] In operation S240 , in response to the i-th source manifest file being inconsistent with each of the target manifest files, the i-th source manifest file is synchronized to the target public cloud.

[0049] In an embodiment of the present disclosure, a corresponding operation entry is provided for the user to choose to agree or reject the automated decision result. That is, before synchronizing the i-th source inventory file to the target public cloud for processing / decision, an instruction of the user to agree or reject the processing / decision input through the corresponding operation entry is obtained. If the user agrees to perform the processing / decision, the i-th source inventory file is synchronized to the target public cloud for processing / decision, that is, step S240 is executed. If the user refuses to perform the processing / decision, the expert decision process is entered.

[0050] In some exemplary embodiments, taking synchronization between two public clouds, i.e., source public cloud A and target public cloud B, as an example, the incremental files newly generated in cloud A are synchronized to cloud B every preset time period. In this process, the access logs stored in cloud A are first collected in real time through the log collection system and transmitted to the log analysis platform. The log analysis platform analyzes the access logs stored in cloud A and uses the pre-written incremental synchronization service to obtain the file information most recently generated within the preset time window, i.e., the i-th source manifest file. Then, the difference comparison algorithm is used to verify the i source manifest files, and finally the incremental synchronization of files between different clouds is realized.

[0051] It can be understood that through the method provided by the embodiment of the present disclosure, incremental synchronization of files between different clouds is achieved, the bandwidth and storage costs of data transmission are reduced, while the security and consistency of data are guaranteed, ensuring that important data is stored in multiple clouds, and the intelligence and efficiency of cross-cloud incremental synchronization are improved through the difference comparison algorithm.

[0052] In an embodiment of the present disclosure, the method also includes: determining the synchronization priority of the access log on the source public cloud based on the basic attributes of the access log; and in response to multiple source public clouds initiating the incremental synchronization request at the same time, comparing the synchronization priorities so that the access log with a high synchronization priority is synchronized first.

[0053] Specifically, the basic attributes of the access log include, but are not limited to, the time the log was generated, the file size, the type of service or application to which it belongs, and the access frequency. By utilizing these attributes, the synchronization priority of the log is preliminarily evaluated. For example, a log that is recently generated, large in size, comes from a critical service, and is frequently accessed will have its synchronization priority set to a higher level. Furthermore, in order to more accurately determine the synchronization priority, intelligent algorithms such as machine learning models or predictive analysis algorithms can be introduced. Intelligent algorithms predict the future access pattern of files based on historical data (such as access log records over a period of time in the past), and thus determine the synchronization priority based on the prediction results. After the synchronization priority is determined, a specific intelligent incremental synchronization strategy is formulated to dynamically adjust the priority of incremental synchronization.

[0054] Exemplarily, by analyzing the access frequency of the log, it is possible to identify which files are frequently accessed. For these files, a higher synchronization priority can be set due to their higher importance. In addition, since large files may contain more information, their synchronization cost is also higher. Therefore, the algorithm determines a reasonable synchronization priority by weighing the relationship between file size and importance. At the same time, logs generated by different services or applications may have different importance. For example, security-related logs may require a higher synchronization priority.

[0055] It can be understood that by setting the synchronization priority and giving priority to access logs with high synchronization priority, it can ensure that key data is synchronized in a timely manner, thereby improving the overall synchronization efficiency. At the same time, through the timely synchronization of key data, the system can reduce the risk of data inconsistency and loss.

[0056] In an embodiment of the present disclosure, the method further includes: verifying the i-th source manifest file to determine the consistency between the i-th source manifest file and the i-1-th source manifest file in two adjacent time windows; and in response to the i-th source manifest file being consistent with the i-1-th source manifest file, skipping the synchronization operation on the i-th source manifest file.

[0057] Specifically, the incremental synchronization service generates source manifest files periodically or according to preset trigger conditions. These manifest files record a list of access log files generated or updated on the source public cloud within a specific time window (such as 10 minutes). Since the manifest file is based on the log analysis platform querying the files generated in the recent period, there may be duplicate file records between two adjacent manifest files generated. Therefore, in order to avoid unnecessary repeated synchronization operations, a verification mechanism for source manifest files is introduced in the embodiment of the present disclosure. Specifically, when the i-th source manifest file is generated, the synchronization service will verify it to determine its consistency with the i-1th source manifest file generated in the previous time window. If the i-th source manifest file is completely consistent with the i-1th source manifest file in the file list, then the two manifest files can be considered to be duplicates. In this case, the synchronization service will skip the synchronization operation on the i-th source manifest file.

[0058] It should be noted that determining whether two adjacent source manifest files are consistent can be achieved by comparing the file lists recorded in the two manifest files.

[0059] It is understandable that by determining whether two adjacent source manifest files are consistent, unnecessary repeated synchronization operations can be reduced, which can not only save network bandwidth and storage resources, but also improve the performance and response speed of the synchronization service. At the same time, it also helps to improve the consistency of data synchronization and reduce data inconsistency problems caused by synchronization errors or omissions.

[0060] Furthermore, obtaining the newly generated access log of the source public cloud according to a preset time window through the incremental synchronization service to generate the i-th source inventory file includes: in response to the log analysis platform receiving a query request of the incremental synchronization service, querying the time stamp of the creation or change of the access log through the log analysis platform; in response to the time stamp of the creation or change being within the preset time window and the access log not being synchronized, determining that the access log is a newly generated access log; and packaging the newly generated access log to form the i-th source inventory file.

[0061] Specifically, the pre-written incremental synchronization service is first connected to the log analysis platform, which collects access logs stored on cloud A. Then, the incremental synchronization service will automatically trigger at a preset time interval (e.g., every 5 minutes) to send a query request to the log analysis platform to obtain the newly generated access log information in the preset time window. In this process, the log analysis platform will retrieve the files that have changed in the past preset time window, and in order to determine whether the file is newly created or changed, the platform will compare the time identifier of the file, such as the timestamp. If the time of the change or creation of the file is within the preset time window and has not appeared in the previous synchronization task, it is regarded as a newly generated access log. Then, based on the query results, the incremental synchronization service will package these log information into a manifest file, namely the i-th source manifest file, which contains information such as the file name, path, and size. At the same time, before generating the manifest file, the service will perform deduplication by comparing the unique identifier of the file (such as the hash value or the file path) to ensure that the manifest does not contain any duplicate files.

[0062] It should be noted that in the embodiment of the present disclosure, a 10-minute time window is selected to ensure that most of the newly generated logs can be captured. At the same time, taking into account the possible processing delays of the log analysis platform, if only logs within 5 minutes are queried, some newly generated logs may be missed due to delays.

[0063] It is understandable that compared with full synchronization, the incremental synchronization service is more efficient and reduces the consumption of network bandwidth and storage resources. At the same time, through regular query and incremental synchronization, the system can dynamically adjust the use of resources according to the actual data changes. When the data changes less, the system can reduce the frequency and scale of synchronization, thereby saving network bandwidth, computing resources and storage space. In the case of more data changes, the system can increase the frequency and scale of synchronization accordingly to ensure the timeliness and integrity of the data.

[0064] Furthermore, checking the consistency of the i-th source manifest file with each target manifest file stored in the target public cloud through a difference comparison algorithm includes: adding a uniqueness tag to the i-th source manifest file and each target manifest file, and comparing whether the uniqueness tag of the i-th source manifest file is the same as the uniqueness tag of the target manifest file; and in response to the uniqueness tag of the i-th source manifest file being different from the uniqueness tag of each target manifest file, determining that the i-th source manifest file is inconsistent with the target manifest file.

[0065] Specifically, when the incremental synchronization service wants to synchronize the newly generated i-th source manifest file to B cloud storage, the difference comparison algorithm will compare the consistency of the source manifest file and each target manifest file. If the source manifest file already exists in the target public cloud or the content is consistent, the incremental synchronization service will skip the file and no longer transfer it. The difference comparison technology determines the changes between files by using a difference comparison algorithm and synchronizes only these changes instead of the entire file.

[0066] It should be noted that before the difference comparison algorithm is used for comparison, a unique tag is added to each manifest file (including the i-th source manifest file and each target manifest file stored in the target public cloud). This tag is usually automatically generated based on the file content or file attributes, such as the file name, hash value or version number, etc., and can also be added manually.

[0067] In the embodiments of the present disclosure, the difference comparison technology may include the following methods:

[0068] Exemplarily, file name comparison: compare the file names of the source manifest file and each target manifest file. If there is already a target manifest file with the same name in the target cloud, it means that the file has been synchronized and does not need to be synchronized again.

[0069] For example, hash value comparison: The hash value is a unique identifier of the file content. In incremental synchronization, the system calculates the hash value of the file, for example, using the SHA-256 algorithm. When files need to be synchronized, the system compares the hash values ​​of the source manifest file and each target manifest file. If the hash values ​​are the same, it means that the file content has not changed and synchronization is not required; if the hash values ​​are different, it means that the file has changed and synchronization is required.

[0070] For example, version number comparison: Each file can be associated with a version number. When a file changes, the version number will increase. In incremental synchronization, the system will compare the version numbers of the source manifest file and each target manifest file. If the version numbers are the same, it means that the file content has not changed and no synchronization is required; if the version numbers are different, it means that the file has changed and synchronization is required.

[0071] It can be understood that by using a more efficient difference comparison algorithm, the efficiency of synchronization is improved, unnecessary data transmission during incremental synchronization is reduced, the accuracy of synchronization is ensured, and data duplication or omission is avoided.

[0072] Figure 3 The flowchart of the method for checking the consistency of file blocks according to the embodiment of the present disclosure is schematically shown.

[0073] Furthermore, if Figure 3As shown, checking the consistency of the i-th source manifest file with each target manifest file stored in the target public cloud through a difference comparison algorithm also includes: obtaining the size of the file block in the i-th source manifest file and the size of the file block in each of the target manifest files; comparing the size of the file block in the i-th source manifest file with the size of the file block in each of the target manifest files; and in response to the size of the file block in the i-th source manifest file being inconsistent with the size of the file block in each of the target manifest files, determining that the i-th source manifest file is inconsistent with the target manifest file.

[0074] Specifically, when using a difference comparison algorithm to check the consistency between the source manifest file and the target manifest file stored in the target public cloud, for some large files, the file can be split into multiple fixed-size or variable-length blocks, and then only the changed blocks are compared instead of the entire file. In this process, the size of each file block in the i-th source manifest file and the size of the corresponding file block in the target manifest file are first obtained. The size of these file blocks can be fixed (for example, 1MB per block) or variable-length (dynamically divided according to the file content or structure). Then, the system compares the size of the file block in the i-th source manifest file with the size of the corresponding file block in the target manifest file one by one. If the sizes of the two blocks are the same, then the contents of the two blocks are further compared to see if they are the same. If the block sizes are different, then it is directly determined that the two blocks are inconsistent and need to be synchronized. In addition, if only some blocks have changed, the system can prioritize synchronizing these changed blocks instead of the entire file.

[0075] It can be understood that the introduction of file block level comparison can significantly reduce the amount of data transmission and further improve the efficiency and accuracy of synchronization.

[0076] In an embodiment of the present disclosure, the method also includes: creating a central server, registering and configuring the multiple source public clouds and the target public clouds on the central server; and in response to creating the incremental synchronization request on the central server, managing and monitoring the synchronization operations of each of the source public clouds and each of the target public clouds through the central server; and adjusting the synchronization operations of each of the source public clouds and each of the target public clouds based on the management and monitoring results of the synchronization operations.

[0077] In order to effectively manage the collaborative synchronization between multiple cloud service providers, the embodiment of the present disclosure creates a central coordinator responsible for coordinating the incremental synchronization tasks between different clouds. As the core component of multi-cloud collaborative synchronization, the central coordinator can be an independent service or module, or it can be integrated into the existing management platform.

[0078] In the central coordinator, users need to register and configure information of different cloud service providers, including but not limited to access credentials (such as API keys, access tokens, etc.), API endpoints, storage areas, and network configuration. Through detailed configuration, the central server can establish a communication connection with the cloud service provider and obtain the necessary resources to perform synchronization tasks. In addition, users can create incremental synchronization requests through the central server. These requests contain information about the source public cloud and the target public cloud, as well as parameters such as the frequency and time interval of synchronization. Then the central server creates corresponding synchronization tasks based on these requests and triggers these tasks according to the scheduled time or event. The central server also has the function of task scheduling. When multiple synchronization tasks are running at the same time, the central server will reasonably allocate computing resources and bandwidth based on factors such as task priority, cloud service provider performance, and network conditions to ensure that all tasks can run efficiently and stably. At the same time, the central server will monitor the status of each incremental synchronization task in real time. If a task is abnormal or erroneous (such as network failure or storage limitation), the central server will immediately record the error information and notify the user by email or SMS. The central server will also try to automatically fix some common errors to ensure the smooth progress of synchronization tasks. In addition, the central server can dynamically adjust the synchronization strategy based on the performance, availability, and network conditions of the cloud service provider. For example, when the performance of a cloud service provider is found to be degraded or the network latency is increased, the central server can reduce the synchronization frequency with the cloud service provider or suspend the synchronization task; when the size of a file block changes, the central server can prioritize the synchronization of these changed blocks instead of the entire file.

[0079] It is understandable that the use of a central coordinator makes cross-cloud incremental synchronization more flexible and scalable. At the same time, users can also centrally manage and monitor incremental synchronization tasks between multiple cloud service providers without having to log in to the management interface of each cloud service provider separately, thereby improving management efficiency.

[0080] Figure 4 The structural block diagram of the device for synchronizing increments across clouds according to an embodiment of the present disclosure is schematically shown.

[0081] like Figure 4 As shown, the cross-cloud incremental synchronization device 400 according to this embodiment includes a log collection module 410, a file generation module 420, a file checking module 430 and a file synchronization module 440.

[0082] The log collection module 410 is used to respond to the incremental synchronization request of the source public cloud accessing the log analysis platform, and collect the access logs on the source public cloud in real time. In one embodiment, the log collection module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0083] The file generation module 420 is used to obtain the access log newly generated by the source public cloud according to a preset time window through the incremental synchronization service to generate the i-th source manifest file. In one embodiment, the file generation module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0084] The file checking module 430 is used to check the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud through a difference comparison algorithm. In one embodiment, the file checking module 430 can be used to perform the operation S230 described above, which will not be described in detail here.

[0085] The file synchronization module 440 is used to synchronize the i-th source manifest file to the target public cloud in response to the i-th source manifest file being inconsistent with each of the target manifest files. In one embodiment, the file synchronization module 440 can be used to perform the operation S240 described above, which will not be described in detail here.

[0086] In an embodiment of the present disclosure, the log collection module 410 can also be used to: determine the synchronization priority of the access log on the source public cloud based on the basic attributes of the access log; and in response to multiple source public clouds initiating the incremental synchronization request at the same time, compare the synchronization priorities so that the access log with a higher synchronization priority is synchronized first.

[0087] In an embodiment of the present disclosure, the file checking module 430 may also be used to: verify the i-th source manifest file to determine the consistency between the i-th source manifest file and the i-1-th source manifest file in two adjacent time windows; and in response to the i-th source manifest file being consistent with the i-1-th source manifest file, skip the synchronization operation for the i-th source manifest file.

[0088] In an embodiment of the present disclosure, the file generation module 420 is specifically used for: in response to the log analysis platform receiving a query request for the incremental synchronization service, querying the time stamp of the creation or modification of the access log through the log analysis platform; in response to the time stamp of the creation or modification being within a preset time window and the access log not being synchronized, determining that the access log is a newly generated access log; and packaging the newly generated access log to form the i-th source manifest file.

[0089] In an embodiment of the present disclosure, the file generation module 420 may also be used to: add a uniqueness tag to the i-th source manifest file and each of the target manifest files, and compare whether the uniqueness tag of the i-th source manifest file is the same as the uniqueness tag of the target manifest file; and in response to the uniqueness tag of the i-th source manifest file being different from the uniqueness tags of each of the target manifest files, determine that the i-th source manifest file is inconsistent with the target manifest file.

[0090] In an embodiment of the present disclosure, the file checking module 430 is specifically used to: obtain the size of the file block in the i-th source manifest file and the size of the file block in each of the target manifest files; compare the size of the file block in the i-th source manifest file with the size of the file block in each of the target manifest files; and in response to the size of the file block in the i-th source manifest file being inconsistent with the size of the file block in each of the target manifest files, determine that the i-th source manifest file is inconsistent with the target manifest file.

[0091] In an embodiment of the present disclosure, the log collection module 410 can also be used to: create a central server, register and configure multiple source public clouds and the target public clouds on the central server; and in response to creating the incremental synchronization request on the central server, manage and monitor the synchronization operations of each source public cloud and each target public cloud through the central server; and adjust the synchronization operations of each source public cloud and each target public cloud based on the management and monitoring results of the synchronization operations.

[0092] According to an embodiment of the present disclosure, any multiple modules of the log collection module 410, the file generation module 420, the file check module 430 and the file synchronization module 440 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the log collection module 410, the file generation module 420, the file check module 430 and the file synchronization module 440 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the log collection module 410 , the file generation module 420 , the file checking module 430 , and the file synchronization module 440 may be at least partially implemented as a computer program module, and when the computer program module is executed, a corresponding function may be performed.

[0093] Figure 5 A block diagram of an electronic device for a method for synchronizing increments across clouds according to an embodiment of the present disclosure is schematically shown.

[0094] like Figure 5 As shown, the electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0095] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0096] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the I / O interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0097] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0098] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0099] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the cross-cloud incremental synchronization method provided by the embodiment of the present disclosure.

[0100] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0101] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0102] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0103] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0104] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0105] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0106] The embodiments of the present disclosure are described above. However, these embodiments are only for the purpose of illustration and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for synchronizing increments across clouds, characterized in that: The method comprises: In response to an incremental synchronization request from a source public cloud accessing a log analysis platform, collecting access logs on the source public cloud in real time; Obtaining the access log newly generated by the source public cloud according to a preset time window through an incremental synchronization service to generate an i-th source manifest file; Checking the consistency of the i-th source manifest file with each target manifest file stored in the target public cloud by using a difference comparison algorithm; and In response to the i-th source manifest file being inconsistent with each of the target manifest files, synchronizing the i-th source manifest file to the target public cloud.

2. The method according to claim 1, characterized in that The method further comprises: Determining a synchronization priority of the access log on the source public cloud based on basic attributes of the access log; and In response to multiple source public clouds initiating the incremental synchronization requests at the same time, the synchronization priorities are compared so that the access logs with higher synchronization priorities are synchronized first.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Verifying the i-th source list file to determine the consistency between the i-th source list file and the i-1-th source list file in two adjacent time windows; and In response to the i-th source manifest file being consistent with the (i-1)-th source manifest file, skipping the synchronization operation on the i-th source manifest file.

4. The method according to claim 1, characterized in that: The acquiring the newly generated access log of the source public cloud according to a preset time window through the incremental synchronization service to generate the i-th source inventory file includes: In response to the log analysis platform receiving a query request for the incremental synchronization service, querying the time stamp of creation or modification of the access log through the log analysis platform; In response to the time stamp of the creation or modification being within a preset time window and the access log not being synchronized, determining that the access log is a newly generated access log; and The newly generated access log is packaged to form the i-th source list file.

5. The method according to claim 1, characterized in that The checking the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud by using a difference comparison algorithm includes: adding a unique tag to the i-th source manifest file and each of the target manifest files, and comparing whether the unique tag of the i-th source manifest file is the same as the unique tag of the target manifest file; and In response to the uniqueness tag of the i-th source manifest file being different from the uniqueness tags of each of the target manifest files, it is determined that the i-th source manifest file is inconsistent with the target manifest file.

6. The method according to claim 1 or 5, characterized in that: The checking the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud by using a difference comparison algorithm further includes: Obtain the size of the file block in the i-th source manifest file and the size of the file block in each of the target manifest files; Comparing the size of the file block in the i-th source manifest file with the size of the file block in each of the target manifest files; and In response to the size of the file block in the i-th source manifest file being inconsistent with the size of the file block in each of the target manifest files, it is determined that the i-th source manifest file is inconsistent with the target manifest file.

7. The method according to claim 1, characterized in that The method further comprises: Creating a central server, registering and configuring the plurality of source public clouds and the target public cloud on the central server; and In response to creating the incremental synchronization request on the central server, managing and monitoring synchronization operations of each of the source public clouds and each of the target public clouds through the central server; and Based on the management and monitoring results of the synchronization operation, the synchronization operation of each of the source public clouds and each of the target public clouds is adjusted.

8. A cross-cloud incremental synchronization device, characterized in that: The device comprises: A log collection module, configured to respond to an incremental synchronization request from a source public cloud accessing a log analysis platform and collect access logs on the source public cloud in real time; A file generation module, configured to obtain the access log newly generated by the source public cloud according to a preset time window through an incremental synchronization service to generate an i-th source inventory file; a file checking module, configured to check the consistency of the i-th source manifest file with each target manifest file stored in the target public cloud through a difference comparison algorithm; and The file synchronization module is configured to synchronize the i-th source manifest file to the target public cloud in response to the i-th source manifest file being inconsistent with each of the target manifest files.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more computer programs, It is characterized in that when the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor executes the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images