Cross-cloud incremental synchronization method, apparatus, device, medium and program product

Abstract

The present disclosure provides a cross-cloud incremental synchronization method, which can be applied to the field of cloud computing or big data. The method comprises: in response to an incremental synchronization request of a source public cloud accessing a log analysis platform, collecting access logs on the source public cloud in real time; through an incremental synchronization service, obtaining newly generated access logs of the source public cloud according to a preset time window to generate an i th source manifest file; through a difference comparison algorithm, checking the consistency of the i th source manifest file with each target manifest file stored in a target public cloud; and in response to the inconsistency between the i th source manifest file and each target manifest file, synchronizing the i th source manifest file to the target public cloud. The present disclosure also provides a cross-cloud incremental synchronization device, equipment, storage medium and program product.

Description

Technical Field

[0001] This disclosure relates to the field of cloud computing or big data, and more specifically to a method, apparatus, device, medium, and program product for cross-cloud incremental synchronization. Background Technology

[0002] In cloud computing environments, to improve data security and reliability, some critical data needs to be stored across multiple cloud service providers. However, challenges arise when it comes to data synchronization between different clouds. Traditional data synchronization methods often only support full synchronization, meaning that each synchronization requires completely copying the data from the source to the target, regardless of whether the data has changed. This approach may be feasible for scenarios with small amounts of data or infrequent changes, but its efficiency and cost become apparent in scenarios with large-scale and high-frequency data changes. Summary of the Invention

[0003] In view of the above problems, embodiments of this disclosure provide a method, apparatus, device, medium, and program product for cross-cloud incremental synchronization.

[0004] According to a first aspect of this disclosure, a cross-cloud incremental synchronization method is provided, the method comprising: in response to an incremental synchronization request from a source public cloud accessing a log analysis platform, collecting access logs on the source public cloud in real time; acquiring newly generated access logs from the source public cloud according to a preset time window through an incremental synchronization service to generate an i-th source manifest file; checking the consistency between the i-th source manifest file and each target manifest file stored in a target public cloud through a difference comparison algorithm; and in response to inconsistencies between the i-th source manifest file and each of the target manifest files, synchronizing the i-th source manifest file to the target public cloud.

[0005] According to embodiments of this disclosure, the method further includes: determining the synchronization priority of the access logs on the source public cloud based on the basic attributes of the access logs; and in response to multiple source public clouds initiating the incremental synchronization request at the same time, comparing the synchronization priorities so that the access logs with higher synchronization priorities are synchronized first.

[0006] According to an embodiment of this disclosure, the method further includes: verifying the i-th source manifest file to determine the consistency between the i-th source manifest file and the (i-1)-th source manifest file in two adjacent time windows; and, in response to the consistency between the i-th source manifest file and the (i-1)-th source manifest file, skipping the synchronization operation of the i-th source manifest file.

[0007] According to an embodiment of this disclosure, the step of obtaining the newly generated access log from the source public cloud through incremental synchronization service according to a preset time window to generate the i-th source manifest file includes: responding to the log analysis platform receiving a query request from the incremental synchronization service, querying the creation or modification time identifier of the access log through the log analysis platform; responding to the creation or modification time identifier being within the preset time window and the access log not being synchronized, determining that the access log is the newly generated access log; and packaging the newly generated access log to form the i-th source manifest file.

[0008] According to embodiments of this disclosure, checking the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud using a difference comparison algorithm includes: adding unique tags to the i-th source manifest file and each of the target manifest files, and comparing whether the unique tags of the i-th source manifest file and the target manifest files are the same; and determining that the i-th source manifest file and the target manifest files are inconsistent if the unique tags of the i-th source manifest file and the target manifest files are different.

[0009] According to embodiments of this disclosure, the step of checking the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud using a difference comparison algorithm further includes: obtaining the size of the file blocks in the i-th source manifest file and the size of the file blocks in each of the target manifest files; comparing the size of the file blocks in the i-th source manifest file with the size of the file blocks in each of the target manifest files; and determining that the i-th source manifest file and the target manifest file are inconsistent in response to the inconsistency between the size of the file blocks in the i-th source manifest file and the size of the file blocks in each of the target manifest files.

[0010] According to embodiments of this disclosure, the method further includes: creating a central server, registering and configuring multiple source public clouds and target public clouds on the central server; and, in response to creating the incremental synchronization request on the central server, managing and monitoring the synchronization operations of each source public cloud and each target public cloud through the central server; and adjusting the synchronization operations of each source public cloud and each target public cloud based on the management and monitoring results of the synchronization operations.

[0011] A second aspect of this disclosure provides a cross-cloud incremental synchronization apparatus, comprising: a log collection module, configured to collect access logs on the source public cloud in real time in response to an incremental synchronization request from a source public cloud accessing a log analysis platform; a file generation module, configured to obtain the newly generated access logs from the source public cloud according to a preset time window through an incremental synchronization service, to generate an i-th source manifest file; a file checking module, configured to check the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud using a difference comparison algorithm; and a file synchronization module, configured to synchronize the i-th source manifest file to the target public cloud in response to inconsistencies between the i-th source manifest file and each of the target manifest files.

[0012] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a storage device for storing one or more programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above-described cross-cloud incremental synchronization method.

[0013] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the above-described cross-cloud incremental synchronization method.

[0014] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described cross-cloud incremental synchronization method.

[0015] In the embodiments of this disclosure, incremental data capture is achieved through incremental synchronization service while ensuring data security, avoiding the overhead of full synchronization and reducing data transmission bandwidth and storage costs. At the same time, the consistency of files is checked using a difference comparison algorithm to quickly determine which data is new and which data has been synchronized, avoiding unnecessary data transmission and improving the intelligence and efficiency of cross-cloud incremental synchronization. Attached Figure Description

[0016] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1 The illustration schematically depicts application scenarios of cross-cloud incremental synchronization methods, apparatuses, devices, media, and program products according to embodiments of this disclosure;

[0018] Figure 2 A flowchart illustrating a cross-cloud incremental synchronization method according to an embodiment of the present disclosure is shown schematically;

[0019] Figure 3A flowchart illustrating a method for checking the consistency of file blocks according to an embodiment of the present disclosure is shown schematically.

[0020] Figure 4 A schematic block diagram of a cross-cloud incremental synchronization device according to an embodiment of the present disclosure is shown; and

[0021] Figure 5 A block diagram of an electronic device for a cross-cloud incremental synchronization method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0022] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0023] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0024] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0025] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0026] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0027] Figure 1 The illustration schematically depicts application scenarios of cross-cloud incremental synchronization methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure.

[0028] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0029] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0030] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0031] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0032] It should be noted that the cross-cloud incremental synchronization method provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the cross-cloud incremental synchronization device provided in this disclosure embodiment can generally be located in server 105. The cross-cloud incremental synchronization method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the cross-cloud incremental synchronization device provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0033] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0034] First, the technical terms used in this article are explained and clarified as follows.

[0035] Log analysis platform: Used to analyze access logs stored on the source public cloud and obtain information about recently generated files.

[0036] Incremental synchronization service: Responsible for synchronizing newly generated files to the target public cloud storage based on the manifest file, avoiding duplicate synchronization.

[0037] Intelligent incremental synchronization strategy: Dynamically adjust the priority of incremental synchronization based on factors such as file access frequency, size, and importance.

[0038] Difference comparison technology: using efficient algorithms, only the parts that have changed are synchronized, reducing the amount of data transmission.

[0039] Multi-cloud collaborative synchronization: Considering the collaborative synchronization between multiple cloud service providers, making cross-cloud incremental synchronization more flexible and scalable.

[0040] Central coordinator: Responsible for coordinating incremental synchronization tasks between different clouds, managing task status and error handling.

[0041] The following will be based on Figure 1 The described scene, through Figure 2 and Figure 3 The cross-cloud incremental synchronization method according to embodiments of this disclosure will be described in detail.

[0042] Figure 2 A flowchart illustrating a cross-cloud incremental synchronization method according to an embodiment of the present disclosure is shown schematically.

[0043] like Figure 2As shown, the cross-cloud incremental synchronization method of this embodiment includes operations S210 to S240.

[0044] In operation S210, in response to the incremental synchronization request from the source public cloud accessing the log analysis platform, access logs on the source public cloud are collected in real time.

[0045] In embodiments of this disclosure, user consent or authorization is obtained before collecting access logs from the source public cloud. For example, a request to collect access logs from the source public cloud is sent to the user before operation S210. Operation S210 is performed if the user consents or authorizes the collection of access logs from the source public cloud.

[0046] In operation S220, the newly generated access logs from the source public cloud are obtained through the incremental synchronization service according to a preset time window to generate the i-th source manifest file.

[0047] In operation S230, the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud is checked by a difference comparison algorithm.

[0048] In operation S240, in response to the inconsistency between the i-th source manifest file and each of the target manifest files, the i-th source manifest file is synchronized to the target public cloud.

[0049] In the embodiments of this disclosure, a corresponding operation entry point is provided to the user, allowing the user to choose to agree to or reject the automated decision-making result. That is, before synchronizing the i-th source manifest file to the target public cloud for processing / decision, the user's instruction to agree to or reject the processing / decision is obtained through the corresponding operation entry point. If the user agrees to the processing / decision, then step S240 is executed before synchronizing the i-th source manifest file to the target public cloud for processing / decision. If the user rejects the processing / decision, the expert decision-making process is initiated.

[0050] In some exemplary embodiments, taking synchronization between two public clouds, namely source public cloud A and target public cloud B, as an example, newly generated incremental files in cloud A are synchronized to cloud B at preset time intervals. In this process, firstly, access logs stored in cloud A are collected in real time through a log collection system and transmitted to a log analysis platform. The log analysis platform analyzes the access logs in cloud A and uses a pre-written incremental synchronization service to obtain the information of the most recently generated files within a preset time window, i.e., the i-th source manifest file. Then, a difference comparison algorithm is used to verify the i-th source manifest file, ultimately achieving incremental file synchronization between different clouds.

[0051] It is understood that the method provided by the embodiments of this disclosure enables incremental synchronization of files between different clouds, reduces the bandwidth and storage costs of data transmission, ensures the security and consistency of data, ensures that important data is stored in multiple clouds, and improves the intelligence and efficiency of cross-cloud incremental synchronization through the difference comparison algorithm.

[0052] In embodiments of this disclosure, the method further includes: determining the synchronization priority of the access logs on the source public cloud based on the basic attributes of the access logs; and in response to multiple source public clouds initiating the incremental synchronization request at the same time, comparing the synchronization priorities so that the access logs with higher synchronization priorities are synchronized first.

[0053] Specifically, the basic attributes of access logs include, but are not limited to, log generation time, file size, type of service or application to which it belongs, and access frequency. These attributes are used to initially assess the log synchronization priority. For example, a recently generated, large log from a critical service that is frequently accessed will have a higher synchronization priority. Furthermore, to more accurately determine the synchronization priority, intelligent algorithms, such as machine learning models or predictive analytics algorithms, can be introduced. Intelligent algorithms predict future file access patterns based on historical data (such as access log records over a past period), and thus determine the synchronization priority based on the prediction results. After determining the synchronization priority, a specific intelligent incremental synchronization strategy is formulated to dynamically adjust the priority of incremental synchronization.

[0054] For example, by analyzing the access frequency of logs, the algorithm identifies which files are frequently accessed. These files, due to their higher importance, can be assigned a higher synchronization priority. Furthermore, while larger files may contain more information, their synchronization cost is also higher. Therefore, the algorithm determines a reasonable synchronization priority by balancing file size and importance. Also, logs generated by different services or applications may have different levels of importance. For example, security-related logs may require a higher synchronization priority.

[0055] Understandably, by setting synchronization priorities and prioritizing the processing of access logs with high synchronization priorities, it is possible to ensure that critical data is synchronized in a timely manner, thereby improving overall synchronization efficiency. At the same time, through the timely synchronization of critical data, the system can reduce the risk of data inconsistency and loss.

[0056] In embodiments of this disclosure, the method further includes: verifying the i-th source manifest file to determine the consistency between the i-th source manifest file and the (i-1)-th source manifest file in two adjacent time windows; and, in response to the consistency between the i-th source manifest file and the (i-1)-th source manifest file, skipping the synchronization operation of the i-th source manifest file.

[0057] Specifically, the incremental synchronization service periodically or based on preset trigger conditions generates source manifest files. These manifest files record a list of access log files generated or updated on the source public cloud within a specific time window (e.g., 10 minutes). Since the manifest files are based on files generated within a recent period as queried by the log analysis platform, duplicate file records may exist between adjacent manifest files. Therefore, to avoid unnecessary duplicate synchronization operations, the embodiments of this disclosure introduce a verification mechanism for the source manifest files. Specifically, when the i-th source manifest file is generated, the synchronization service verifies it to determine its consistency with the (i-1)-th source manifest file generated in the previous time window. If the i-th source manifest file and the (i-1)-th source manifest file are completely identical in the file list, then the two manifest files can be considered duplicates. In this case, the synchronization service skips the synchronization operation for the i-th source manifest file.

[0058] It should be noted that determining whether two adjacent source manifest files are identical can be achieved by comparing the file lists recorded in the two manifest files.

[0059] Understandably, checking the consistency of two adjacent source manifest files reduces unnecessary duplicate synchronization operations, saving network bandwidth and storage resources while improving the performance and response speed of the synchronization service. It also helps improve data synchronization consistency, reducing data inconsistencies caused by synchronization errors or omissions.

[0060] Furthermore, the step of obtaining the newly generated access logs from the source public cloud through incremental synchronization service according to a preset time window to generate the i-th source manifest file includes: in response to the log analysis platform receiving a query request from the incremental synchronization service, querying the creation or modification time identifier of the access log through the log analysis platform; in response to the creation or modification time identifier being within the preset time window and the access log not being synchronized, determining that the access log is the newly generated access log; and packaging the newly generated access log to form the i-th source manifest file.

[0061] Specifically, the pre-written incremental synchronization service is first connected to a log analysis platform, which collects access logs stored on Cloud A. Then, the incremental synchronization service automatically triggers at preset time intervals (e.g., every 5 minutes), sending query requests to the log analysis platform to retrieve newly generated access log information within a preset time window. During this process, the log analysis platform retrieves files that have changed within the preset time window. To determine if a file is newly created or modified, the platform compares the file's timestamp. If the file's modification or creation time falls within the preset time window and has not appeared in previous synchronization tasks, it is considered a newly generated access log. Based on the query results, the incremental synchronization service packages this log information into a manifest file, the i-th source manifest file, which contains information such as filename, path, and size. Before generating the manifest file, the service performs deduplication by comparing the file's unique identifier (e.g., hash value or file path) to ensure that the manifest does not contain any duplicate files.

[0062] It should be noted that in the embodiments disclosed herein, a 10-minute time window is selected to ensure that most newly generated logs can be captured. However, considering the potential processing delays of the log analysis platform, if only logs within the last 5 minutes are queried, some newly generated logs may be missed due to the delay.

[0063] Understandably, incremental synchronization is more efficient than full synchronization, reducing network bandwidth and storage resource consumption. Furthermore, through periodic queries and incremental synchronization, the system can dynamically adjust resource usage based on actual data changes. When data changes are minimal, the system can reduce the frequency and scale of synchronization, thereby saving network bandwidth, computing resources, and storage space. Conversely, when data changes are significant, the system can correspondingly increase the frequency and scale of synchronization to ensure data timeliness and integrity.

[0064] Furthermore, the step of checking the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud through the difference comparison algorithm includes: adding unique tags to the i-th source manifest file and each of the target manifest files, and comparing whether the unique tags of the i-th source manifest file and the target manifest files are the same; and determining that the i-th source manifest file and the target manifest files are inconsistent in response to the unique tags of the i-th source manifest file and the target manifest files being different.

[0065] Specifically, when the incremental synchronization service needs to synchronize the newly generated i-th source manifest file to cloud storage B, the difference comparison algorithm compares the consistency between the source manifest file and each target manifest file. If the source manifest file already exists on the target public cloud or its content is identical, the incremental synchronization service will skip that file and will not transfer it. The difference comparison technology uses a difference comparison algorithm to identify changes between files and synchronizes only these changes, rather than the entire file.

[0066] It should be noted that before comparing using the difference comparison algorithm, a unique tag is added to each manifest file (including the i-th source manifest file and each target manifest file stored in the target public cloud). This tag is usually automatically generated based on the file content or file attributes, such as filename, hash value, or version number, but it can also be added manually.

[0067] In this disclosure, the difference comparison technique may include the following methods:

[0068] For example, filename comparison: compare the filenames of the source manifest file and each target manifest file. If a target manifest file with the same name already exists in the target cloud, it means that the file has already been synchronized and does not need to be synchronized again.

[0069] For example, hash value comparison: A hash value is a unique identifier for the file content. In incremental synchronization, the system calculates the hash value of the file, for example, using the SHA-256 algorithm. When files need to be synchronized, the system compares the hash values ​​of the source manifest file and each target manifest file. If the hash values ​​are the same, it means the file content has not changed and synchronization is not required; if the hash values ​​are different, it means the file has changed and synchronization is required.

[0070] For example, version number comparison: Each file can be associated with a version number. As a file changes, the version number increments. In incremental synchronization, the system compares the version numbers of the source manifest file and each target manifest file. If the version numbers are the same, it means the file content has not changed and synchronization is unnecessary; if the version numbers are different, it means the file has changed and synchronization is required.

[0071] Understandably, by using a more efficient difference comparison algorithm, the efficiency of synchronization is improved, unnecessary data transmission during incremental synchronization is reduced, and the accuracy of synchronization is ensured, avoiding data duplication or omission.

[0072] Figure 3 A flowchart illustrating a method for checking the consistency of file blocks according to an embodiment of the present disclosure is shown.

[0073] Furthermore, such as Figure 3As shown, checking the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud through the difference comparison algorithm further includes: obtaining the size of the file blocks in the i-th source manifest file and the size of the file blocks in each of the target manifest files; comparing the size of the file blocks in the i-th source manifest file with the size of the file blocks in each of the target manifest files; and determining that the i-th source manifest file and the target manifest file are inconsistent in response to the inconsistency between the size of the file blocks in the i-th source manifest file and the size of the file blocks in each of the target manifest files.

[0074] Specifically, when using a difference comparison algorithm to check the consistency between a source manifest file and a target manifest file stored in the target public cloud, for some large files, the file can be split into multiple fixed-size or variable-length blocks. Then, only the changed blocks are compared, rather than the entire file. In this process, the size of each file block in the i-th source manifest file and the size of the corresponding file block in the target manifest file are first obtained. These file blocks can be fixed-size (e.g., 1MB each) or variable-length (dynamically divided based on file content or structure). Then, the system compares the size of each file block in the i-th source manifest file with the size of the corresponding file block in the target manifest file. If the two blocks are the same size, the content of the two blocks is further compared. If the block sizes are different, the two blocks are directly determined to be inconsistent and need to be synchronized. Furthermore, if only some blocks have changed, the system can prioritize synchronizing these changed blocks rather than the entire file.

[0075] Understandably, introducing file block-level comparisons can significantly reduce data transfer volume and further improve the efficiency and accuracy of synchronization.

[0076] In embodiments of this disclosure, the method further includes: creating a central server, registering and configuring multiple source public clouds and target public clouds on the central server; and, in response to creating the incremental synchronization request on the central server, managing and monitoring the synchronization operations of each source public cloud and each target public cloud through the central server; and adjusting the synchronization operations of each source public cloud and each target public cloud based on the management and monitoring results of the synchronization operations.

[0077] To effectively manage collaborative synchronization among multiple cloud service providers, embodiments of this disclosure create a central coordinator responsible for coordinating incremental synchronization tasks between different clouds. This central coordinator, as a core component of multi-cloud collaborative synchronization, can be a standalone service or module, or it can be integrated into an existing management platform.

[0078] In the central coordinator, users need to register and configure information about different cloud service providers, including but not limited to access credentials (such as API keys, access tokens, etc.), API endpoints, storage regions, and network configurations. Through detailed configuration, the central server can establish communication connections with cloud service providers and obtain the necessary resources to execute synchronization tasks. Furthermore, users can create incremental synchronization requests through the central server. These requests include information about the source and target public clouds, as well as parameters such as synchronization frequency and time intervals. The central server then creates corresponding synchronization tasks based on these requests and triggers these tasks according to predetermined times or events. The central server also has task scheduling capabilities. When multiple synchronization tasks run simultaneously, the central server will rationally allocate computing resources and bandwidth based on task priority, cloud service provider performance, and network conditions to ensure all tasks run efficiently and stably. Simultaneously, the central server monitors the status of each incremental synchronization task in real time. If a task encounters an anomaly or error (such as network failure or storage limitations), the central server will immediately record the error information and notify the user via email or SMS. The central server will also attempt to automatically correct some common errors to ensure the smooth progress of synchronization tasks. In addition, the central server can dynamically adjust its synchronization strategy based on factors such as the performance, availability, and network conditions of the cloud service provider. For example, if it detects a performance degradation or increased network latency of a particular cloud service provider, the central server can reduce the synchronization frequency with that provider or pause synchronization tasks; when the size of a file block changes, the central server can prioritize synchronizing those changed blocks rather than the entire file.

[0079] Understandably, the use of a central coordinator makes cross-cloud incremental synchronization more flexible and scalable. At the same time, users can centrally manage and monitor incremental synchronization tasks between multiple cloud service providers without having to log in to the management interface of each cloud service provider separately, thus improving management efficiency.

[0080] Figure 4 A schematic block diagram of a cross-cloud incremental synchronization device according to an embodiment of the present disclosure is shown.

[0081] like Figure 4 As shown, the cross-cloud incremental synchronization device 400 according to this embodiment includes a log collection module 410, a file generation module 420, a file inspection module 430, and a file synchronization module 440.

[0082] The log collection module 410 is used to collect access logs on the source public cloud in real time in response to incremental synchronization requests from the source public cloud accessing the log analysis platform. In one embodiment, the log collection module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0083] The file generation module 420 is used to obtain the newly generated access logs from the source public cloud according to a preset time window through an incremental synchronization service, so as to generate the i-th source manifest file. In one embodiment, the file generation module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0084] The file inspection module 430 is used to check the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud using a difference comparison algorithm. In one embodiment, the file inspection module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0085] The file synchronization module 440 is used to synchronize the i-th source manifest file to the target public cloud in response to inconsistencies between the i-th source manifest file and each of the target manifest files. In one embodiment, the file synchronization module 440 can be used to perform the operation S240 described above, which will not be repeated here.

[0086] In embodiments of this disclosure, the log collection module 410 can also be used to: determine the synchronization priority of the access logs on the source public cloud based on the basic attributes of the access logs; and in response to multiple source public clouds initiating the incremental synchronization request at the same time, compare the synchronization priorities so that the access logs with higher synchronization priorities are synchronized first.

[0087] In embodiments of this disclosure, the file inspection module 430 can also be used to: verify the i-th source manifest file, determine the consistency between the i-th source manifest file and the (i-1)-th source manifest file in two adjacent time windows; and, in response to the consistency between the i-th source manifest file and the (i-1)-th source manifest file, skip the synchronization operation of the i-th source manifest file.

[0088] In the embodiments of this disclosure, the file generation module 420 is specifically configured to: in response to the log analysis platform receiving a query request from the incremental synchronization service, query the creation or modification time identifier of the access log through the log analysis platform; in response to the creation or modification time identifier being within a preset time window and the access log not being synchronized, determine that the access log is a newly generated access log; and package the newly generated access log to form the i-th source manifest file.

[0089] In embodiments of this disclosure, the file generation module 420 may further be used to: add unique tags to the i-th source manifest file and each of the target manifest files, and compare whether the unique tags of the i-th source manifest file and the target manifest files are the same; and in response to the unique tags of the i-th source manifest file and the target manifest files being different, determine that the i-th source manifest file and the target manifest files are inconsistent.

[0090] In embodiments of this disclosure, the file inspection module 430 is specifically configured to: obtain the size of file blocks in the i-th source manifest file and the size of file blocks in each of the target manifest files; compare the size of file blocks in the i-th source manifest file with the size of file blocks in each of the target manifest files; and, in response to a discrepancy between the size of file blocks in the i-th source manifest file and the size of file blocks in each of the target manifest files, determine that the i-th source manifest file and the target manifest file are inconsistent.

[0091] In embodiments of this disclosure, the log collection module 410 can also be used to: create a central server, register and configure multiple source public clouds and target public clouds on the central server; and in response to creating the incremental synchronization request on the central server, manage and monitor the synchronization operations of each source public cloud and each target public cloud through the central server; and adjust the synchronization operations of each source public cloud and each target public cloud based on the management and monitoring results of the synchronization operations.

[0092] According to embodiments of this disclosure, any multiple modules among the log collection module 410, file generation module 420, file inspection module 430, and file synchronization module 440 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the log collection module 410, file generation module 420, file inspection module 430, and file synchronization module 440 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the log collection module 410, file generation module 420, file inspection module 430, and file synchronization module 440 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0093] Figure 5 A block diagram of an electronic device for a cross-cloud incremental synchronization method according to an embodiment of the present disclosure is shown schematically.

[0094] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0095] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0096] According to an embodiment of the present invention, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to a bus 504. The electronic device 500 may also include one or more of the following components connected to the I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0097] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0098] According to embodiments of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, the computer-readable storage medium may include ROM 502 and / or RAM 503 and / or one or more memories other than ROM 502 and RAM 503 described above.

[0099] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the cross-cloud incremental synchronization method provided in embodiments of this disclosure.

[0100] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0101] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0102] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0103] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0105] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0106] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A cross-cloud incremental synchronization method, characterized in that, The method includes: In response to incremental synchronization requests from the source public cloud accessing the log analysis platform, access logs on the source public cloud are collected in real time. The incremental synchronization service obtains the newly generated access logs from the source public cloud according to a preset time window to generate the i-th source manifest file. The source manifest file is a list of access log files generated or updated by the source public cloud within the time window. The consistency between the i-th source manifest file and each target manifest file stored in the target public cloud is checked using a difference comparison algorithm; and In response to the inconsistency between the i-th source manifest file and each of the target manifest files, the i-th source manifest file is synchronized to the target public cloud; The method further includes: The i-th source manifest file is validated to determine its consistency with the (i-1)-th source manifest file within two adjacent time windows; and In response to the fact that the i-th source manifest file is consistent with the (i-1)-th source manifest file, the synchronization operation of the i-th source manifest file is skipped.

2. The method according to claim 1, characterized in that, The method further includes: Based on the basic attributes of the access logs, determine the synchronization priority of the access logs on the source public cloud; and In response to multiple source public clouds initiating incremental synchronization requests at the same time, the synchronization priorities are compared so that the access logs with higher synchronization priorities are synchronized first.

3. The method according to claim 1, characterized in that, The step of obtaining the newly generated access logs from the source public cloud through incremental synchronization service according to a preset time window to generate the i-th source manifest file includes: In response to the log analysis platform receiving a query request from the incremental synchronization service, the log analysis platform queries the time identifier of the creation or modification of the access log. If the created or modified timestamp falls within a preset time window and the access log has not been synchronized, the access log is determined to be a newly generated access log; and The newly generated access logs are packaged to form the i-th source manifest file.

4. The method according to claim 1, characterized in that, The step of checking the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud using a difference comparison algorithm includes: Add unique tags to the i-th source manifest file and each of the target manifest files, and compare whether the unique tags of the i-th source manifest file and the target manifest files are the same; and In response to the fact that the unique tags of the i-th source manifest file and the unique tags of each of the target manifest files are different, it is determined that the i-th source manifest file and the target manifest file are inconsistent.

5. The method according to claim 1 or 4, characterized in that, The step of checking the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud using a difference comparison algorithm also includes: Obtain the size of the file block in the i-th source manifest file and the size of the file block in each of the target manifest files; Compare the size of the file blocks in the i-th source manifest file with the size of the file blocks in each of the target manifest files; and In response to the inconsistency between the size of the file blocks in the i-th source manifest file and the size of the file blocks in each of the target manifest files, it is determined that the i-th source manifest file and the target manifest file are inconsistent.

6. The method according to claim 1, characterized in that, The method further includes: Create a central server, register and configure the multiple source public clouds and the target public cloud on the central server; and In response to the creation of the incremental synchronization request on the central server, the central server manages and monitors the synchronization operations of each of the source public clouds and each of the target public clouds; and Based on the management and monitoring results of the synchronization operations, adjustments are made to the synchronization operations of each source public cloud and each target public cloud.

7. A cross-cloud incremental synchronization device, characterized in that, The device includes: The log collection module is used to collect access logs on the source public cloud in real time in response to incremental synchronization requests from the source public cloud that accesses the log analysis platform. The file generation module is used to obtain the newly generated access logs of the source public cloud according to a preset time window through incremental synchronization service, so as to generate the i-th source manifest file. The source manifest file is the access log file generated or updated by the source public cloud within the time window. The file inspection module is used to check the consistency between the i-th source manifest file and each target manifest file stored in the target public cloud using a difference comparison algorithm; it is also used to verify the i-th source manifest file and determine the consistency between the i-th source manifest file and the (i-1)-th source manifest file in two adjacent time windows; in response to the i-th source manifest file being consistent with the (i-1)-th source manifest file, the synchronization operation for the i-th source manifest file is skipped; and The file synchronization module is used to synchronize the i-th source manifest file to the target public cloud in response to the inconsistency between the i-th source manifest file and each of the target manifest files.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more computer programs. The characteristic is that, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having executable instructions stored thereon, characterized in that, When executed by the processor, this instruction causes the processor to perform the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 6.

Images