A multi-cloud heterogeneous object storage eventual consistency method, system, medium and product based on operation log

Abstract

The application discloses a multi-cloud heterogeneous object storage eventual consistency method and system based on operation logs, a medium and a product, relates to the multi-cloud storage disaster recovery field, and comprises the following steps: utilizing multi-cloud heterogeneous object storage and performing synchronous double writing; when any storage fails, suspending double writing, marking the degraded storage, only reading and writing the normal storage, and saving all write operations during the failure into a log and storing the log in a specified path of the normal storage. After the failure is recovered, the sequential double writing operation is performed. According to the log, the degraded storage is subjected to a double reading check mechanism for asynchronous synchronization, so that the old data is prevented from covering the new data. After the synchronization is completed, the original synchronous double writing configuration is restored, and the storage consistency is realized. The application can reduce the switching and recovery time when the object storage fails, realizes zero data loss, prevents the old data from covering the new data, and improves the reliability and disaster recovery capability in the data storage process.

Description

Technical Field

[0001] This application relates to the field of multi-cloud storage disaster recovery, and in particular to a method, system, medium and product for eventual consistency of multi-cloud heterogeneous object storage based on operation logs. Background Technology

[0002] To meet business continuity, data security, and compliance requirements, storing file objects across multiple cloud vendors' object storage systems has become essential for many critical core systems. However, the industry currently lacks a comprehensive disaster recovery solution for high availability and zero data loss across multi-cloud heterogeneous object storage. First, cross-region replication provided by cloud vendors can only replicate between multiple regions within that vendor and cannot achieve zero data loss disaster recovery. Second, while gateway-level dual-write can achieve multi-cloud dual-write, it cannot solve the data recovery problem after disaster recovery. Finally, existing site replication solutions are simply data copies, lacking version control and conflict retention mechanisms. They cannot address the issue of new data being overwritten by old data during data replication and business synchronization, and due to numerous invalid comparison or copy operations, recovery times are long in scenarios with very large historical data volumes.

[0003] In summary, existing technologies have drawbacks such as long cloud storage switching and recovery times, easy data loss during failures, lack of version control during recovery, inconsistency between old and new data, and insufficient reliability and disaster recovery capabilities. Summary of the Invention

[0004] The purpose of this application is to provide a method, system, medium, and product for eventual consistency of multi-cloud heterogeneous object storage based on operation logs, which can reduce switching and recovery time in the event of object storage failure and achieve zero data loss; at the same time, it can prevent old data from overwriting new data and improve the reliability and disaster recovery capability of data storage.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for eventual consistency of multi-cloud heterogeneous object storage based on operation logs. The method includes: configuring multi-cloud heterogeneous object storage through a configuration center, and performing synchronous dual-write operations on target data using each object storage; the synchronous dual-write operation involves simultaneously writing to each object storage; when any object storage fails, the synchronous dual-write operation is suspended, and after a degradation judgment is performed on the failed object storage, it is marked as a degraded object storage; read and write operations are stopped on the degraded object storage, and read and write operations are only performed on the normal object storage, while simultaneously updating the entire failed object storage... During the failure phase, write operation records at each moment are compiled into a write operation log and saved to a designated path in the normal object storage. When the degraded object storage recovers from the failure, a sequential dual-write operation is performed through the configuration center. The sequential dual-write operation is as follows: write to the normal object storage first, then write to the degraded object storage. Read operations are only performed in the normal object storage. Asynchronous data synchronization is performed on the degraded object storage according to the write operation log and a dual-read verification synchronization mechanism. After the asynchronous data synchronization operation is completed, the configuration is restored to the synchronous dual-write operation state of the multi-cloud heterogeneous object storage before the failure through the configuration center.

[0006] In a second aspect, this application also provides a computer system, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the operation log-based multi-cloud heterogeneous object storage eventual consistency method described in the first aspect.

[0007] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for eventual consistency of multi-cloud heterogeneous object storage based on operation logs.

[0008] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described operation log-based method for eventual consistency of multi-cloud heterogeneous object storage.

[0009] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application utilizes multi-cloud heterogeneous object storage and performs synchronous dual-write. When any storage fails, dual-write is suspended, the affected storage is marked as degraded, and reads and writes are only performed on the normal storage. All write operations during the failure period are compiled into a log and saved to a designated path on the normal storage. After recovery, sequential dual-write operations are performed. Based on the log, a dual-read verification mechanism is used to asynchronously synchronize the degraded storage to prevent old data from overwriting new data. After synchronization is complete, the original synchronous dual-write configuration is restored to achieve storage consistency. This application can reduce switchover and recovery time during object storage failures, achieving zero data loss; at the same time, it can prevent old data from overwriting new data, improving the reliability and disaster recovery capabilities of data storage. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating the eventual consistency method for multi-cloud heterogeneous object storage based on operation logs provided in an embodiment of this application.

[0012] Figure 2 This is an internal structure diagram of a computer system provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] Example 1, as Figure 1 As shown, this embodiment provides an eventual consistency method for multi-cloud heterogeneous object storage based on operation logs, the method comprising: S1. Configure multi-cloud heterogeneous object storage through the configuration center, and perform synchronous dual-write operations on the target data using each object storage; the synchronous dual-write operation is to perform write operations to each object storage simultaneously.

[0016] Furthermore, step S1 specifically includes: S11. Configure multi-cloud heterogeneous object storage through the configuration center.

[0017] S12. Use the client proxy component to read the configuration from the configuration center.

[0018] S13. Perform synchronous dual-write operations on multiple heterogeneous object stores according to the configuration.

[0019] Furthermore, the object storage includes at least Alibaba Cloud Object Storage and Tencent Cloud Object Storage.

[0020] Optionally, in addition to Alibaba Cloud Object Storage and Tencent Cloud Object Storage, it can be extended to any other type of object storage.

[0021] S2. When any object storage fails, pause the synchronous dual-write operation, perform a degradation judgment on the failed object storage, and mark it as a degraded object storage.

[0022] Further step S2 specifically includes: S21. When any object storage fails, pause the execution of synchronous double write operations.

[0023] S22. Use the circuit breaker mechanism to determine whether the failed object storage needs to be downgraded. If it is determined that downgrading is necessary, notify the configuration center to mark the failed object storage as downgraded object storage; otherwise, do not mark it.

[0024] Optionally, the circuit breaker mechanism includes: if the number of failed accesses to an object storage and the failure rate reach a specified threshold within a specified time period, then the object storage is determined to need to be downgraded.

[0025] In practical applications, when an object storage fails, the circuit breaker mechanism will determine that the object storage should be downgraded, and then automatically notify the configuration center to turn on the downgrade switch; or the configuration center can be manually turned on the downgrade switch based on the alarm information.

[0026] S3. Stop performing read and write operations on the degraded object storage, and only perform read and write operations on the normal object storage. At the same time, compile the write operation records at each moment of the entire fault phase into a write operation log and save it to the specified path of the normal object storage.

[0027] Furthermore, the specified path for normal object storage is specifically: a unique path in a storage bucket with a fixed bucket name, containing the operation time, the name of the operated bucket, and the sequence number.

[0028] S4. When the degraded object storage recovers from the failure, a sequential double-write operation is performed through the configuration center. The sequential double-write operation is as follows: first write to the normal object storage, then write to the degraded object storage, and read operations are only performed in the normal object storage.

[0029] S5. Based on the write operation log, perform asynchronous data synchronization operations on the degraded object storage according to the dual-read verification synchronization mechanism.

[0030] Furthermore, step S5 specifically includes: S51. Read the status of the file to be synchronized in the normal object storage during the fault period from the write operation log and its corresponding file to be synchronized, and use the status of the file to be synchronized as the comparison status.

[0031] S52. Copy the content and status of the file to be synchronized to the degraded object storage. If the status of the file to be synchronized is non-existent, delete the corresponding file to be synchronized in the degraded object storage.

[0032] S53. Reread the status of the file to be synchronized in the normal object storage. If it matches the comparison status, the synchronization process of the file to be synchronized ends. Otherwise, return to copy the content and status of the file to be synchronized to the degraded object storage. If the status of the file to be synchronized does not exist, delete the corresponding file to be synchronized in the degraded object storage (step S52) until it matches the comparison status.

[0033] In practical applications, if the comparison states are inconsistent, the reread state will be used as the new comparison state. The content and state of the file to be synchronized will then be copied to the degraded object storage. The state of the file to be synchronized in the normal object storage will then be read again. This process will be repeated until the copy operation is completed and the state of the file to be synchronized in the normal object storage is consistent with the comparison state.

[0034] S6. After the asynchronous data synchronization operation is completed, restore the configuration to the synchronous dual-write operation state of the multi-cloud heterogeneous object storage before the failure through the configuration center.

[0035] Furthermore, when the synchronous dual-write operation state is restored, the degradation flag of the degraded object storage is canceled.

[0036] The technical effects of this application are as follows: This application utilizes multi-cloud heterogeneous object storage and performs synchronous dual-write. When any storage fails, dual-write is suspended, the affected storage is marked as degraded, and reads and writes are only performed on the normal storage. All write operations during the failure period are compiled into a log and saved to a designated path on the normal storage. After recovery, sequential dual-write operations are performed. Based on the log, a dual-read verification mechanism is used to asynchronously synchronize the degraded storage to prevent old data from overwriting new data. After synchronization is complete, the original synchronous dual-write configuration is restored to achieve storage consistency. This application can reduce switchover and recovery time during object storage failures, achieving zero data loss; at the same time, it can prevent old data from overwriting new data, improving the reliability and disaster recovery capabilities of data storage.

[0037] Example 2: This example provides a computer system, which can be a server or a terminal, and its internal structure diagram can be as follows. Figure 2 As shown, the computer system includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores forced oscillation samples and sub / supersynchronous oscillation samples. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. The computer program, when executed by the processor, implements the aforementioned methods.

[0038] Those skilled in the art will understand that Figure 2 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer system to which the present application is applied. A specific computer system may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0039] Example 3: This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0040] Example 4: This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described above.

[0041] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0042] The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited thereto.

[0043] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0044] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for eventual consistency in multi-cloud heterogeneous object storage based on operation logs, characterized in that, The method includes: Configure multi-cloud heterogeneous object storage through the configuration center, and perform synchronous dual-write operations on the target data using each object storage; the synchronous dual-write operation is to perform write operations to each object storage simultaneously. When any object storage fails, the synchronous dual-write operation is suspended, and the failed object storage is marked as a degraded object storage after a degradation assessment. Stop reading and writing operations on degraded object storage, and only perform reading and writing operations on normal object storage. At the same time, compile the write operation records at each moment of the entire failure phase into a write operation log and save it to the specified path of normal object storage. Once the degraded object storage recovers from the failure, a sequential double-write operation is performed through the configuration center. The sequential double-write operation is as follows: first write to the normal object storage, then write to the degraded object storage, and read operations are only performed in the normal object storage. Based on the write operation log, asynchronous data synchronization operations are performed on the degraded object storage according to the dual-read verification synchronization mechanism. Once the asynchronous data synchronization operation is complete, the configuration will be restored to the state of synchronous dual-write operation of the multi-cloud heterogeneous object storage before the failure through the configuration center.

2. The method for eventual consistency of multi-cloud heterogeneous object storage based on operation logs according to claim 1, characterized in that, Configure multi-cloud heterogeneous object storage through the configuration center, and perform synchronous dual-write operations on the target data using each object storage system, specifically including: Configure multi-cloud heterogeneous object storage through the configuration center; Use the client-side proxy component to read the configuration from the configuration center; Perform synchronous dual-write operations on multiple heterogeneous object stores according to the configuration.

3. The method for eventual consistency of multi-cloud heterogeneous object storage based on operation logs according to claim 1, characterized in that, When any object storage fails, synchronous dual-write operations are suspended. After a degradation assessment is performed on the failed object storage, it is marked as a degraded object storage. Specifically, this includes: If any object storage fails, the synchronous dual-write operation is suspended. The circuit breaker mechanism is used to determine whether the object storage that has failed needs to be downgraded. If it is determined that downgrading is necessary, the configuration center is notified to mark the object storage that failed as downgraded. Otherwise, no marking is performed.

4. The method for eventual consistency of multi-cloud heterogeneous object storage based on operation logs according to claim 1, characterized in that, Based on the write operation logs, asynchronous data synchronization operations are performed on the degraded object storage using a dual-read verification synchronization mechanism, specifically including: Read the status of the files to be synchronized in the normal object storage during the fault period from the write operation log and its corresponding files to be synchronized, and use the status of the files to be synchronized as the comparison status. Copy the content and status of the file to be synchronized to the degraded object storage. If the status of the file to be synchronized is non-existent, delete the corresponding file to be synchronized in the degraded object storage. Reread the status of the file to be synchronized in the normal object storage. If it matches the comparison status, the synchronization process of the file to be synchronized ends. Otherwise, return to copy the content and status of the file to be synchronized to the degraded object storage. If the status of the file to be synchronized does not exist, delete the corresponding file to be synchronized in the degraded object storage until it matches the comparison status.

5. The method for eventual consistency of multi-cloud heterogeneous object storage based on operation logs according to claim 1, characterized in that, The object storage includes at least: Alibaba Cloud Object Storage and Tencent Cloud Object Storage.

6. The method for eventual consistency of multi-cloud heterogeneous object storage based on operation logs according to claim 1, characterized in that, The specified path for normal object storage is specifically: a unique path in a storage bucket with a fixed bucket name, containing the operation time, the name of the bucket being operated on, and the sequence number.

7. The method for eventual consistency of multi-cloud heterogeneous object storage based on operation logs according to claim 1, characterized in that, When the synchronous dual-write operation resumes, the degradation flag of the degraded object storage is canceled.

8. A computer system, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the eventual consistency method for multi-cloud heterogeneous object storage based on operation logs as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the eventual consistency method for multi-cloud heterogeneous object storage based on operation logs as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the eventual consistency method for multi-cloud heterogeneous object storage based on operation logs as described in any one of claims 1-7.

Images