Data bidirectional synchronization method and device, equipment and medium

Abstract

The embodiment of the invention discloses a data bidirectional synchronization method and device, equipment and a medium. The method comprises the following steps: receiving a data object operation request of a main cluster, and processing the main cluster based on the data object operation request to obtain to-be-synchronized object information; storing the to-be-synchronized object information to a first-level message queue; placing object information to be synchronized in a primary message queue into a secondary message queue based on the state of the main cluster; and synchronizing the to-be-synchronized object information in the secondary message queue to a standby cluster. According to the technical scheme, limitations such as bottom layer flow control and timing scanning in the prior art are avoided, the data synchronization speed is effectively increased, and the data missing condition of the main cluster and the standby cluster is prevented.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and medium for bidirectional data synchronization. Background Art

[0002] With the rapid development of internet technology, data volumes have exploded, ushering in the era of big data. Much of this exponentially growing data consists of unstructured data, such as images, videos, audio, and documents. Object storage, as a scalable, low-cost, and highly reliable cloud storage service, is particularly well-suited for storing unstructured data and is gradually becoming mainstream in the industry.

[0003] While existing object storage data synchronization technologies achieve one-way synchronization of primary and standby cluster data at the underlying layer, the underlying asynchronous data transmission is subject to flow control and is triggered by scheduled scans, resulting in slow data synchronization. If an exception occurs in the primary application and the standby application switches to access the standby cluster, some data may be lost and inaccessible, impacting upper-layer services. Summary of the Invention

[0004] The present invention provides a method, device, equipment and medium for bidirectional data synchronization, which avoids the limitations of the existing technology such as underlying flow control and timed scanning, effectively improves the data synchronization speed, and prevents data loss in the master-slave cluster.

[0005] According to one aspect of the present invention, a method for bidirectional data synchronization is provided, comprising:

[0006] receiving a data object operation request from a primary cluster, processing the primary cluster based on the data object operation request, and obtaining information of objects to be synchronized;

[0007] Storing the information of the object to be synchronized in the first-level message queue;

[0008] Placing the object information to be synchronized in the primary message queue into the secondary message queue based on the state of the primary cluster;

[0009] Synchronize the information of the objects to be synchronized in the secondary message queue to the standby cluster.

[0010] According to another aspect of the present invention, there is provided a bidirectional data synchronization device, comprising:

[0011] A data acquisition module, configured to receive a data object operation request from a primary cluster, process the primary cluster based on the data object operation request, and obtain information about objects to be synchronized;

[0012] A first-level queue storage module, used to store the information of the object to be synchronized in the first-level message queue;

[0013] A secondary queue storage module is used to place the information of the objects to be synchronized in the primary message queue into the secondary message queue based on the status of the primary cluster;

[0014] The data synchronization module is used to synchronize the information of the objects to be synchronized in the secondary message queue to the standby cluster.

[0015] According to another aspect of the present invention, an electronic device is provided, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to execute the bidirectional data synchronization method described in any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the bidirectional data synchronization method described in any embodiment of the present invention when executed.

[0020] The technical solution of an embodiment of the present invention receives a data object operation request from a primary cluster, processes the primary cluster based on the data object operation request, and obtains information about objects to be synchronized; stores the information about objects to be synchronized in a primary message queue; places the information about objects to be synchronized in the primary message queue into a secondary message queue based on the status of the primary cluster; and synchronizes the information about objects to be synchronized in the secondary message queue to the backup cluster. This technical solution avoids the limitations of existing technologies, such as underlying flow control and timed scanning, effectively improving data synchronization speed and preventing data loss between the primary and backup clusters.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1This is a flowchart of a bidirectional data synchronization method provided according to the first embodiment of the present invention;

[0024] Figure 2 This is a flow chart of a method for bidirectional data synchronization provided according to a second embodiment of the present invention;

[0025] Figure 3 This is a structural diagram of a bidirectional data synchronization device provided according to a third embodiment of the present invention;

[0026] Figure 4 It is a structural diagram of an electronic device provided according to the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Example 1

[0030] Figure 1 This is a flow chart of a method for bidirectional data synchronization provided according to the first embodiment of the present invention. This embodiment is applicable to bidirectional data synchronization in an object-oriented storage disaster recovery scenario. The method can be executed by a bidirectional data synchronization device. The bidirectional data synchronization device can be implemented in the form of hardware and / or software. The bidirectional data synchronization device can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0031] S110 : Receive a data object operation request from the primary cluster, process the primary cluster based on the data object operation request, and obtain information of objects to be synchronized.

[0032] This embodiment can be executed by a data bidirectional synchronization system in an object-oriented storage disaster recovery scenario. The system can be composed of four parts: a message production unit, a message placement unit, a message processing unit, and a data synchronization unit. Among them, the message production unit is used to put the object information that needs to be synchronized into the first-level message queue; the message placement unit is used to extract the message from the first-level message queue and place it into the second-level message queue; the message processing unit is used to process and parse the messages in the second-level consumer queue; the data synchronization unit concurrently consumes the messages in the second-level consumer queue to perform the actual data synchronization operation. In this embodiment, the object storage (Object Storage Service, OSS), also called object-based storage, has only two levels: data bucket (Bucket, equivalent to a directory or folder) and object (Object, equivalent to a file). The data bucket is a container for storing each object, and the structure inside the same data bucket is flat, without a directory hierarchy.

[0033] The data object operation request may be a request to perform a specific operation on a data object. In this embodiment, the data object operation request may include a request to upload or delete a data object. In this embodiment, the data object operation request may be initiated by an upper-layer consumer system through a primary application to the primary cluster, thereby accessing the primary cluster to perform an object upload or deletion operation. The object information to be synchronized can be understood as the object information obtained after performing a specific operation on a data object in the primary cluster via the data object operation request.

[0034] In this embodiment, the data bidirectional synchronization system receives a data object operation request initiated by the upper-layer consumption system to the main cluster through the main application, and then accesses the main cluster based on the data object operation request to perform object upload or deletion operations to obtain the object information to be synchronized.

[0035] S120: Store the information of the object to be synchronized in the first-level message queue.

[0036] Among them, the message queue is a technology used for communication between applications. Different applications can asynchronously transmit or receive messages through the message queue. Different types of message queues have different effects. In this embodiment, it can include a first-level message queue and a second-level message queue. Among them, the first-level message queue can be a heavyweight message queue that can store tens of millions of messages; the second-level message queue can be lightweight and can only store tens of thousands of messages, but with better performance. In this embodiment, the first-level message queue can include a "master-slave" first-level message queue and a "slave-master" first-level message queue. It can be understood that the "master-slave" first-level message queue is the message queue corresponding to the object to be synchronized from the master cluster to the backup cluster; the "slave-master" first-level message queue is the message queue corresponding to the object to be synchronized from the backup cluster to the master cluster.

[0037] In this embodiment, whether the status of the primary cluster is normal can determine whether the information of the object to be synchronized is stored in the "primary-backup" level 1 message queue or the "backup-primary" level 1 message queue. Furthermore, in this embodiment, if the status of the primary cluster is normal, the information of the object to be synchronized is stored in the "primary-backup" level 1 message queue.

[0038] S130: placing the information of the objects to be synchronized in the primary message queue into the secondary message queue based on the state of the primary cluster.

[0039] The status of the primary cluster may refer to the operating status of the primary cluster. The status of the primary cluster may include both normal and abnormal operating states of the primary cluster. The secondary message queue may consume messages from the corresponding message queue. It is understood that in this embodiment, the primary message object may be used to store tens of millions of data objects, and the secondary message queue may be used to process messages for the acquired data objects. The secondary message queue in this embodiment has storage space limitations, and therefore the queue length of the secondary message queue has corresponding limitations. In this embodiment, the secondary message queue may include a "primary-backup" secondary message queue and a "backup-primary" secondary message queue. It is understood that the "primary-backup" secondary message queue is the message queue corresponding to objects to be synchronized from the primary cluster to the backup cluster; the "backup-primary" secondary message queue is the message queue corresponding to objects to be synchronized from the backup cluster to the primary cluster. In this embodiment, based on the operating status of the primary cluster and the queue length limit requirements of the secondary message queue, the information on objects to be synchronized in the corresponding primary message queue may be placed in the secondary message queue.

[0040] S140: Synchronize the information of the objects to be synchronized in the secondary message queue to the standby cluster.

[0041] In this embodiment, when both the primary application and the primary cluster are operating normally, the secondary message objects can be consumed through the routing link from the primary object storage cluster to the backup object storage cluster, thereby synchronizing the object information to be synchronized in the secondary message queue to the backup cluster.

[0042] In this embodiment, optionally, the information of the object to be synchronized in the secondary message queue is synchronized to the backup cluster, including: locking the information of the object to be synchronized in the secondary message queue to obtain the locked information of the object to be synchronized; determining the position of the locked information of the object to be synchronized in the backup cluster and performing synchronization processing to complete synchronization to the backup cluster.

[0043] Locking involves locking the information about the object to be synchronized within the secondary message object, ensuring that only one consumer can process the information about a specific object at a time. Locking within the secondary message queue uses distributed or local lock mechanisms to serialize concurrent operations on the same object. The information about the object to be synchronized after locking can be the information about the object to be synchronized obtained after locking.

[0044] In this embodiment, in the secondary message queue, the purpose of locking the information of the object to be synchronized is to solve the data consistency problem during concurrent consumption and prevent multiple consumers or threads from processing the same object at the same time, causing conflicts (such as repeated operations or data overwriting, etc.). In this embodiment, the data synchronization unit consists of a concurrent controller and a data synchronizer. The concurrent controller is used to process multiple messages in multiple threads, and at the same time, to prevent contention for the same message, a locking process is performed; after the thread competes for the message processing lock, the data synchronizer performs the actual data synchronization between the master and standby clusters.

[0045] In this embodiment, the object information to be synchronized in the secondary message queue can be locked through a distributed lock or a local lock mechanism to obtain the locked object information to be synchronized. Then, the message is parsed to obtain the object data stream information to obtain the location of the locked object information to be synchronized in the standby cluster and copy the object information to be synchronized to the standby cluster, thereby performing synchronization processing to complete synchronization to the standby cluster.

[0046] Specifically, data synchronization in this embodiment can be performed through a thread pool. In this embodiment, the specific method of synchronizing the object information to be synchronized in the secondary message queue to the backup cluster may include: step 1, judging whether the thread pool is idle, if so, proceeding to step 2; otherwise, proceeding to step 3; step 2, locking the object information to be synchronized in the secondary message queue, and judging whether the locking is successful through the code function, if successful, proceeding to step 4; otherwise, ending; step 3, putting the message back into the secondary message queue, ending; step 4, judging whether it is a synchronization for deleting data, if not, proceeding to step 6; otherwise, proceeding to step 7; step 5, locating the location of the data in the source cluster according to message parsing; step 6, obtaining the object data stream information from the source object storage cluster, transmitting it to the destination object storage cluster, entering step 8; step 7, locating the location of the data in the destination cluster according to message parsing, and deleting the data; step 8, releasing the lock, ending.

[0047] Through such a setting in this embodiment, bidirectional synchronization of data deletion operations in the object storage master and backup clusters can be achieved at the application layer, solving the problem of data synchronization failure in some scenarios of the existing technology, and effectively adapting to the scenario of object storage disaster recovery construction.

[0048] The technical solution of an embodiment of the present invention receives data object operation requests from the primary cluster, processes the primary cluster based on the data object operation requests, obtains information about objects to be synchronized, stores the information in a primary message queue, places the information in the primary message queue into a secondary message queue based on the status of the primary cluster, and synchronizes the information in the secondary message queue to the backup cluster. This technical solution avoids the limitations of existing technologies, such as underlying flow control and timed scanning, effectively improves data synchronization speed, and prevents data loss between the primary and backup clusters.

[0049] Example 2

[0050] Figure 2 This is a flow chart of a bidirectional data synchronization method provided according to the second embodiment of the present invention. This embodiment is optimized based on the above embodiment. The specific optimization is as follows: the first-level message queue includes the first message queue and the second message queue; accordingly, the information of the object to be synchronized is stored in the first-level message queue, including: judging whether the main cluster is in normal working state; if the main cluster is in normal working state, the information of the object to be synchronized is placed in the first message queue. Figure 2 As shown, the method includes:

[0051] S210: Receive a data object operation request from the primary cluster, process the primary cluster based on the data object operation request, and obtain information of objects to be synchronized.

[0052] In this embodiment, the first-level message queue may include a first message queue and a second message queue; wherein the first message queue may be a "master-backup" first-level message queue, and the second message queue may be a "backup-master" first-level message queue.

[0053] S220: Determine whether the main cluster is in a normal working state.

[0054] In this embodiment, a flag is stored in the primary cluster's storage database. Based on this flag, it can be used to determine whether the primary cluster is operating normally. As can be understood, in this embodiment, the flag is a field. Normal indicates a normal field state. If the primary cluster switches, the field will switch to an abnormal or switched state.

[0055] S230: If the main cluster is in a normal working state, the information of the object to be synchronized is put into the first message queue.

[0056] In this embodiment, if the master cluster is in a normal working state, the information of the object to be synchronized may be put into the first message queue, that is, the "master-backup" level one message queue.

[0057] In this embodiment, the message production unit is composed of a data accessor and a first-level message queue processor. The data accessor is used to process the data upload or deletion operation request of the consumer system to the application; the first-level message queue processor puts the message generated after the consumer system data operation is successful into the correct first-level message queue. Specifically, in this embodiment, the specific method of storing the object information to be synchronized into the first-level message queue can be: Step 1, the consumer system accesses the application to upload or delete the object. If the operation is successful, go to step 2; otherwise, end directly; Step 2, determine whether the data synchronization switch is turned on. If not, end directly; otherwise, go to step 3; Step 3, determine whether the main application and the main cluster are normal. If both are normal, go to step 4; otherwise, go to step 5; Step 4, put the object information to be synchronized into the "master-slave" first-level message queue and end; Step 5, determine whether the backup application and the main cluster are normal. If both are normal, go to step 6; otherwise, end; Step 6, put the object information to be synchronized into the "slave-master" first-level message queue and end.

[0058] In this embodiment, optionally, it also includes: if the main cluster is in an abnormal working state, storing the information of the object to be synchronized in the second message queue; placing the information of the object to be synchronized in the second message queue into the secondary message queue based on the state of the backup cluster; and synchronizing the information of the object to be synchronized in the secondary message queue to the main cluster.

[0059] Among them, the abnormal working state can be considered as the state in which the current main cluster has been switched to the backup cluster due to a failure or other reasons. In this embodiment, the abnormal working state may refer to the situation where the application failure cannot be accessed normally, and the subsequent problems caused. In this embodiment, if the main cluster is in an abnormal working state, it will continue to determine whether the backup cluster is in a normal working state. If the cluster is in a normal working state, the information of the object to be synchronized can be placed in the second message queue, that is, the "backup-master" first-level message queue. In this embodiment, after the information of the object to be synchronized is stored in the second message queue, the information of the object to be synchronized in the corresponding second message queue can be placed in the secondary message queue, that is, the "backup-master" secondary message queue, based on the working state of the backup cluster and the queue length limit requirements of the secondary message queue. Then, when the backup application and the backup cluster are working normally, the secondary message object is consumed through the routing link from the backup object storage cluster to the primary object storage cluster, so that the information of the object to be synchronized in the secondary message queue is synchronized to the main cluster.

[0060] Specifically, in this embodiment, when the primary application and the primary object storage cluster are functioning normally, the upper-layer consumer system accesses the primary cluster through the primary application to upload or delete objects. If the data operation request is successful, the primary application synchronizes data with the backup object storage cluster. If a production failure occurs in the platform's primary application, the upper-layer consumer system accesses the backup cluster through the backup application to perform object operations. If the operation is successful, the backup application synchronizes data with the primary object storage cluster. When the primary application returns to normal, the upper-layer consumer system re-accesses the primary cluster through the primary application.

[0061] In this embodiment, such a setting is used to achieve bidirectional data synchronization in the object storage disaster recovery construction scenario, significantly reduce the RPO, and ensure the continuity of upper-layer services.

[0062] S240: placing the information of the objects to be synchronized in the primary message queue into the secondary message queue based on the state of the primary cluster.

[0063] In this embodiment, optionally, the information of the object to be synchronized in the primary message queue is placed in the secondary message queue based on the status of the primary cluster, including: when the working status of the primary cluster is normal, obtaining the information of the object to be synchronized from the primary message queue; determining the current queue length of the secondary message queue; and placing the information of the object to be synchronized in the secondary message queue according to the current queue length.

[0064] The information about the object to be synchronized may include information such as the application system number, data bucket number, and object number. The current queue length can be understood as the length and capacity of the data objects currently stored in the secondary message queue. In this embodiment, the current queue length of the secondary message queue can be determined to determine whether the secondary message queue still has capacity to store the data objects to be synchronized in the primary message queue.

[0065] In this embodiment, it is possible to first determine whether the working status of the main cluster is normal. Then, if the working status of the main cluster is normal, obtain the object message from the "master-backup" first-level message queue in the first-level message queue, and then determine the current queue length of the secondary message queue. According to the current queue length of the secondary message queue, the length of the data objects that can be stored in the current secondary message queue is determined, and then the object information to be synchronized is placed in the secondary message queue according to the length of the data objects that can be stored in the current secondary message queue.

[0066] In this embodiment, through such a setting, a method for bidirectional replication between clusters can be implemented based on the primary message queue messages and the secondary message queue of the application and cluster status.

[0067] In this embodiment, optionally, the object information to be synchronized is placed in the secondary message queue based on the current queue length, including: determining whether the current queue length reaches a set threshold; if the current queue length does not reach the set threshold, placing the object information to be synchronized in the secondary message queue.

[0068] The set threshold value may be a pre-set threshold value. The set threshold value in this embodiment is a threshold value set for the current queue length and may be set according to actual needs. In this embodiment, it is determined whether the current queue length has reached the set threshold value; if the current queue length has not reached the set threshold value, that is, there is still space available to store data objects, the information of the object to be synchronized may be placed in the secondary message queue.

[0069] In this embodiment, by such a setting, the length of the secondary message queue is limited, and the information of the object to be synchronized is obtained based on the corresponding length limit, thereby ensuring the efficiency and high efficiency of real-time data processing.

[0070] In this embodiment, optionally, it also includes: if the current queue length reaches the set threshold, then after waiting for a set time, it is determined again whether the current queue length of the secondary message queue reaches the set threshold until the object information to be synchronized is placed in the secondary message queue.

[0071] The set duration can be pre-set duration information. In this embodiment, the set duration can be 5 seconds, and can also be set according to actual needs.

[0072] In this embodiment, if the current queue length reaches the set threshold, that is, the current secondary message queue has no space to store data objects, it is necessary to wait for the set time and then determine again whether the current queue length of the secondary message queue reaches the set threshold until the secondary message queue has storage space, so that the object information to be synchronized can be placed in the secondary message queue.

[0073] It can be understood that the data in the secondary message queue in this embodiment is always being processed in a real-time loop through the task control of the thread pool. Therefore, after determining that the current secondary message queue has no space to store data objects, after waiting for a certain period of time, there will be space to store data objects in the secondary message queue.

[0074] Through such a setting in this embodiment, the data objects of the first-level message queue can be read and processed when there is storage space in the second-level message queue, avoiding the limitations of the underlying flow control, timed scanning, etc. of the existing technology, effectively improving the data synchronization speed, and greatly reducing the RPO of the active-standby cluster.

[0075] In this embodiment, messages in the primary message queue are taken out and placed into the corresponding secondary message queue. A thread pool is used to scan and read the data in the primary message queue in real time. However, since the capacity of the secondary message queue is limited, if the data capacity in the secondary message queue is full, no more data objects will be placed. It is necessary to wait for the secondary message queue to be idle before continuing to place more data objects. It can be understood that in this embodiment, the secondary message queue is processed in real time, and then the data read in real time from the primary message queue is placed into the secondary message queue. The secondary message queue processes the data in real time and then continues to read the data.

[0076] In this embodiment, the message placement unit is composed of a message placer, which is used to take out the message from the first-level message queue and place it into the corresponding second-level message queue. Specifically, in this embodiment, the specific operation steps of placing the information of the object to be synchronized in the first-level message queue into the second-level message queue based on the status of the main cluster include: step 1, judging whether the main application and the main cluster are both normal, if both are normal, proceeding to step 2; otherwise, proceeding to step 3; step 2, obtaining the information of the object to be synchronized from the "master-slave" first-level message queue, the message content including the application system number, data bucket number, object number, etc., proceeding to step 5; step 3, judging whether the backup application and the main cluster are both normal, if both are normal, proceeding to step 4; otherwise, directly ending; step 4, obtaining the message of the object to be synchronized from the "slave-master" first-level message queue, proceeding to step 5; step 5, judging whether the storage capacity of the corresponding second-level message queue is full, if the storage capacity is full, proceeding to step 6; otherwise, proceeding to step 7; step 6, sleeping for 5 seconds and then re-entering step 5; step 7, placing the information of the object to be synchronized into the second-level message queue, and ending.

[0077] Furthermore, in this embodiment, the message processing unit consists of a link resolver and a secondary message queue processor. The link resolver determines the specific object storage cluster link based on the application and object storage cluster status and establishes the corresponding object storage connection. After link resolution is established, the secondary message queue processor consumes messages from the corresponding message queue. The message processing unit workflow may include: Step 1, determining whether the primary application and the primary cluster are both operating normally, if so, proceeding to Step 2; otherwise, proceeding to Step 4; Step 2, determining whether a corresponding link from the primary object storage cluster to the backup object storage cluster already exists in the cache, if so, proceeding to Step 7; otherwise, proceeding to Step 3; Step 3, creating a routing link from the primary object storage cluster to the backup object storage cluster, and proceeding to Step 7; Step 4, determining whether the backup application and the primary cluster are both operating normally, if not, terminating; otherwise, proceeding to Step 5; Step 5, determining whether a corresponding link from the backup object storage cluster to the primary object storage cluster already exists in the cache, if so, proceeding to Step 8; otherwise, proceeding to Step 6; Step 6, creating a routing link from the backup object storage cluster to the primary object storage cluster, and proceeding to Step 8; Step 7, consuming the "primary-backup" secondary message queue, and terminating; Step 8, consuming the "backup-primary" secondary message queue, and terminating. The object storage cluster status may refer to the specific access status of the object storage cluster, thereby determining whether a routing link is required to access the "primary-backup" secondary message queue or the "backup-primary" secondary message queue.

[0078] S250: Synchronize the information of the objects to be synchronized in the secondary message queue to the standby cluster.

[0079] The solution of this embodiment implements bidirectional near-real-time synchronization of object storage master and backup cluster data at the application layer, effectively shortening the RPO, minimizing the impact of application failures, and ensuring the data integrity of the underlying object storage cluster and the business continuity of the upper-layer consumer system.

[0080] The technical solution of the embodiment of the present invention receives a data object operation request from the primary cluster, processes the primary cluster based on the data object operation request, and obtains the object information to be synchronized; the first-level message queue includes a first message queue and a second message queue; determines whether the primary cluster is in normal working state; if the primary cluster is in normal working state, puts the object information to be synchronized into the first message queue; puts the object information to be synchronized in the first-level message queue into the second-level message queue based on the state of the primary cluster; and synchronizes the object information to be synchronized in the second-level message queue to the backup cluster. This technical solution avoids the limitations of the underlying flow control, timed scanning, etc. of the prior art, effectively improves the data synchronization speed, and prevents data loss in the primary and backup clusters. This embodiment realizes bidirectional synchronization of data deletion operations of the object storage primary and backup clusters at the application layer, solves the problem that some scenarios of the prior art cannot synchronize data, and is effectively adapted to the scenario of object storage disaster recovery construction.

[0081] Example 3

[0082] Figure 3 1 is a schematic diagram of the structure of a bidirectional data synchronization device provided according to the third embodiment of the present invention. Figure 3 As shown, the device includes:

[0083] The data acquisition module 310 is configured to receive a data object operation request from the primary cluster, process the primary cluster based on the data object operation request, and obtain information about objects to be synchronized;

[0084] The first-level queue storage module 320 is used to store the information of the object to be synchronized in the first-level message queue;

[0085] The secondary queue storage module 330 is used to place the information of the objects to be synchronized in the primary message queue into the secondary message queue based on the status of the primary cluster;

[0086] The data synchronization module 340 is used to synchronize the information of the objects to be synchronized in the secondary message queue to the standby cluster.

[0087] Optionally, the first-level message queue includes a first message queue and a second message queue;

[0088] Accordingly, the first-level queue storage module 320 is specifically configured to:

[0089] Determine whether the main cluster is in normal working status;

[0090] If the main cluster is in normal working state, the information of the object to be synchronized is put into the first message queue.

[0091] Optionally, the first-level queue storage module 320 is further configured to:

[0092] If the main cluster is in an abnormal working state, the information of the object to be synchronized is stored in the second message queue;

[0093] Based on the status of the standby cluster, the information of the objects to be synchronized in the second message queue is placed in the secondary message queue;

[0094] Synchronize the information of objects to be synchronized in the secondary message queue to the main cluster.

[0095] Optionally, the secondary queue storage module 330 includes:

[0096] An information acquisition unit is used to obtain information about objects to be synchronized from the primary message queue when the main cluster is in normal working condition;

[0097] A queue length determination unit, used to determine the current queue length of the secondary message queue;

[0098] The information placement unit is used to place the information of the object to be synchronized into the secondary message queue according to the current queue length.

[0099] Optional information placement unit, specifically used for:

[0100] Determine whether the current queue length reaches the set threshold;

[0101] If the current queue length does not reach the set threshold, the information of the object to be synchronized will be placed in the secondary message queue.

[0102] Optionally, the information placement unit is further used to:

[0103] If the current queue length reaches the set threshold, the system will wait for a set period of time and then determine again whether the current queue length of the secondary message queue reaches the set threshold, until the object information to be synchronized is placed in the secondary message queue.

[0104] Optionally, the data synchronization module 340 is specifically configured to:

[0105] Lock the information of the object to be synchronized in the secondary message queue to obtain the locked information of the object to be synchronized;

[0106] The location of the locked object information in the standby cluster is determined and synchronization is performed to complete synchronization to the standby cluster.

[0107] A bidirectional data synchronization device provided by an embodiment of the present invention can execute a bidirectional data synchronization method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects of the execution method.

[0108] Example 4

[0109] Figure 41 is a schematic diagram of the structure of an electronic device provided according to embodiment four of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0110] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0111] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0112] The processor 11 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 performs the various methods and processes described above, such as the data bidirectional synchronization method. The data bidirectional synchronization method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data bidirectional synchronization method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the data bidirectional synchronization method in any other appropriate manner (e.g., by means of firmware).

[0113] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0115] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0116] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0117] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0118] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0119] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0120] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for bidirectional data synchronization, characterized in that: include: receiving a data object operation request from a primary cluster, processing the primary cluster based on the data object operation request, and obtaining information of objects to be synchronized; Storing the information of the object to be synchronized in the first-level message queue; Placing the object information to be synchronized in the primary message queue into the secondary message queue based on the state of the primary cluster; Synchronize the information of the objects to be synchronized in the secondary message queue to the standby cluster.

2. The method according to claim 1, characterized in that The first-level message queue includes a first message queue and a second message queue; Accordingly, the information of the object to be synchronized is stored in the first-level message queue, including: Determine whether the main cluster is in normal working state; If the main cluster is in a normal working state, the information of the object to be synchronized is put into the first message queue.

3. The method according to claim 2, characterized in that Also includes: If the main cluster is in an abnormal working state, storing the information of the object to be synchronized in the second message queue; placing the information of the object to be synchronized in the second message queue into the secondary message queue based on the state of the standby cluster; Synchronize the information of the objects to be synchronized in the secondary message queue to the main cluster.

4. The method according to claim 1, wherein Placing the object information to be synchronized in the primary message queue to the secondary message queue based on the state of the primary cluster includes: When the working status of the main cluster is normal, obtaining information of the object to be synchronized from the first-level message queue; Determining the current queue length of the secondary message queue; The object information to be synchronized is placed in the secondary message queue according to the current queue length.

5. The method according to claim 4, characterized in that Placing the object information to be synchronized into the secondary message queue according to the current queue length, including: Determine whether the current queue length reaches a set threshold; If the current queue length does not reach the set threshold, the object information to be synchronized is placed in the secondary message queue.

6. The method according to claim 5, characterized in that Also includes: If the current queue length reaches the set threshold, the process waits for a set period of time and then determines again whether the current queue length of the secondary message queue reaches the set threshold, until the object information to be synchronized is placed in the secondary message queue.

7. The method according to claim 1, characterized in that Synchronizing the information of the objects to be synchronized in the secondary message queue to the standby cluster includes: Locking the information of the object to be synchronized in the secondary message queue to obtain the locked information of the object to be synchronized; The location of the locked object information in the standby cluster is determined and synchronization is performed to complete synchronization to the standby cluster.

8. A bidirectional data synchronization device, characterized in that: include: A data acquisition module, configured to receive a data object operation request from a primary cluster, process the primary cluster based on the data object operation request, and obtain information about objects to be synchronized; A first-level queue storage module, used to store the information of the object to be synchronized in the first-level message queue; A secondary queue storage module is used to place the information of the objects to be synchronized in the primary message queue into the secondary message queue based on the status of the primary cluster; The data synchronization module is used to synchronize the information of the objects to be synchronized in the secondary message queue to the standby cluster.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the bidirectional data synchronization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the bidirectional data synchronization method according to any one of claims 1 to 7 when executed.

Images