A system and method for implementing minio real-time disaster recovery

Abstract

The application provides a system and method for realizing MinIO real-time disaster recovery, and belongs to the technical field of data storage. The system for realizing MinIO real-time disaster recovery comprises a connection processing module, a full-data comparison module, an event listening module, a bucket event processing module and an object event processing module. The connection processing module is used for maintaining the connection of a disaster recovery service with a source MinIO service and a disaster recovery MinIO service, ensuring that the connection of the disaster recovery service with the services at both ends is normal and available, and providing the connection for other modules to obtain. The application realizes real-time backup of the MinIO service, avoids the loss of all data when more than half of the MinIO storage nodes suffer physical damage, and prevents a disastrous production accident. Meanwhile, the application can select equal deployment or unequal deployment, reasonably utilizes resources, and when a fault occurs at the source end, can quickly switch to the disaster recovery end MinIO for use, thereby ensuring the robustness and high availability of the service.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data storage, and particularly relates to a system and method for realizing real-time disaster recovery of MinIO. BACKGROUND

[0002] MinIO is a high-performance, distributed object storage system. Unlike traditional storage and other object storage, it is designed for higher performance private cloud standards from the beginning, and is designed in a more user-friendly way to achieve all the functions required by object storage. It is also more powerful in performance and does not compromise usability and efficiency for more business functions. It can more simply realize the elastic scaling capability of native object storage services.

[0003] To ensure the high availability of MinIO, MinIO can be deployed in a distributed deployment mode, in which MinIO uses a distributed algorithm to ensure high availability and data redundancy. MinIO uses Reed-Solomon code to split objects into N / 2 data and N / 2 parity blocks. This means that if there are 12 disks, an object will be divided into 6 data blocks and 6 parity blocks. In the case of losing any 6 disks (whether they are data blocks or parity blocks), the data can still be recovered from the remaining disks. When a node is down, other nodes can automatically take over the data blocks of the node and continue to provide services, and the node will automatically recover data after it is back online, thereby ensuring the high availability of the service.

[0004] However, when more than half of the MinIO storage nodes are physically damaged, the entire MinIO instance will become unavailable, resulting in the loss of all data and causing a disastrous production accident. When real-time synchronization of MinIO services is needed for disaster recovery, MinIO provides MinIO Client, which can realize real-time synchronization of data to the disaster recovery MinIO. However, when MinIO Client performs file parallel writing, inconsistencies between the source data and the disaster recovery data may occur due to network reasons. For example, if the source MinIO writes the A.pdf file twice, the two changes of MinIO Client will be synchronized to the disaster recovery MinIO. However, it cannot be guaranteed that the final file version of the disaster recovery MinIO is the A.pdf file uploaded the second time. Instead, it depends on which file is uploaded later when MinIO Client synchronizes twice. If the second upload is completed later than the first upload, it will cause data inconsistency. SUMMARY

[0005] The application aims to provide a system and method for realizing MinIO real-time disaster recovery, aiming to solve the problem that in the prior art, when more than half of the nodes of a MinIO service encounter physical damage, the instance becomes unavailable, leading to a disastrous production accident.

[0006] To achieve the above-mentioned purpose, the application provides the following technical solutions.

[0007] A system for realizing MinIO real-time disaster recovery comprises a connection processing module, a full-amount data comparison module, an event listening module, a bucket event processing module, and an object event processing module. The connection processing module is used to maintain the connection of a disaster recovery service with a source MinIO service and a disaster recovery MinIO service, to ensure that the connection of the disaster recovery service with the two services is normally available, and to provide the connection for other modules to obtain the connection. The full-amount data comparison module is used to compare the bucket data and object data of the MinIO service on the source end and the disaster recovery end when initiating disaster recovery. The event listening module is used to listen to the bucket events and object events of the source MinIO service in real time. The object event processing module is used to process the object event synchronization tasks submitted by the event listening module and the full-amount data comparison module in parallel.

[0008] As a preferred scheme of the application, the priority of the bucket data is higher than that of the object data when the full-amount data comparison module compares the bucket data and the object data.

[0009] As a preferred scheme of the application, the event listening module is further used to convert the bucket event content into a task of the bucket event processing module and submit the task to the bucket event processing module, and convert the object event content into a task of the object event processing module and submit the task to the object event processing module.

[0010] A method for realizing MinIO real-time disaster recovery comprises a data synchronization process, a data comparison execution process, an object synchronization event processing process, and a thread execution task processing process.

[0011] As a preferred scheme of the application, the data synchronization process specifically comprises the following steps.

[0012] Step S11: A user applies to initiate a MinIO synchronization task, and submits source MinIO service information and disaster recovery end MinIO service information.

[0013] Step S12: A synchronization service acquires a Client object of the source MinIO with event listening through an SDK according to the source MinIO service information.

[0014] Step S13: The synchronization service acquires a Client object of the disaster recovery end MinIO through an SDK according to the disaster recovery end MinIO service information.

[0015] Step S14, through the listBuckets method of the SDK, all bucket information of the source MinIO service and the disaster recovery end MinIO service is respectively acquired.

[0016] Step S15, the bucket information of the source MinIO service is circularly traversed, and the bucket information of the disaster recovery end MinIO is compared, and the bucket data existing in the source end MinIO and not existing in the disaster recovery end MinIO is found out;

[0017] Step S16, the makeBucket method of the SDK is used to create the bucket data existing in the source end MinIO and not existing in the disaster recovery end MinIO in the disaster recovery end MinIO;

[0018] Step S17, the event listening of the source MinIO service is started (bucket event listening + object event listening) ;

[0019] Step S18, the existing object difference comparison and synchronization task of the source MinIO service and the disaster recovery end MinIO service is synchronously performed, and the listening event object synchronization task is synchronously performed, at this time, the full data synchronization and the incremental data synchronization are simultaneously performed in the synchronization service, and the data consistency of the source end MinIO and the disaster recovery end MinIO is guaranteed;

[0020] Step S19, the synchronization queue continuously consumes the message in the queue, and according to the object information, the object data is acquired from the source MinIO service, and the data is synchronized to the disaster recovery end MinIO service.

[0021] As a preferred scheme of the application, in the object difference comparison and synchronization of the source MinIO service and the disaster recovery end MinIO service in the step S18, the listObjects() method of the SDK is used to respectively acquire all object data of the source end and the disaster recovery end; all objects of the source end are circularly compared according to the information of the object of the source end; when the object does not exist in the disaster recovery end or the object data is inconsistent, the information of the object is pressed into the task queue of the synchronization service to be synchronized.

[0022] As a preferred scheme of the application, the data comparison execution process specifically includes the following steps:

[0023] Step S21, when the source end MinIO object data is traversed, the object meta information of the source end is acquired;

[0024] Step S22, the object of the disaster recovery end MinIO is traversed, and the object meta information is acquired;

[0025] Step S23, the object data of the disaster recovery end with the same name is taken as the reference of the source end object:

[0026] Step S231, if the object with the same name exists, then continue to compare.

[0027] Step S232, if there is no same name object, it means that the disaster end does not have the information of the current object of the source end, and the object data of the current source end is pushed into the object synchronization queue;

[0028] Step S24, if the source end object does not have the version number and the disaster end has the same name object with the modification time greater than that of the source end object, it is considered that the object currently processed by the source end is the previous version of the object of the disaster end, that is, the current processed object does not need to be synchronized to the disaster end; if it is not the case, the judgment continues;

[0029] Step S25, if the disaster end object satisfies the condition, it is considered that it is the same object and the current source end object does not need to be synchronized;

[0030] Step S26, if it does not satisfy the condition in step S25, if the source end has the version number and it is judged that it is not the same object, the disaster end has the object data of the version number, at this time, the abnormal processing strategy is triggered.

[0031] As a preferred scheme of the present application, in the step S25 of judging whether the disaster end object satisfies the condition, the condition includes the conditions of equal size, equal user metadata and equal object modification time.

[0032] As a preferred scheme of the present application, the object synchronization event processing flow specifically includes the following steps:

[0033] Step S31, the queue data is derived from the object event received from the full synchronization and incremental event step;

[0034] Step S32, it is judged whether the thread pool for processing the object event has been created, if not, the thread pool for processing the object event is first created, the thread pool queue selects ArrayBlockingQueue, the rejection policy selects CallerRunsPolicy, when the thread pool is full, the thread for event listening is used to execute the object event, and the processing efficiency is improved;

[0035] Step S33, it is judged whether the current object has the version number:

[0036] Step S331, if there is the version number, the task is directly submitted to the thread pool, and the flow is ended;

[0037] Step S332, if there is no version number: it is judged whether the same type of task has been submitted: if the same type of task has been submitted, the current event information is added to the waiting queue in the ConcurrentHashMap, and the flow is ended; if not, the task is submitted to the thread pool, and after the information of the current event is submitted to the ConcurrentHashMap, the flow is ended.

[0038] As a preferred scheme of the application, the:

[0039] Step S41, the thread pool starts to continuously process the line synchronization task;

[0040] Step S42, after consuming an object synchronization task, it is judged whether the current object has a version number:

[0041] Step S421, if there is a version number, the current task processing is completed and the flow is ended;

[0042] Step S422, if there is no version number, it is judged whether there is a waiting queue with the same name in ConcurrentHashMap:

[0043] Step S4221, if there is no waiting queue, the current task processing is completed and the flow is ended;

[0044] Step S4222, if there is a waiting queue, the object synchronization task with the latest time is taken into the flow recursion until all the tasks in the waiting queue are executed, and the flow is ended.

[0045] Compared with the prior art, the application has the beneficial effects that:

[0046] The application realizes real-time backup of MinIO service, avoids all data loss when more than half of MinIO storage nodes are physically damaged, and causes a difficult production accident; at the same time, the problem of data inconsistency caused by MinIO Client parallel upload is solved; the application can be deployed in a peer-to-peer manner or not, resources are reasonably utilized, and when a source end fails, the application can quickly switch to a disaster recovery end MinIO for use, thereby ensuring service robustness and high availability. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the application, and constitute a part of the specification, together with the embodiments of the application, to explain the application, and do not constitute a limitation on the application. In the drawings:

[0048] Figure 1 A flow chart for monitoring real-time disaster recovery MinIO service in the system and method for realizing real-time disaster recovery of MinIO of the application;

[0049] Figure 2 A data synchronization flowchart for realizing real-time disaster recovery of MinIO of the application;

[0050] Figure 3 A data comparison execution flowchart for realizing real-time disaster recovery of MinIO of the application;

[0051] Figure 4 FIG. 1 is a schematic diagram of an object synchronization event processing flow in a system and method for realizing real-time disaster recovery of MinIO according to the present application;

[0052] Figure 5 FIG. 2 is a schematic diagram of a thread execution task processing flow in a system and method for realizing real-time disaster recovery of MinIO according to the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0054] Embodiment 1

[0055] Referring to Figures 1-5 The present application provides the following technical solutions:

[0056] A system for realizing real-time disaster recovery of MinIO comprises a connection processing module, a full-data comparison module, an event listening module, a bucket event processing module and an object event processing module. The connection processing module is used to maintain the connection of the disaster recovery service with the source MinIO service and the disaster recovery MinIO service, to ensure that the connection of the disaster recovery service with the two services is normal and available, and to provide the connection for other modules. The full-data comparison module is used to compare the bucket data and object data of the MinIO service on the source side and the disaster recovery side when initiating disaster recovery. The event listening module is used to listen to the bucket events and object events of the source MinIO service in real time. The object event processing module is used to process the object event synchronization tasks submitted by the event listening module and the full-data comparison module in parallel.

[0057] Specifically, the priority of the bucket data is higher than that of the object data when the full-data comparison module compares the bucket data and the object data.

[0058] Specifically, the event listening module is further used to convert the content of the bucket events into tasks of the bucket event processing module and submit the tasks to the bucket event processing module, and convert the content of the object events into tasks of the object event processing module and submit the tasks to the object event processing module.

[0059] In the specific embodiments of the present application, the SDK is used to listen to the bucket events and object events of the source MinIO service in real time, and the contents of the events are reproduced on the MinIO service of the disaster recovery side through the query, download, upload and deletion methods of the SDK, so as to realize the real-time disaster recovery of the MinIO service.

[0060] This system mainly consists of five modules: connection processing module, full data comparison module, event listening module, bucket event processing module, and object event processing module.

[0061] The connection processing module is responsible for maintaining the connection between the disaster recovery service and the source MinIO service and the disaster recovery MinIO service, ensuring that the connection between the disaster recovery service and the services at both ends is normal and available, so that other modules can obtain the connection and call the SDK methods to process data in the MinIO service.

[0062] The full data comparison module is responsible for performing a full comparison of existing bucket and object data in the MinIO services on both the source and backup ends when initiating disaster recovery. First, it compares the bucket data; if inconsistencies are found, the bucket data is synchronized to the backup end. After completing the full bucket data comparison, the incremental synchronization event listening module is activated before performing a full comparison of the object data to ensure that no change events from the source MinIO are lost during the full object data comparison (both full and incremental synchronization are performed simultaneously). During the full object data comparison, the module checks if the object's name, version number, last modified time, size, and userMetadata are the same. If all are the same, the object data does not need to be synchronized; if any one of these is inconsistent, the object data synchronization task is submitted to the object synchronization event handling module.

[0063] The event listening module is responsible for real-time monitoring of bucket events (create, delete, modify) and object events (create, delete, modify) of the source MinIO service, and submitting the bucket event content to the bucket event handling module and the object event content to the object event handling module.

[0064] The bucket event handling module is responsible for processing bucket event synchronization tasks submitted by the event listening module in parallel. To improve data synchronization efficiency, a bucket event handling thread pool is used, with multiple threads synchronizing data in parallel. The thread pool queue is selected from ArrayBlockingQueue (thread-safe and bounded), and the rejection policy is CallerRunsPolicy (when the thread pool is full, the submitted thread is used to execute the task).

[0065] The object event handling module is responsible for processing object event synchronization tasks submitted by the event listening module and the full data comparison module in parallel. To improve data synchronization efficiency, an object event handling thread pool is used for multi-threaded parallel data synchronization. The thread pool queue is selected as ArrayBlockingQueue (thread-safe and bounded), and the rejection policy is CallerRunsPolicy (when the thread pool is full, the submitted thread is used to execute the task).

[0066] Specifically, a method for implementing MinIO real-time disaster recovery, the method comprising: a data synchronization process, a data comparison execution process, an object synchronization event processing process, and a thread execution task processing process.

[0067] In this embodiment: since it is a multi-threaded processing object data synchronization, in order to avoid the problem of data inconsistency caused by parallel modification described in the background, the object processing thread in this scheme is designed as follows: when the source MinIO service has opened the object version number, since there is a version number as an identifier, there is no problem of data inconsistency (synchronization version number); when the source MinIO service does not open the object version number, a ConcurrentHashMap (key is synchronization data id + source bucket name + source object name) is used to save the key of the executing unopened version number and the corresponding task list, and the program cannot operate this ConcurrentHashMap in parallel, and reading, modifying and deleting can only be executed in series. When the thread takes the synchronization task of the unopened version number, the key value of the current object data is taken, and it is first judged whether the ConcurrentHashMap has the data of the key, if yes, the current task is added to the task list corresponding to the key, and if not, a data of the key is added to the ConcurrentHashMap; when the thread pool executes the task, if it is a synchronization task of the unopened version number, after the current task is executed, it will try to get data from the ConcurrentHashMap with the key corresponding to the current task, if the data is obtained, the value of the key in the ConcurrentHashMap is cleared, the current thread continues to execute the task list obtained, and after the list obtained is executed, the data is again tried to be obtained from the ConcurrentHashMap with the key corresponding to the current task, and so on, until the data of the key cannot be obtained, and the current synchronization task of the object is completed. In this way, the problem of data inconsistency caused by concurrent uploading is solved.

[0068] For details, please refer to Figure 2 When initiating a synchronization task, the synchronization process (including bucket data synchronization processing) is as follows:

[0069] Step S11, a user applies to initiate a MinIO synchronization task, and submits source MinIO service information (login URL, account, password), disaster recovery end MinIO service information (login URL, account, password);

[0070] Step S12, the synchronization service obtains a Client object with event listening of the source MinIO according to the source MinIO service information (login URL, account, password) through the SDK;

[0071] Step S13, the synchronization service obtains the Client object of the disaster recovery end MinIO according to the disaster recovery end MinIO service information (login URL, account, and password) through the SDK.

[0072] Step S14, all bucket information of the source MinIO service and the disaster recovery end MinIO service is respectively obtained through the listBuckets() method of the SDK.

[0073] Step S15, the bucket information of the source MinIO service is looped and traversed, and is compared with the bucket information of the disaster recovery end MinIO, to find out the bucket data that exists in the source end MinIO and does not exist in the disaster recovery end MinIO.

[0074] Step S16, the makeBucket() method of the SDK is used to create the bucket data that exists in the source end MinIO and does not exist in the disaster recovery end MinIO in the disaster recovery end MinIO.

[0075] Step S17, the event listening of the source MinIO service is started (bucket event listening + object event listening).

[0076] Step S18, the existing object difference comparison and synchronization of the source MinIO service and the disaster recovery end MinIO service are synchronously performed, and the listening event object synchronization task is performed, so that the synchronization service simultaneously performs the full data synchronization and the incremental data synchronization, to ensure the data consistency of the source end MinIO and the disaster recovery end MinIO.

[0077] Step S181, the object difference comparison and synchronization of the source MinIO service and the disaster recovery end MinIO service: all object data of the source end and the disaster recovery end is respectively obtained through the listObjects() method of the SDK; all objects of the source end are looped, and the object data of the disaster recovery end MinIO is compared according to the information of the object of the source end; when the object does not exist in the disaster recovery end or the object data is inconsistent, the information of the object is pressed into the task queue of the synchronization service to be synchronized.

[0078] Step S182, the object change event of the source MinIO service is continuously listened to, and when the change event is received, the information of the object is pressed into the task queue of the synchronization service to be synchronized.

[0079] Step S19, the synchronization queue continuously consumes the messages in the queue, and according to the object information, the object data is obtained from the source MinIO service and is synchronized to the disaster recovery end MinIO service.

[0080] For details, please refer to Figure 3 After the synchronization is initiated, the comparison between the existing source end MinIO object data and the existing disaster recovery end MinIO object data is performed according to the following procedure:

[0081] Step S21, when traversing the source MinIO object data, the object meta information of the source is obtained;

[0082] Step S22, traverse the objects of the disaster recovery end MinIO to obtain the object meta information;

[0083] Step S23, take the object data of the same name of the disaster recovery end as the reference of the source object:

[0084] Step S231, if there is a same name object, continue to compare;

[0085] Step S232, if there is no same name object, it means that the disaster recovery end does not have the information of the current object of the source end, and the object data of the current source end is pushed into the object synchronization queue;

[0086] Step S24, if the source object does not start the version number and the disaster recovery end has the same name object with the modification time greater than the modification time of the source object, it is considered that the object currently processed by the source is the previous version of the object of the disaster recovery end, that is, the current processed object does not need to be synchronized to the disaster recovery end; if it is not the case, continue to judge;

[0087] Step S25, judge whether the disaster recovery end object satisfies 1. size equal, 2. user metadata equal, 3. object modification time equal, if yes, it is considered as the same object, and the current source object does not need to be synchronized;

[0088] Step S26, if step S25 is not satisfied, if the source starts the version number and is judged as not the same object, the disaster recovery end has the object data of the version number, at this time, the abnormal processing strategy is triggered.

[0089] For details, please refer to Figure 4 , the object synchronization event processing flow is as follows:

[0090] Step S31, the queue data is derived from the object event submitted from the full synchronization, incremental event step;

[0091] Step S32, judge whether the thread pool for object event processing has been created, if not, create the thread pool for object event processing (lazy loading) first, the thread pool queue selects ArrayBlockingQueue (multi-thread safe, bounded), the rejection policy selects CallerRunsPolicy (when the thread pool is full, use the submission thread to execute the task), when the thread pool is full, the thread for event listening is also used to execute the object event, improving the processing efficiency;

[0092] Step S33, judge whether the current object has no version number (to handle the scene of inconsistent data caused by one-way synchronization of the same name object):

[0093] Step S331, if there is a version number, directly submit the task to the thread pool, and end the process;

[0094] Step S332, if there is no version number:

[0095] Step S3321, judge whether the same type of task has been submitted (store the task record submitted to the thread pool in ConcurrentHashMap with key as the unique value of the object file): if the same type of task has been submitted, only need to add the current event information to the waiting queue in the ConcurrentHashMap, end the process; if not, submit the task to the thread pool, and after submitting the current event information to ConcurrentHashMap, end the process.

[0096] For details, please refer to Figure 5 , the thread execution task processing flow when the object synchronization task is not started in the version is as follows:

[0097] Step S41, the thread pool starts to continuously process the object synchronization task;

[0098] Step S42, after consuming an object synchronization task, judge whether the current object has no version number:

[0099] Step S421, if there is a version number, the current task processing is completed and the process is ended;

[0100] Step S422, if there is no version number, take whether there is a waiting queue with the same name in ConcurrentHashMap:

[0101] Step S4221, if there is no waiting queue, the current task processing is completed and the process is ended;

[0102] Step S4222, if there is a waiting queue, take the object synchronization task with the latest time into the process recursively until all the tasks in the waiting queue are executed, and end the process.

[0103] Embodiment 2

[0104] The application also provides a specific implementation step, which is specifically as follows:

[0105] S1: deploy a source MinIO service in a source environment, for example, execute

[0106]

[0107] S2: deploy a disaster recovery MinIO service in a disaster recovery environment, for example, execute

[0108]

[0109] S3: Deploy the MinIO synchronization service system, and ensure that the MinIO synchronization service is in smooth network connection with the source end and the disaster recovery end;

[0110] S4: A MinIO disaster recovery request is initiated to the MinIO synchronization service system, and source MinIO service information (login URL, account, password) and disaster recovery end MinIO service information (login URL, account, password) are submitted;

[0111] S5: After the MinIO synchronization service system is connected with the source MinIO and the disaster recovery end MinIO, full + incremental synchronization of data is started;

[0112] S6: The MinIO synchronization service is in a running state, and the source MinIO service is real-time disaster recovery.

[0113] The application realizes real-time backup of the MinIO service, avoids the loss of all data when more than half of the MinIO storage nodes are physically damaged, and causes a difficult production accident; meanwhile, the problem of inconsistent data caused by parallel uploading of the MinIO Client is solved; the peer-to-peer deployment or non-peer-to-peer deployment can be selected, resources are reasonably utilized, and when a fault occurs in the source end, the disaster recovery end MinIO can be quickly switched to for use, thereby ensuring the robustness and high availability of the service.

[0114] Finally, it should be noted that: the above only describes the preferred embodiments of the application and is not used to limit the application, although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A method for implementing MinIO real-time disaster recovery, characterized in that, The method comprises a data synchronization process, a data comparison execution process, an object synchronization event processing process, and a thread execution task processing process. The data synchronization process specifically comprises the following steps: Step S11, a user applies to initiate a MinIO synchronization task, and submits source MinIO service information and disaster recovery end MinIO service information; Step S12, the synchronization service acquires a Client object with event listening of the source MinIO according to the source MinIO service information through an SDK; Step S13, the synchronization service acquires a Client object of the disaster recovery end MinIO according to the disaster recovery end MinIO service information through an SDK; Step S14, all bucket information of the source MinIO service and the disaster recovery end MinIO service is respectively acquired through a listBuckets method of the SDK; Step S15, the bucket information of the source MinIO service is looped and traversed, and compared with the bucket information of the disaster recovery end MinIO to find out the bucket data that exists in the source end MinIO and does not exist in the disaster recovery end MinIO; Step S16, the makeBucket method of the SDK is used to create the bucket data that exists in the source end MinIO and does not exist in the disaster recovery end MinIO in the disaster recovery end MinIO; Step S17, event listening of the source MinIO service is started; Step S18, existing object difference comparison and synchronization and listening event object synchronization tasks of the source MinIO service and the disaster recovery end MinIO service are synchronously performed, at this time, the synchronization service simultaneously performs full-amount data synchronization and incremental data synchronization to ensure data consistency of the source end MinIO and the disaster recovery end MinIO; Step S19, the synchronization queue continuously consumes messages in the queue, acquires object data from the source MinIO service according to the object information, and synchronizes the data to the disaster recovery end MinIO service; The method for implementing MinIO real-time disaster recovery is applicable to a system for implementing MinIO real-time disaster recovery, which comprises a connection processing module, a full-amount data comparison module, an event listening module, a bucket event processing module, and an object event processing module.

2. The method for implementing MinIO real-time disaster recovery according to claim 1, characterized in that, The connection processing module is used for maintaining connection of a disaster recovery service with a source MinIO service and a disaster recovery MinIO service, ensuring that the connection of the disaster recovery service with the two services is normally available, and providing the connection for other modules to acquire. The full-amount data comparison module is used for comparing existing bucket data and object data of the source end and the disaster recovery end MinIO service in full amount when initiating disaster recovery. The event listening module is responsible for real-time listening of bucket events and object events of the source end MinIO service. The object event processing module is responsible for parallel processing of object event synchronization tasks submitted by the event listening module and the full-amount data comparison module. When the full-amount data comparison module compares the bucket data and the object data in full amount, the priority of the bucket data is higher than that of the object data.

3. The method for implementing MinIO real-time disaster recovery according to claim 2, characterized in that, The event listening module is further configured to convert the bucket event content into a task of the bucket event processing module and submit the task to the bucket event processing module, and convert the object event content into a task of the object event processing module and submit the task to the object event processing module.

4. The method for implementing MinIO real-time disaster recovery according to claim 3, characterized in that, In the object difference comparison and synchronization of the source MinIO service and the standby MinIO service in the step S18, all object data of the source and the standby is acquired through the listObjects() method of the SDK; all objects of the source are looped, and the object data of the standby MinIO is compared according to the information of the object of the source; when the object does not exist in the standby or the object data is inconsistent, the information of the object is pressed into the task queue of the synchronization service to be synchronized.

5. The method for implementing MinIO real-time disaster recovery according to claim 4, characterized in that, The data comparison execution process specifically includes the following steps: In step S21, the object meta information of the source is acquired when the source MinIO object data is traversed; In step S22, the object meta information is acquired when the object of the standby MinIO is traversed; In step S23, the object data of the standby object with the same name is taken as the reference of the object of the source: In step S231, if there is an object with the same name, the comparison continues; In step S232, if there is no object with the same name, it is indicated that the information of the current object of the source does not exist in the standby, and the object data of the current source is pressed into the object synchronization queue; In step S24, whether the object of the source does not start the version number and the object with the same name of the standby exists and the modification time of the object of the standby is greater than the modification time of the object of the source is judged, and if so, the object of the source currently processed is considered as the version before the object of the standby, that is, the object of the source currently processed does not need to be synchronized to the standby; if not, the judgment continues; In step S25, whether the object of the standby satisfies the condition is judged, and if so, the object is considered as the same object and the object of the source currently processed does not need to be synchronized; In step S26, if the condition is not satisfied in step S25, if the source starts the version number and is judged as not the same object, the object data of the version number exists in the standby, and an abnormal processing strategy is triggered.

6. The method for implementing MinIO real-time disaster recovery according to claim 5, characterized in that, In the step S25 of judging whether the object of the standby satisfies the condition, the condition includes the conditions of equal size, equal user metadata and equal object modification time.

7. The method for implementing MinIO real-time disaster recovery according to claim 6, characterized in that, The object synchronization event processing process specifically includes the following steps: In step S31, the queue data is derived from the object event received from the full synchronization and incremental event step; In step S32, whether the thread pool of the object event processing has been created is judged, and if not, the thread pool of the object event processing is created first, the thread pool queue selects ArrayBlockingQueue, the rejection policy selects CallerRunsPolicy, and when the thread pool is full, the thread of the event listening is used to execute the object event, so that the processing efficiency is improved; In step S33, whether the current object has a version number is judged: In step S331, if the version number exists, the task is directly submitted to the thread pool, and the process is ended. Step S332, if there is no version number, it is judged whether the same kind of task has been submitted, a ConcurrentHashMap with key as the unique value of the object file is used to store the record of the task submitted to the thread pool, if the same kind of task has been submitted, the current event information is added to the waiting queue in the ConcurrentHashMap, and the flow ends; If not, the task is submitted to the thread pool, and the information of the current event is submitted to the ConcurrentHashMap, and then the flow ends.

8. The method for implementing MinIO real-time disaster recovery according to claim 7, characterized in that, The: Step S41, the thread pool starts to continuously process the object synchronization task; Step S42, after consuming an object synchronization task, it is judged whether the current object has no version number: Step S421, if there is a version number, the current task processing is completed and the flow ends; Step S422, if there is no version number, it is judged whether there is a waiting queue with the same name in the ConcurrentHashMap: Step S4221, if there is no waiting queue, the current task processing is completed and the flow ends; Step S4222, if there is a waiting queue, the object synchronization task with the latest time is taken to enter the recursive flow, and the flow ends until all the tasks in the waiting queue are executed.

Images