A data processing method, device, equipment and storage medium
By monitoring the resource information of the Kubernetes cluster in real time and synchronizing resource events in sequence, the problem of data change information loss in disaster recovery backup between Kubernetes clusters is solved, real-time backup and recovery of data change information is realized.
Patent Information
- Application Number
- CN202010757060.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-31
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-07-31
AI Technical Summary
In the prior art, the disaster recovery backup method between Kubernetes clusters cannot effectively backup the data change information generated by data operations during the two backup time intervals, resulting in the loss of data change information.
By monitoring the resource information on the first cluster in real time, determining whether resource events are generated, and synchronizing them to the second cluster in the order of resource events to ensure real-time backup of data change information.
It effectively avoids the loss of resource information caused by backup time intervals, realizes real-time monitoring and backup of data change information, and improves the reliability of data recovery.
Smart Images

Figure CN114064349B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to data processing technology, and in particular to a data processing method, device, electronic equipment and computer storage medium. Background Art
[0002] In the related art, the disaster recovery backup method across Kubernetes clusters mainly realizes disaster recovery backup between Kubernetes clusters by backing up the data of the Etcd component in the Kubernetes cluster. Therefore, it is necessary to regularly perform snapshot backups of the Etcd component. Whether manually backing up the Etcd component of the Kubernetes cluster to restore a failed Kubernetes cluster, or restoring a failed Kubernetes cluster through a synchronization tool, it may not be possible to back up the data change information generated by data operations within the time interval between two backups, resulting in the loss of data change information. Summary of the invention
[0003] The embodiment of the present invention is intended to provide a technical solution for data processing to solve the problem in the related art that data change information generated by data operations within the time interval between two backups cannot be backed up, resulting in loss of data change information.
[0004] An embodiment of the present invention provides a data processing method, the method comprising:
[0005] Monitoring resource information on a first cluster in real time, where the first cluster represents a main cluster based on container resources;
[0006] Determine whether to generate at least one resource event based on the resource information. If it is determined that at least one resource event is generated, synchronize the at least one resource event to a second cluster, where the resource event represents resource change information caused by resource operations, and the second cluster represents a backup cluster corresponding to the main cluster based on container resources.
[0007] Optionally, synchronizing the at least one resource event to the second cluster includes:
[0008] When the number of the at least one resource event is greater than 1, the at least one resource event is synchronized to the second cluster according to the sequence in which the at least one resource event occurs.
[0009] Optionally, synchronizing the at least one resource event to the second cluster according to the order in which the at least one resource event occurs includes:
[0010] storing the at least one resource event in a message queue according to the order in which the at least one resource event occurs;
[0011] The at least one resource event is obtained from the message queue in the order, and the at least one resource event obtained is synchronized to the second cluster.
[0012] Optionally, the real-time monitoring of resource information on the first cluster includes:
[0013] Determine a synchronization resource type, where the synchronization resource type represents a type of resource that needs to be monitored on the first cluster;
[0014] The resource information corresponding to the synchronization resource type is monitored in real time.
[0015] Optionally, the method further comprises:
[0016] When it is determined that the at least one resource event is successfully synchronized to the second cluster, synchronization progress information is updated, where the synchronization progress information indicates synchronization status information corresponding to the at least one resource event.
[0017] Optionally, the real-time monitoring of resource information on the first cluster includes: monitoring resource information on the first cluster using a synchronization component,
[0018] The method further comprises:
[0019] When the synchronization component is restarted, the synchronization progress information is obtained, and resource events that need to be synchronized are determined according to the synchronization progress information.
[0020] Optionally, updating the synchronization progress information includes: updating the synchronization progress information to a storage database,
[0021] The obtaining of the synchronization progress information includes:
[0022] The synchronization progress information is obtained from the storage database.
[0023] Optionally, the method further comprises:
[0024] In the event of failure in synchronizing the at least one resource event to the second cluster, controlling the synchronization component to restart.
[0025] The embodiment of the present invention further provides a data processing device, the device comprising: a monitoring module and a synchronization module, wherein:
[0026] The monitoring module is used to monitor resource information on a first cluster in real time, where the first cluster represents a main cluster based on container resources;
[0027] The synchronization module is used to determine whether to generate at least one resource event based on the resource information. If it is determined that at least one resource event is generated, the at least one resource event is synchronized to a second cluster, where the resource event represents resource change information caused by resource operations, and the second cluster represents a backup cluster corresponding to the main cluster based on container resources.
[0028] Optionally, the synchronization module is used to synchronize the at least one resource event to the second cluster, including:
[0029] When the number of the at least one resource event is greater than 1, the at least one resource event is synchronized to the second cluster according to the sequence in which the at least one resource event occurs.
[0030] Optionally, the synchronization module is used to synchronize the at least one resource event to the second cluster according to the order in which the at least one resource event occurs, including:
[0031] storing the at least one resource event in a message queue according to the order in which the at least one resource event occurs;
[0032] The at least one resource event is obtained from the message queue in the order, and the at least one resource event obtained is synchronized to the second cluster.
[0033] Optionally, the monitoring module is used to monitor resource information on the first cluster in real time, including:
[0034] Determine a synchronization resource type, where the synchronization resource type represents a type of resource that needs to be monitored on the first cluster;
[0035] The resource information corresponding to the synchronization resource type is monitored in real time.
[0036] Optionally, the synchronization module is further configured to update synchronization progress information when it is determined that the at least one resource event is successfully synchronized to the second cluster, wherein the synchronization progress information indicates synchronization status information corresponding to the at least one resource event.
[0037] Optionally, the monitoring module is used to monitor the resource information on the first cluster in real time, including: using the synchronization component to monitor the resource information on the first cluster, and the monitoring module is also used to obtain the synchronization progress information when the synchronization component is restarted, and determine the resource events that need to be synchronized based on the synchronization progress information.
[0038] Optionally, the synchronization module is used to update the synchronization progress information, including: updating the synchronization progress information to a storage database,
[0039] The monitoring module is used to obtain the synchronization progress information, including:
[0040] The synchronization progress information is obtained from the storage database.
[0041] Optionally, the monitoring module is further configured to control the synchronization component to restart when synchronization of the at least one resource event to the second cluster fails.
[0042] An embodiment of the present invention further provides an electronic device, comprising a processor and a memory for storing a computer program that can be run on the processor; wherein:
[0043] When the processor is used to run the computer program, it executes any one of the data processing methods described above.
[0044] An embodiment of the present invention further provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, any one of the above-mentioned data processing methods is implemented.
[0045] It can be seen that the data processing method in the embodiment of the present invention monitors the resource information on the first cluster in real time, the first cluster represents the main cluster based on the container resources; determines whether to generate at least one resource event according to the resource information, and when it is determined that the at least one resource event is generated, synchronizes the at least one resource event to the second cluster, the resource event represents the resource change information caused by the resource operation, and the second cluster represents the backup cluster corresponding to the main cluster based on the container resources; because in the embodiment of the present invention, the resource information on the first cluster can be monitored in real time, and all resource events occurring on the first cluster can be monitored, therefore, it is helpful to avoid the loss of resource information caused by the backup time interval.
[0046] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present invention and, together with the specification, are used to explain the technical solutions of the present invention.
[0048] Figure 1 A flowchart of a data processing method according to an embodiment of the present invention;
[0049] Figure 2 This is a schematic diagram of the architecture of Kubernetes resource backup according to an embodiment of the present invention;
[0050] Figure 3 A schematic diagram of the internal structure of a synchronization component according to an embodiment of the present invention;
[0051] Figure 4 A schematic diagram of the workflow of the synchronization component of an embodiment of the present invention;
[0052] Figure 5 A schematic diagram of the structure of a data processing device according to an embodiment of the present invention;
[0053] Figure 6 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0054] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the embodiments provided below are partial embodiments for implementing the present invention, rather than providing all embodiments for implementing the present invention. In the absence of conflict, the technical solutions recorded in the embodiments of the present invention can be implemented in any combination.
[0055] It should be noted that, in the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or apparatus including a series of elements includes not only the elements explicitly recorded, but also includes other elements not explicitly listed, or also includes elements inherent to the implementation of the method or apparatus. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other related elements (such as steps in a method or units in a device, for example, a unit may be a part of a circuit, a part of a processor, a part of a program or software, etc.) in the method or apparatus including the element.
[0056] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.
[0057] For example, the data processing method provided by the embodiment of the present invention includes a series of steps, but the data processing method provided by the embodiment of the present invention is not limited to the recorded steps. Similarly, the data processing device provided by the embodiment of the present invention includes a series of modules, but the device provided by the embodiment of the present invention is not limited to including the modules explicitly recorded, and may also include modules required to obtain relevant information or perform processing based on information.
[0058] The embodiments of the present invention can be applied to hardware such as servers or computer systems composed of hardware, and can operate with many other general or special computing system environments or configurations, or can be implemented by a processor running computer executable code. Here, the server can be a server computer system, a small computer system, a large computer system, and a distributed cloud computing technology environment including any of the above systems, etc.
[0059] Servers may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Servers may be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.
[0060] As the demand for containerization increases, Kubernetes is increasingly used as a container orchestration engine, and high availability and disaster recovery in production environments have become a feature that must be considered. Currently, disaster recovery for containers in Kubernetes is mainly achieved within a Kubernetes cluster through multiple copies or by restarting the Pod through probe detection. As for the disaster recovery between different Kubernetes clusters, in related technologies, a more common disaster recovery method is to take a snapshot of the Etcd component in Kubernetes. When the main Kubernetes cluster fails to work properly, you can manually restore the Etcd component to the backup Kubernetes cluster by taking a snapshot, or use the Etcd component synchronization tool, such as Etcdctl make-mirror, to synchronize data in the Etcd component.
[0061] The above-mentioned disaster recovery method within a Kubernetes cluster only supports disaster recovery when the master node of the Kubernetes cluster can work normally but the business container cannot work normally. When the master node of the Kubernetes cluster cannot work, the management components on the master node will also not work, and the containers on the business node cannot be managed. Therefore, when a problem occurs in the business container, the problem in the business cluster cannot be recovered.
[0062] Regarding the above-mentioned method of implementing disaster recovery backup between Kubernetes clusters by backing up the data of the Etcd component in the Kubernetes cluster, since this method requires regular snapshot backup of the Etcd component in the Kubernetes cluster, whether manually backing up the Etcd component of the Kubernetes cluster to restore the failed Kubernetes cluster or restoring the failed Kubernetes cluster through a synchronization tool, it may be impossible to back up the data change information generated by data operations within the time interval between two backups, resulting in the loss of data change information.
[0063] In order to solve the above technical problems, an embodiment of the present invention proposes a data processing method. Figure 1 is a flow chart of a data processing method according to an embodiment of the present invention, such as Figure 1 As shown, the process may include:
[0064] Step 101: monitor resource information on a first cluster in real time, where the first cluster represents a main cluster based on container resources.
[0065] In one example, container resources may be represented as resource units with resource containers as units. Here, the computing resources of the system may be uniformly divided and encapsulated into many resource units.
[0066] In one implementation, the first cluster may be a Kubernetes master cluster or another type of master cluster based on container resources, and the resource information may represent the change information of resources in the Kubernetes cluster. The implementation method of real-time monitoring of resource information on the first cluster may be to determine a synchronization resource type, where the synchronization resource type represents the type of resources that need to be monitored on the first cluster, and then monitor the resource information corresponding to the synchronization resource type in real time.
[0067] In one example, the synchronization resource type may include one resource type or multiple resource types. For example, the synchronization resource type may include one resource type, ConfigMap, or three resource types, ConfigMap, Depleyment, and Namespace. The implementation method for determining the synchronization resource type may be to determine the resources that need to be synchronized on the first cluster according to user needs, and determine the corresponding resource types according to the determined resources that need to be synchronized. For example, when the user needs to synchronize ConfigMap and Depleyment resources, the corresponding resource types determined are ConfigMap and Depleyment. At the same time, when it is determined that the resources of the first cluster need to be synchronized, a synchronization component may be created. The synchronization component may be a SyncComponent component. Further, the synchronization component may be considered as a Kubernetes custom resource (Custom Resource Definition, CRD), wherein the CRD of the synchronization component may be defined as:
[0068]
[0069] After the CRD of the synchronization component is created, the control component (Controller) corresponding to the CRD will monitor it and create a service instance corresponding to the synchronization component. The control component (Controller) is located in the Kubernetes cluster.
[0070] When the second cluster represents the backup cluster corresponding to the first cluster and the first cluster and the second cluster need to be synchronized, the following settings need to be performed in the synchronization component:
[0071]
[0072]
[0073] Among them, the name of the synchronization component SyncComponent can be set to the unique identifier of the second cluster (backup cluster), and the setting information corresponding to spec includes the settings of the unique identifiers of the first cluster (main cluster) and the second cluster (backup cluster) and the relevant certificates of the second cluster (backup cluster). Through the setting of this information, the synchronization component can operate the first cluster and the second cluster at the same time.
[0074] For the implementation method of real-time monitoring of resource information corresponding to the synchronization resource type, the monitoring module (Watcher) in the synchronization component can be used to monitor the resource information corresponding to the synchronization resource type in real time. Specifically, the monitoring module (Watcher) can monitor the changes of various resource information on the first cluster, and convert the data corresponding to the resource changes into data in yaml format, and store the converted data in the message queue of the synchronization component SyncComponent; specifically, two different monitoring modules (Watchers) corresponding to the two resource types ConfigMap and Depleyment can be used to achieve real-time monitoring of resource information. For example, the monitoring module Watcher1 corresponds to the ConfigMap resource, and the monitoring module Depleyment2 corresponds to the Depleyment resource to monitor the resource information in real time.
[0075] It can be seen that by determining the resource types corresponding to the resources that need to be synchronized according to the needs, and then only monitoring the resource information corresponding to the determined resource types, it is unnecessary to monitor the entire Kubernetes cluster, thus reducing the loss of resource monitoring.
[0076] Step 102: Determine whether to generate at least one resource event based on the resource information. If it is determined that at least one resource event is generated, synchronize the at least one resource event to a second cluster, where the resource event represents resource change information caused by resource operations, and the second cluster represents a backup cluster corresponding to the main cluster based on container resources.
[0077] As an implementation method, a resource operation may be an operation on a certain resource, such as deletion, creation, and modification of a ConfigMap, Namespace, or Depleyment resource, which is not specifically limited here; a resource event may indicate resource change information caused by creating a ConfigMap resource instance or a Depleyment resource instance. Specifically, a certain type of resource may contain multiple resource instances of the same type.
[0078] A resource event can be resource change information caused by an operation on a resource instance in the type of resource. Of course, a resource event can also be resource change information caused by an operation on one resource instance in multiple types of resources. Specifically, a ConfigMap resource can contain many ConfigMap resource instances, for example, it can contain resource instance A, resource instance B, resource instance C, and resource instance D. A resource event can be resource change information caused by deleting resource instance A, or it can be resource change information caused by creating resource instance B. For ConfigMap resources and Depleyment resources, and the Depleyment resource contains resource instance E, resource instance F, resource instance G, and resource instance H, a resource event can be resource change information caused by deleting resource instance A, modifying resource instance B, creating resource instance G, or deleting resource F.
[0079] In one example, determining whether to generate at least one resource event according to the resource information may be determining whether to generate resource event 1 and resource event 2 according to the detected resource information changes caused by deleting resource instance A and modifying resource instance B. In the case where it is determined that at least one resource event is generated, the implementation method of synchronizing the at least one resource event to the second cluster may be synchronizing the resource event 1 and resource event 2 determined to be generated to the second cluster when it is determined that at least one resource event is generated.
[0080] In practical applications, steps 101 to 102 can be implemented using a processor in an electronic device, and the processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), an FPGA, a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.
[0081] It can be seen that an embodiment of the present invention proposes a data processing method, which monitors resource information on a first cluster in real time, where the first cluster represents a main cluster based on container resources; determines whether to generate at least one resource event based on the resource information, and synchronizes the at least one resource event to a second cluster when it is determined that at least one resource event is generated, wherein the resource event represents resource change information caused by resource operations, and the second cluster represents a backup cluster corresponding to the main cluster based on container resources; since in an embodiment of the present invention, the resource information on the first cluster can be monitored in real time, all resource events occurring on the first cluster can be monitored, and therefore, it is helpful to avoid the loss of resource information due to backup time intervals.
[0082] In one implementation, synchronizing the at least one resource event to the second cluster includes: when the number of the at least one resource event is greater than 1, synchronizing the at least one resource event to the second cluster in the order in which the at least one resource event occurs.
[0083] In one example, synchronizing the at least one resource event to the second cluster in the order in which the at least one resource event occurred may be storing the at least one resource event in a message queue in the order in which the at least one resource event occurred, obtaining the at least one resource event from the message queue in the order, and synchronizing the obtained at least one resource event to the second cluster.
[0084] Specifically, for the case where at least one resource event is resource event 1 and resource event 2, and resource event 1 occurs before resource event 2, the implementation method of storing the at least one resource event in the message queue in the order of the at least one resource event may be to first store resource event 1 in the message queue and then store resource event 2 in the message queue, obtain the at least one resource event from the message queue in the order, and synchronize the obtained at least one resource event to the second cluster, or to first obtain resource event 1 from the message queue and synchronize resource event 1 to the second cluster, then obtain resource event 2 from the message queue, and synchronize resource event 2 to the second resource cluster. Here, the message queue may be located in the synchronization component and used to store various resource events to ensure that events can be executed in order. The synchronization component also includes
[0085] In one example, synchronizing the at least one resource event to the second cluster can be performed by obtaining at least one resource event in the message queue through a processing module Handler in the synchronization component, and synchronizing the obtained at least one resource event to the second cluster. Specifically, the processing module Handler can obtain data in the yaml format in the message queue, deserialize the obtained yaml format data, parse the corresponding resource event, and synchronize the second cluster according to the parsed resource event.
[0086] The implementation method of synchronizing the at least one resource event to the second cluster may be, for example, determining the specific resource instance corresponding to the resource event for each resource event, and identifying the determined resource instance, and the processing module Handler only performs synchronization operations on the identified resource instance, and does not perform synchronization operations on the unidentified resource instance. For example, for a case where at least one resource event is resource event 1 and resource event 2, resource event 1 represents resource change information caused by deleting ConfigMap resource instance A, and resource event 2 represents resource change information caused by creating Depleyment resource instance B, and resource event 1 occurs before resource event 2, ConfigMap resource instance A and Depleyment resource instance B are identified respectively, and other ConfigMap resource instances and Depleyment resource instances other than ConfigMap resource instance A and Depleyment resource instance B are not identified, and the processing module Handler only performs synchronization operations on the identified ConfigMap resource instance A and Depleyment resource instance B, and does not perform synchronization operations on other ConfigMap resource instances and Depleyment resource instances other than the unidentified ConfigMap resource instance A and Depleyment resource instance B.
[0087] It can be seen that the above synchronization method can refine the granularity of backup to the resource instance level. There is no need to back up all resources in the entire first cluster (Kubernetes main cluster). The specific synchronization resources can be determined according to user needs for synchronization, which will not affect the synchronization logic of other resources, making synchronization more flexible. In addition, through the specific implementation of the processing module Handle, customizable backup can be performed for different types of resources, reducing resource loss during the backup process.
[0088] In one embodiment, when it is determined that the at least one resource event is successfully synchronized to the second cluster, synchronization progress information is updated, where the synchronization progress information indicates synchronization status information corresponding to the at least one resource event. The updating of the synchronization progress information includes: updating the synchronization progress information to a storage database.
[0089] In one example, the synchronization progress information may be relationship information used to represent the events before and after a resource event occurs. Specifically, the synchronization progress information may be the version number of the resource event. For example, the first synchronization information may be version number 1 and version number 2, wherein the resource event corresponding to version number 1 occurs before the resource event corresponding to version number 2. The storage database may be an Etcd component, wherein the Etcd component is a highly available and strongly consistent service discovery storage database that may be responsible for saving the current synchronization status of various resources.
[0090] In one example, when it is determined that at least one resource event is successfully synchronized to the second cluster, the synchronization progress information is updated. The synchronization progress information may be updated for the first time after it is determined that the first resource event is successfully synchronized to the second cluster, and the synchronization progress information may be updated for the second time after it is determined that the later resource event is successfully synchronized to the second cluster. For example, for resource event A and resource event B, where resource event A occurs before resource event B, the version number 1 may be updated to version number 2 after resource event A is successfully updated to the second cluster, and the version number 2 may be updated to version number 3 after resource event B is successfully updated to the second cluster.
[0091] The implementation method of updating the synchronization progress information into the storage database may be to update the version number of the resource event into the storage database, for example, to update the version number 1 into the data storage library.
[0092] In one implementation, when the synchronization component is restarted, the synchronization progress information is obtained, and the resource events that need to be synchronized are determined according to the synchronization progress information. The synchronization progress information is obtained by obtaining the synchronization progress information from the storage database.
[0093] In one example, when the synchronization component is restarted, the synchronization progress information is obtained, and the resource events that need to be synchronized are determined based on the synchronization progress information. The synchronization component may be restarted after a failure, and the latest version number of the resource event is obtained from the storage database. The resource synchronization status is determined based on the obtained latest version number, and then the resource events that need to be synchronized are determined. Specifically, the first synchronization information corresponding to resource event A is version number 5, and resource event A cannot be synchronized to the second cluster due to a failure in the synchronization component. At this time, the version number in the storage database should be version number 4 corresponding to the previous resource event corresponding to resource event A that was successfully synchronized to the second cluster, and the message queue in the corresponding synchronization component will not delete the data corresponding to resource event A that was not successfully synchronized to the second cluster. After the synchronization component is restarted, it will first obtain the latest version number 4 of the resource event from the Etcd component, and obtain resource event A from the message queue again through the processing module Handler, and synchronize resource event A to the second cluster. At the same time, the resource events monitored after the restart of the synchronization component are synchronized sequentially. In this way, by saving the first synchronization information, the synchronization event can continue to be synchronized according to the synchronization progress after the restart, and repeated synchronization of the same resource event will not occur.
[0094] In one implementation, in the event of failure to synchronize the at least one resource event to the second cluster, the synchronization component may be controlled to restart.
[0095] As an implementation method, when the synchronization of resource events to the second cluster fails, the synchronization component can be controlled to restart according to the cause of the synchronization failure, when the solution corresponding to the failure cause is to restart the synchronization component. Specifically, an error notification can be issued when the synchronization of resource events to the second cluster fails. The specific form of issuing the error notification is not limited here, and the error notification can be issued through other methods such as voice prompts and dialog box displays.
[0096] Figure 2 The schematic diagram of the architecture of Kubernetes resource backup in an embodiment of the present invention is as follows: Figure 2 As shown, the business clusters in the primary data center are all primary clusters, the business clusters in the backup data center are all backup clusters, and the primary clusters in the primary data center correspond to the backup clusters in the backup data center respectively. The synchronization component can monitor the resource information on the primary business cluster and implement resource synchronization operations on the backup cluster.
[0097] Figure 3 FIG. 1 is a schematic diagram of the internal structure of the synchronization component of an embodiment of the present invention. Figure 3As shown, the synchronization component consists of a monitoring module, a message queue and a processing module, wherein the monitoring module can read the synchronization progress information in the Etcd component and monitor the resource information in the main cluster. When it is determined that a resource event is generated according to the monitored resource information, the generated resource event is formatted and stored in the internal message queue. The processing module can obtain the formatted resource event in the message queue, deserialize and parse the formatted resource event, synchronize the corresponding resource event to the backup cluster according to the parsing result, and update the synchronization progress information in the Etcd component when the resource event is successfully synchronized to the backup cluster.
[0098] Figure 4 FIG. 1 is a schematic diagram of the workflow of the synchronization component of an embodiment of the present invention. Figure 4 As shown, when the synchronization component is started, it will determine the corresponding resource type according to the resource to be synchronized, and initialize the monitoring module and processing module corresponding to each resource type according to the type of resource type. After the monitoring module is started, the synchronization progress information can be obtained from the Etcd component first, and the resource information corresponding to the resources to be synchronized in the main cluster can be monitored. When it is determined to generate a resource event based on the monitored resource information, the generated resource event will be formatted and the resource event after format conversion will be placed in the message queue. Then the monitoring module will wait for the next resource event to occur. When a monitoring error occurs in the monitoring module, the synchronization progress will be re-obtained from the Etcd component and a new monitoring will be started. After the processing module is started, it will start waiting for the resource event in the message queue. After obtaining the resource event, the resource event will be deserialized and parsed, and the parsed resource event will be synchronized to the backup cluster. When the resource event is successfully synchronized to the backup cluster, the synchronization progress information in the Etcd component is updated. When the resource event fails to be synchronized to the backup cluster, the synchronization progress information in the Etcd component is not updated, so that the synchronization component can continue to perform synchronization work after restarting.
[0099] Based on the data processing method proposed in the above-mentioned embodiment, an embodiment of the present invention proposes a data processing device.
[0100] Figure 5 FIG. 1 is a schematic diagram of the structure of a data processing device according to an embodiment of the present invention. Figure 5 As shown, the device may include: a monitoring module 501 and a synchronization module 502, wherein:
[0101] The monitoring module 501 is used to monitor resource information on the first cluster in real time, where the first cluster represents a main cluster based on container resources;
[0102] The synchronization module 502 is used to determine whether to generate at least one resource event based on the resource information. If it is determined that at least one resource event is generated, the at least one resource event is synchronized to the second cluster, where the resource event represents resource change information caused by resource operations, and the second cluster represents a backup cluster corresponding to the main cluster based on container resources.
[0103] Optionally, the synchronization module 502 is used to synchronize the at least one resource event to the second cluster, including:
[0104] When the number of the at least one resource event is greater than 1, the at least one resource event is synchronized to the second cluster according to the sequence in which the at least one resource event occurs.
[0105] Optionally, the synchronization module 502 is used to synchronize the at least one resource event to the second cluster according to the order in which the at least one resource event occurs, including:
[0106] storing the at least one resource event in a message queue according to the order in which the at least one resource event occurs;
[0107] The at least one resource event is obtained from the message queue in the order, and the at least one resource event obtained is synchronized to the second cluster.
[0108] Optionally, the monitoring module 501 is used to monitor resource information on the first cluster in real time, including:
[0109] Determine a synchronization resource type, where the synchronization resource type represents a type of resource that needs to be monitored on the first cluster;
[0110] The resource information corresponding to the synchronization resource type is monitored in real time.
[0111] Optionally, the synchronization module 502 is further configured to update synchronization progress information when it is determined that the at least one resource event is successfully synchronized to the second cluster, where the synchronization progress information indicates synchronization status information corresponding to the at least one resource event.
[0112] Optionally, the monitoring module 501 is used to monitor the resource information on the first cluster in real time, including: using the synchronization component to monitor the resource information on the first cluster, and the monitoring module is also used to obtain the synchronization progress information when the synchronization component is restarted, and determine the resource events that need to be synchronized based on the synchronization progress information.
[0113] Optionally, the synchronization module 502 is used to update the synchronization progress information, including: updating the synchronization progress information to a storage database,
[0114] The monitoring module is used to obtain the synchronization progress information, including:
[0115] The synchronization progress information is obtained from the storage database.
[0116] Optionally, the monitoring module 501 is further configured to control the synchronization component to restart when synchronization of the at least one resource event to the second cluster fails.
[0117] In practical applications, the monitoring module 501 and the synchronization module 502 can be implemented using a processor in an electronic device, and the processor can be at least one of an ASIC, a DSP, a DSPD, a PLD, a FPGA, a CPU, a controller, a microcontroller, and a microprocessor.
[0118] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.
[0119] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the relevant technology or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.
[0120] Specifically, a computer program instruction corresponding to a data processing method in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a USB flash drive. When the computer program instruction corresponding to a video processing method in the storage medium is read or executed by an electronic device, any data processing method in the aforementioned embodiments is implemented.
[0121] Based on the same technical concept as the above embodiments, see Figure 6 , which shows an electronic device provided by an embodiment of the present invention, which may include: a memory 601 and a processor 602; wherein,
[0122] The memory 601 is used to store computer programs and data;
[0123] The processor 602 is used to execute the computer program stored in the memory to implement any one of the data processing methods in the foregoing embodiments.
[0124] In practical applications, the memory 601 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 602.
[0125] The processor 602 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understandable that for different electronic devices, the electronic device used to implement the processor function may also be other, which is not specifically limited in the embodiment of the present invention.
[0126] In some embodiments, the functions or modules included in the device provided by the embodiments of the present invention can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0127] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.
[0128] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.
[0129] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.
[0130] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.
[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network equipment, etc.) to execute the methods described in each embodiment of the present invention.
[0132] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.
Claims
1. A data processing method, characterized in that: The method comprises: Monitor resource information on the first cluster in real time, where the first cluster represents a main cluster based on container resources; the resource information is information corresponding to a synchronized resource type, where the synchronized resource type includes a configuration map ConfigMap, a deployment Depleyment, and a namespace Namespace; Determine whether to generate at least one resource event based on the resource information. If it is determined that at least one resource event is generated, synchronize the at least one resource event to a second cluster, where the resource event represents resource change information caused by resource operations, and the second cluster represents a backup cluster corresponding to the main cluster based on container resources.
2. The method according to claim 1, characterized in that The synchronizing the at least one resource event to the second cluster includes: When the number of the at least one resource event is greater than 1, the at least one resource event is synchronized to the second cluster according to the order in which the at least one resource event occurs.
3. The method according to claim 2, characterized in that The synchronizing the at least one resource event to the second cluster according to the order in which the at least one resource event occurs includes: storing the at least one resource event in a message queue according to the order in which the at least one resource event occurs; The at least one resource event is obtained from the message queue in the order, and the at least one resource event obtained is synchronized to the second cluster.
4. The method according to claim 1, characterized in that: The real-time monitoring of resource information on the first cluster includes: Determine a synchronization resource type, where the synchronization resource type represents a type of resource that needs to be monitored on the first cluster; The resource information corresponding to the synchronization resource type is monitored in real time.
5. The method according to claim 1, characterized in that The method further comprises: When it is determined that the at least one resource event is successfully synchronized to the second cluster, synchronization progress information is updated, where the synchronization progress information indicates synchronization status information corresponding to the at least one resource event.
6. The method according to claim 5, characterized in that The real-time monitoring of the resource information on the first cluster includes: using a synchronization component to monitor the resource information on the first cluster, The method further comprises: When the synchronization component is restarted, the synchronization progress information is obtained, and resource events that need to be synchronized are determined according to the synchronization progress information.
7. The method according to claim 6, characterized in that The updating of synchronization progress information includes: updating the synchronization progress information to a storage database; The obtaining of the synchronization progress information includes: The synchronization progress information is obtained from the storage database.
8. The method according to claim 6, characterized in that The method further comprises: In the event of failure in synchronizing the at least one resource event to the second cluster, controlling the synchronization component to restart.
9. A data processing device, characterized in that: The device comprises: a monitoring module and a synchronization module, wherein: The monitoring module is used to monitor resource information on the first cluster in real time, where the first cluster represents a main cluster based on container resources; the resource information is information corresponding to a synchronized resource type, where the synchronized resource type includes a configuration map ConfigMap, a deployment Depleyment, and a namespace Namespace; The synchronization module is used to determine whether to generate at least one resource event based on the resource information. If it is determined that at least one resource event is generated, the at least one resource event is synchronized to a second cluster, where the resource event represents resource change information caused by resource operations, and the second cluster represents a backup cluster corresponding to the main cluster based on container resources.
10. An electronic device, characterized in that: comprising a processor and a memory for storing a computer program capable of running on the processor; wherein, When the processor is used to run the computer program, it executes the data processing method according to any one of claims 1 to 8.
11. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 8 is implemented.
