Data synchronization method and electronic device

By monitoring data changes in the first cluster and assembling data, and synchronizing the message queue to different clusters, the problems of inefficient and inaccurate data synchronization across clusters are solved, and more efficient and accurate data synchronization is achieved.

CN118520051BActive Publication Date: 2024-12-06深圳市灵智数字科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410980740.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-22
Publication Date
2024-12-06
Estimated Expiration
2044-07-22

AI Technical Summary

Technical Problem

The prior art is inefficient and inaccurate in data synchronization across clusters, especially when the amount of log files is large or the cluster version is inconsistent, real-time data synchronization cannot be effectively achieved.

Method used

By monitoring data changes in the first cluster and performing data assembly, the assembled data is synchronized to a different second cluster using a pre-built message queue, including cluster identification information, to ensure accurate synchronization.

Benefits of technology

Improve the efficiency and accuracy of cross-cluster data synchronization, avoiding the inaccurate synchronization problems caused by the large amount of log file data and inconsistent cluster versions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118520051B_ABST
    Figure CN118520051B_ABST
Patent Text Reader

Abstract

The present application is applicable to the field of data synchronization technology, and provides a data synchronization method and an electronic device, including: when a first cluster is running, monitoring whether data changes occur in the first cluster, and when monitoring data changes in the first cluster, assembling the changed data in the first cluster, and pushing the assembled data to a pre-built message queue, wherein the assembled data includes identification information of the cluster to be synchronized with the data, and synchronizing the assembled data to a second cluster corresponding to the identification information according to the message queue, wherein the second cluster is a cluster different from the first cluster. The present application can improve the efficiency of cross-cluster data synchronization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the technical field of data synchronization, and in particular, relates to a data synchronization method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the development of digital technology, real-time retrieval and analysis of data often need to be achieved through distributed clusters. For example, the Es (Elaticsearch) search engine can be expanded to hundreds of servers, and distributed clusters are built through different servers to achieve real-time retrieval and analysis of massive data. It is widely used in e-commerce product search, mobile application search and other fields. In order to ensure the normal operation of the business, the data of different clusters needs to be synchronized.

[0003] Currently, cross-cluster data synchronization can be achieved by parsing log files. However, the above method has low data synchronization efficiency when the amount of log file data is large; at the same time, because the versions of different clusters may be inconsistent, parsing log files may lead to inaccurate data synchronization. Summary of the invention

[0004] The embodiments of the present application provide a data synchronization method, device, electronic device, and computer-readable storage medium, which can improve the efficiency of cross-cluster data synchronization.

[0005] In a first aspect, an embodiment of the present application provides a data synchronization method, comprising:

[0006] When the first cluster is running, monitoring whether data changes occur in the first cluster;

[0007] When monitoring that data changes occur in the first cluster, assembling the changed data in the first cluster;

[0008] Pushing the assembled data to a pre-built message queue, wherein the assembled data includes identification information of the cluster to be synchronized with data;

[0009] The assembled data is synchronized to a second cluster corresponding to the identification information according to the message queue; the second cluster is a cluster different from the first cluster.

[0010] In a second aspect, an embodiment of the present application provides a data synchronization device, comprising:

[0011] A monitoring module, used for monitoring whether data changes occur in the first cluster when the first cluster is running;

[0012] A data assembly module, configured to assemble the changed data in the first cluster when monitoring data changes in the first cluster;

[0013] A message queue construction module, used to push the assembled data to a pre-constructed message queue, wherein the assembled data includes identification information of the cluster to be synchronized with data;

[0014] A data synchronization module is used to synchronize the assembled data to a second cluster corresponding to the identification information according to the message queue; the second cluster is a cluster different from the first cluster.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data synchronization method described in the first aspect when executing the computer program.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data synchronization method described in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the data synchronization method described in any one of the first aspects above.

[0018] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0019] In an embodiment of the present application, when the first cluster is running, the first cluster is monitored for data changes, and when the data in the first cluster changes, the changed data in the first cluster is assembled, and then the assembled data is pushed to the pre-built message queue, and the assembled data is synchronized to the second cluster through the message queue. Since the second cluster is a cluster different from the first cluster, it means that while the first cluster is running, the changed data in the first cluster can be assembled and synchronized to a cluster different from the first cluster. Compared with the method of parsing data in the log file, the problem of low data synchronization efficiency caused by the log file including a large amount of data can be avoided, and therefore, the efficiency of cross-cluster data synchronization can be greatly improved. At the same time, since the assembled data includes the identification information of the cluster to be synchronized, it means that the cluster to be synchronized by the assembled data in the first cluster can be accurately determined by the identification information, so the assembled data in the message queue can be more accurately synchronized with the cluster different from the first cluster, so as to better adapt to different data synchronization scenarios, and improve the accuracy of cross-cluster data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a flowchart of a data synchronization method provided by an embodiment of the present application;

[0022] Figure 2 It is a schematic diagram of data assembly provided by an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of cross-cluster data synchronization provided by an embodiment of the present application;

[0024] Figure 4 is a structural diagram of a data synchronization device provided in an embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0028] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0029] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0030] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0032] With the rapid development of digital technology, real-time retrieval and analysis of massive amounts of data often need to be achieved through distributed clusters. In order to ensure the real-time and stability of data retrieval and analysis, data synchronization across clusters is required.

[0033] For example, after the ES cluster establishes an association with the document data through the index, it can perform real-time query through the index. Taking the ES cluster in the e-commerce field as an example, the ES cluster is deployed in two or more data centers (such as the main computer room and the backup computer room), and provides business services (such as product retrieval, etc.) to the outside world through two or more data centers at the same time, so as to effectively utilize cluster resources and improve system service capabilities, especially during peak trading periods, and can meet sudden demands through flexible and fast resource scheduling. Assuming that the data center includes the main computer room and the backup computer room, and the clusters corresponding to the main computer room and the backup computer room undertake different business services, in order to ensure data consistency, the data of the main computer room and the backup computer room need to be synchronized, that is, the data of the main cluster (the cluster corresponding to the main computer room) needs to be synchronized to the backup cluster (the cluster corresponding to the backup computer room), and the data of the backup cluster needs to be synchronized to the main cluster.

[0034] At present, cross-cluster data synchronization is mainly achieved by reading and parsing log files. However, this method requires recording the last read time and then continuing to read from the last read log. If the log file becomes larger and larger, it will cause data backlog, resulting in a decrease in the reading speed, thus affecting the efficiency of cross-cluster data synchronization. At the same time, since the versions of different clusters may be inconsistent, parsing log files may lead to inaccurate data synchronization. In addition, cross-cluster synchronization can also be performed through some official functions. For example, in the cross-cluster replication function provided by Es, the x-park advanced function can be enabled through kibana to synchronize data from different clusters. However, this method charges a high fee, most functions are not required, and the configuration is cumbersome. A kibana needs to be equipped for each cluster, which makes data synchronization inefficient and cannot meet actual business needs.

[0035] Therefore, none of the above methods can accurately and efficiently achieve cross-cluster data synchronization.

[0036] In order to improve the accuracy and efficiency of cross-cluster data synchronization, an embodiment of the present application provides a data synchronization method based on a monitoring plug-in. In this method, a monitoring plug-in is used to monitor whether data changes occur in the first cluster, and when data changes occur in the first cluster, data is assembled according to the data type of the changed data in the first cluster, and the assembled data is synchronized to a cluster different from the first cluster according to a message queue.

[0037] Figure 1A schematic diagram of a data synchronization method provided by an embodiment of the present application is shown, and is described in detail as follows:

[0038] S11 . When the first cluster is running, monitor whether data changes occur in the first cluster.

[0039] The first cluster is a cluster constructed by servers in a data center (or computer room). The first cluster can be a cluster constructed by servers in a single computer room or a cluster constructed by servers in different computer rooms. The data change is a data change caused by user operations. The user operations include one or more operations such as adding, deleting, modifying, batch processing, and updating data. Correspondingly, data changes can include adding data, deleting data, modifying data, batch processing data, and updating data.

[0040] In an optional embodiment of the present application, the above-mentioned cluster may be a Remote Dictionary Server (Redis) cluster or an ES cluster, etc.

[0041] Specifically, a plug-in may be deployed in the first cluster, the plug-in includes a method for monitoring data changes, and the plug-in is synchronously started when the first cluster is started, so that when the first cluster is running, the plug-in is used to monitor whether data changes occur in the first cluster. Alternatively, the encrypted file in the data buffer may be obtained, and the content change of the encrypted file at a preset time interval may be used to monitor whether data changes occur in the first cluster.

[0042] For example, taking the ES cluster deployed in a remote computer room as an example, assuming that the main computer room corresponds to the primary cluster and the backup computer room corresponds to the backup cluster, the plug-in can be installed in the ES clusters corresponding to the servers in the main and backup computer rooms respectively, so that when the main cluster and / or backup cluster is running, the main cluster and / or backup cluster can be synchronously monitored for data changes.

[0043] In the embodiment of the present application, by monitoring whether data changes occur in the first cluster when the first cluster is running, data changes can be detected in real time, thereby improving the efficiency of data synchronization.

[0044] S12: When it is monitored that data changes occur in the first cluster, assemble the changed data in the first cluster.

[0045] The changed data refers to data that has changed due to user operations, including but not limited to added data, deleted data, updated data, etc. Exemplarily, taking the ES cluster as an example, data changes may include changes in index models (i.e., indexes) and / or index data (i.e., document data), wherein the index model includes one or more of index name, index field, index structure, index configuration, etc.; the index data (i.e., document data) includes data corresponding to different index models.

[0046] Specifically, since different information may need to be synchronized when different data changes, in the process of monitoring the first cluster, the data that has changed in the first cluster can be determined first, and then different data fields can be constructed for different changed data, and the changed data can be assembled by assembling the above different data fields.

[0047] For example, taking the ES cluster as an example, if the above-mentioned changed data is determined to be document data through the plug-in, the corresponding data fields may include document id, document version, cluster name, timestamp, etc., and the changed document data is assembled by assembling the above-mentioned data fields.

[0048] In the embodiment of the present application, by directly assembling the changed data, the consistency of the assembled data and the changed data can be ensured, thereby improving the accuracy of data synchronization. At the same time, different data fields are constructed for different changed data, and data assembly is performed by assembling different data fields, which also improves the flexibility of data synchronization.

[0049] S13: Push the assembled data to a pre-built message queue, wherein the assembled data includes identification information of the cluster to be synchronized with data.

[0050] The message queue (MQ) is a container for storing the assembled data during data synchronization. The identification information may be a tag value of a tag, where different tag values ​​may represent different identification information. For example, if the tag value of a tag of assembled data in the message queue is set to prod, it means that the assembled data is synchronized to the main cluster.

[0051] Specifically, for different assembled data, identification information can be set according to actual business synchronization requirements (such as active-active or upgrade), and then the assembled data with the identification information added can be pushed to a pre-built message queue. Optionally, when pushing data, the assembled data can be pushed to a pre-built message queue according to the order of data assembly (for example, according to the order of assembly time corresponding to the assembled data). Of course, the above-mentioned message queue can also be constructed according to the type of data (for example, according to different types of preset priorities), the size of the data (for example, larger data is pushed first), etc., which is not limited here.

[0052] It should also be noted that the number of the above-mentioned message queues can be determined according to the number of clusters to be synchronized. For example, the number of message queues can be set to be equal to the number of clusters to be synchronized. At this time, the message queues and the clusters to be synchronized correspond one to one. Alternatively, when there is a lot of data to be synchronized, the number of message queues can be set to be greater than the number of clusters to be synchronized; when there is less data to be synchronized, the number of message queues can be set to be greater than the number of clusters to be synchronized, so as to adapt to different data synchronization scenarios and improve the efficiency of data synchronization. Of course, the number of the above-mentioned message queues can also be determined according to the changed data, which is not limited here. For example, in the ES cluster, the changed data includes index models and / or index data, then the index model and index data can be pushed to two different message queues respectively.

[0053] In an optional embodiment of the present application, the above-mentioned message queue can be pre-built through message middleware (such as Kafka, ActiveMQ, RabbitMQ, RocketMQ, etc.).

[0054] S14. Synchronize the assembled data to a second cluster corresponding to the identification information according to the message queue; the second cluster is a cluster different from the first cluster.

[0055] Specifically, since the assembled data in the message queue contains the identification information of the cluster to be synchronized with the data, the assembled data can be accurately synchronized to the second cluster according to the identification information. The second cluster can be a cluster of the same version or a different version as the first cluster. If the second cluster is a cluster of a different version from the first cluster, assuming that the first cluster is a cluster of a lower version and the second cluster is a cluster of a higher version, through the above data synchronization method, the cluster of the lower version can be upgraded while the first cluster is running, thereby realizing seamless version upgrade; if the second cluster is a cluster of the same version as the first cluster, real-time data synchronization between the first cluster and the second cluster can be realized while the first cluster is running, thereby improving the efficiency of data synchronization.

[0056] It should also be noted that, in order to avoid data redundancy caused by the identification information or affecting subsequent data processing, after the second cluster to be synchronized is identified by the identification information, the identification information can be deleted. Alternatively, in order to avoid data redundancy, only part of the assembled data can be synchronized.

[0057] In an embodiment of the present application, when the first cluster is running, the first cluster is monitored for data changes, and when the data in the first cluster changes, the changed data in the first cluster is assembled, and then the assembled data is pushed to the pre-built message queue, and the assembled data is synchronized to the second cluster through the message queue. Since the second cluster is a cluster different from the first cluster, it means that while the first cluster is running, the changed data in the first cluster can be assembled and synchronized to a cluster different from the first cluster. Compared with the method of parsing data in the log file, the problem of low data synchronization efficiency caused by the log file including a large amount of data can be avoided, and therefore, the efficiency of cross-cluster data synchronization can be greatly improved. At the same time, since the assembled data includes the identification information of the cluster to be synchronized, it means that the cluster to be synchronized by the assembled data in the first cluster can be accurately determined by the identification information, so the assembled data in the message queue can be more accurately synchronized with the cluster different from the first cluster, so as to better adapt to different data synchronization scenarios, and improve the accuracy of cross-cluster data synchronization.

[0058] In an optional embodiment of the present application, taking an ES cluster as an example, the data changed in the ES cluster may be document data or indexes. Since different configurations are required for monitoring document data or indexes, before monitoring whether data changes occur in the first cluster, the following is further included:

[0059] A monitoring plug-in is constructed according to preset plug-in configuration information; the preset plug-in configuration information includes monitoring configuration information of document data and monitoring configuration information of index.

[0060] Among them, the preset plug-in configuration information includes the configuration information required for the monitoring plug-in to monitor the first cluster. The monitoring configuration information of the document data is used to monitor the changes of the document data in the ES cluster, and the monitoring configuration information of the index is used to monitor the changes of the index in the ES cluster. The preset plug-in configuration information can also include plug-in control configuration information for controlling the monitoring plug-in.

[0061] Specifically, when building a monitoring plug-in, you can first customize the plug-in class (such as MonitorPlugin) and inherit the official basic plug-in class (such as Plugin), that is, the above-mentioned custom plug-in class (such as MonitorPlugin) will obtain the properties and methods defined in the official basic plug-in class (such as Plugin), and then add the preset plug-in configuration information in the custom plug-in class (such as MonitorPlugin), so that you can add new functions or modify existing functions based on the official basic plug-in class (such as Plugin), and finally follow the specifications of the official interface (such as ActionPlugin), so as to realize the extension and customization of the monitoring plug-in through inheritance and interface.

[0062] Among them, in order to enable the monitoring plug-in to better monitor the data changes of the ES cluster, you can rewrite the component creation method (such as createComponents) of the official interface (such as ActionPlugin) in the above custom plug-in class (such as MonitorPlugin), and then add a plug-in configuration custom class (such as MonitorConfiguration) in this method, and add a configuration file (such as ElasticDump.cfg.xml) in the corresponding resource (such as resource) directory, and store the above preset plug-in configuration information in the above file, thereby initializing the plug-in configuration.

[0063] In an optional embodiment of the present application, the plug-in control configuration information in the above-mentioned preset plug-in configuration information may include a remote management interface, etc. When constructing a listening plug-in according to the above-mentioned plug-in control configuration information, the method for obtaining the remote implementation class (e.g., getRestHandlers) of the official interface (e.g., ActionPlugin) may be rewritten in the custom plug-in class (e.g., MonitorPlugin), and then in the method, a custom remote implementation class (e.g., MonitorRestHandler) is added, and the official remote abstract class method (e.g., BaseRestHandler) is implemented. The remote management interface (e.g., http interface) is customized through the BaseRestHandler method, so as to control the start or close of the listening plug-in and data synchronization through the above-mentioned remote management interface; the listening configuration information of the document data in the above-mentioned preset plug-in configuration information may include: monitoring operation instructions that change document data (such as operation instructions such as adding, updating, and deleting), judging whether to enable the listening configuration, collecting document data (including document ID, document content, document version, cluster name, operation type (update / add / delete), index name), etc. When constructing a listening plug-in according to the above-mentioned document data listening configuration information, the listening plug-in class may be first configured with the listener configuration information in the custom plug-in class. In the MonitorPlugin class, rewrite the custom index structure method (e.g. onIndexModule) of the official basic plug-in class (e.g. Plugin). In this method, add a custom document listener implementation class (e.g. DocOperationListen) and implement the official listening interface (IndexingOperationListener). Then rewrite the listening methods, such as the postIndex method (used to listen to the operation instructions of adding and updating document data) and the postDelete method (used to listen to the operation instructions of deleting document data). Finally, add the listening configuration information of document data, such as whether to enable data listening configuration, whether to judge whether it is the main cluster, filtering the index data of the system itself (that is, not collecting the index of the system itself), collecting document data, adding to the local message queue (e.g. BlockingQueue), triggering the local queue listening program, and sending data to the remote data receiving program.The monitoring configuration information of the index in the above-mentioned preset plug-in configuration information may include: monitoring the operation instructions that cause the index to change (such as adding, updating, deleting, creating and other operation instructions), collecting indexes (including index configuration, operation type (update / add / delete / create), index name), etc. When building a monitoring plug-in according to the above-mentioned index monitoring configuration information, you can first rewrite the filtering method (such as getActionFilters) of the official interface (such as ActionPlugin) in the custom plug-in class (such as MonitorPlugin), add a request monitoring custom implementation class (such as MonitorActionFilter) in the method, and implement the official filtering interface (such as ActionFilter), and then rewrite the application method (such as apply method), add monitoring in this method and intercept the operation instructions that cause the relevant index to change, for example, it can include: a. "indices:admin / settings / update" (update setting); b. "indices:admin / mapping / put" (add mapping field); c. "indices:admin / delete" (delete index); d. "indices:admin / create" (Create index), etc., and finally set the monitoring configuration information for reporting indexes. By building a monitoring plug-in through the above method, the above monitoring plug-in can monitor in a targeted manner when the data type is document data or index data changes, thereby improving the accuracy of ES cluster monitoring. ;

[0064] The monitoring of whether data changes occur in the first cluster includes:

[0065] The monitoring plug-in is used to monitor whether data changes occur in the first cluster, wherein the data changes include data changes caused by operation instructions corresponding to different operation types.

[0066] The above operation types include one or more operations such as update, add, delete, create, etc.

[0067] In an embodiment of the present application, when the above-mentioned monitoring plug-in monitors the above-mentioned first cluster, it can determine whether data changes occur in the first cluster by monitoring and intercepting operation instructions corresponding to different operation types. If an operation instruction corresponding to the configured operation type is monitored, it is determined that a data change occurs; otherwise, it is determined that no data change occurs. In this way, it is possible to accurately determine in real time whether data changes occur in the first cluster.

[0068] In some embodiments, taking an ES cluster as an example, the changed data includes index and / or document data, and the data assembly of the changed data in the first cluster includes:

[0069] Determine whether the first cluster includes transmission configuration information; the transmission configuration information includes configuration parameters required for the first cluster and the second cluster to perform data transmission;

[0070] In the case where the first cluster includes transmission configuration information, the changed data is assembled according to a data assembly strategy corresponding to the changed data.

[0071] Among them, the above-mentioned transmission configuration information may include: cluster name (for example, when the main and backup computer rooms push data, the corresponding main and backup cluster names will be included), whether to receive data (for example, when it is not turned on, the data reported by the cluster will be discarded), whether dual active (for example, the versions are inconsistent and dual active is not possible), whether to perform cluster switching (for example, the cluster can switch between main and backup, the backup cluster becomes the main cluster, and the main cluster becomes the backup cluster), etc.

[0072] Specifically, for data that has changed (including document data and / or indexes), the configuration bound to the first cluster can be obtained according to a preset interface, and then a determination can be made as to whether the configuration includes transmission configuration information. If the first cluster includes transmission configuration information, the changed data can be assembled according to the transmission configuration information and the data assembly strategy corresponding to the changed data, wherein the preset interface can correspond to the changed data.

[0073] Exemplarily, if the changed data is document data, the preset interface may be “interface / gzip / doc-change”; if the changed data is an index, the preset interface may be “interface / indices-changed”.

[0074] It should also be noted that, in the case where the first cluster does not include transmission configuration information, data synchronization may be terminated directly or prompted through an alarm.

[0075] In an embodiment of the present application, since the data to be assembled for different changed data (such as document data and indexes) may be different when the ES cluster performs data synchronization, and the server status may be inconsistent with the transmission configuration information due to operational errors, for the changed data, first determine whether the first cluster includes the transmission configuration information. If the first cluster includes the transmission configuration information, then use the transmission configuration information and the data assembly strategy corresponding to the changed data to perform data assembly. This can ensure that the transmission configuration information is used as the basis for data synchronization, thereby improving the accuracy of data assembly.

[0076] In some embodiments, the data assembly of the changed data according to the data assembly strategy corresponding to the changed data includes:

[0077] In the case where the changed data is the document data, data assembling is performed on the document change information corresponding to the document data and the document operation type corresponding to the document data;

[0078] or,

[0079] In the case where the changed data is the index, data assembly is performed according to the index operation type corresponding to the index.

[0080] The document change information refers to information about changes in document data, which may include fields such as document ID, document version, and document data. In addition, in order to ensure the integrity of document data assembly and the traceability of assembled data, the document change information may also include fields such as cluster name, index name, and timestamp. The document operation types include, but are not limited to, document data update, addition, and deletion operations.

[0081] Specifically, in the case where the changed data is document data, the changed information can be assembled in the form of fields to obtain the above-mentioned document change information, and then assembled with the document operation type corresponding to the document data to obtain the assembled data; in the case where the changed data is an index, since the change of the index may cause the index structure in the ES cluster to change, in order to improve the accuracy of data assembly, you can first determine whether the index operation type corresponding to the index causes the change of the index structure. When the index structure changes, the changed index structure will also be assembled, thereby improving the accuracy of data assembly.

[0082] In some embodiments, the data assembly according to the index operation type corresponding to the index includes:

[0083] In the case where the index operation type corresponding to the above index is a deletion type, data assembly is performed according to the index change information of the above index and the above index operation type;

[0084] or,

[0085] In the case where the index operation type corresponding to the above index is an addition type or a modification type, the above index change information is updated, and data assembly is performed using the updated index change information and the above index operation type.

[0086] The index change information refers to information about changes in the index, which may include fields such as cluster name, index name, alias, etc. Of course, the index change information may also include fields that reflect changes in the index structure, including fields such as setting and mappings. The index operation types include, but are not limited to, index update, add, delete, create, and other operation types.

[0087] Specifically, when the index operation type corresponding to the index is a deletion type, the changed information can be assembled in the form of fields to obtain the above-mentioned index change information, and then the data can be assembled with the index operation type corresponding to the index to obtain the assembled data; when the index operation type corresponding to the index is an addition type or a modification type, since the index structure may change, the index structure of the first cluster can be queried in real time, the corresponding index structure can be obtained and assembled to obtain the above-mentioned updated index change information, and then the updated index change information and the above-mentioned index operation type can be used to perform data assembly to obtain the assembled data.

[0088] It should also be noted that for index operation types such as index update and creation that may cause changes in the index structure, the index structure of the first cluster can also be queried in real time before data assembly, which will not be repeated here.

[0089] In the embodiment of the present application, corresponding data assembly processing is performed for different index operation types, which can improve the accuracy of data assembly when the index changes, thereby improving the accuracy of cross-cluster data synchronization.

[0090] In some embodiments, the transmission configuration information includes a data synchronization type, and the step of pushing the assembled data to a pre-built message queue includes:

[0091] The assembled data is identified according to the data synchronization type to obtain synchronization identification information; the synchronization identification information is used to indicate the cluster to which the assembled data is to be synchronized;

[0092] The identified assembled data is pushed to the pre-built message queue.

[0093] The data synchronization type is used to indicate the purpose of data synchronization of the current cluster, which may include active-active or cluster upgrade (ie, non-active-active), etc. The synchronization identification information is the identification information of the data assembled in the message queue.

[0094] Specifically, for the assembled data, different labels can be set according to the above-mentioned data synchronization type to obtain the corresponding synchronization identification information, and then the identified assembled data can be pushed to the pre-built message queue. Since the above-mentioned synchronization identification information is used to indicate the cluster to which the assembled data is to be synchronized, the cluster to be synchronized can directly consume the data in the above-mentioned message queue according to the above-mentioned synchronization identification information, thereby improving the accuracy of cross-cluster data synchronization.

[0095] In an optional embodiment of the present application, when data is synchronized in a cluster of two computer rooms, it is assumed that the first cluster is the cluster of the main computer room (i.e., the main cluster). If the data synchronization type is a cluster upgrade (i.e., non-active-active), the tag of the assembled data can be set to prod, indicating that the assembled data is used for upgrading the current version of the cluster; if the data synchronization type is active-active, the tag can be set to idc, indicating that the assembled data is used for synchronization to other clusters of the same version.

[0096] In order to better describe the data assembly process, the following Figure 2 To illustrate, take the first cluster as the main cluster as an example. During the operation of the main cluster, the data changes of the main cluster are monitored through the monitoring plug-in. When data changes are detected, the different changed data are processed through the API interface provided by the data receiving program. If the changed data is document data, the above API interface is "interface / gzip / doc-change"; if the changed data is an index, the above API interface is "interface / indices-changed"; then the configuration information bound to the main cluster is obtained. If the transmission configuration information is included, subsequent data assembly is performed. If not, the data synchronization is terminated. During data synchronization, if the changed data is document data, the tag tag can be set to prod or idc after the data is assembled. If the tag (tag) is set to prod, it means that the assembled data is used to upgrade the current version of the cluster, and if the tag (tag) is set to idc, it means that the assembled data is used to synchronize to other clusters of the same version. If the changed data is an index, different processing can be performed according to the index operation type. When the index operation type is deletion, the tag tag can be set to prod or idc after data assembly. If the index operation type is addition or modification, the index structure can be queried in real time, and then the changed index structure is assembled, and the tag tag is set to prod or idc. Finally, a message queue is built based on the marked data.

[0097] In some embodiments, the synchronization identification information includes first synchronization identification information or second synchronization identification information, and synchronizing the assembled data to the second cluster corresponding to the identification information according to the message queue includes:

[0098] In the case where the synchronization identification information of the assembled data in the message queue is the first synchronization identification information, synchronizing the assembled data to a cluster of a different version from the first cluster corresponding to the first synchronization identification information;

[0099] or,

[0100] When the synchronization identification information of the assembled data in the message queue is the second synchronization identification information, the assembled data is synchronized to a cluster of the same version as the first cluster corresponding to the second synchronization identification information.

[0101] Specifically, the first synchronization identification information is used to indicate that the assembled data is synchronized to a higher version cluster, and the second synchronization identification information is used to indicate that the assembled data is synchronized to other clusters of the same version. During the data synchronization process, in order to improve the efficiency of data synchronization, different synchronization methods can be used according to the changed data. In the case where the changed data is document data, since the amount of data may be large, the assembled data can be first stored in the database, and then the data can be synchronized in batches from the database; in the case where the changed data is an index, the assembled data can be directly synchronized to the corresponding cluster. Of course, for the above-mentioned index, the assembled data can also be first stored in the database, and then the data can be synchronized in batches from the database.

[0102] It should also be noted that since the data can be synchronized to a higher version cluster during the operation of the first cluster through the above-mentioned first synchronization identification information, there is no need for reinitialization and the like. The higher version cluster after data synchronization can be used directly, thereby achieving seamless version upgrade.

[0103] Exemplarily, taking the active-active cluster of the primary and standby clusters as an example, the assembled data corresponding to the primary and standby clusters will be marked with the second synchronization identification information, which is used to indicate that the assembled data will be synchronized to other clusters of the same version. Assuming that the assembled second synchronization identification information corresponding to the primary cluster is idc, it indicates that the assembled data will be synchronized to the standby cluster, and the assembled second synchronization identification information corresponding to the standby cluster is prod, which indicates that the assembled data will be synchronized to the primary cluster.

[0104] Among them, for the assembled data in the data queue, the server in the main computer room will listen to the data with the tag prod, and the server in the backup computer room will listen to the data with the tag idc. Then, for the data with the tag prod, the main computer room stores the data of the document data type that is listened to in the database. When the main cluster synchronizes the document data, the data synchronization can be performed according to the following steps: 1. Batch obtain the data to be synchronized from the database (for example, scheduled batch acquisition); 2. Perform data deduplication (for example, for the data with the same document ID, keep the one with the highest document version); 3. Assemble the data packets of the batch request; 4. Synchronize the assembled data packets to the main cluster; 5. Clean up the synchronized data. Similarly, the backup cluster corresponding to the backup computer room can perform data synchronization according to the above method, which will not be repeated here.

[0105] To better describe cross-cluster data synchronization, the following is combined Figure 3 For example, the monitoring plug-in is installed in the servers corresponding to the primary and backup clusters. The primary and backup clusters are monitored simultaneously through asynchronous processing. When user operations (including deletion, addition, batch processing, update, etc.) cause data changes in the primary and backup clusters, the data receiving processing program will assemble data and set tags according to different user operations (see Figure 2 And related descriptions, no longer repeated here), and then build the message queue of the corresponding computer room. If the data synchronization type is active-active, for the assembled data in the data queue corresponding to the standby computer room, the main computer room will listen to the data with the tag prod, and for the assembled data in the data queue corresponding to the main computer room, the standby computer room will listen to the data with the tag idc, and then the main and standby computer rooms will synchronize the monitored data to the corresponding clusters through data consumption, thereby realizing active-active synchronization of the main and standby clusters; if the data synchronization type is not active-active (such as version upgrade, refer to the dotted line part), for the assembled data in the data queue corresponding to the main computer room, the main computer room will listen to the data with the tag prod (at this time, the data with the tag prod is the data assembled by the low-version main cluster), and then synchronize the monitored data to the high-version main cluster, thereby realizing seamless version upgrade of the main cluster. Among them, when the servers in each computer room synchronize the monitored data to the corresponding cluster, different methods of data synchronization can be performed for the changed data. That is, if the changed data is document data, the corresponding assembled data can be stored in the database, and then synchronized to the corresponding cluster in batches through scheduled scheduling. If the changed data is an index, the corresponding assembled data can be directly synchronized to the corresponding cluster in batches through scheduled scheduling.

[0106] It should also be noted that in the above Figure 3 In the process of upgrading the standby cluster, the data of the active cluster has been synchronized to the standby cluster during the active-active synchronization. Therefore, the standby cluster can be upgraded directly based on the synchronized data, that is, the standby cluster does not need to use its own synchronized data for version upgrade. Of course, if a failure occurs and the active and standby clusters are switched, the standby cluster becomes the active cluster. At this time, the active cluster (actually the original standby cluster) can upgrade the version through its own synchronized data.

[0107] In another optional embodiment of the present application, since the monitoring plug-in is extended after the running first cluster, the data before the first cluster applies the monitoring plug-in may not be synchronized through the monitoring plug-in. Before monitoring whether the data of the first cluster changes, the following is further included:

[0108] The data of the first cluster is backed up to a cluster of the same version as the first cluster through a data snapshot.

[0109] Specifically, the first cluster and a cluster of the same version as the first cluster can be bound to the same address first, and then a mirror of the first cluster can be generated in the form of a data snapshot. The mirror of the first cluster can be restored in a cluster of the same version as the first cluster, thereby backing up the data of the first cluster to a cluster of the same version as the first cluster through the data snapshot.

[0110] Assuming that the first cluster is the active cluster and the above cluster of the same version as the first cluster is the standby cluster, you can first make the servers corresponding to the active and standby clusters bind to the same Network Attached Storage (NAS) address, that is, the active and standby clusters use the same NAS address. For example, you can bind the same NAS address through the following steps: 1. Create a directory: mkdir-p / es / backups; 2. Authorize the directory: chown-R elasticsearch:elasticsearch / directory; 3. Mount the directory: mount-t nfs IP: / nasdirectory / directory. When generating the image of the first cluster in the form of data snapshot, you can first enable the mirror function in the active and standby clusters (for example, under the config of the active and standby clusters, edit the elasticsearch.yml file, add the configuration: path.repo: ["directory"], and enable the mirror function after restarting to take effect), then create snapshot directories for the active and standby clusters, and then generate the image of the active cluster. The image of the active cluster is restored in the standby cluster (for example, restore the image through the method of POST _snapshot / directory / image backup name / _restore?wait_for_completion=true), thereby backing up the data of the active cluster to the standby cluster through data snapshot.

[0111] In an embodiment of the present application, the data of the first cluster can be backed up to a cluster of the same version as the above-mentioned first cluster through the data snapshot method, so that data that cannot be synchronized through the monitoring plug-in can be synchronized, which can improve the accuracy and completeness of cross-cluster data synchronization.

[0112] In another optional embodiment of the present application, during the data synchronization process, since the first cluster may fail, it is necessary to switch the first cluster to another cluster for data synchronization. Therefore, after synchronizing the assembled data to the second cluster corresponding to the identification information according to the message queue, the method further includes:

[0113] Monitoring a preset domain name, wherein the preset domain name is a domain name associated with the first cluster;

[0114] When it is monitored that the cluster associated with the preset domain name changes, the first cluster is switched to a cluster of the same version as the first cluster.

[0115] Specifically, you can first apply for a set of preset domain names, associate the preset domain names with the above-mentioned first cluster by resolving the preset domain names to the first cluster, and then monitor whether the domain name resolution result of the above-mentioned preset domain names changes. When the domain name resolution result of the above-mentioned preset domain names changes, resolve the above-mentioned preset domain names to a cluster of the same version as the above-mentioned first cluster, and then modify the corresponding configuration information to switch the above-mentioned first cluster to a cluster of the same version as the above-mentioned first cluster.

[0116] Also assume that the first cluster is the primary cluster, and the above cluster of the same version as the first cluster is the standby cluster. First, apply for a set of domain names, resolve the domain names to the primary cluster, and then monitor whether the resolution result of the domain name changes. When the resolution result of the domain name changes, first resolve the domain name to the standby cluster, and then modify the transmission configuration information again, set the primary cluster to the standby cluster, set the standby cluster to the primary cluster, and notify the ES cluster to modify the local configuration, including: i. Notify through the interface of the monitoring plug-in; ii. Enable data transmission of the standby cluster and disable data transmission of the primary cluster, so as to switch the primary cluster to the standby cluster.

[0117] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0118] Corresponding to the data synchronization method described in the above embodiment, Figure 4 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0119] Reference Figure 4 The device may be a data synchronization device 41 , and the data synchronization device 41 may include a monitoring module 411 , a data assembly module 412 , a message queue construction module 413 and a data synchronization module 414 .

[0120] Reference Figure 4 , the data synchronization device 41 comprises:

[0121] The monitoring module 411 is used to monitor whether data changes occur in the first cluster when the first cluster is running;

[0122] The data assembly module 412 is used to assemble the changed data in the first cluster when monitoring the data change in the first cluster;

[0123] The message queue construction module 413 is used to push the assembled data to the pre-constructed message queue, wherein the assembled data includes the identification information of the cluster to be synchronized with the data;

[0124] The data synchronization module 414 is used to synchronize the assembled data to the second cluster corresponding to the identification information according to the message queue; the second cluster is a cluster different from the first cluster.

[0125] In some other embodiments, the data synchronization device 41 further includes a plug-in construction module, and the plug-in construction module is used to monitor whether data changes occur in the first cluster, including:

[0126] A monitoring plug-in is constructed according to preset plug-in configuration information; the preset plug-in configuration information includes monitoring configuration information of document data and monitoring configuration information of index.

[0127] In some embodiments, when monitoring whether data changes occur in the first cluster, the monitoring module 411 is used to:

[0128] The monitoring plug-in is used to monitor whether data changes occur in the first cluster, wherein the data changes include data changes caused by operation instructions corresponding to different operation types.

[0129] In some embodiments, the changed data includes index and / or document data. When the data assembly module 412 assembles the changed data in the first cluster, it is used to:

[0130] Determine whether the first cluster includes transmission configuration information; the transmission configuration information includes configuration parameters required for the first cluster and the second cluster to perform data transmission;

[0131] In the case where the first cluster includes transmission configuration information, the changed data is assembled according to a data assembly strategy corresponding to the changed data.

[0132] In some embodiments, when the data assembly module 412 assembles the changed data according to the data assembly strategy corresponding to the changed data, it is used to:

[0133] In the case where the changed data is the document data, data assembling is performed on the document change information corresponding to the document data and the document operation type corresponding to the document data;

[0134] or,

[0135] In the case where the changed data is the index, data assembly is performed according to the index operation type corresponding to the index.

[0136] In some embodiments, when the data assembly module 412 performs data assembly according to the index operation type corresponding to the index, it is used to:

[0137] In the case where the index operation type corresponding to the above index is a deletion type, data assembly is performed according to the index change information of the above index and the above index operation type;

[0138] or,

[0139] In the case where the index operation type corresponding to the above index is an addition type or a modification type, the above index change information is updated, and data assembly is performed using the updated index change information and the above index operation type.

[0140] In some embodiments, the transmission configuration information includes a data synchronization type, and the message queue construction module 413 is used to:

[0141] The assembled data is identified according to the data synchronization type to obtain synchronization identification information; the synchronization identification information is used to indicate the cluster to which the assembled data is to be synchronized;

[0142] The identified assembled data is pushed to the pre-built message queue.

[0143] In some embodiments, the synchronization identification information includes first synchronization identification information or second synchronization identification information, and when the data synchronization module 414 synchronizes the assembled data to the second cluster corresponding to the identification information according to the message queue, it is used to:

[0144] In the case where the synchronization identification information of the assembled data in the message queue is the first synchronization identification information, synchronizing the assembled data to a cluster of a different version from the first cluster corresponding to the first synchronization identification information;

[0145] or,

[0146] When the synchronization identification information of the assembled data in the message queue is the second synchronization identification information, the assembled data is synchronized to a cluster of the same version as the first cluster corresponding to the second synchronization identification information.

[0147] In another optional embodiment, since the monitoring plug-in is extended after the running first cluster, the data before the monitoring plug-in is applied to the first cluster may not be synchronized through the monitoring plug-in, and the data synchronization device 41 further includes a data snapshot module, which is used for monitoring whether the data of the first cluster changes before the data changes, including:

[0148] The data of the first cluster is backed up to a cluster of the same version as the first cluster through a data snapshot.

[0149] In another optional embodiment, during the data synchronization process, since the first cluster may fail, it is necessary to switch the first cluster to another cluster for data synchronization. The data synchronization device 41 further includes a switching module, which is used to synchronize the assembled data to the second cluster corresponding to the identification information according to the message queue, including:

[0150] Monitoring a preset domain name, wherein the preset domain name is a domain name associated with the first cluster;

[0151] When it is monitored that the cluster associated with the preset domain name changes, the first cluster is switched to a cluster of the same version as the first cluster.

[0152] It should be noted that the information interaction, execution process, etc. between the devices / units are based on the same concept as the method embodiments of the present application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0153] Figure 5 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 5 As shown, the electronic device 5 of this embodiment includes: at least one processor 50 ( Figure 5 Only one is shown in the figure), a memory 51, and a computer program 52 stored in the memory 51 and executable on the at least one processor 50. When the processor 50 executes the computer program 52, the steps in any of the method embodiments are implemented.

[0154] The electronic device 5 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will appreciate that Figure 5 It is only an example of the electronic device 5 and does not constitute a limitation of the electronic device 5. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include an input sending device, a network access device, a bus, etc.

[0155] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0156] In some embodiments, the memory 51 may be an internal storage unit of the electronic device 5, such as a hard disk or memory of the electronic device 5. The memory 51 may also be an external storage device of the electronic device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. Further, the memory 51 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 51 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 51 may also be used to temporarily store data that has been sent or is to be sent.

[0157] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the functional units and modules is used as an example. In practical applications, the function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0158] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the method embodiments when executing the computer program.

[0159] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the various method embodiments can be implemented.

[0160] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the various method embodiments when executing the computer program product.

[0161] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0162] In the embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0163] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0164] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0165] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data synchronization method, It is characterized in that The method comprises: When the first cluster is running, monitoring whether data changes occur in the first cluster; When it is monitored that data changes occur in the first cluster, data assembling is performed on the changed data in the first cluster by field assembly, wherein the data fields in the assembled data correspond to the changed data; Pushing the assembled data to a pre-built message queue, wherein the assembled data includes identification information of the cluster to be synchronized with data; the identification information is used to indicate the data synchronization requirement and the cluster corresponding to the data synchronization requirement, the data synchronization requirement at least includes cluster active-active and / or cluster upgrade, and the identification information can be applied to both the cluster active-active and the cluster upgrade; The assembled data in the message queue is synchronized to a second cluster corresponding to the identification information according to the identification information; the second cluster is a cluster different from the first cluster, and a version of the second cluster is the same as or different from a version of the first cluster based on the data synchronization requirement.

2. The data synchronization method according to claim 1, It is characterized in that Before monitoring whether data change occurs in the first cluster, the method further includes: Constructing a monitoring plug-in according to preset plug-in configuration information; the preset plug-in configuration information includes monitoring configuration information of document data and monitoring configuration information of index; The monitoring whether data change occurs in the first cluster includes: The monitoring plug-in is used to monitor whether data changes occur in the first cluster, wherein the data changes include data changes caused by operation instructions corresponding to different operation types.

3. The data synchronization method according to claim 2, It is characterized in that The changed data includes index and / or document data, and the step of assembling the changed data in the first cluster by field assembly includes: Determining whether the first cluster includes transmission configuration information; the transmission configuration information includes configuration parameters required for the first cluster and the second cluster to perform data transmission; In the case where the first cluster includes transmission configuration information, the data field corresponding to the changed data is determined according to the data assembly strategy corresponding to the changed data, and the changed data is assembled according to the determined data field.

4. The data synchronization method according to claim 3, It is characterized in that The step of determining a data field corresponding to the changed data according to a data assembly strategy corresponding to the changed data, and assembling the changed data according to the determined data field, comprises: In the case where the changed data is the document data, data assembling is performed on the document change information corresponding to the document data and the document operation type corresponding to the document data; the document change information includes the data field corresponding to the document data; or, In the case where the changed data is the index, the data field corresponding to the changed data is determined according to the index operation type corresponding to the index, and data assembly is performed according to the determined data field.

5. The data synchronization method according to claim 4, It is characterized in that The step of determining the data field corresponding to the changed data according to the index operation type corresponding to the index, and assembling the data according to the determined data field, comprises: In the case where the index operation type corresponding to the index is a deletion type, data assembly is performed according to the index change information of the index and the index operation type; the index change information includes a data field corresponding to the index; or, In the case where the index operation type corresponding to the index is an addition type or a modification type, the index change information is updated, and data assembly is performed using the updated index change information and the index operation type.

6. The data synchronization method according to any one of claims 3 to 5, It is characterized in that The transmission configuration information includes a data synchronization type, and the step of pushing the assembled data to a pre-built message queue includes: The assembled data is identified according to the data synchronization type to obtain synchronization identification information; the synchronization identification information is used to indicate the cluster to be synchronized with the assembled data; The identified assembled data is pushed to the pre-built message queue.

7. The data synchronization method according to claim 6, It is characterized in that The synchronization identification information includes first synchronization identification information or second synchronization identification information, and synchronizing the assembled data in the message queue to the second cluster corresponding to the identification information according to the identification information includes: When the synchronization identification information of the assembled data in the message queue is the first synchronization identification information, synchronizing the assembled data to a cluster of a different version from the first cluster corresponding to the first synchronization identification information; or, When the synchronization identification information of the assembled data in the message queue is the second synchronization identification information, the assembled data is synchronized to a cluster corresponding to the second synchronization identification information and having the same version as the first cluster.

8. The data synchronization method according to any one of claims 1 to 5, It is characterized in that Before monitoring whether data changes occur in the first cluster, the method further includes: The data of the first cluster is backed up to a cluster of the same version as the first cluster through a data snapshot.

9. The data synchronization method according to any one of claims 1 to 5, It is characterized in that After synchronizing the assembled data in the message queue to the second cluster corresponding to the identification information according to the identification information, the method further includes: Monitoring a preset domain name, wherein the preset domain name is a domain name associated with the first cluster; When it is monitored that the cluster associated with the preset domain name changes, the first cluster is switched to a cluster of the same version as the first cluster.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.