Data synchronization storage method and device, processor and electronic equipment

Abstract

The application provides a data synchronization storage method and device, a processor and an electronic device. The storage method comprises: generating an ESSink operator; obtaining target data from Kafka, executing a first operator to store the target data to a first cluster, and executing a second operator to store the target data to a second cluster; generating a first receipt instruction after executing the first operator, and generating a second receipt instruction after executing the second operator; determining whether the target data is successfully stored to the first cluster and the second cluster according to the first receipt instruction and the second receipt instruction, storing the target data to a first compensation table of the first cluster in the case that the target data is not successfully stored to the first cluster, and storing the target data to a second compensation table of the second cluster in the case that the target data is not successfully stored to the second cluster. Through the application, the problem that the ESSink operator cannot write data to double clusters simultaneously is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular, to a data synchronization storage method and device, a computer readable storage medium, a processor and an electronic device. BACKGROUND

[0002] Apache Flink is a framework and distributed processing engine for stateful computation over unbounded and bounded data streams. A native connector for connecting Elasticsearch (a search server based on Lucene, referred to as ES) is provided in the Flink distributed processing engine, which is used to write stream data into ES in real time. For a specific ES cluster, the ESSink (ES gateway node, Elasticsearch Sink, referred to as ESSink) operator of the Flink distributed processing engine uses a batch processor to complete the data writing function, and the batch processor submits 1000 pieces of data at a time by default.

[0003] Although the Apache Flink native ESSink has provided a method for writing stream data into ES, this solution has many disadvantages: a gateway node can only write to a single cluster, and cannot support the demand for simultaneous data synchronization writing to double clusters; and when writing to a cluster fails, only data retry can be performed, and after data retry fails, data compensation processing cannot be performed, resulting in data loss.

[0004] Therefore, there is an urgent need for a method that can write data to double clusters at the same time. SUMMARY

[0005] The main purpose of the present application is to provide a data synchronization storage method and device, a computer readable storage medium, a processor and an electronic device, to at least solve the problem that the ESSink operator in the prior art cannot write data to double clusters at the same time.

[0006] To achieve the above object, according to one aspect of the present application, a data synchronization storage method is provided, which is applied in an Apache Flink processing engine, and includes: generating an ESSink operator, wherein the ESSink operator includes a first operator and a second operator; obtaining target data from Kafka, executing the first operator to store the target data to a first cluster, and executing the second operator to store the target data to a second cluster; after executing the first operator, generating a first receipt instruction, and after executing the second operator, generating a second receipt instruction, wherein the first receipt instruction includes a first execution result identifier and a first cluster identifier, the first execution result identifier is used to represent whether the target data is successfully written to the first cluster, and the first cluster identifier is used to represent the first cluster, the second receipt instruction includes a second execution result identifier and a second cluster identifier, the second execution result identifier is used to represent whether the target data is successfully written to the second cluster, and the second cluster identifier is used to represent the second cluster; according to the first receipt instruction and the second receipt instruction, determining whether the target data is successfully stored to the first cluster and the second cluster, in the case that the target data is not successfully stored to the first cluster, storing the target data to a first compensation table of the first cluster, and in the case that the target data is not successfully stored to the second cluster, storing the target data to a second compensation table of the second cluster.

[0007] Optionally, according to the first receipt instruction and the second receipt instruction, determining whether the target data is successfully stored to the first cluster and the second cluster includes: in the case that the first receipt instruction contains the first cluster identifier and the first execution result identifier is successful, determining that the target data is successfully stored to the first cluster, and in the case that the second receipt instruction contains the second cluster identifier and the second execution result identifier is successful, determining that the target data is successfully stored to the second cluster; in the case that the first receipt instruction contains the first cluster identifier and the first execution result identifier is failed, determining that the target data is not successfully stored to the first cluster, and in the case that the second receipt instruction contains the second cluster identifier and the second execution result identifier is failed, determining that the target data is not successfully stored to the second cluster.

[0008] Optionally, after storing the target data into the first compensation table of the first cluster, the method further comprises: storing the target data into the first compensation table of the first cluster, generating a third receipt instruction; determining whether the target data is successfully stored into the first compensation table of the first cluster according to the third receipt instruction, and storing the target data into a third compensation table of the first cluster in the case that the target data is not successfully stored into the first compensation table of the first cluster; after storing the target data into the second compensation table of the second cluster, the method further comprises: storing the target data into the second compensation table of the second cluster, generating a fourth receipt instruction; determining whether the target data is successfully stored into the second compensation table of the second cluster according to the fourth receipt instruction, and storing the target data into a fourth compensation table of the second cluster in the case that the target data is not successfully stored into the second compensation table of the second cluster.

[0009] Optionally, determining whether the target data is successfully stored into the first compensation table of the first cluster according to the third receipt instruction comprises: in the case that the first cluster identifier is contained in the third receipt instruction and the third execution result identifier is successful, determining that the target data is successfully stored into the first compensation table of the first cluster; and in the case that the first cluster identifier is contained in the third receipt instruction and the third execution result identifier is failed, determining that the target data is not successfully stored into the first compensation table of the first cluster; and determining whether the target data is successfully stored into the second compensation table of the second cluster according to the fourth receipt instruction comprises: in the case that the second cluster identifier is contained in the fourth receipt instruction and the fourth execution result identifier is successful, determining that the target data is successfully stored into the second compensation table of the second cluster; and in the case that the second cluster identifier is contained in the fourth receipt instruction and the fourth execution result identifier is failed, determining that the target data is not successfully stored into the second compensation table of the second cluster.

[0010] Optionally, after storing the target data into the third compensation table of the first cluster, the method further comprises: storing the target data into the third compensation table of the first cluster, generating a fifth receipt instruction; determining whether the target data is successfully stored into the third compensation table of the first cluster according to the fifth receipt instruction, and generating a log file of the target data in the case that the target data is not successfully stored into the third compensation table of the first cluster; and after storing the target data into the fourth compensation table of the second cluster, the method further comprises: storing the target data into the fourth compensation table of the second cluster, generating a sixth receipt instruction; determining whether the target data is successfully stored into the fourth compensation table of the second cluster according to the sixth receipt instruction, and generating a log file of the target data in the case that the target data is not successfully stored into the fourth compensation table of the second cluster.

[0011] Optionally, determining whether the target data was successfully stored in the third compensation table of the first cluster according to the fifth receipt instruction includes: if the fifth receipt instruction contains the first cluster identifier and the fifth execution result identifier is successful, determining that the target data was successfully stored in the third compensation table of the first cluster; if the fifth receipt instruction contains the first cluster identifier and the fifth execution result identifier is failure, determining that the target data was not successfully stored in the third compensation table of the first cluster. Determining whether the target data was successfully stored in the fourth compensation table of the second cluster according to the sixth receipt instruction includes: if the sixth receipt instruction contains the second cluster identifier and the sixth execution result identifier is successful, determining that the target data was successfully stored in the fourth compensation table of the second cluster; if the fourth receipt instruction contains the second cluster identifier and the sixth execution result identifier is failure, determining that the target data was not successfully stored in the fourth compensation table of the second cluster.

[0012] Optionally, after executing the first operator, a first receipt instruction is generated, and after executing the second operator, a second receipt instruction is generated, including: after executing the first operator, receiving the first receipt instruction generated by the first cluster, and after executing the second operator, receiving the second receipt instruction generated by the second cluster.

[0013] According to another aspect of the present application, there is provided a data synchronization storage apparatus applied in an Apache Flink processing engine, comprising: a first generating unit configured to generate an ESSink operator, wherein the ESSink operator comprises a first operator and a second operator; a first storing unit configured to acquire target data from a Kafka, execute the first operator to store the target data to a first cluster, and execute the second operator to store the target data to a second cluster; a second generating unit configured to generate a first receipt instruction after executing the first operator, and generate a second receipt instruction after executing the second operator, wherein the first receipt instruction comprises a first execution result identifier and a first cluster identifier, the first execution result identifier is used to represent whether the target data is successfully written to the first cluster, and the first cluster identifier is used to represent the first cluster, the second receipt instruction comprises a second execution result identifier and a second cluster identifier, the second execution result identifier is used to represent whether the target data is successfully written to the second cluster, and the second cluster identifier is used to represent the second cluster; and a second storing unit configured to determine whether the target data is successfully stored to the first cluster and the second cluster according to the first receipt instruction and the second receipt instruction, store the target data to a first compensation table of the first cluster in a case that the target data is not successfully stored to the first cluster, and store the target data to a second compensation table of the second cluster in a case that the target data is not successfully stored to the second cluster.

[0014] According to still another aspect of the present application, there is provided a computer-readable storage medium, comprising a stored program, wherein the computer-readable storage medium is caused to perform any of the data synchronization storage methods when the program is executed.

[0015] According to yet another aspect of the present application, there is provided a processor configured to execute a program, wherein the processor is caused to perform any of the data synchronization storage methods when the program is executed.

[0016] According to still another aspect of the present application, there is provided an electronic device, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs comprise instructions for performing any of the data synchronization storage methods.

[0017] The technical scheme of the application is applied to generate an ESSink operator, wherein the ESSink operator comprises a first operator and a second operator; target data is acquired from Kafka, the first operator is executed to store the target data to a first cluster, and the second operator is executed to store the target data to a second cluster; after the execution of the first operator, a first receipt instruction is generated, and after the execution of the second operator, a second receipt instruction is generated, wherein the first receipt instruction comprises a first execution result identifier and a first cluster identifier, the first execution result identifier is used to represent whether the target data is successfully written to the first cluster, and the first cluster identifier is used to represent the first cluster, the second receipt instruction comprises a second execution result identifier and a second cluster identifier, the second execution result identifier is used to represent whether the target data is successfully written to the second cluster, and the second cluster identifier is used to represent the second cluster; according to the first receipt instruction and the second receipt instruction, it is determined whether the target data is successfully stored to the first cluster and the second cluster, in the case that the target data is not successfully stored to the first cluster, the target data is stored to a first compensation table of the first cluster, and in the case that the target data is not successfully stored to the second cluster, the target data is stored to a second compensation table of the second cluster. Compared with the prior art in which the ESSink operator can only write data to a single cluster, the application simultaneously writes target data to two clusters by defining the ESSink operator comprising the first operator and the second operator, adds a receipt instruction in the operator, generates a corresponding receipt instruction after the execution of the first operator and the second operator, and judges whether the two clusters are successfully written by the cluster identifier and the execution result identifier in the receipt instruction. When the writing of a certain cluster fails, the target data is stored to a compensation table of the corresponding cluster to ensure the integrity of the data, thereby solving the problem that the ESSink operator cannot simultaneously write data to two clusters. BRIEF DESCRIPTION OF DRAWINGS

[0018] The drawings accompanying the specification of the present application form a part thereof, serve to provide further understanding of the present application, and together with the specification explain the application. The use of these drawings in explaining the application does not imply that the present application should be limited to these embodiments. In the drawings:

[0019] Figure 1 A hardware structure block diagram of a mobile terminal for executing a data synchronization storage method is shown according to an embodiment of the present application;

[0020] Figure 2 A flowchart of a data synchronization storage method is shown according to an embodiment of the present application;

[0021] Figure 3 A flowchart of a specific data synchronization storage method is shown according to an embodiment of the present application;

[0022] Figure 4A flow diagram of another specific data synchronization storage method provided by an embodiment of the present application is shown.

[0023] Figure 5 A structural block diagram of a data synchronization storage device provided by an embodiment of the present application is shown.

[0024] Among the above figures, the following reference signs are included:

[0025] 102, processor; 104, memory; 106, transmission device; 108, input / output device. DETAILED DESCRIPTION

[0026] It should be noted that the embodiments and features in the present application can be combined with each other without conflict. The technical solutions in the embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0028] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] For the convenience of description, the following describes some nouns or terms related to the embodiments of the present application:

[0030] Flink: Apache Flink is an open source stream processing framework developed by Apache Software Foundation, and its core is a distributed stream data flow engine written in Java and Scala. Flink executes any stream data program in a data parallel and pipelined manner, and the pipelined runtime system of Flink can execute batch processing and stream processing programs. In addition, the runtime of Flink itself also supports the execution of iterative algorithms.

[0031] Elasticsearch: Elasticsearch is a Lucene-based search server, abbreviated as ES. It provides a distributed multi-user capability full-text search engine based on the RESTful web interface. Elasticsearch is developed in Java language and released as open source under the Apache license, and is a popular enterprise search engine. Elasticsearch is used in cloud computing, which can achieve real-time search, stability, reliability, speed, and easy installation and use.

[0032] ESSink operator: Elasticsearch gateway node operator for writing streaming data to an ES cluster;

[0033] Kafka: an open source stream processing platform developed by the Apache Software Foundation, which is a high-throughput distributed publish / subscribe message system.

[0034] As introduced in the background, the ESSink operator in the prior art can only write to a single cluster. To solve the problem that the ESSink operator cannot write data to dual clusters at the same time, the embodiments of the present application provide a data synchronization storage method, device, computer readable storage medium, processor and electronic equipment.

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application.

[0036] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking the case of running on a mobile terminal, Figure 1 is a hardware structure block diagram of a mobile terminal of a data synchronization storage method according to an embodiment of the present application. As shown in Figure 1 , the mobile terminal can include one or more (only one is shown in Figure 1 ) processor 102 (the processor 102 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the above-mentioned mobile terminal can further include a transmission device 106 for communication function and an input / output device 108. Those skilled in the art can understand that Figure 1 the structure shown is only schematic, which does not limit the structure of the above-mentioned mobile terminal. For example, the mobile terminal can further include more or fewer components than those shown in Figure 1 , or have a different configuration from Figure 1 .

[0037] The memory 104 can be used to store computer programs, such as software programs of application software and modules, such as a computer program corresponding to the data synchronization storage method of the embodiments of the present application. The processor 102 can execute various functional applications and data processing, i.e., implement the above-mentioned method, by running the computer programs stored in the memory 104. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include memories remotely arranged with respect to the processor 102, which can be connected to the mobile terminal through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. The transmission device 106 is used to receive or send data via a network. The specific examples of the above-mentioned network can include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet in a wireless manner.

[0038] In the present embodiment, a data synchronization storage method running on a mobile terminal, a computer terminal or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] Figure 2 is a flowchart of the data synchronization storage method according to the embodiments of the present application. As shown in Figure 2 , the method includes the following steps:

[0040] Step S201, generating an ESSink operator, wherein the ESSink operator includes a first operator and a second operator.

[0041] Specifically, the ESSink operator is used to write target data, i.e., streaming data, into a corresponding cluster, i.e., an ES cluster. Since a native Sink operator provided in an Apache Flink processing engine can only write data into one ES cluster, i.e., an ES single cluster, when two native Sink operators are used to implement simultaneous writing of data into double clusters, when writing into a certain cluster fails, data retry can only be performed on a single cluster, and after data retry fails, compensation processing cannot be performed on the data, resulting in data loss. The present application generates a new ESSink operator by self-definition. The above ESSink operator includes a first operator and a second operator. In the specific implementation process, the ESSink operator can also be a corresponding operator in other processing engines. The first operator stores target data into a first cluster, and the second operator stores target data into a second cluster. Of course, in some optional embodiments, the first operator can also store target data into the second cluster, and the second operator can store target data into the first cluster.

[0042] Step S202, target data is obtained from Kafka, the first operator is executed to store the target data into a first cluster, and the second operator is executed to store the target data into a second cluster.

[0043] Specifically, a database is arranged in each ES cluster, which is used to store target data obtained from a Kafka platform. The Kafka platform is a target database for storing target data, and in some embodiments, it can also be other feasible database platforms. When target data needs to be written into an ES cluster, a Kafka topic task is newly created, target data is obtained from the Kafka platform, the first operator in the ESSink operator is used to store, i.e., write, the target data into a first cluster, and the second operator in the ESSink operator is used to store, i.e., write, the target data into a second cluster, thereby achieving the purpose of simultaneously writing target data into double clusters.

[0044] Step S203, after the first operator is executed, a first receipt instruction is generated, and after the second operator is executed, a second receipt instruction is generated. The first receipt instruction includes a first execution result identifier and a first cluster identifier. The first execution result identifier is used to represent whether the target data is successfully written into the first cluster, and the first cluster identifier is used to represent the first cluster. The second receipt instruction includes a second execution result identifier and a second cluster identifier. The second execution result identifier is used to represent whether the target data is successfully written into the second cluster, and the second cluster identifier is used to represent the second cluster.

[0045] Specifically, when a write failure occurs in a certain cluster during the implementation of the double-cluster data simultaneous writing using the native Sink operator provided in the two Apache Flink processing engines, only the data retry can be performed on the single cluster, and after the data retry fails, the compensation processing cannot be performed on the data, which causes the data loss. Therefore, the application adds a receipt instruction in the ESSink operator, and the receipt instruction at least includes a cluster identifier and an execution result identifier to represent whether the target data is successfully written into the double cluster. After the first operator is executed, the first receipt instruction is generated, and after the second operator is executed, the second receipt instruction is generated. In this way, the staff can determine which cluster the target data is successfully written into and which cluster the target data is failed to be written into through the first execution result identifier and the first cluster identifier in the first receipt instruction and the second execution result identifier and the second cluster identifier in the second receipt instruction.

[0046] In step S204, whether the target data is successfully stored in the first cluster and the second cluster is determined according to the first receipt instruction and the second receipt instruction. In the case that the target data is not successfully stored in the first cluster, the target data is stored in the first compensation table of the first cluster. In the case that the target data is not successfully stored in the second cluster, the target data is stored in the second compensation table of the second cluster.

[0047] Specifically, a database compensation table is also provided in each cluster. When the target data fails to be stored in the database of a certain cluster, the target data is stored in the corresponding database compensation table, i.e., the first or second compensation table. If the target data is successfully stored in the corresponding database compensation table, the target data in the database compensation table is sent to the original Kafka topic task for consumption by the compensation task, so as to perform the compensation storage operation on the target data of the failed cluster, thereby ensuring the integrity of the data.

[0048] Through the embodiment, an ESSink operator is generated, wherein the ESSink operator comprises a first operator and a second operator; target data is acquired from Kafka, the first operator is executed to store the target data to a first cluster, and the second operator is executed to store the target data to a second cluster; after the execution of the first operator, a first receipt instruction is generated, and after the execution of the second operator, a second receipt instruction is generated, wherein the first receipt instruction comprises a first execution result identifier and a first cluster identifier, the first execution result identifier is used to represent whether the target data is successfully written to the first cluster, and the first cluster identifier is used to represent the first cluster, the second receipt instruction comprises a second execution result identifier and a second cluster identifier, the second execution result identifier is used to represent whether the target data is successfully written to the second cluster, and the second cluster identifier is used to represent the second cluster; according to the first receipt instruction and the second receipt instruction, it is determined whether the target data is successfully stored to the first cluster and the second cluster, in the case that the target data is not successfully stored to the first cluster, the target data is stored to a first compensation table of the first cluster, and in the case that the target data is not successfully stored to the second cluster, the target data is stored to a second compensation table of the second cluster. Compared with the prior art that the ESSink operator can only write data to a single cluster, the ESSink operator comprising the first operator and the second operator is defined in the application, the target data is written to two clusters at the same time, the receipt instruction is added in the operator, the corresponding receipt instruction is generated after the execution of the first operator and the second operator, the cluster identifier and the execution result identifier in the receipt instruction are used to determine whether all the double clusters are successfully written, when the writing of a cluster fails, the target data is stored to the compensation table of the corresponding cluster, so as to ensure the integrity of the data, thereby solving the problem that the ESSink operator cannot write data to double clusters at the same time.

[0049] In the implementation process, in order to confirm whether the target data is successfully stored in the dual cluster, according to the first receipt instruction and the second receipt instruction, it is determined whether the target data is successfully stored in the first cluster and the second cluster, comprising: in the case that the first receipt instruction contains the first cluster identifier and the first execution result identifier is successful, it is determined that the target data is successfully stored in the first cluster, and in the case that the second receipt instruction contains the second cluster identifier and the second execution result identifier is successful, it is determined that the target data is successfully stored in the second cluster; in the case that the first receipt instruction contains the first cluster identifier and the first execution result identifier is failed, it is determined that the target data is not successfully stored in the first cluster, and in the case that the second receipt instruction contains the second cluster identifier and the second execution result identifier is failed, it is determined that the target data is not successfully stored in the second cluster. The method determines whether the target data is successfully stored in the first cluster and the second cluster by judging the cluster identifier and the execution result identifier in the receipt instruction, so that it can be accurately judged which cluster is successfully stored and which cluster is failed, and further compensation operation is performed on the data of the cluster which is failed to store.

[0050] Specifically, assuming that the first cluster identifier is 01, the second cluster identifier is 02, and the execution result identifier is an identifier indicating whether the process of storing the target data into the cluster is successful or failed, for example, it can be represented by "success" or "failure", that is, the first execution result identifier, the second execution result identifier, the third execution result identifier, the fourth execution result identifier, the fifth execution result identifier and the sixth execution result identifier in the following are successful or failed. There are three cases: if the first receipt instruction is 01 success and the second receipt instruction is 02 failure, it indicates that the target data is successfully stored in the first cluster, and the target data is failed to store in the second cluster; if the first receipt instruction is 01 success and the second receipt instruction is 02 success, it indicates that the target data is successfully stored in the first cluster and the second cluster; if the first receipt instruction is 01 failure and the second receipt instruction is 02 failure, it indicates that the target data is failed to store in the first cluster and the second cluster.

[0051] In order to ensure the integrity of the target data that is not successfully stored, after the target data is stored into the first compensation table of the first cluster, the method further comprises: storing the target data into the first compensation table of the first cluster, generating a third receipt instruction; determining whether the target data is successfully stored into the first compensation table of the first cluster according to the third receipt instruction, and in the case that the target data is not successfully stored into the first compensation table of the first cluster, storing the target data into a third compensation table of the first cluster; after the target data is stored into the second compensation table of the second cluster, the method further comprises: storing the target data into the second compensation table of the second cluster, generating a fourth receipt instruction; determining whether the target data is successfully stored into the second compensation table of the second cluster according to the fourth receipt instruction, and in the case that the target data is not successfully stored into the second compensation table of the second cluster, storing the target data into a fourth compensation table of the second cluster. The method again determines whether the target data is successfully stored through the returned receipt instruction, and in the case that the target data is not successfully stored into the first compensation table of the first cluster or the second compensation table of the second cluster, the target data that is not successfully stored is again stored into the third compensation table or the fourth compensation table of the corresponding cluster, further ensuring the integrity of the target data and ensuring that the target data will not be lost.

[0052] Specifically, the third compensation table and the fourth compensation table are Kafka topic task compensation tables, that is, in the case that the target data that is not successfully stored is failed to be stored into the corresponding first or second compensation table (database compensation table), the target data that is failed to be stored is again stored into the Kafka topic task compensation table, and in the case of successful storage, the target data is waited to be taken out from the Kafka topic task compensation table by a compensation task, a Kafka topic task is re-established, and then the target data is re-stored into the corresponding failed cluster by the re-established Kafka topic task.

[0053] In some embodiments, determining whether the target data is successfully stored in the first compensation table of the first cluster according to the third receipt instruction includes: when the first cluster identifier is included in the third receipt instruction and the third execution result identifier is success, determining that the target data is successfully stored in the first compensation table of the first cluster; and when the first cluster identifier is included in the third receipt instruction and the third execution result identifier is failure, determining that the target data is not successfully stored in the first compensation table of the first cluster. Determining whether the target data is successfully stored in the second compensation table of the second cluster according to the fourth receipt instruction includes: when the second cluster identifier is included in the fourth receipt instruction and the fourth execution result identifier is success, determining that the target data is successfully stored in the second compensation table of the second cluster; and when the second cluster identifier is included in the fourth receipt instruction and the fourth execution result identifier is failure, determining that the target data is not successfully stored in the second compensation table of the second cluster. The method continues to determine whether the target data is successfully stored in the first compensation table or the second compensation table of the corresponding cluster through the receipt instruction, which can further accurately and effectively determine whether the data is successfully stored.

[0054] Specifically, after the target data that is not successfully stored in the first cluster or the second cluster is stored in the compensation table of the corresponding cluster, the compensation table of the corresponding cluster returns a receipt instruction. As described above, the third execution result identifier and the fourth execution result identifier are success or failure, for example, the target data is stored in the first compensation table of the first cluster, and the first compensation table of the first cluster returns the third receipt instruction. When the third receipt instruction is 01 failure, it indicates that the target data is not successfully stored in the first compensation table of the first cluster. When the third receipt instruction is 01 success, it indicates that the target data is successfully stored in the first compensation table of the first cluster. Similarly, the fourth receipt instruction returned by the second compensation table of the second cluster can be used to determine whether the target data is successfully stored in the second compensation table of the second cluster.

[0055] In some embodiments, after storing the target data into the third compensation table of the first cluster, the method further comprises: storing the target data into the third compensation table of the first cluster, generating a seventh receipt instruction; determining whether the target data is successfully stored into the third compensation table of the first cluster according to the seventh receipt instruction, and in the case that the target data is not successfully stored into the third compensation table of the first cluster, generating a log file of the target data; after storing the target data into the fourth compensation table of the second cluster, the method further comprises: storing the target data into the fourth compensation table of the second cluster, generating an eighth receipt instruction; determining whether the target data is successfully stored into the fourth compensation table of the second cluster according to the eighth receipt instruction, and in the case that the target data is not successfully stored into the fourth compensation table of the second cluster, generating a log file of the target data. After the target data is not successfully stored into the first compensation table of the first cluster or the second compensation table of the second cluster, the method stores the target data into the third compensation table of the first cluster or the fourth compensation table of the second cluster, and determines whether the storage is successful through the receipt instructions generated by the third compensation table and the fourth compensation table. In the case that the storage is not successful, a log file of the target data is generated. In this way, when the target data is not successfully stored, a log file of the target data is generated to record the data that is not successfully stored, thereby further ensuring the integrity of the target data.

[0056] Specifically, the log file can contain the time of event occurrence, events, etc. When the target data fails to be stored into the cluster and the compensation table, the time of storage failure and the target data of storage failure are recorded through the log file, for example: when the target data fails to be stored into the first cluster, and then fails to be stored into the first compensation table of the first cluster, and then fails to be stored into the third compensation table of the first cluster, a log file is generated to record the time of storage failure and the data, thereby ensuring the integrity of the data.

[0057] In some embodiments, the determining whether the target data is successfully stored in the third compensation table of the first cluster according to the fifth feedback instruction comprises: when the fifth feedback instruction contains the first cluster identifier and the fifth execution result identifier is success, determining that the target data is successfully stored in the third compensation table of the first cluster; and when the fifth feedback instruction contains the first cluster identifier and the fifth execution result identifier is failure, determining that the target data is not successfully stored in the third compensation table of the first cluster. The determining whether the target data is successfully stored in the fourth compensation table of the second cluster according to the sixth feedback instruction comprises: when the sixth feedback instruction contains the second cluster identifier and the sixth execution result identifier is success, determining that the target data is successfully stored in the fourth compensation table of the second cluster; and when the fourth feedback instruction contains the second cluster identifier and the sixth execution result identifier is failure, determining that the target data is not successfully stored in the fourth compensation table of the second cluster. This method still determines whether the target data is successfully stored in the third compensation table and the fourth compensation table according to the feedback instruction, so that it can be more convenient and accurate to determine whether the target data is successfully stored.

[0058] Specifically, when the target data is stored in the third compensation table, the third compensation table returns the fifth feedback instruction; and when the target data is stored in the fourth compensation table, the fourth compensation table returns the sixth feedback instruction. The fifth execution result identifier and the sixth execution result identifier are represented by success or failure, for example: when the target data is stored in the third compensation table, the third compensation table returns the fifth feedback instruction, if the fifth feedback instruction is 01 success, it indicates that the target data is successfully stored in the third compensation table, if the fifth feedback instruction is 02 failure, it indicates that the target data is not successfully stored in the third compensation table. Similarly, whether the target data is successfully stored in the fourth compensation table can be determined by the sixth feedback instruction.

[0059] In some embodiments, after the first operator is executed, the first feedback instruction is generated, and after the second operator is executed, the second feedback instruction is generated, which comprises: after the first operator is executed, the first feedback instruction generated by the first cluster is received, and after the second operator is executed, the second feedback instruction generated by the second cluster is received. After the target data is stored in the first cluster by the first operator, whether the target data is successfully stored in the first cluster is determined by the first feedback instruction returned by the first cluster, and after the target data is stored in the second cluster by the second operator, whether the target data is successfully stored in the first cluster is determined by the second feedback instruction returned by the second cluster, so that the cluster which is not successfully stored can be identified, and the data of the cluster which is not successfully stored can be compensated in the subsequent process.

[0060] Specifically, after the target data is stored to the first cluster, the first cluster returns a first receipt instruction, the first receipt instruction returned by the first cluster is received, and whether the target data is successfully stored can be determined according to the content of the first receipt instruction; after the target data is stored to the second cluster, the second cluster returns a second receipt instruction, the second receipt instruction returned by the second cluster is received, and whether the target data is successfully stored can be determined according to the content of the second receipt instruction.

[0061] In order to enable those skilled in the art to more clearly understand the technical solutions of the present application, the implementation process of the data synchronization storage method of the present application will be described in detail below in combination with specific embodiments.

[0062] The present embodiment relates to a specific data synchronization storage method, as shown in Figure 3 and Figure 4 The method comprises the following steps:

[0063] Step S1: a Kafka topic task is created, and target data is obtained from Kafka;

[0064] Step S2: the target data is written to A center (first cluster) through a first operator in ESSink operator, and the target data is written to B center (second cluster) through a second operator in ESSink operator;

[0065] Step S3: whether the target data is successfully written to A center (first cluster) is determined through a first receipt instruction returned by the first cluster, and whether the target data is successfully written to B center (second cluster) is determined through a second receipt instruction returned by the second cluster; in the case that the target data is successfully written to both the first cluster and the second cluster, the process is ended; in the case that the target data is unsuccessfully written to the first cluster or the second cluster, the compensation process of the corresponding center is entered;

[0066] Step S4: the target data of the center that fails to write is written to the database compensation table (first compensation table of the first cluster or second compensation table of the second cluster) of the corresponding center, whether the target data is successfully written to the database compensation table of the corresponding center is determined through a receipt instruction returned by the database compensation table; in the case that the target data is successfully written to the database compensation table of the corresponding center, the target data in the above database compensation table is sent to the original Kafka topic task for consumption, and the process is ended; in the case that the target data fails to be written to the database compensation table of the corresponding center, the target data is written to the compensation Topic (third compensation table of the first cluster or fourth compensation table of the second cluster);

[0067] Step S5: judging whether the target data is successfully written into the compensation Topic through the receipt instruction, if yes, re-establishing a Kafka topic task, re-storing the target data into the corresponding failed cluster, ending the process, if not, printing a Log log (generating a log file) of the target data.

[0068] The embodiment of the present application further provides a data synchronization storage device. It should be noted that the data synchronization storage device of the embodiment of the present application can be used to execute the data synchronization storage method provided by the embodiment of the present application. The device is used to realize the above-mentioned embodiment and preferred embodiment, and details are not repeated. As used below, the term "module" can be a combination of software and / or hardware that realizes a predetermined function. Although the device described in the following embodiment is preferably realized in software, realization of hardware, or a combination of software and hardware, is also possible and contemplated.

[0069] The data synchronization storage device provided by the embodiment of the present application is introduced below.

[0070] Figure 5 is a schematic diagram of the data synchronization storage device according to the embodiment of the present application. As shown in Figure 5 , the device includes:

[0071] The first generation unit 10 is configured to generate an ESSink operator, wherein the ESSink operator includes a first operator and a second operator;

[0072] The first storage unit 20 is configured to obtain target data from Kafka, execute the first operator to store the target data to a first cluster, and execute the second operator to store the target data to a second cluster;

[0073] The second generation unit 30 is configured to generate a first receipt instruction after executing the first operator, and generate a second receipt instruction after executing the second operator, wherein the first receipt instruction includes a first execution result identifier and a first cluster identifier, the first execution result identifier is used to represent whether the target data is successfully written to the first cluster, and the first cluster identifier is used to represent the first cluster, the second receipt instruction includes a second execution result identifier and a second cluster identifier, the second execution result identifier is used to represent whether the target data is successfully written to the second cluster, and the second cluster identifier is used to represent the second cluster;

[0074] The second storage unit 40 is configured to determine whether the target data is successfully stored in the first cluster and the second cluster according to the first receipt instruction and the second receipt instruction, and store the target data in a first compensation table of the first cluster when the target data is not successfully stored in the first cluster, and store the target data in a second compensation table of the second cluster when the target data is not successfully stored in the second cluster.

[0075] In a specific implementation process, in order to confirm whether the target data is successfully stored in the dual clusters, the second storage unit includes a first determination module and a second determination module. The first determination module is configured to determine that the target data is successfully stored in the first cluster when the first cluster identifier is included in the first receipt instruction and the first execution result identifier is successful, and determine that the target data is successfully stored in the second cluster when the second cluster identifier is included in the second receipt instruction and the second execution result identifier is successful. The second determination module is configured to determine that the target data is not successfully stored in the first cluster when the first cluster identifier is included in the first receipt instruction and the first execution result identifier is failed, and determine that the target data is not successfully stored in the second cluster when the second cluster identifier is included in the second receipt instruction and the second execution result identifier is failed. The device determines whether the target data is successfully stored in the first cluster and the second cluster by judging the cluster identifier and the execution result identifier in the receipt instruction, which can accurately determine which cluster is successfully stored and which cluster is failed, and further compensate the data of the cluster that is failed to store.

[0076] Specifically, assuming that the first cluster identifier is 01 and the second cluster identifier is 02, there are three cases: if the first receipt instruction is 01 success and the second receipt instruction is 02 failure, it indicates that the target data is successfully stored in the first cluster and the target data is failed to store in the second cluster; if the first receipt instruction is 01 success and the second receipt instruction is 02 success, it indicates that the target data is successfully stored in the first cluster and the second cluster; if the first receipt instruction is 01 failure and the second receipt instruction is 02 failure, it indicates that the target data is failed to store in the first cluster and the second cluster.

[0077] In order to ensure the integrity of the target data which is not stored successfully, the second storage unit further comprises a first generating module and a first storage module, wherein the first generating module is configured to store the target data into the first compensation table of the first cluster and generate a third receipt instruction; the first storage module is configured to determine whether the target data is successfully stored into the first compensation table of the first cluster according to the third receipt instruction, and in the case that the target data is not successfully stored into the first compensation table of the first cluster, store the target data into a third compensation table of the first cluster; the second storage unit further comprises a second generating module and a second storage module, wherein the second generating module is configured to store the target data into the second compensation table of the second cluster and generate a fourth receipt instruction; the second storage module is configured to determine whether the target data is successfully stored into the second compensation table of the second cluster according to the fourth receipt instruction, and in the case that the target data is not successfully stored into the second compensation table of the second cluster, store the target data into a fourth compensation table of the second cluster. The device determines whether the target data is stored successfully again through the returned receipt instruction, and in the case that the target data is not successfully stored into the first compensation table of the first cluster or the second compensation table of the second cluster, stores the target data which is not stored successfully into the third compensation table or the fourth compensation table of the corresponding cluster again, thereby further ensuring the integrity of the target data and ensuring that the target data will not be lost.

[0078] Specifically, the third compensation table and the fourth compensation table are Kafka topic task compensation tables, that is, in the case that the target data which is not stored successfully is failed to be stored into the corresponding first or second compensation table (database compensation table), the target data which is failed to be stored is stored into the Kafka topic task compensation table again, and in the case of successful storage, the target data is taken out from the Kafka topic task compensation table by a compensation task, a Kafka topic task is re-established, and then the target data is re-stored into the corresponding failed cluster by the re-established Kafka topic task.

[0079] In some embodiments, the first storage module comprises: a first determination submodule configured to determine that the target data is successfully stored in the first compensation table of the first cluster if the first cluster identifier is contained in the third receipt instruction and the third execution result identifier is success, and determine that the target data is not successfully stored in the first compensation table of the first cluster if the first cluster identifier is contained in the third receipt instruction and the third execution result identifier is failure. The second storage module comprises: a second determination submodule configured to determine that the target data is successfully stored in the second compensation table of the second cluster if the second cluster identifier is contained in the fourth receipt instruction and the fourth execution result identifier is success, and determine that the target data is not successfully stored in the second compensation table of the second cluster if the second cluster identifier is contained in the fourth receipt instruction and the fourth execution result identifier is failure. The device continues to determine whether the target data is successfully stored in the first compensation table or the second compensation table of the corresponding cluster through the receipt instruction, which can further accurately and effectively determine whether the data is successfully stored.

[0080] Specifically, after the target data that is not successfully stored in the first cluster or the second cluster is stored in the compensation table of the corresponding cluster, the compensation table of the corresponding cluster returns a receipt instruction. For example, the target data is stored in the first compensation table of the first cluster, at this time, the first compensation table of the first cluster returns the third receipt instruction. If the third receipt instruction is 01 failure, it indicates that the target data is not successfully stored in the first compensation table of the first cluster. If the third receipt instruction is 01 success, it indicates that the target data is successfully stored in the first compensation table of the first cluster. Similarly, the fourth receipt instruction returned by the second compensation table of the second cluster can be used to determine whether the target data is successfully stored in the second compensation table of the second cluster.

[0081] In some embodiments, the first storage module further comprises a first generating submodule and a second generating submodule, wherein the first generating submodule is configured to store the target data into the third compensation table of the first cluster to generate a first feedback instruction; and the second generating submodule is configured to determine whether the target data is successfully stored into the third compensation table of the first cluster according to the first feedback instruction, and generate a log file of the target data in the case that the target data is not successfully stored into the third compensation table of the first cluster. The second storage module further comprises a third generating submodule and a fourth generating submodule, wherein the third generating submodule is configured to store the target data into the fourth compensation table of the second cluster to generate a sixth feedback instruction; and the fourth generating submodule is configured to determine whether the target data is successfully stored into the fourth compensation table of the second cluster according to the sixth feedback instruction, and generate a log file of the target data in the case that the target data is not successfully stored into the fourth compensation table of the second cluster. After the target data is not successfully stored into the first compensation table of the first cluster or the second compensation table of the second cluster, the device stores the target data into the third compensation table of the first cluster or the fourth compensation table of the second cluster, and determines whether the storage is successful through the feedback instructions generated by the third compensation table and the fourth compensation table. In the case that the storage is not successful, the device generates a log file of the target data. In this way, the log file of the target data can be generated to record the data that is not successfully stored when the target data is not successfully stored, and the integrity of the target data is further ensured.

[0082] Specifically, the log file can contain the time of event occurrence, events, etc. When the target data fails to be stored into the cluster and the compensation table, the log file is used to record the time of storage failure and the target data of storage failure, for example: when the target data fails to be stored into the first cluster, and then fails to be stored into the first compensation table of the first cluster, and then fails to be stored into the third compensation table of the first cluster, a log file is generated to record the time of storage failure and the data, so as to ensure the integrity of the data.

[0083] In some embodiments, the first generating sub-module comprises: a third determining sub-module and a fourth determining sub-module. The third determining sub-module is configured to determine that the target data is successfully stored in the third compensation table of the first cluster if the first feedback instruction contains the first cluster identifier and the fifth execution result identifier is successful, and determine that the target data is not successfully stored in the third compensation table of the first cluster if the first feedback instruction contains the first cluster identifier and the fifth execution result identifier is unsuccessful. The fourth determining sub-module is configured to determine that the target data is successfully stored in the fourth compensation table of the second cluster if the fourth feedback instruction contains the second cluster identifier and the sixth execution result identifier is successful, and determine that the target data is not successfully stored in the fourth compensation table of the second cluster if the fourth feedback instruction contains the second cluster identifier and the sixth execution result identifier is unsuccessful. The device still determines whether the target data is successfully stored in the third compensation table and the fourth compensation table according to the feedback instruction, so that it is more convenient and accurate to determine whether the target data is successfully stored.

[0084] Specifically, the third compensation table returns the first feedback instruction after the target data is stored in the third compensation table, and the fourth compensation table returns the sixth feedback instruction after the target data is stored in the fourth compensation table. For example, the target data is stored in the third compensation table, the third compensation table returns the first feedback instruction, if the first feedback instruction is 01 successful, it indicates that the target data is successfully stored in the third compensation table, if the first feedback instruction is 02 unsuccessful, it indicates that the target data is not successfully stored in the third compensation table. Similarly, whether the target data is successfully stored in the fourth compensation table can be determined by the sixth feedback instruction.

[0085] In some embodiments, the second generating unit comprises: receiving the first feedback instruction generated by the first cluster after the first operator is executed, and receiving the second feedback instruction generated by the second cluster after the second operator is executed. The device determines whether the target data is successfully stored in the first cluster by the first feedback instruction returned by the first cluster after the target data is stored in the first cluster by the first operator, and determines whether the target data is successfully stored in the first cluster by the second feedback instruction returned by the second cluster after the target data is stored in the first cluster by the second operator, so as to identify the cluster which is not successfully stored, and to ensure that the data of the cluster which is not successfully stored is compensated.

[0086] Specifically, after the first operator is executed and the target data is stored to the first cluster, the first cluster returns a first receipt instruction, the first receipt instruction returned by the first cluster is received, and whether the target data is stored successfully can be judged according to the content of the first receipt instruction; after the second operator is executed and the target data is stored to the second cluster, the second cluster returns a second receipt instruction, the second receipt instruction returned by the second cluster is received, and whether the target data is stored successfully can be judged according to the content of the second receipt instruction.

[0087] Compared with the device in the prior art in which the ESSink operator can only write data to a single cluster, the data synchronization storage device of the embodiment simultaneously writes target data to two clusters through the self-defined ESSink operator containing the first operator and the second operator, adds a receipt instruction in the operator, generates a corresponding receipt instruction after the first operator and the second operator are executed, judges whether the two clusters are all written successfully through the cluster identifier and the execution result identifier in the receipt instruction, stores the target data to a compensation table of the corresponding cluster when a cluster fails to write, so as to ensure the integrity of the data, and thus the problem that the ESSink operator cannot simultaneously write data to two clusters is solved.

[0088] The above data synchronization storage device includes a processor and a memory, the first generation unit, the first storage unit, the second generation unit and the second storage unit and the like are stored in the memory as program units, and the corresponding functions are realized by the processor executing the above program units stored in the memory. The above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0089] The processor contains a core, and the corresponding program unit is called from the memory by the core. One or more than one core can be set, and the data is simultaneously written to two clusters through the ESSink operator by adjusting the core parameters.

[0090] The memory can include a non-permanent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM), and the memory includes at least one memory chip.

[0091] The embodiment of the application provides a computer readable storage medium, and the computer readable storage medium includes a stored program, wherein when the program runs, the device where the computer readable storage medium is located executes the data synchronization storage method.

[0092] Specifically, the data synchronization storage method includes:

[0093] Step S201, generating an ESSink operator, wherein the ESSink operator comprises a first operator and a second operator.

[0094] Step S202, obtaining target data from Kafka, executing the first operator to store the target data to a first cluster, and executing the second operator to store the target data to a second cluster.

[0095] Step S203, generating a first receipt instruction after executing the first operator, and generating a second receipt instruction after executing the second operator, wherein the first receipt instruction comprises a first execution result identifier and a first cluster identifier, the first execution result identifier is used to represent whether the target data is successfully written to the first cluster, and the first cluster identifier is used to represent the first cluster, the second receipt instruction comprises a second execution result identifier and a second cluster identifier, the second execution result identifier is used to represent whether the target data is successfully written to the second cluster, and the second cluster identifier is used to represent the second cluster.

[0096] Step S204, determining whether the target data is successfully stored to the first cluster and the second cluster according to the first receipt instruction and the second receipt instruction, storing the target data to a first compensation table of the first cluster in a case that the target data is not successfully stored to the first cluster, and storing the target data to a second compensation table of the second cluster in a case that the target data is not successfully stored to the second cluster.

[0097] An embodiment of the present application provides a processor, which is used for running a program, wherein the program is used to execute the data synchronization storage method.

[0098] Specifically, the data synchronization storage method comprises:

[0099] Step S201, generating an ESSink operator, wherein the ESSink operator comprises a first operator and a second operator.

[0100] Step S202, obtaining target data from Kafka, executing the first operator to store the target data to a first cluster, and executing the second operator to store the target data to a second cluster.

[0101] Step S203, after the first operator is executed, a first receipt instruction is generated, and after the second operator is executed, a second receipt instruction is generated, wherein the first receipt instruction includes a first execution result identifier and a first cluster identifier, the first execution result identifier is used to represent whether the target data is successfully written to the first cluster, and the first cluster identifier is used to represent the first cluster, the second receipt instruction includes a second execution result identifier and a second cluster identifier, the second execution result identifier is used to represent whether the target data is successfully written to the second cluster, and the second cluster identifier is used to represent the second cluster.

[0102] Step S204, according to the first receipt instruction and the second receipt instruction, it is determined whether the target data is successfully stored to the first cluster and the second cluster, in the case that the target data is not successfully stored to the first cluster, the target data is stored to a first compensation table of the first cluster, and in the case that the target data is not successfully stored to the second cluster, the target data is stored to a second compensation table of the second cluster.

[0103] The embodiment of the application provides a device, the device comprises a processor, a memory and a program stored on the memory and executable on the processor, and at least the following steps are implemented when the processor executes the program:

[0104] Step S201, an ESSink operator is generated, wherein the ESSink operator includes a first operator and a second operator;

[0105] Step S202, target data is obtained from Kafka, the first operator is executed to store the target data to a first cluster, and the second operator is executed to store the target data to a second cluster;

[0106] Step S203, after the first operator is executed, a first receipt instruction is generated, and after the second operator is executed, a second receipt instruction is generated, wherein the first receipt instruction includes a first execution result identifier and a first cluster identifier, the first execution result identifier is used to represent whether the target data is successfully written to the first cluster, and the first cluster identifier is used to represent the first cluster, the second receipt instruction includes a second execution result identifier and a second cluster identifier, the second execution result identifier is used to represent whether the target data is successfully written to the second cluster, and the second cluster identifier is used to represent the second cluster;

[0107] Step S204, according to the first receipt instruction and the second receipt instruction, determine whether the target data is successfully stored to the first cluster and the second cluster, in the case of the target data is not successfully stored to the first cluster, the target data is stored to the first compensation table of the first cluster, in the case of the target data is not successfully stored to the second cluster, the target data is stored to the second compensation table of the second cluster.

[0108] The device herein can be a server, a PC, a PAD, a mobile phone, etc.

[0109] The application also provides a computer program product, when executed on a data processing device, is adapted to execute the program of initializing at least the following method steps:

[0110] Step S201, generating ESSink operator, wherein the ESSink operator includes a first operator and a second operator;

[0111] Step S202, obtaining target data from Kafka, executing the first operator to store the target data to the first cluster, and executing the second operator to store the target data to the second cluster;

[0112] Step S203, after executing the first operator, generating a first receipt instruction, and after executing the second operator, generating a second receipt instruction, wherein the first receipt instruction includes a first execution result identifier and a first cluster identifier, the first execution result identifier is used to represent whether the target data is successfully written to the first cluster, and the first cluster identifier is used to represent the first cluster, the second receipt instruction includes a second execution result identifier and a second cluster identifier, the second execution result identifier is used to represent whether the target data is successfully written to the second cluster, and the second cluster identifier is used to represent the second cluster;

[0113] Step S204, according to the first receipt instruction and the second receipt instruction, determine whether the target data is successfully stored to the first cluster and the second cluster, in the case of the target data is not successfully stored to the first cluster, the target data is stored to the first compensation table of the first cluster, in the case of the target data is not successfully stored to the second cluster, the target data is stored to the second compensation table of the second cluster.

[0114] It should be apparent to those skilled in the art that the modules or steps of the application described above can be implemented with a general purpose computer, and can be centralized in a single computer or distributed among a network of computers, and can be implemented with program code executable by a computer, and thus can be stored in a storage device and executed by a computer, and in some cases, the steps shown or described can be executed in a different order than shown or described, or can be implemented as separate integrated circuit modules or as a single integrated circuit module, and thus the application is not limited to any particular combination of hardware and software.

[0115] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can be in the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage media, etc.) embodying computer readable program code.

[0116] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 The flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products described herein can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 The flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products described herein can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. The flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products described herein can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams.

[0117] The flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products described herein can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 The flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products described herein can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 The flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products described herein can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. The flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products described herein can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams.

[0118] The flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products described herein can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams.Figure 1 one or more processes and / or functions specified in one or more blocks Figure 1 one or more processes and / or functions specified in one or more blocks

[0119] In one typical arrangement, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0120] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, such as read-only memory (ROM), EPROM, and / or flash memory, etc. The memory is an example of computer readable media.

[0121] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0122] It should also be noted that the terms "comprising", "containing", or any other variant thereof, are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

[0123] From the above description, it can be seen that the above-mentioned embodiments of the present application achieve the following technical effects:

[0124] 1)、the data synchronization storage method of the application, generate ESSink operator, get target data from Kafka, execute the first operator to store the target data to the first cluster, execute the second operator to store the target data to the second cluster; after executing the first operator, generate the first receipt instruction, and after executing the second operator, generate the second receipt instruction; according to the first receipt instruction and the second receipt instruction, determine whether the target data is successfully stored to the first cluster and the second cluster, in the case that the target data is not successfully stored to the first cluster, store the target data to the first compensation table of the first cluster, and in the case that the target data is not successfully stored to the second cluster, store the target data to the second compensation table of the second cluster. Compared with the prior art that the ESSink operator can only write data to a single cluster, the application simultaneously writes the target data to two clusters by defining the ESSink operator containing the first operator and the second operator, adds the receipt instruction in the operator, generates the corresponding receipt instruction after executing the first operator and the second operator, judges whether all the double clusters are successfully written through the cluster identifier and the execution result identifier in the receipt instruction, and stores the target data to the compensation table of the corresponding cluster when the writing of a cluster fails, so as to ensure the integrity of the data, thereby solving the problem that the ESSink operator cannot simultaneously write data to double clusters.

[0125] 2)、the data synchronization storage device of the application, generate ESSink operator, get target data from Kafka, execute the first operator to store the target data to the first cluster, execute the second operator to store the target data to the second cluster; after executing the first operator, generate the first receipt instruction, and after executing the second operator, generate the second receipt instruction; according to the first receipt instruction and the second receipt instruction, determine whether the target data is successfully stored to the first cluster and the second cluster, in the case that the target data is not successfully stored to the first cluster, store the target data to the first compensation table of the first cluster, and in the case that the target data is not successfully stored to the second cluster, store the target data to the second compensation table of the second cluster. Compared with the prior art that the ESSink operator can only write data to a single cluster, the application simultaneously writes the target data to two clusters by defining the ESSink operator containing the first operator and the second operator, adds the receipt instruction in the operator, generates the corresponding receipt instruction after executing the first operator and the second operator, judges whether all the double clusters are successfully written through the cluster identifier and the execution result identifier in the receipt instruction, and stores the target data to the compensation table of the corresponding cluster when the writing of a cluster fails, so as to ensure the integrity of the data, thereby solving the problem that the ESSink operator cannot simultaneously write data to double clusters.

[0126] The above descriptions are only the preferred embodiments of the present application, and are not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data synchronization storage method, characterized by, The application is applied in an Apache Flink processing engine, and includes: generating an ESSink operator, wherein the ESSink operator includes a first operator and a second operator; obtaining target data from Kafka, executing the first operator to store the target data to a first cluster, and executing the second operator to store the target data to a second cluster; generating a first receipt instruction after executing the first operator, and generating a second receipt instruction after executing the second operator, wherein the first receipt instruction includes a first execution result identifier and a first cluster identifier, the first execution result identifier is used to represent whether the target data is successfully stored to the first cluster, and the first cluster identifier is used to represent the first cluster, the second receipt instruction includes a second execution result identifier and a second cluster identifier, the second execution result identifier is used to represent whether the target data is successfully stored to the second cluster, and the second cluster identifier is used to represent the second cluster; determining whether the target data is successfully stored to the first cluster and the second cluster according to the first receipt instruction and the second receipt instruction, storing the target data to a first compensation table of the first cluster in a case where the target data is not successfully stored to the first cluster, and storing the target data to a second compensation table of the second cluster in a case where the target data is not successfully stored to the second cluster.

2. The storage method according to claim 1, wherein, The method further includes: in a case where the first receipt instruction includes the first cluster identifier and the first execution result identifier is successful, it is determined that the target data is successfully stored to the first cluster, and in a case where the second receipt instruction includes the second cluster identifier and the second execution result identifier is successful, it is determined that the target data is successfully stored to the second cluster; in a case where the first receipt instruction includes the first cluster identifier and the first execution result identifier is failed, it is determined that the target data is not successfully stored to the first cluster, and in a case where the second receipt instruction includes the second cluster identifier and the second execution result identifier is failed, it is determined that the target data is not successfully stored to the second cluster.

3. The storage method of claim 1, wherein after the target data is stored to the first compensation table of the first cluster, the method further includes: storing the target data to the first compensation table of the first cluster, and generating a third receipt instruction; determining whether the target data is successfully stored to the first compensation table of the first cluster according to the third receipt instruction, and storing the target data to a third compensation table of the first cluster in a case where the target data is not successfully stored to the first compensation table of the first cluster. after the target data is stored to the second compensation table of the second cluster, the method further includes: ​ store the target data into a second compensation table of the second cluster to generate a fourth receipt instruction; determine whether the target data is successfully stored into the second compensation table of the second cluster according to the fourth receipt instruction, and store the target data into a fourth compensation table of the second cluster in a case that the target data is not successfully stored into the second compensation table of the second cluster.

4. The storage method of claim 3, wherein determining whether the target data is successfully stored into the first compensation table of the first cluster according to the third receipt instruction comprises: in a case that the first cluster identifier is included in the third receipt instruction and the third execution result identifier is successful, determining that the target data is successfully stored into the first compensation table of the first cluster; and in a case that the first cluster identifier is included in the third receipt instruction and the third execution result identifier is failed, determining that the target data is not successfully stored into the first compensation table of the first cluster; determining whether the target data is successfully stored into the second compensation table of the second cluster according to the fourth receipt instruction comprises: in a case that the second cluster identifier is included in the fourth receipt instruction and the fourth execution result identifier is successful, determining that the target data is successfully stored into the second compensation table of the second cluster; and in a case that the second cluster identifier is included in the fourth receipt instruction and the fourth execution result identifier is failed, determining that the target data is not successfully stored into the second compensation table of the second cluster.

5. The storage method of claim 3, wherein after storing the target data into the third compensation table of the first cluster, the method further comprises: storing the target data into the third compensation table of the first cluster to generate a fifth receipt instruction; determining whether the target data is successfully stored into the third compensation table of the first cluster according to the fifth receipt instruction, and generating a log file of the target data in a case that the target data is not successfully stored into the third compensation table of the first cluster. after storing the target data into the fourth compensation table of the second cluster, the method further comprises: storing the target data into the fourth compensation table of the second cluster to generate a sixth receipt instruction; determining whether the target data is successfully stored into the fourth compensation table of the second cluster according to the sixth receipt instruction, and generating a log file of the target data in a case that the target data is not successfully stored into the fourth compensation table of the second cluster.

6. The storage method of claim 5, wherein determining whether the target data is successfully stored into the third compensation table of the first cluster according to the fifth receipt instruction comprises: in a case that the first cluster identifier is included in the fifth receipt instruction and the fifth execution result identifier is successful, determining that the target data is successfully stored into the third compensation table of the first cluster; and in a case that the first cluster identifier is included in the fifth receipt instruction and the fifth execution result identifier is failed, determining that the target data is not successfully stored into the third compensation table of the first cluster; and determining whether the target data is successfully stored into the fourth compensation table of the second cluster according to the sixth receipt instruction comprises: in a case that the second cluster identifier is included in the sixth receipt instruction and the sixth execution result identifier is successful, determining that the target data is successfully stored into the fourth compensation table of the second cluster; and in a case that the second cluster identifier is included in the sixth receipt instruction and the sixth execution result identifier is failed, determining that the target data is not successfully stored into the fourth compensation table of the second cluster. The determining whether the target data is successfully stored into the fourth compensation table of the second cluster according to the sixth feedback instruction comprises: in a case that the second cluster identifier is included in the sixth feedback instruction and the sixth execution result identifier is successful, it is determined that the target data is successfully stored into the fourth compensation table of the second cluster; and in a case that the second cluster identifier is included in the fourth feedback instruction and the sixth execution result identifier is failed, it is determined that the target data is not successfully stored into the fourth compensation table of the second cluster.

7. The storage method of claim 1, wherein, The first feedback instruction is generated after the first operator is executed, and the second feedback instruction is generated after the second operator is executed, comprising: The first feedback instruction generated by the first cluster is received after the first operator is executed, and the second feedback instruction generated by the second cluster is received after the second operator is executed.

8. A data synchronization storage device, comprising: The application is applied in an Apache Flink processing engine, comprising: A first generation unit is configured to generate an ESSink operator, wherein the ESSink operator comprises a first operator and a second operator. A first storage unit is configured to acquire target data from Kafka, execute the first operator to store the target data into a first cluster, and execute the second operator to store the target data into a second cluster. A second generation unit is configured to generate a first feedback instruction after the first operator is executed and generate a second feedback instruction after the second operator is executed, wherein the first feedback instruction comprises a first execution result identifier and a first cluster identifier, the first execution result identifier is used to represent whether the target data is successfully stored into the first cluster, and the first cluster identifier is used to represent the first cluster; the second feedback instruction comprises a second execution result identifier and a second cluster identifier, the second execution result identifier is used to represent whether the target data is successfully stored into the second cluster, and the second cluster identifier is used to represent the second cluster. A second storage unit is configured to determine whether the target data is successfully stored into the first cluster and the second cluster according to the first feedback instruction and the second feedback instruction, store the target data into a first compensation table of the first cluster in a case that the target data is not successfully stored into the first cluster, and store the target data into a second compensation table of the second cluster in a case that the target data is not successfully stored into the second cluster.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored program, wherein the program controls the device where the computer readable storage medium is located to execute the data synchronization storage method in any one of claims 1 to 7 when the program is running.

10. A processor, comprising: The processor is configured to run a program, wherein the program executes the data synchronization storage method in any one of claims 1 to 7 when the program is running.

11. An electronic device, comprising: Comprising: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including programs for performing the data synchronization storage method of any one of claims 1 to 7.

Images