Data processing method and device

Abstract

The invention discloses a data processing method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of creating a current check point in response to a preset check point triggering condition, and determining a writing mode of to-be-written data; determining a data writing strategy and a target table corresponding to the writing mode, and writing the data into the target table through the data writing strategy to obtain an execution result; under the condition that the execution result is abnormal, responding to a rewriting condition, and re-writing the data into the target table through the data writing strategy; and in response to the situation that the rewriting condition is not met, canceling the writing operation, processing the data written into the target table through a rollback strategy corresponding to the writing mode, and triggering a task restarting operation to recover the task from the previous check point. When the implementation mode is abnormal, the rewriting or rollback operation is automatically performed according to the preset condition, so that data loss and errors are reduced, and the reliability and recovery capability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method and device. Background Art

[0002] When Flink is used in combination with Clickhouse, although both have powerful distributed processing capabilities and fault tolerance mechanisms, they face some challenges in big data processing scenarios, mainly including data inconsistency caused by restarts and data system unavailability caused by exceptions. When data streams are written to Clickhouse through Flink tasks, most existing scenarios do not implement overall fault tolerance for the writing process, and mainly rely on the default fault tolerance of the two systems. This may cause some nodes to write successfully while other nodes fail, causing data duplication or system stagnation. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a data processing method and device, which can at least solve the problem of data duplication or system stagnation existing in the combination of Flink and Clickhouse in the prior art.

[0004] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, there is provided a data processing method, comprising: in response to reaching a preset checkpoint trigger condition, creating a current checkpoint, determining a write mode of data to be written through the current checkpoint; determining a data write strategy and a target table corresponding to the write mode, writing the data into the target table through the data write strategy, and obtaining an execution result; in the case where the execution result is abnormal, in response to meeting a re-write condition, re-writing the data into the target table through the data write strategy; in response to not meeting the re-write condition, canceling the current write operation, processing the data written to the target table through the rollback strategy corresponding to the write mode, and triggering a restart task operation to restore the task from the previous checkpoint.

[0005] Optionally, determining whether the re-write condition is met includes: determining the exception code according to the exception type, and in response to the exception code being included in the retryable exception, obtaining the maximum number of retries pre-configured for the exception code; counting the current exception retry times, and determining that the re-write condition is met when the current exception retry times is less than or equal to the maximum retry times; and determining that the re-write condition is not met when the exception code is not included in the retryable exception, or the current exception retry times is greater than the maximum retry times.

[0006] Optionally, in the case where the write mode is to write to a distributed table, determining the data write strategy and target table corresponding to the write mode, and writing the data to the target table through the data write strategy includes: writing the data to the distributed table of Clickhouse through the Flink node cluster; and processing the data written to the target table through the rollback strategy corresponding to the write mode, including: deleting the data successfully written to the Clickhouse distributed table by the Flink node cluster.

[0007] Optionally, the method further includes: synchronizing the data to a local table of the Clickhouse shard.

[0008] Optionally, in the case where the write mode is to write to a local table, determining the data write strategy and target table corresponding to the write mode, and writing the data to the target table through the data write strategy, includes: determining the data allocated to each node according to the number of nodes in the Flink node cluster; transmitting the data to the shards through the nodes according to the mapping relationship between the nodes in Flink and the shards in the Clickhouse cluster shards, so that the shards write the data to the local table; in response to meeting the re-write condition, re-writing the data to the target table through the data write strategy, including: in response to the existence of an available backup copy of the current abnormal shard, disconnecting the connection between the Flink node and the current abnormal shard of Clickhouse, marking the copy of the current abnormal shard as unavailable, switching the current abnormal shard to any other available backup copy, so as to write the data to the local table through any other available backup copy; processing the data written to the target table through the rollback strategy corresponding to the write mode, includes: deleting the data that has been successfully written to the Clickhouse shard by other Flink nodes.

[0009] Optionally, the creating of the current checkpoint includes: creating the current checkpoint when the metadata of the previous checkpoint is normal; the method also includes: setting the metadata of the current checkpoint to normal when the execution result is successful, or the execution result is failed but the rollback is successful.

[0010] Optionally, the method also includes: splitting the write operation into multiple subtasks according to the number of nodes in the Flink node cluster; and determining the amount of data allocated to each subtask according to the amount of data to be written; determining the execution result of each subtask, and in response to the execution results of all subtasks being successful, setting the metadata of the current checkpoint to normal; in response to the execution results of one or more subtasks being failed and the remaining subtasks being successful being rolled back successfully, setting the metadata of the current checkpoint to normal; in response to the execution results of one or more subtasks being failed and the remaining subtasks being successful being rolled back unsuccessfully, setting the metadata of the current checkpoint to abnormal.

[0011] Optionally, the method also includes: constructing a key-value pair based on the identifier of the subtask and the identifier of the current checkpoint; wherein the key-value pair represents the progress of the subtask processing checkpoints; for any subtask, in response to the execution result being successful, switching the identifier of the current checkpoint in the key-value pair to the identifier of the next checkpoint; in response to the execution result being successful but there are other subtasks that fail to execute and trigger a rollback strategy, in the event of a successful rollback, switching the identifier of the next checkpoint in the key-value pair to the identifier of the current checkpoint.

[0012] Optionally, the method further includes: setting, by the idempotent table engine, a switch state for verifying whether the metadata of the previous checkpoint is normal, and in response to the switch state being an off state, determining that the metadata of the previous checkpoint is normal.

[0013] Optionally, the method further includes: setting a primary key for each data to be written in the memory of Flink; wherein, when the data includes a partition number and an offset, creating a primary key based on the partition number and the offset; when the data does not include a partition number and an offset, hashing the message to obtain the primary key; before exiting the current Flink task, calling a cleanup callback function, obtaining the primary key from the memory, building a cleanup task based on the primary key and submitting it to Clickhouse, so that Clickhouse performs a cleanup operation on the data matching the primary key.

[0014] Optionally, the method further includes: in the target table of Clickhouse, setting a flag record field for each data written; wherein the flag record field is used to mark the correspondence between the data and the current checkpoint; determining the flag record field according to the identifier of the current checkpoint through the rollback strategy, and then cleaning the data marked in the flag record field in the target table.

[0015] To achieve the above-mentioned purpose, according to another aspect of an embodiment of the present invention, there is provided a data processing device, including: a determination module, for creating a current checkpoint in response to reaching a preset checkpoint trigger condition, and determining a write mode of data to be written through the current checkpoint; a write module, for determining a data write strategy and a target table corresponding to the write mode, and writing the data into the target table through the data write strategy to obtain an execution result; a retry module, for, in the case where the execution result is abnormal, in response to meeting the re-write condition, re-writing the data into the target table through the data write strategy; a task restart module, for, in response to not meeting the re-write condition, canceling the current write operation, processing the data written to the target table through the rollback strategy corresponding to the write mode, and triggering the restart of the task operation to restore the task from the previous checkpoint.

[0016] Optionally, determining whether the re-write condition is met includes: determining the exception code according to the exception type, and in response to the exception code being included in the retryable exception, obtaining the maximum number of retries pre-configured for the exception code; counting the current exception retry times, and determining that the re-write condition is met when the current exception retry times is less than or equal to the maximum retry times; and determining that the re-write condition is not met when the exception code is not included in the retryable exception, or the current exception retry times is greater than the maximum retry times.

[0017] Optionally, when the write mode is to write to a distributed table, the write module is used to write data to the distributed table of Clickhouse through the Flink node cluster; the task restart module is used to delete the data successfully written into the Clickhouse distributed table by the Flink node cluster.

[0018] Optionally, the device also includes: synchronizing the data to a local table of the Clickhouse shard.

[0019] Optionally, when the write mode is to write to a local table, the write module is used to: determine the data allocated to each node according to the number of nodes in the Flink node cluster; transmit the data to the shard through the node according to the mapping relationship between the nodes in Flink and the shards in the Clickhouse cluster shards, so that the shards write the data to the local table; the retry module is used to: in response to the existence of an available backup copy of the current abnormal shard, disconnect the connection between the Flink node and the current abnormal shard of Clickhouse, mark the copy of the current abnormal shard as unavailable, and switch the current abnormal shard to any other available backup copy, so as to write the data to the local table through any other available backup copy; the task restart module is used to: delete the data that has been successfully written to the Clickhouse shard by other Flink nodes.

[0020] Optionally, the determination module is used to: create the current checkpoint when the metadata of the previous checkpoint is normal; the device also includes: setting the metadata of the current checkpoint to normal when the execution result is successful, or the execution result is failed but the rollback is successful.

[0021] Optionally, the device also includes a metadata update module, which is used to: split the write operation into multiple subtasks according to the number of nodes in the Flink node cluster; and determine the amount of data allocated to each subtask according to the amount of data to be written; determine the execution result of each subtask, and in response to the execution results of all subtasks being successful, set the metadata of the current checkpoint to normal; in response to the execution results of one or more subtasks being failed and the remaining subtasks being successful being rolled back successfully, set the metadata of the current checkpoint to normal; in response to the execution results of one or more subtasks being failed and the remaining subtasks being successful being rolled back failed, set the metadata of the current checkpoint to abnormal.

[0022] Optionally, the device also includes a key-value pair module, which is used to: construct a key-value pair based on the identifier of the subtask and the identifier of the current checkpoint; wherein the key-value pair represents the progress of the subtask processing checkpoint; for any subtask, in response to the execution result being successful, the identifier of the current checkpoint in the key-value pair is switched to the identifier of the next checkpoint; in response to the execution result being successful but there are other subtasks that fail to execute and trigger a rollback strategy, in the event of a successful rollback, the identifier of the next checkpoint in the key-value pair is switched to the identifier of the current checkpoint.

[0023] Optionally, the device function management module is used to: set a switch state for verifying whether the metadata of the previous checkpoint is normal through an idempotent table engine, and determine that the metadata of the previous checkpoint is normal in response to the switch state being off.

[0024] Optionally, the device also includes a primary key module, which is used to: set a primary key for each data to be written in the memory of Flink; wherein, when the data includes a partition number and an offset, create a primary key based on the partition number and the offset; when the data does not include a partition number and an offset, perform hashing on the message to obtain the primary key; before exiting the current Flink task, call a cleanup callback function, obtain the primary key from the memory, build a cleanup task based on the primary key and submit it to Clickhouse, so that Clickhouse performs a cleanup operation on the data that matches the primary key.

[0025] Optionally, the device also includes a marking module, which is used to: set a flag record field for each data written in the target table of Clickhouse; wherein the flag record field is used to mark the correspondence between the data and the current checkpoint; determine the flag record field according to the identifier of the current checkpoint through the rollback strategy, and then clean up the data marked in the flag record field in the target table.

[0026] To achieve the above objective, according to another aspect of an embodiment of the present invention, a data processing electronic device is provided.

[0027] The electronic device of an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned data processing methods.

[0028] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, any of the above data processing methods is implemented.

[0029] To achieve the above object, according to another aspect of an embodiment of the present invention, a computer program product is provided. A computer program product of an embodiment of the present invention includes a computer program, and when the program is executed by a processor, the data processing method provided by the embodiment of the present invention is implemented.

[0030] According to the solution provided by the present invention, an embodiment of the above invention has the following advantages or beneficial effects: determining the write mode, performing the write operation according to the data write strategy of the write mode, and in the case of write exceptions, by distinguishing between retryable and non-retryable exceptions and implementing corresponding retry strategies, unnecessary system restarts are reduced and the possibility of data system unavailability is reduced; even if retry is not possible, the operation can be rolled back to avoid data duplication, thereby improving the reliability and recovery capability of the system, thereby solving the existing data duplication or system stagnation problems, thereby improving the robustness and accuracy of data writing.

[0031] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0033] Figure 1 This is a schematic diagram of the existing process of importing data into Clickhouse through Flink;

[0034] Figure 2 is a main flow chart of a data processing method according to an embodiment of the present invention;

[0035] Figure 3 is a flowchart of an optional data processing method according to an embodiment of the present invention;

[0036] Figure 4 is a flowchart of another optional data processing method according to an embodiment of the present invention;

[0037] Figure 5This is a diagram of Clickhouse connection management in Flink;

[0038] Figure 6 is a flowchart of another optional data processing method according to an embodiment of the present invention;

[0039] Figure 7 This is an example diagram of the data rollback process;

[0040] Figure 8 is a flowchart of another optional data processing method according to an embodiment of the present invention;

[0041] Fig. 9 This is a schematic diagram of the Clickhouse data synchronization process that supports fault tolerance through Flink in this solution;

[0042] Fig.10 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0043] Fig.11 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0044] Fig.12 It is a schematic diagram of the structure of a computer system of a mobile device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0046] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0047] As an open source stream processing framework and distributed processing engine, Flink can provide exactly-once state consistency guarantee. With its high throughput, low latency and strong fault tolerance, it is widely used in the field of real-time data processing and analysis. Clickhouse is a column-based database management system (DBMS) designed for online analytical processing (OLAP). With its high performance, high reliability and high scalability, it has been widely used in the field of big data analysis and real-time query. Clickhouse adopts a distributed architecture, supports the deployment of multiple nodes, and can horizontally expand data to meet the storage and query requirements of large-scale data.

[0048] In the process of combining Flink with Clickhouse to achieve data writing, there are usually two ways: one is to write directly to the distributed table and distribute the data to different cluster shards through Clickhouse; the other is for users to independently implement the data distribution logic and directly connect to the local table for writing. The former method is simple but it is easy to generate more small files, which puts more pressure on Clickhouse metadata management and occupies a high quota for the basic account. The latter method is not easy to generate small files. Although the writing performance is higher and the quota is lower, the operation and maintenance is relatively complex. It is currently the preferred mode for importing large amounts of data in the production environment.

[0049] However, no matter which method is used, in the big data distributed computing scenario, single point failure is an inevitable and non-negligible form of error, especially in the docking process of two distributed systems. First, let's look at the corresponding fault tolerance methods: When a Flink real-time task encounters a single point failure in any link link, it generally restarts from the last checkpoint. This exception is generally triggered by capturing it at the task level. It is a very heavyweight operation, involving a series of processes such as cancellation of the current task, redistribution of operators, and reading and loading of status. Clickhouse itself is also a fault-tolerant system. In the production environment, a replicated table engine is generally used for high availability. For this type of engine, data services can still be provided normally after problems occur in some replicas.

[0050] However, when data flows through Flink tasks and is written to ClickHouse, existing solutions (see Figure 1As shown in the figure, the overall fault tolerance requirements are often not fully considered, and in most cases only rely on the default fault tolerance mechanisms of the two systems. For example, when the existing solution restarts a task, some nodes may fail to write while some nodes may have successfully written. If a rollback is not performed, the written data will be repeated when returning to the previous checkpoint. In addition, there is no subdivision of the error type of writing, such as network jitter, sharding anomalies, replica anomalies, etc. When an error occurs when writing to a certain node, the error is generally thrown out and the Flink task is restarted, which will cause the data link to be temporarily unavailable. Therefore, there is an urgent need for a new solution to improve the robustness and accuracy of data writing.

[0051] The main disadvantages of the prior art are reflected in two aspects:

[0052] 1. Data inconsistency caused by restart: Although the atomicity of a single batch of Clickhouse statements ensures that there will be no partial success and partial failure, the semantic atomicity is broken in a distributed environment. The writing of a batch of data is executed synchronously through different nodes with different degrees of parallelism. It is very likely that some nodes will succeed in writing and some will fail in writing. At this time, if the Flink write task is directly restarted and restored from the last checkpoint without additional processing, the data that has been successfully written before the restart will be repeated. Unless it is an idempotent table engine (such as ReplicatedReplacingMergeTree), other table engines cannot avoid this problem. Therefore, a new solution needs to be introduced to ensure that data is not repeated after restart. Since Clickhouse does not support the transaction function of replicated tables so far, it is necessary to rely on a third-party cache (Redis) to achieve idempotence of writes and cooperate with Flink's checkpoint mechanism to prevent data duplication.

[0053] 2. The occurrence of errors causes data system unavailability: When the data stream is written to Clickhouse through the Flink task, an exception will cause the real-time task to restart, causing the data to be temporarily unavailable. The duration of this unavailability varies with the task scale and the complexity of the topology. It usually lasts from tens of seconds to tens of minutes. If operations such as reallocation of container resources are encountered, the time will be even longer. For some occasional network communication failures, retries may be enough to resolve them; but for some non-occasional failures, especially in the local table direct connection write mode, a Clickhouse replica node has a problem, and the task often restarts continuously and cannot heal itself, because once the restart is completed, the exception information is lost. Even if the task reconnects, it is still possible to randomly select the problematic node, requiring manual intervention to recover. Therefore, a more intelligent error handling mechanism is needed to improve the stability and availability of the system.

[0054] This solution makes the following optimizations to avoid frequent task restarts and failure to self-heal:

[0055] 1. Differentiate between different types of exceptions, which are "retryable exceptions" and "non-retryable exceptions". For "retryable exceptions", a retry mechanism needs to be supported;

[0056] 2. For "non-retryable exceptions" and "retryable exceptions" that cannot be recovered, a replica switching mechanism needs to be supported;

[0057] 3. For exceptions that cannot be recovered after switching copies, they need to be thrown to the task as a whole for restart.

[0058] See also Figure 2 , which shows a main flow chart of a data processing method provided by an embodiment of the present invention, comprising the following steps:

[0059] S201: In response to reaching a preset checkpoint trigger condition, creating a current checkpoint, and determining a writing mode of data to be written through the current checkpoint;

[0060] S202: Determine a data writing strategy and a target table corresponding to the writing mode, write the data into the target table according to the data writing strategy, and obtain an execution result;

[0061] S203: When the execution result is abnormal, in response to meeting the re-writing condition, re-write the data into the target table using the data writing strategy;

[0062] S204: In response to not meeting the re-write condition, canceling the current write operation, processing the data written to the target table through a rollback strategy corresponding to the write mode, and triggering a restart task operation to restore the task from the last checkpoint.

[0063] In the above implementation, for step S201, the entire scheme is in the Flink write task, and the Flink write task is initialized, that is, the Flink write task is started, which is responsible for collecting and accumulating the data to be written from the data source. In the process of data stream processing, for the data to be written, Flink will first cache the data instead of writing it immediately to the target storage system, which can reduce the frequency of writing to the storage system and improve efficiency. For example, Flink obtains the data to be written from the upstream kafka, preprocesses the data (such as filtering, conversion, etc.), and then writes the preprocessed data to the cache. Through the cache mechanism, Flink can accumulate a certain amount of data before batch writing, which can significantly reduce the number of interactions with the target storage system, reduce I / O operations, and thus improve the overall processing efficiency. In addition, the cache mechanism can also improve the fault tolerance of the system to a certain extent. If the target storage system is temporarily unavailable, Flink can continue to receive and process data, and continue to complete the write operation after the storage system is restored.

[0064] Flink will write the cached data to the target storage system, such as Clickhouse, in batches according to the configuration, periodically or when certain conditions are met. For example, it sets the checkpoint period, such as 3 minutes, determines the execution time of the last checkpoint, and determines whether the interval between the current time and the execution time reaches 3 minutes. If so, it triggers the start of the write operation on the cached data. Flink supports fault tolerance mechanisms, one of the core mechanisms of which is checkpoints. Checkpoints ensure that in the event of a failure, the state can be restored from the most recent checkpoint to ensure data consistency and accuracy. When a checkpoint is triggered, Flink will suspend data processing and take a state snapshot.

[0065] It should be noted that checkpoints are a mechanism for persistent storage of intermediate states in stream data processing. For example, taking the data volume as an example, for data streams ABCDEF, assuming that the system receives one data from the upstream every minute, and the checkpoint is set to be triggered every 3 minutes, the first checkpoint will be triggered after receiving A, B, and C. While making a state snapshot, the three data of ABC will be collected together and written into Clickhouse. After the writing is completed, the first checkpoint is truly completed; then the system will continue to receive D, E, and F. If the entire system crashes when writing E, the task will be processed again from the previous checkpoint after restarting, that is, it will start processing from D instead of A.

[0066] This solution adds an additional mechanism after the checkpoint is triggered, that is, to check whether the metadata of the previous checkpoint is complete (normal). If not, the process ends; complete metadata means that the previous checkpoint was successfully executed, or rolled back normally after an exception. If the metadata is complete, continue to the next step; if it is incomplete, the task is terminated immediately and an exception is reported, indicating that manual intervention is required. For updates on whether the metadata of the current checkpoint is complete, see the subsequent Figure 6 As shown in the description. In addition, in the existing solution, if the Flink write task is directly restarted and restored from the last checkpoint without additional processing, the data that has been successfully written before the restart will be repeated. Unless it is an idempotent table engine, other table engines cannot avoid this problem. Therefore, this solution preferably uses an idempotent table engine to set the switch state of verifying the integrity of the metadata of the last checkpoint, which can be set to the off state, that is, all complete by default here.

[0067] For step S202, conventional Flink writing to Clickhouse can be done by directly writing to a distributed table, and completing data distribution to different cluster nodes through Clickhouse; or the user can independently implement data distribution logic and directly connect to the local table for writing. However, these two methods are simple but prone to generate more small files, which puts more pressure on Clickhouse metadata management and occupies a higher quota of the basic account. The latter method is not easy to generate small files. Although the writing performance is higher and the quota occupancy is lower, the operation and maintenance is relatively complex. It is the preferred mode for importing large amounts of data in the current production environment. This solution configures and encapsulates these two methods, and what is exposed to the outside is the choice of writing mode.

[0068] Compared with this Figure 1 As shown in the figure, this solution has two modes: distributed table and local table. Distributed table itself does not store any data, and the Clickhouse cluster automatically completes data writing, distribution, query, routing and other tasks. The connection address is the same when users write to distributed tables. Therefore, when triggering a write, Clickhouse needs to first calculate how to route and distribute the data, and then synchronize the data to the local table node corresponding to the remote shard.

[0069] When writing to a local table, you usually first get the cluster shard information from the Clickhouse system table (system.clusters), including the IP address, port, shard number, replica number, etc., and then establish connections between the Flink nodes and the shards to write data directly. Sharding represents the vertical segmentation of data, and replicas represent the backup of data. After a piece of data is divided into different shards, you only need to write to one of the replicas, and the data synchronization between the remaining replicas is completed by Clickhouse.

[0070] In the first embodiment, the write mode is to write to a distributed table. Through the node cluster of Flink, the data to be written is written to the distributed table of Clickhouse. The distributed table can be regarded as the middleware between Flink and the local table of Clickhouse. It does not store data and is used to synchronize data to the local table node of the corresponding remote shard. That is, writing to the distributed table is actually forwarding directly to the local table. In principle, whether writing to a distributed table or a local table, it will eventually be written to the local table.

[0071] Assume that there are 10,000 data records to be written, and there are 100 nodes in the node cluster of Flink. Then, each node may be assigned 100 data records, and these 10,000 data records are written into the distributed table through these 100 nodes. In this mode, Flink only communicates with the distributed table. However, if the write fails, the data successfully written into the distributed table by these 100 nodes needs to be deleted. Assuming that 8,000 records are successfully written, these 8,000 records need to be deleted. Therefore, the deletion operation here only needs to be submitted to the distributed table connection to which the current node is connected.

[0072] Example 2: The write mode is to write to the local table. Assuming that there are 10,000 data to be written, and the node cluster in Flink has 100 nodes, first determine the data assigned to each node, for example, each node is assigned 100 data, and then based on the mapping relationship between the nodes in Flink and the shards in the Clickhouse system table (system.clusters) cluster shards, the data is transferred to the shards through the nodes, so that the shards write the data to the local table. The entire write task is a many-to-many distributed task. Assuming that Clickhouse has 50 shards, there will be 2 Flink nodes connected to the same Clickhouse shard for write operations, that is, one shard stores 200 data. However, if one or more of the 100 nodes fail to execute, the data of the remaining successful nodes needs to be deleted. The deletion operation only needs to be submitted to the local table connection to which the current node is connected.

[0073] For step S203, if the execution result is successful, or the execution fails but the rollback is normal, the value of the metadata of the current checkpoint is set to complete, and in other cases it is set to incomplete. In the case of failed execution, it is determined whether the re-write condition is met. If it is met, the data is re-written to the target table through the data write strategy. However, if it is not met, it is necessary to cancel the current write operation, process the data written to the target table through the rollback strategy corresponding to the write mode, and trigger the restart process operation to restore the task from the previous checkpoint.

[0074] For example, the previous checkpoint writes ABC and is executed, and the current checkpoint writes DEF, but an exception occurs during the writing of E. Even if the task is restarted, it is processed again based on the previous checkpoint, that is, it restarts from D instead of A. This solution reserves a flag record field in the Clickhouse data table in advance to distinguish which checkpoint the current row data was written from, which can be used as a basis for deletion during manual intervention, such as determining the corresponding flag record field based on the identifier of the current checkpoint, and then determining the corresponding data to perform the deletion operation.

[0075] The method provided in the above embodiment can effectively manage the data writing process and ensure the integrity and consistency of the data by setting the checkpoint and writing mode. When an abnormality occurs, the system can automatically rewrite or roll back according to the preset conditions, thereby reducing data loss and errors and improving the reliability and recovery ability of the system. In addition, the automated processing mechanism reduces the need for manual intervention, improves operational efficiency, and ensures the efficiency and accuracy of data processing.

[0076] See also Figure 3 , shows a schematic flow chart of an optional data processing method according to an embodiment of the present invention, comprising the following steps:

[0077] S301: In response to reaching a preset checkpoint trigger condition, creating a current checkpoint and obtaining a write mode of data to be written;

[0078] S302: When the write mode is to write to a distributed table, the data is written to the distributed table of Clickhouse through the Flink node cluster, and the data to be written is synchronized to the local table of the Clickhouse shard;

[0079] S303: when the execution result is abnormal, determining an exception code according to the exception type, and in response to the exception code being included in the retryable exception, obtaining a maximum number of retries pre-configured for the exception code;

[0080] S304: Count the current number of exception retries. If the current number of exception retries is less than or equal to the maximum number of retries, determine that the re-write condition is met, and re-write the data into the distributed table of Clickhouse through the Flink node cluster;

[0081] S305: When the retryable exception does not include the exception code, or the current exception retry count is greater than the maximum retry count, it is determined that the re-write condition is not met, the write operation is canceled, the data successfully written to the Clickhouse distributed table by the Flink node cluster is deleted, and the task restart operation is triggered to restore the task from the previous checkpoint.

[0082] In the above implementation, the current batch of data is attempted to be written into the distributed table of Clickhouse through the Flink node cluster. If the writing process goes smoothly, the task continues to execute in a loop to accumulate new data and wait for the checkpoint to be triggered for writing. If an exception occurs during the writing process, exception handling is required. First, the exception code is determined by the returned exception type to determine whether the exception code belongs to a "retryable exception". If it does, the maximum number of retries configured in advance for the exception code is obtained. The maximum number of retries for different exception codes may be different. The classification of exceptions and the number of retries are dynamically configurable. Generally speaking, exceptions such as network exceptions, connection timeouts, write quotas exceeded, and tables entering read-only status are defined as "retryable exceptions"; table structure errors, field type errors, etc. are defined as "non-retryable exceptions".

[0083] Determine whether the number of retries for the current exception (for example, if the current batch includes data ABC, then only the current exception retries for the current batch data ABC) exceeds the configured maximum number of retries. If not, retry writing, that is, retry writing the current batch of data to the distributed table of Clickhouse through the Flink node cluster. If the maximum number of retries has been exceeded or it is a non-retryable exception, perform unrecoverable exception handling. Throwing an exception means that the write has failed and cannot be recovered, and delete the data that has been successfully written to the node to maintain data consistency. Restart the task, restore the task status from the most recent checkpoint, return to the checkpoint trigger step, and continue processing data. This process ensures that during the data writing process, even if an exception occurs, the data can be recovered and processed as much as possible, while ensuring data consistency and integrity.

[0084] The method provided in the above embodiment ensures the reliability and stability of data when writing to the Clickhouse distributed table through a refined exception handling mechanism and retry mechanism. When encountering a retryable exception, the system can automatically count the number of retries and rewrite within the set maximum number of retries, minimizing the risk of data loss; and for non-retryable exceptions, the system can cancel the write operation and perform data cleanup in time, triggering the task restart to recover from the last checkpoint, ensuring the continuity and consistency of the entire data processing process. This design not only improves the efficiency of data processing, but also enhances the fault tolerance of the system.

[0085] See also Figure 4 , shows a schematic flow chart of an optional data processing method according to an embodiment of the present invention, comprising the following steps:

[0086] S401: In response to reaching a preset checkpoint trigger condition, creating a current checkpoint and obtaining a write mode of data to be written;

[0087] S402: When the write mode is to write to a local table, determine the data allocated to each node according to the number of nodes in the Flink node cluster;

[0088] S403: According to the mapping relationship between the nodes in Flink and the shards in the Clickhouse cluster shards, the data is transmitted to the shards through the nodes, so that the shards write the data into the local table;

[0089] S404: when the execution result is abnormal, determining an exception code according to the exception type, and in response to the exception code being included in the retryable exception, obtaining a maximum number of retries pre-configured for the exception code;

[0090] S405: Count the current abnormal retry times. If the current abnormal retry times are less than or equal to the maximum retry times, determine that the re-writing condition is met, and return to step S202;

[0091] S406: When the retryable exception does not include an exception code or the current exception retry count is greater than the maximum retry count, in response to the existence of an available standby copy for the current abnormal shard, disconnect the Flink node from the current abnormal shard of Clickhouse, mark the copy of the current abnormal shard as unavailable, and switch the current abnormal shard to any other available standby copy, so as to write data into the local table through any other available standby copy;

[0092] S407: In response to the absence of an available backup copy for the current abnormal shard, cancel the current write operation, delete the data that has been successfully written to the Clickhouse shard by other Flink nodes, and trigger a restart task operation to restore the task from the last checkpoint.

[0093] In the above implementation, writing to a local table is slightly different from writing to a distributed table. When writing to a distributed table, the replica availability is managed by Clickhouse, so users do not need to worry about switching replicas. However, when writing to a local table directly, if a shard fails, you need to handle the switch yourself to use other standby replicas that are still available.

[0094] The concept of backup copy is at the Clickhouse level. Clickhouse has the concepts of shard and replica. Sharding is horizontal expansion, and replica is redundant storage. For example, for a Clickhouse cluster with 3 shards and 2 replicas, ideally, the data will be evenly distributed to the 3 shards when written, that is, each shard stores 1 / 3 of the data, and each shard also stores 2 copies of the data (because of 2 replicas). The replicas themselves do not have a master-slave relationship, and they are all equal. When writing to one of the replicas, Clickhouse will automatically synchronize the same data to the other replica. Therefore, in theory, it is only necessary to ensure that one replica of each shard is written successfully.

[0095] Determine whether the number of retries for the current exception exceeds the configured maximum number of retries. If not, determine the data allocated to each node based on the number of nodes in the Flink node cluster; based on the mapping relationship between nodes in Flink and shards in the cluster shard of the Clickhouse system table, transfer data to the shards through the nodes so that the shards write the data to the local table. If the maximum number of retries has been exceeded or it is a non-retryable exception, then:

[0096] Step 1: Determine whether there is an available backup copy for the current abnormal shard. If yes, go to step 2; if not, go to step 3.

[0097] Step 2: Disconnect the connection of the current abnormal shard, mark the copy of the current abnormal shard as unavailable, switch to any available standby copy, and try writing again; see Figure 5 As shown in the figure, it is a shard-copy diagram for this solution. The number of retries for each shard-copy will be recorded in the connection management. Once the number of retries is exceeded, it will be marked as "unavailable". If a flag is set, this status will not change until the task is restarted.

[0098] Step 3: Throw an exception, write fails and cannot be recovered, delete the data that has been successfully written to the Clickhouse shard by other Flink nodes. Then trigger the restart process operation, restore the task status from the most recent checkpoint, return to the checkpoint trigger step, and continue processing data. For example, if ABC is executed and an exception occurs during the execution of E of DEF, it will be restored from D. This process ensures that even if an exception occurs during the data writing process, the data can be recovered and processed as much as possible, while ensuring the consistency and integrity of the data.

[0099] In actual operation, the structure is set to Shard 1-Copy 1, Shard 1-Copy 2, Shard 1-Copy 3, etc. Assuming that Shard 1-Copy 1 fails to write, determine whether the maximum number of retries exceeds the maximum number of retries. If it exceeds, if it is determined that there are still available standby replicas Shard 1-Copy 2 and Shard 1-Copy 3, disconnect the link with Shard 1-Copy 1 and mark Copy 1 in Shard 1-Copy 1 as unavailable. Assuming that it switches to Shard 1-Copy 2, re-write the data through Shard 1-Copy 2. However, assuming that Shard 1-Copy 2 also fails to write and the number of retries reaches the maximum number of retries, continue to switch to Shard 1-Copy 3 for re-writing. Similarly, assuming that Shard 1-Copy 3 also fails to write and the number of retries reaches the maximum number of retries, return failure and end this write operation. Assuming the maximum number of retries is 3, Shard 1 can retry 3*3=9 times at most.

[0100] The method provided in the above embodiment significantly improves the reliability of data writing and the fault tolerance of the system through a flexible exception handling mechanism and a replica switching mechanism. When an exception occurs, the system can automatically identify the exception type and handle it according to the retry mechanism, ensuring that data writing is attempted as much as possible when it can be retried; at the same time, by switching the backup replica, global write failures caused by a single shard failure are avoided, thereby ensuring the continuous availability and consistency of data. In addition, the design of this method enables the system to recover quickly under abnormal circumstances, reducing the need for manual intervention and improving overall processing efficiency.

[0101] See also Figure 6 , shows a schematic flow chart of another optional data processing method according to an embodiment of the present invention, comprising the following steps:

[0102] S601: splitting the write operation into multiple subtasks according to the number of nodes in the Flink node cluster; and determining the amount of data allocated to each subtask according to the amount of data to be written;

[0103] S602: Determine the execution result of each subtask, and in response to the execution results of all subtasks being successful, set the metadata of the current checkpoint to normal;

[0104] S603: In response to one or more subtasks failing and the remaining subtasks succeeding in rolling back, the metadata of the current checkpoint is set to normal;

[0105] S604: In response to the failure of the rollback of the subtasks whose execution results of one or more subtasks are failures and the remaining subtasks whose execution results are successes, the metadata of the current checkpoint is set as abnormal.

[0106] In the above implementation, it is assumed that there are 100 nodes in the Flink node cluster, and the write operation is split into 100 subtasks, each of which represents a write task performed by each Flink node. For a large number of data, each node will be assigned a part of the data to perform writing. Assuming that there are 10,000 data to be written in this batch, each node is allocated 100 data to store these 100 data in a distributed table or Clickhouse shard. The connection management of the local table is integrated in Flink's Sink operator, which is managed separately by subtasks with different degrees of parallelism. Different connection retries are isolated from each other, that is, the data processed by different subtasks are different.

[0107] See also Figure 7 As shown, a key-value pair can be constructed based on the identifier of the subtask and the identifier of the current checkpoint, and the key-value pair indicates the progress of the subtask processing checkpoint. Verifying integrity and deleting data that has been successfully written to the node requires external storage. This solution prefers to use Redis's Hash data structure to store the successful execution of the task's checkpoint (i.e., metadata). The writing process is completely dependent on the checkpoint. Only when 1) the batch writing of data to Clickhouse is completed; 2) the task-related metadata in Redis is successfully updated, the checkpoint of the currently executing subtask will be considered successful. Each time the data is written to Clickhouse, it will trigger the update of the current checkpoint ID of the current subtask ID to the Hash structure. In the normal writing process, the checkpoint ID will continue to increase (such as Figure 7 chk-1 is changed to chk-2) and updated periodically.

[0108] It should be noted that in addition to Redis's Hash data structure, you can also choose a variety of other storage methods to record the progress of subtask processing checkpoints, including but not limited to other NoSQL databases (such as MongoDB, HBase document storage), relational databases (such as MySQL or PostgreSQL), memory databases (such as Memcached), and distributed file systems (such as Hadoop HDFS). Each storage method has its applicable scenarios and advantages and disadvantages, and you can choose the most appropriate solution based on specific needs.

[0109] From another perspective, metadata can indicate whether the work progress of all subtasks is consistent. If they are consistent, the metadata is normal. If they are inconsistent, the metadata is abnormal. Assuming the number of subtasks is i, subtask-i in the figure is the i-th subtask, chk-1 indicates the first checkpoint, and chk-2 indicates the second checkpoint; subtask-i:chk-1 indicates that the i-th subtask executes the first checkpoint, and subtask-i:chk-2 indicates that the i-th subtask executes the second checkpoint. The ID of the checkpoint is incremented, such as 1 is followed by 2, and 2 is followed by 3. Figure 7 For example, if the execution results of all subtasks are successful, the metadata of the first checkpoint is set to normal. At this time, for each subtask, the checkpoint is automatically switched to the next checkpoint, such as chk-2. However, if the execution result of a subtask is a failure, the checkpoint ID remains unchanged, such as Figure 7 The second line is only partially successful, and the execution result of subtask 2 is a failure, which is still subtask-i:chk-1. In this case, the rollback logic needs to be executed, that is, the remaining subtasks except subtask 2 are rolled back, such as switching subtask-1:chk-2 to subtask-1:chk-1. However, if the rollback fails, for example, after the rollback, it is still subtask-1:chk-2, the metadata of the first checkpoint 1 is set to abnormal.

[0110] If the task exits normally, the checkpoint ID in the metadata will be rolled back to the last successful state, such as subtask-1:chk-2 switching to subtask-1:chk-1. If the Flink node is disconnected and other abnormal exits occur, the rollback cannot be normal, and the abnormal situation of inconsistent checkpoint IDs of each subtask will occur, such as subtask-1:chk-2, subtask-2:chk-1. When the Flink task is restarted to check this metadata again, an exception will be thrown directly, prompting that manual intervention is required.

[0111] The method provided in the above embodiment significantly improves the parallel processing capability and resource utilization efficiency of the system by splitting the write operation into multiple subtasks and reasonably allocating them according to the number of nodes and the amount of data. When all subtasks are executed successfully, the metadata of the current checkpoint can be set to normal in time, and when some subtasks fail, the system can flexibly perform rollback processing to ensure that the execution progress of all subtasks is consistent, further enhancing the fault tolerance and reliability of the system. In addition, through dynamic monitoring and management of execution results, it is possible to respond quickly to abnormal situations and improve the stability and efficiency of overall data processing.

[0112] See also Figure 8, shows a schematic flow chart of another optional data processing method according to an embodiment of the present invention, comprising the following steps:

[0113] S801: In Flink's memory, a primary key is set for each data to be written. If the data includes a partition number and an offset, the primary key is created based on the partition number and the offset. If the data does not include a partition number and an offset, the message is hashed to obtain the primary key.

[0114] S802: Before exiting the current Flink task, call the cleanup callback function, obtain the primary key from the memory, build a cleanup task based on the primary key and submit it to Clickhouse, so that Clickhouse performs a cleanup operation on the data matching the primary key.

[0115] In the above implementation, when a task restart occurs, the corresponding cleanup callback function will be called before the task exits as a whole, in which the logic of deleting the currently written batch data (stored in the JVM) will be executed. In order to characterize the uniqueness of the message, the message primary key (pk) is generally constructed as the basis for deletion, and DELETE FROM table_a WHERE pk ='xxx' is submitted for execution. In scenarios such as kafka, kafka is used as upstream data, and the data transmitted to Flink comes with the partition number partition number + message offset offset. Therefore, for this scenario, this solution preferably uses the partition number partition number + message offset offset as the message primary key. If there is no such information, the overall hash of the message can be used to construct it. Therefore, for other scenarios, if there is no partition number and offset, the overall hash value of the message can be used as the primary key, such as DELETE FROM table_aWHERE pk = 'message_hash'.

[0116] The method provided in the above embodiment ensures the uniqueness and traceability of data in Flink and Clickhouse by setting a primary key for each data to be written, thereby effectively preventing data duplication or conflict. In the presence of partition numbers and offsets, creating a primary key directly based on this information can more accurately locate data; in the absence of this information, generating a primary key through hashing still ensures the uniqueness of the data. In addition, the calling mechanism of the cleanup callback function enables timely cleaning of data matching the primary key before the task exits, keeping the Clickhouse target table clean and efficient, avoiding potential data redundancy and storage waste, and improving the overall performance and management efficiency of the system.

[0117] See also Fig. 9, shows a schematic diagram of the Clickhouse data synchronization process that supports fault tolerance through Flink according to this solution, including the following steps:

[0118] 1. Initialize the Flink write task: Start the Flink write task, which is responsible for collecting and accumulating the data to be written from the data source to obtain the current batch of data.

[0119] 2. Checkpoint triggering and metadata verification: When the write task reaches the preset checkpoint trigger condition, the system first verifies the metadata integrity of the previous checkpoint (idempotent table engines can turn off this function to improve performance). If the metadata is complete, proceed to the next step; if the metadata is incomplete, the task is terminated immediately and an exception is reported, indicating that manual intervention is required.

[0120] 3. Data writing attempt: Try to write the current batch of data to ClickHouse. If the writing process goes smoothly, the task continues to loop and returns to step 2 to accumulate new data. If an exception occurs during the writing process, jump to step 4 for exception handling.

[0121] 4. Exception type judgment: According to the returned exception type and exception code, judge whether the exception is a "retryable exception". If it is a retryable exception, jump to step 5; if not, jump to step 6.

[0122] 5. Determine the number of retries: For retryable exceptions, the system checks whether the number of retries for the current exception has reached the configured maximum number of retries. If it has not reached the maximum, return to step 3 and try writing again; if it has reached the maximum, jump to step 6.

[0123] 6. Write mode judgment:

[0124] 1) When the write mode is to write to the local table, the system checks whether there is an available backup replica node for the current abnormal shard. If there is a backup replica node, jump to step 7; if not, jump to step 8.

[0125] 2) If the write mode is to write to a distributed table, jump directly to step 8.

[0126] 7. Switching the backup replica node: For a local table write exception, the system determines whether there is an available backup replica for the current abnormal shard. If so, the connection to the current abnormal shard is disconnected and its replica is marked as unavailable. Subsequently, the system switches to any available backup replica and returns to step 3 to retry writing.

[0127] As long as one of the different replicas of the same shard is successfully written, it is considered successful. The first replica is selected by default. If it fails, the second replica is selected, and so on. Therefore, when the standby replica is switched, if the standby replica of the direct connection to the local table fails to be written, the task can still be directly restarted by switching to the standby replica and trying again. And before switching, the previous replica will be marked as unavailable to avoid cyclic attempts.

[0128] 8. Write failure processing: If the write cannot be restored, the system throws an exception and deletes the data that has been successfully written to the target table through the rollback strategy to prevent duplication. In the target table of Clickhouse, a flag record field is set for each written data; the flag record field is used to mark the correspondence between the data and the current checkpoint; the rollback strategy determines the flag record field according to the identifier of the current checkpoint, and then cleans up the data marked in the flag record field in the target table.

[0129] 9. Task restart and recovery: After a write fails, the system automatically restarts the Flink task and restores the task status from the previous checkpoint, and then returns to step 2 to continue the write process.

[0130] In a specific embodiment, the scenario is that the browsing data of an e-commerce APP is written into a Clickhouse cluster in real time. During the specific operation process, the system of the present invention guarantees an availability time of 99.99%. Even if the network card of a physical machine of Clickhouse is damaged, causing some shard replica nodes to be abnormal, the Flink write task can still run normally after switching the replica, and there is basically no need for manual intervention in operation and maintenance during the overall life cycle of the task.

[0131] The embodiment of the present invention provides a Flink-based method for writing data to Clickhouse that supports fault tolerance. The method introduces a fault-tolerant mechanism that supports idempotent writing, and uses an external cache to record whether the metadata of the previous checkpoint is normal, so as to determine whether the current checkpoint can be executed. The method introduces a retry mechanism, and can first determine the type of exception before making a decision when an exception occurs when Flink writes to Clickhouse, thereby avoiding restarts caused by most exceptions, reducing the unavailable time of the data system, enhancing the robustness of the system, and alleviating the pressure of operation and maintenance. The method introduces a replica switching mechanism, and can automatically switch to another available standby replica for writing when a shard fails, thereby reducing the number of abnormal restarts of tasks and improving the stability and reliability of the overall system.

[0132] See also Fig.10, showing a schematic diagram of the main modules of a data processing device 1000 provided by an embodiment of the present invention, including: a determination module 1001, used to create a current checkpoint in response to reaching a preset checkpoint trigger condition, and determine the write mode of the data to be written through the current checkpoint; a write module 1002, used to determine the data write strategy and target table corresponding to the write mode, write the data into the target table through the data write strategy, and obtain an execution result; a retry module 1003, used to, in the case of an abnormal execution result, in response to meeting the re-write condition, re-write the data into the target table through the data write strategy; a task restart module 1004, used to cancel the current write operation in response to not meeting the re-write condition, process the data written to the target table through the rollback strategy corresponding to the write mode, and trigger the restart task operation to restore the task from the previous checkpoint.

[0133] In the implementation device of the present invention, judging whether the re-writing condition is met includes: determining the exception code according to the exception type, and in response to the exception code being included in the retryable exception, obtaining the maximum number of retries pre-configured for the exception code; counting the current exception retry times, and determining that the re-writing condition is met when the current exception retry times is less than or equal to the maximum retry times; and determining that the re-writing condition is not met when the exception code is not included in the retryable exception, or the current exception retry times is greater than the maximum retry times.

[0134] In the implementation device of the present invention, when the write mode is to write to a distributed table, the write module 1002 is used to: write data to the distributed table of Clickhouse through the Flink node cluster; the task restart module 1004 is used to: delete the data successfully written into the Clickhouse distributed table by the Flink node cluster.

[0135] The implementation device of the present invention includes: synchronizing data to a local table of a Clickhouse shard.

[0136] In the implementation device of the present invention, when the write mode is to write to a local table, the write module 1002 is used to: determine the data allocated to each node according to the number of nodes in the Flink node cluster; transmit the data to the shard through the node according to the mapping relationship between the nodes in Flink and the shards in the Clickhouse cluster shards, so that the shards write the data to the local table; the retry module 1003 is used to: in response to the existence of an available backup copy of the current abnormal shard, disconnect the connection between the Flink node and the current abnormal shard of Clickhouse, mark the copy of the current abnormal shard as unavailable, and switch the current abnormal shard to any other available backup copy, so as to write the data to the local table through any other available backup copy; the task restart module 1004 is used to: delete the data that has been successfully written to the Clickhouse shard by other Flink nodes.

[0137] In the implementation device of the present invention, the determination module 1001 is used to: create the current checkpoint when the metadata of the previous checkpoint is normal; the device also includes: setting the metadata of the current checkpoint to normal when the execution result is successful, or the execution result is failed but the rollback is successful.

[0138] The implementation device of the present invention also includes a metadata update module, which is used to: split the write operation into multiple subtasks according to the number of nodes in the Flink node cluster; and determine the amount of data allocated to each subtask according to the amount of data to be written; determine the execution result of each subtask, and in response to the execution results of all subtasks being successful, set the metadata of the current checkpoint to normal; in response to the execution results of one or more subtasks being failed and the remaining subtasks being successful being rolled back successfully, set the metadata of the current checkpoint to normal; in response to the execution results of one or more subtasks being failed and the remaining subtasks being successful being rolled back failed, set the metadata of the current checkpoint to abnormal.

[0139] The implementation device of the present invention also includes a key-value pair module, which is used to: construct a key-value pair based on the identifier of the subtask and the identifier of the current checkpoint; wherein the key-value pair represents the progress of the subtask processing checkpoint; for any subtask, in response to the execution result being successful, the identifier of the current checkpoint in the key-value pair is switched to the identifier of the next checkpoint; in response to the execution result being successful but there are other subtasks that fail to execute and trigger a rollback strategy, in the case of a successful rollback, the identifier of the next checkpoint in the key-value pair is switched to the identifier of the current checkpoint.

[0140] The function management module of the device implemented in the present invention is used to: set a switch state for checking whether the metadata of the previous checkpoint is normal through an idempotent table engine, and determine that the metadata of the previous checkpoint is normal in response to the switch state being the closed state.

[0141] The implementation device of the present invention also includes a primary key module, which is used to: set a primary key for each data to be written in the memory of Flink; wherein, when the data includes a partition number and an offset, create a primary key based on the partition number and the offset; when the data does not include a partition number and an offset, perform hash processing on the message to obtain the primary key; before exiting the current Flink task, call a cleanup callback function, obtain the primary key from the memory, build a cleanup task based on the primary key and submit it to Clickhouse, so that Clickhouse performs a cleanup operation on the data matching the primary key.

[0142] The implementation device of the present invention also includes a marking module, which is used to: set a flag record field for each data written in the target table of Clickhouse; wherein the flag record field is used to mark the correspondence between the data and the current checkpoint; determine the flag record field according to the identifier of the current checkpoint through the rollback strategy, and then clean up the data marked in the flag record field in the target table.

[0143] In addition, the specific implementation content of the device described in the embodiment of the present invention has been described in detail in the method described above, so the repeated content will not be described again here.

[0144] Fig.11 An exemplary system architecture 1100 to which embodiments of the present invention may be applied is shown, including terminal devices 1101 , 1102 , 1103 , a network 1104 , and a server 1105 (only an example).

[0145] Terminal devices 1101, 1102, 1103 can be various electronic devices with display screens and supporting web browsing, and various communication client applications are installed. Users can use terminal devices 1101, 1102, 1103 to interact with server 1105 through network 1104 to receive or send messages, etc.

[0146] The network 1104 is used to provide a medium for communication links between the terminal devices 1101, 1102, 1103 and the server 1105. The network 1104 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0147] The server 1105 may be a server that provides various services, such as a backend management server that provides support for shopping websites browsed by users using terminal devices 1101, 1102, and 1103 (for example only). The backend management server may analyze and process the received product information query request and other data, and feed back the processing results (for example, target push information, product information - for example only) to the terminal device. It should be noted that the method provided in the embodiment of the present invention is generally executed by the server 1105, and accordingly, the device is generally set in the server 1105.

[0148] It should be understood that Fig.11 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0149] Reference below Fig.12 , which shows a schematic diagram of the structure of a computer system 1200 of a terminal device suitable for implementing an embodiment of the present invention. Fig.12 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0150] like Fig.12 As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage part 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the system 1200 are also stored. The CPU 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0151] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed, so that a computer program read therefrom is installed into the storage section 1208 as needed.

[0152] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, the above-mentioned functions defined in the system of the present invention are executed.

[0153] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0154] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0155] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor, for example, may be described as: a processor includes a determination module, a write module, a retry module, and a task restart module. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the retry module may also be described as a "retry write module".

[0156] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device executes any of the above data processing methods.

[0157] The computer program product of the present invention comprises a computer program, and when the computer program is executed by a processor, the data processing method in the embodiment of the present invention is implemented.

[0158] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: In response to reaching a preset checkpoint trigger condition, creating a current checkpoint, and determining a write mode of data to be written through the current checkpoint; Determine a data writing strategy and a target table corresponding to the writing mode, write the data into the target table according to the data writing strategy, and obtain an execution result; In the case where the execution result is abnormal, in response to meeting the re-writing condition, re-writing the data into the target table through the data writing strategy; In response to not meeting the re-write condition, the write operation is canceled, the data written to the target table is processed through a rollback strategy corresponding to the write mode, and a restart task operation is triggered to restore the task from the last checkpoint.

2. The method according to claim 1, characterized in that Determine whether the re-write conditions are met, including: Determine an exception code according to the exception type, and in response to the exception code being included in the retryable exception, obtain a maximum number of retries pre-configured for the exception code; Count the current number of abnormal retries, and if the current number of abnormal retries is less than or equal to the maximum number of retries, determine that the re-writing condition is met; When the retryable exception does not include an exception code, or the current exception retry count is greater than the maximum retry count, it is determined that the re-write condition is not met.

3. The method according to claim 1 or 2, characterized in that: In the case where the write mode is to write to a distributed table, determining a data write strategy and a target table corresponding to the write mode, and writing the data to the target table through the data write strategy, includes: writing the data to a distributed table of Clickhouse through a Flink node cluster; The processing of the data written to the target table by the rollback strategy corresponding to the write mode includes: deleting the data successfully written to the Clickhouse distributed table by the Flink node cluster.

4. The method according to claim 3, characterized in that The method also includes: synchronizing the data to a local table of the Clickhouse shard.

5. The method according to claim 1 or 2, characterized in that: In the case where the write mode is to write to a local table, determining a data write strategy and a target table corresponding to the write mode, and writing the data into the target table by using the data write strategy includes: Determine the data to be allocated to each node based on the number of nodes in the Flink node cluster; According to the mapping relationship between nodes in Flink and shards in Clickhouse cluster shards, data is transferred to shards through nodes, so that shards write data to local tables; In response to meeting the re-write condition, re-writing the data into the target table through the data writing strategy includes: in response to the existence of an available standby copy of the current abnormal shard, disconnecting the connection between the Flink node and the current abnormal shard of Clickhouse, marking the copy of the current abnormal shard as unavailable, switching the current abnormal shard to any other available standby copy, so as to write the data into the local table through any other available standby copy; The data written to the target table is processed by a rollback strategy corresponding to the write mode, including: deleting data that has been successfully written to the Clickhouse shard by other Flink nodes.

6. The method according to claim 1, characterized in that The creating of the current checkpoint includes: Create the current checkpoint when the metadata of the previous checkpoint is normal; The method further includes: in a case where the execution result is successful, or the execution result is failed but the rollback is successful, setting the metadata of the current checkpoint to normal.

7. The method according to claim 6, characterized in that The method further comprises: Split the write operation into multiple subtasks based on the number of nodes in the Flink node cluster, and determine the amount of data to be allocated to each subtask based on the amount of data to be written. Determine the execution result of each subtask, and in response to the execution results of all subtasks being successful, set the metadata of the current checkpoint to normal; In response to one or more subtasks having failed execution results and the remaining subtasks having successful execution results being rolled back successfully, setting the metadata of the current checkpoint to normal; In response to the failure of the rollback of the subtasks in which the execution results of one or more subtasks are failures and the remaining subtasks are successful, the metadata of the current checkpoint is set as abnormal.

8. The method according to claim 7, characterized in that The method further comprises: Construct a key-value pair according to the identifier of the subtask and the identifier of the current checkpoint; wherein the key-value pair indicates the progress of the subtask in processing the checkpoint; For any subtask, in response to the execution result being successful, the identifier of the current checkpoint in the key-value pair is switched to the identifier of the next checkpoint; In response to the execution result being successful but other subtasks failing to execute triggering a rollback strategy, in the case of a successful rollback, the identifier of the next checkpoint in the key-value pair is switched to the identifier of the current checkpoint.

9. The method according to any one of claims 6 to 8, characterized in that: The method further comprises: Through the idempotent table engine, a switch state is set to verify whether the metadata of the previous checkpoint is normal, and in response to the switch state being the closed state, it is determined that the metadata of the previous checkpoint is normal.

10. The method according to claim 1, characterized in that The method further comprises: In Flink's memory, a primary key is set for each piece of data to be written. If the data includes the partition number and offset, the primary key is created based on the partition number and offset. If the data does not include the partition number and offset, the message is hashed to obtain the primary key. Before exiting the current Flink task, the cleanup callback function is called to obtain the primary key from the memory, build a cleanup task based on the primary key, and submit it to Clickhouse, so that Clickhouse performs cleanup operations on the data that matches the primary key.

11. The method according to claim 1, characterized in that: The method further comprises: In the target table of Clickhouse, a flag record field is set for each data written; the flag record field is used to mark the correspondence between the data and the current checkpoint; The rollback strategy determines the flag record field according to the identifier of the current checkpoint, and then cleans up the data in the flag record field in the target table.

12. A data processing device, characterized in that: include: A determination module, configured to create a current checkpoint in response to reaching a preset checkpoint trigger condition, and determine a write mode of data to be written through the current checkpoint; A writing module is used to determine a data writing strategy and a target table corresponding to the writing mode, write the data into the target table according to the data writing strategy, and obtain an execution result; A retry module is used to re-write the data into the target table through the data writing strategy in response to meeting the re-writing condition when the execution result is abnormal; The task restart module is used to cancel the current write operation in response to not meeting the re-write conditions, process the data written to the target table through the rollback strategy corresponding to the write mode, and trigger the restart task operation to restore the task from the last checkpoint.

13. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 11.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images