Data processing method, device and storage medium

Abstract

Embodiments of the present application provide a data processing method, device and storage medium. In the data processing method, a data source node can send stream data in a message queue to a data conversion node for processing; at least one operator node in the data conversion node can asynchronously back up a state and send a state backup progress to the data source node. After the stream processing system is abnormally recovered, the data source node can determine a target backup progress for recovering the system state from the received state backup progress, and re-send stream data corresponding to the target backup progress to the data conversion node. Thus, the operator node in the data conversion node can recover the processing operation according to the stream data corresponding to the target backup progress. In this implementation, the operator node in the stream computing system can asynchronously back up the respective state, reducing the blocking formed by the processing process of the stream data, thereby improving the stream processing efficiency of the data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data computing, and in particular to a data processing method, device and storage medium. BACKGROUND

[0002] A streaming process is a continuous processing manner for unbounded data or events, and a process of the streaming process can be described by using a Directed Acyclic Graph (DAG). In the DAG, each edge can represent a data or event stream, and each vertex can represent a data conversion node.

[0003] In some architectures of the streaming process, there is a checkpoint mechanism, by which a state of a streaming process runtime can be saved, and in the event of a fault, the running state before the fault can be restored from the saved state. However, the existing checkpoint mechanism will block the processing of a data stream when executed, thereby reducing the efficiency of the streaming process. Therefore, a new solution needs to be proposed. SUMMARY

[0004] Aspects of the present application provide a data processing method, device and storage medium, to reduce the risk of blocking the processing of a data stream when a checkpoint mechanism is executed, thereby improving the efficiency of the streaming process of data.

[0005] The present application provides a data processing method, comprising: sending streaming data in a message queue to a data conversion node for processing; the data conversion node comprising at least one operator node; obtaining state backup progress respectively sent by the at least one operator node, the state backup progress of any operator node comprising: a queue identifier and an offset of a message queue in which streaming data that has been processed completely is located when a state is backed up; after the streaming processing system is abnormally recovered, determining a target backup progress for restoring the system state according to the respective state backup progress of the at least one operator node; and resending streaming data corresponding to the target backup progress to the data conversion node, so that the at least one operator node resumes processing operations according to the streaming data corresponding to the target backup progress.

[0006] Further optionally, determining a target backup progress for restoring the system state according to the respective state backup progress of the at least one operator node comprises: determining at least one group of offsets from the offsets in the respective state backup progress of the at least one operator node, any group of offsets corresponding to the same queue identifier; selecting a smaller offset from the any group of offsets, and taking the smaller offset and the queue identifier corresponding to the any group of offsets as the target backup progress of the operator node corresponding to the any group of offsets.

[0007] Further optionally, the state backup progress sent by the at least one operator node is acquired by: sending a state backup progress acquisition request to the at least one operator node according to a set inquiry period; and receiving the state backup progress returned by the at least one operator node according to the state backup progress acquisition request.

[0008] The embodiments of the present application also provide a data processing method, comprising: in a process of processing stream data, backing up a current state according to a set state backup rule; sending a state backup progress to a data source node in a stream processing system, the state backup progress comprising: a queue identifier and an offset of a message queue in which stream data processed at a backup state is completed; after the stream processing system is abnormally recovered, receiving stream data corresponding to a target backup progress for recovering a system state sent by the data source node, the target backup progress being less than or equal to the state backup progress; recovering the backed up state of the operator node according to the target backup progress, and performing a recovery processing operation on the stream data corresponding to the target backup progress according to the recovered state.

[0009] Further optionally, in the process of processing stream data, the current state is backed up according to the set state backup rule, comprising: when a specified time interval after the last backup arrives, backing up the current state to a remote database; or, when a state amount in a local cache is greater than a set threshold, backing up the current state to the remote database; or, in response to a backup instruction sent by the data source node, backing up the current state to the remote database.

[0010] Further optionally, the state backup progress is sent to the data source node in the stream processing system, comprising: if the stream data processed at the backup state is split into multiple stream data in a processing process, selecting multiple non-integer offsets between the offset of the stream data processed at the backup state and the offset of the next stream data as offsets of the multiple stream data obtained by splitting.

[0011] Further optionally, the current state is backed up, comprising: backing up the current state to a remote database, and backing up the queue identifier and the offset of the stream data processed at the backup state to the remote database.

[0012] Further optionally, the operator node is restored to the backed-up state according to the target backup progress, and the stream data corresponding to the target backup progress is processed according to the restored state, including: reading a larger state backup progress with the same queue identifier as the target backup progress from the remote database; performing deduplication processing on the stream data corresponding to the target backup progress according to the offset in the larger state backup progress, to obtain stream data that has not been processed completely before the stream processing system abnormally; and performing processing operation on the stream data that has not been processed completely according to the state corresponding to the larger backup progress.

[0013] Further optionally, performing processing operation on the stream data that has not been processed completely according to the state corresponding to the larger backup progress, including: reading the state corresponding to the larger backup progress from the remote database, and performing processing operation on the stream data that has not been processed completely according to the read state; and synchronously downloading the state corresponding to the larger backup progress to the local storage space of the operator node; after the state corresponding to the larger backup progress is downloaded, performing processing operation on the stream data that has not been processed completely according to the state corresponding to the larger backup progress stored in the local storage space.

[0014] Further optionally, the method further includes: when the current state is backed up to the remote database, using an asynchronous thread to process the stream data received in real time in real time.

[0015] Embodiments of the present application also provide a server, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to perform the steps in the method provided by the embodiments of the present application.

[0016] Embodiments of the present application also provide a computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps in the method provided by the embodiments of the present application.

[0017] In the embodiment of the present application, the data source node in the stream computing system can send the stream data in the message queue to the data conversion node for processing; at least one operator node in the data conversion node can asynchronously backup the state and send the state backup progress to the data source node. When the stream processing system is abnormally recovered, the data source node can determine the target backup progress for recovering the system state from the received state backup progress, and resend the stream data corresponding to the target backup progress to the data conversion node. Thus, the operator node in the data conversion node can recover the processing operation according to the stream data corresponding to the target backup progress. In this implementation, the operator node in the stream computing system can asynchronously backup the respective state, which reduces the blocking formed by the processing process of the stream data, thereby improving the stream processing efficiency of the data. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the present application, illustrate embodiments of the present application and together with the description given below, serve to explain the present application. The present application is not limited by the illustrative embodiments given hereinafter.

[0019] Figure 1a Structure schematic diagram of the stream data processing system provided for an exemplary embodiment of the present application;

[0020] Figure 1b Stream computing schematic diagram before failure provided for an exemplary embodiment of the present application;

[0021] Figure 1c Stream computing schematic diagram after failure provided for an exemplary embodiment of the present application;

[0022] Figure 2 Flowchart of the data processing method provided for an exemplary embodiment of the present application when executed at the data source node side;

[0023] Figure 3 Flowchart of the data processing method provided for an exemplary embodiment of the present application when executed at the operator node side;

[0024] Figure 4 Structure schematic diagram of the server provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0025] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0026] Stream processing is a continuous processing mode for unbounded data or events, and in the stream processing process, data flows between different data conversion nodes. In some stream processing architectures, there is a checkpoint mechanism, through which the runtime state of the data conversion node can be saved, and in the event of a failure, the running state before the failure can be restored from the backup state. In the existing checkpoint mechanism, when executing, any node needs to wait for all upstream nodes to store the state after receiving the checkpoint task sent by the checkpoint coordinator, and then store the state of the node. During the waiting process, each data conversion node needs to block the processing of the real-time streaming data flowing into the node, thereby reducing the efficiency of stream processing.

[0027] To solve the above technical problems, in some embodiments of the present application, a solution is provided, and the technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0028] Figure 1a The structural schematic diagram of a stream data processing system provided by an exemplary embodiment of the present application is shown in FIG. 1, and the stream data processing system 100 includes a data source node 10, a data conversion node 20, and a sink node 30. The data source node 10 is configured to read external data or events into the stream data processing system 100; the data conversion node 20 is configured to perform processing operations on the incoming data by setting an algorithm; and the sink node 30 is configured to collect the results of stream processing and output.

[0029] In the stream data processing system 100, the data conversion node 20 is usually composed of a plurality of operator nodes 201 which form a topology structure. The data conversion node 20 includes stateful operator nodes and stateless operator nodes. The stateless operator nodes are mainly used for stateless computation, which means that the operator only needs to process the current data to obtain the computation result. The stateful operator nodes are mainly used for stateful computation, which means that the operator needs to combine some historical states (such as historical computation results, intermediate computation results, etc.) to perform relevant computation, so as to obtain the computation result. The present embodiment relates to state backup and recovery of operator nodes, and therefore, the stateful data conversion node implemented by the operator nodes will be taken as an example for illustrative description. The number of operator nodes can be multiple, which is set according to actual requirements. Different operator nodes can perform different data processing operations by using different operators. The upstream node of an operator node can be one or more nodes, and the downstream node can also be one or more nodes, which is not limited in the present embodiment. One data source node can be connected to one or more operator nodes, and one sink node can also be connected to one or more operator nodes.

[0030] The data source node 10, the data conversion node 20 and the sink node 30 can run on one server, or can run on different servers respectively, which is not limited in the present embodiment.

[0031] In the stream data processing system 100, the data source node 10 is mainly used for sending the received external data to the data consumer (i.e. the data conversion node). In the embodiment, the data on the side of the data source node 10 is cached in the form of a message queue. The data source node 10 can send the continuously incoming data to the data conversion node through the message queue.

[0032] In the stream data processing system 100, any operator node 201 in the data conversion node 20 is mainly used for backing up the current state according to the set state backup rule in the process of processing the received stream data. The state of the operator node 201 refers to the state of the computation task in the data conversion node 20, which can include the stream data cached by the data conversion node 20, the historical computation result and the current intermediate result data, etc. The remote database refers to the database which is not local to the operator node 201, and has high reliability, and can be used for backup storage of the state of the operator node 201.

[0033] In the embodiment, the operation of the state backup performed by the operator node 201 can be an active operation, which can not be controlled by other nodes (e.g., the coordinator node). The operator node 201 can actively perform the state backup according to a set period, or can actively perform the state backup after processing a specified amount of stream data. Alternatively, the operator node 201 can also perform passive backup under the instruction of the data source node 10, which is not limited in the present embodiment. When there are multiple operator nodes in the data conversion node 20, the multiple operator nodes can perform the state backup asynchronously, which reduces the mutual dependence between the operator nodes. Compared with the scheme in which each operator node waits for the upstream node to complete the backup before performing the state backup, the multiple operator nodes performing the state backup asynchronously can greatly reduce the waiting time and improve the efficiency of stream data processing.

[0034] The set state backup rule can be a rule of performing backup according to a set period, or a rule of performing backup according to a cache amount. In some embodiments, the operator node 201 can perform backup on the current state when a specified time interval after the last backup arrives. The specified time interval can be a period length, which can be 5 minutes, 10 minutes, etc., which is not limited in the present embodiment. In other embodiments, the operator node 201 can perform backup on the current state when the amount of local cache state is greater than a set threshold. The size of the set threshold can be set according to the size of the local cache space. For example, the set threshold can be set to 80%, 90% of the local cache space, etc., which is not limited in the present embodiment. In yet other embodiments, the operator node 201 can perform backup on the current state to the remote database 40 in response to the backup instruction sent by the data source node 10, such as the DB (Data Base) or KV (Key-value) shown in FIG. 1. Figure 1a

[0035] The state backup progress sent by the operator node 201 to the data source node 10 in the stream processing system 100 can include the queue identifier (queue_ID) and the offset of the message queue in which the stream data processed by the operator node 201 during the state backup is located. For example, the state backup progress can include the queue identifier of the message queue A in which the data a1 is located and the offset offset(a1) of the data a1 in the message queue A, when the operator node 201 performs the state backup operation after processing the data a1. Based on the state backup progress, the operator node 201 can save the state backup progress to the data source node 10, so as to implement the fault tolerance mechanism of the stream processing system 100.

[0036] ​In some embodiments, the operator node 201 can actively send the state backup progress to the data source node 10 for storage after each backup state. In other embodiments, the operator node 201 can send the latest state backup progress to the data source node 10 upon the trigger of the data source node 10. In this implementation, the data source node 10 can send a state backup progress acquisition request to at least one operator node connected thereto according to a set inquiry period, and receive the state backup progress returned by the at least one operator node according to the state backup progress acquisition request.

[0037] After the stream processing system 100 is recovered from an exception, the data source node 10 can determine a target backup progress for restoring the system state according to the state backup progress of each of the at least one operator node. The target backup progress can be any one of the state backup progress of each of the at least one operator node.

[0038] The processing progress of different operator nodes is different, and thus the state backup progress sent by different operator nodes to the data source node 10 is also different. When determining the target backup progress, the smaller one of the at least one backup progress can be selected as the target backup progress according to the principle of "at least once" of each data conversion node for stream data.

[0039] Alternatively, the state backup progress contains the queue identifier and the offset of the message queue, and the data source node 10 can determine at least one group of offsets from the offset of each of the at least one operator node when determining the target backup progress for restoring the system state according to the state backup progress of each of the at least one operator node. The queue identifier of the message queue corresponding to any group of offsets is the same. That is, the offsets of the message queue with the same queue identifier are divided into a group. For any group of offsets, the smaller offset can be selected from the any group of offsets, and the smaller offset and the queue identifier of the message queue corresponding to the any group of offsets are taken as the target backup progress of the operator node corresponding to the any group of offsets. The smaller offset can be the minimum offset to ensure that there is no missing data for each data conversion node. The following will be described in combination with specific examples.

[0040] Suppose that the state backup progress received by the data source node 10 includes:

[0041] The backup progress 1 of the operator node R1: queue_ID=10, offset1=1;

[0042] The backup progress 2 of the operator node R2: queue_ID=10, offset2=2;

[0043] Backup progress 3 of operator node R3: queue_ID = 10, offset3 = 3;

[0044] Backup progress 4 of operator node R4: queue_ID = 11, offset4 = 2;

[0045] Backup progress 4 of operator node R5: queue_ID = 11, offset5 = 3.

[0046] offset1, offset2 and offset3 are a group of offsets with the same queue ID of the message queue, and the smaller offset (i.e., offset1 = 1) can be selected from the group of offsets, and queue_ID = 10 and offset1 = 1 are taken as the target backup progress of the operator node R1, the operator node R2 and the operator node R3.

[0047] offset4 and offset5 are a group of offsets with the same queue ID of the message queue, and the smaller offset (i.e., offset1 = 2) can be selected from the group of offsets, and queue_ID = 11 and offset1 = 2 are taken as the target backup progress of the data conversion node R4 and the data conversion node R5.

[0048] The target backup progress can be used as a backup progress for restoring the state of the stream data processing system 100 before the exception. After determining the target backup progress, the data source node 10 can resend the stream data corresponding to the target backup progress to the data conversion node 20. At least one operator node in the data conversion node 20 can receive the stream data corresponding to the target backup progress sent by the data source node 10 for restoring the state of the system after the stream processing system 100 is recovered from the exception, restore the backed-up state of the data conversion node according to the target backup progress, and perform the processing operation on the stream data corresponding to the target backup progress according to the restored state.

[0049] For any operator node, the target backup progress is less than or equal to the state backup progress sent by the data conversion node to the data source node 10. Based on the target backup progress, the operator node can be restored to the operation state before the system exception.

[0050] Continuing with the above example, the data source node 10 can resend the stream data after offset 1 = 1 in the message queue with queue_ID = 10 to the operator node R1, the operator node R2 and the operator node R3. Taking the operator node R1 as an example, the target backup progress equals to the state backup progress sent by the operator node R1 to the data source node 10, and the operator node R1 has pre-stored the state corresponding to the target backup progress. Therefore, the operator node R1 can re-compute the stream data after offset 1 = 1 in the message queue with queue_ID = 10 according to the state corresponding to the target backup progress. Taking the data conversion node R2 as an example, the target backup progress is less than the state backup progress sent by the data conversion node R2 to the data source node 10, and the data conversion node R2 has pre-stored the state corresponding to the target backup progress. Therefore, the data conversion node R2 can re-compute the stream data after offset 1 = 1 in the message queue with queue_ID = 10 according to the state corresponding to the target backup progress. Thus, the stream computing system 100 can be restored to the state before the system abnormality.

[0051] In the embodiment, the data source node in the stream computing system can send the stream data in the message queue to the data conversion node for processing; at least one operator node in the data conversion node can backup the state asynchronously and send the state backup progress to the data source node. When the stream processing system is recovered from abnormality, the data source node can determine the target backup progress for restoring the system state from the received state backup progress, and resend the stream data corresponding to the target backup progress to the data conversion node. Thus, the operator nodes in the data conversion node can resume the processing operation according to the stream data corresponding to the target backup progress. In this implementation, the operator nodes in the stream computing system can backup the respective states asynchronously, which reduces the blocking of the processing operation of the stream data, and thus improves the efficiency of the stream processing of the data.

[0052] In the above and the following embodiments of the present application, the operator node 201 can perform the state backup operation and the stream data processing operation by using different threads. That is, the operator node 201 can optionally perform the real-time processing of the real-time received stream data by using the asynchronous thread while backing up the current state. Based on this implementation, the risk of blocking the processing operation of the current incoming data by the state backup operation can be reduced, and the efficiency of the stream data processing can be improved.

[0053] In the foregoing embodiments, it is described that the operator node 201 sends the queue identifier and the offset of the message queue where the current backup state corresponds to the stream data to the data source node 10. In some possible embodiments, the operator node 201 is configured to perform splitting processing on the received stream data. In this case, the stream data will be split into different sub-data, and the sub-data will be processed one by one. In some optional embodiments, in order to more accurately record the state of the operator node 201 from the dimension of the sub-data, the offset for marking the processing progress can be generated for each sub-data packet. Optionally, if the splitting processing has been performed on the stream data corresponding to the current state, a plurality of non-integer offsets can be selected between the offset of the stream data that has been completed when the state is backed up and the offset of the next stream data, as the offsets of the plurality of stream data obtained by splitting. The following will be described with reference to specific examples.

[0054] It is assumed that when a stream data a1 flows into the operator node 201, the operator node 201 splits the stream data a1 into stream data a11, stream data a12, stream data a13 and stream data a14. It is assumed that the offset of the stream data a1 in the message queue A is offset(a1)=1, and the offset of the next stream data a2 of the stream data a1 in the message queue A is offset(a2)=2, the offsets of the stream data a11, the stream data a12, the stream data a13 and the stream data a14 obtained by splitting the stream data a1 can be set as offset(a11)=1.1, offset(a12)=1.2, offset(a13)=1.3 and offset(a13)=1.4 respectively.

[0055] Based on the foregoing offset setting manner, after the stream data is split and processed, each data obtained by splitting can be more accurately marked, thereby facilitating subsequent accurate recovery of the processing progress.

[0056] For the data source node 10, when it receives a non-integer offset, the non-integer offset can be rounded down for subsequent progress recovery. Continuing with the above example, assume that the operator node 201 performs a state backup operation after processing the stream data a13, and sends the state backup progress to the data source node 10 as: the queue identifier A of the message queue and offset(a13) = 1.3. The data source node 10 can round down the received offset to obtain the state backup progress as: the queue identifier A of the message queue and offset = 1. If the data source 10 has not completed processing the stream data a14 when the stream processing system 100 fails, after recovery from the failure, the data source node 10 can distribute the stream data again according to the state backup progress of the operator node 201. In this embodiment, the data source node 10 can send the stream data with offset = 1 in the message queue with identifier A to the operator node 201. The operator node 201 can split the stream data with offset = 1 into the stream data a11, the stream data a12, the stream data a13, and the stream data a14, and then reprocess the stream data a13 according to the offset(a13) = 1.3 in the actual backup progress. Compared with the method of reprocessing the entire stream data a1, this method can avoid repeated calculation in a smaller dimension (i.e., the dimension of the split sub-data), and further improve the recovery efficiency after the failure.

[0057] As described in the foregoing embodiments, the target backup progress for recovering the system state determined by the data source node 10 can be the smallest state backup progress in the state backup progress of each of the at least one operator node in the data conversion node 20. For the operator node 201, the target backup progress can be equal to or smaller than the latest state backup progress thereof. In some embodiments, the operator node 201 can recover the calculation according to the target backup progress. In this calculation method, there is a possibility that part of the data is calculated repeatedly.

[0058] In other embodiments, the operator node 201 can deduplicate the target backup progress according to the actual state backup progress thereof, and recover the calculation according to the deduplication result. In this implementation, the operator node 201 can store the state backup progress thereof in a remote database, and can perform deduplication processing on the stream data corresponding to the target progress node sent by the data source node 10 after the abnormal end, to reduce the amount of repeated calculation.

[0059] Optionally, when the operator node 201 backs up the current state to the remote database, the operator node 201 can further backup the queue identifier and the offset of the message queue in which the stream data completed at the backup state to the remote database.

[0060] Accordingly, when operator node 201 restores the backed-up state of the operator node according to the target backup progress, it can read the larger state backup progress that is the same as the queue identifier in the target backup progress from the remote database; according to the offset in the larger state backup progress, it performs deduplication processing on the streaming data corresponding to the target backup progress to obtain the streaming data that has not been processed before the streaming processing system malfunctioned.

[0061] Next, operator node 201 can perform processing operations on the unprocessed streaming data based on the status corresponding to the larger backup progress. The following will illustrate this with specific examples.

[0062] Assuming, such as Figure 1a As shown, data source node 10 sends four messages from message queue A: message 1, message 2, message 3, and message 4. After sending message 1, operator node 201 performs a state backup, backing up the state of message 1 in the corresponding location of data source node 10.<queue_ID,offset=1> The backup progress is sent to data source node 10 and synchronously backed up to the remote database (DB / KV). Data source node 10 can also store the received backup progress in the remote database. When operator node 201 receives messages 2 and 3 and completes the calculation, it saves the intermediate calculation results corresponding to messages 2 and 3, as well as the offset of message 3 (offset=3), in the remote database. At this time, the streaming data processing system 100 fails.

[0063] For data source node 10, assuming that the status backup progress of the three data transformation nodes it received before the system failure was as follows:<queue_ID=A,offset=1> ,<queue_ID=A,offset=1> ,<queue_ID=A,offset=3> Based on the minimum offset in the above-mentioned backup progress, the target backup progress used for recovery calculation can be determined as follows:<queue_ID=A,offset=1> When the streaming data processing system 100 recovers from the fault, such as Figure 1b The data source node shown can be based on<queue_ID=A,offset=1> Messages 2, 3, and 4 are resent. For operator node 201, the maximum offset of the data it processes, offset = 3, can be obtained from the remote database. Therefore, operator node 201 can discard messages in message queue A sent by data source node 10 with an offset less than 3. That is, operator node 201 can perform calculations on the received message 4. This achieves the effect of exactly-once calculation after system recovery, which helps ensure the accuracy of the state calculation results.

[0064] In some optional embodiments, the operator node 201 can restore the backed-up state from the remote database after receiving the stream data corresponding to the target backup progress sent by the data source node 10, and wait until the state restoration is completed before performing the calculation according to the restored state.

[0065] In some other optional embodiments, the operator node 201 can restore the backed-up state from the remote database and directly perform the calculation using the stored state in the remote database in an asynchronous thread after receiving the stream data corresponding to the target backup progress sent by the data source node 10. That is, when the operator node 201 performs the processing operation on the stream data that has not been processed completely according to the state corresponding to a larger backup progress, the operator node 201 can read the state corresponding to the larger backup progress from the remote database, perform the processing operation on the stream data that has not been processed completely according to the read state, and simultaneously download the state corresponding to the larger backup progress to the local storage space of the operator node. After the downloading of the state corresponding to the larger backup progress is completed, the operator node 201 performs the processing operation on the stream data that has not been processed completely according to the state corresponding to the larger backup progress stored in the local storage space.

[0066] In this embodiment, the operation of the operator node 201 to read the latest backed-up state from the remote database can be implemented based on the remote storage interface of the remote database. The remote database can be implemented as a MySQL or KV (Key-Value) database, and the present embodiment is not limited thereto.

[0067] In this implementation, the operator node 201 can restore the calculation in a short time after the system exception recovery, and switch to the local calculation after the local storage is restored, without waiting for the local storage to be restored, thereby reducing the exception recovery obstruction and improving the performance of the stream computing system.

[0068] In addition to the stream data processing system 100 provided in the foregoing embodiments, the present embodiment also provides a data processing method, which will be described below with reference to the accompanying drawings.

[0069] Figure 2 The flowchart of the data processing method provided in an exemplary embodiment of the present application is shown in FIG. 2. When the method is executed at the data source node side in the stream processing system, the method mainly includes the following steps. Figure 2

[0070] Step 201: Send the stream data in the message queue to the data conversion node for processing; the data conversion node includes at least one operator node.

[0071] ​In step 202, the state backup progress sent by each of the at least one operator node is acquired, and the state backup progress of any operator node comprises a queue identifier and an offset of a message queue in which the processed stream data is stored.

[0072] In step 203, after the stream processing system is recovered, the target backup progress for recovering the system state is determined according to the state backup progress of each of the at least one operator node.

[0073] In step 204, the stream data corresponding to the target backup progress is re-sent to the data conversion node, so that the at least one operator node resumes the processing operation according to the stream data corresponding to the target backup progress.

[0074] In some example embodiments, the way of determining the target backup progress for recovering the system state according to the state backup progress of each of the at least one operator node can comprise: determining at least one group of offsets from the offsets in the state backup progress of each of the at least one operator node, wherein the queue identifiers of the message queues corresponding to any group of offsets are the same; selecting a smaller offset from the at least one group of offsets, and taking the smaller offset and the queue identifier of the message queue corresponding to the at least one group of offsets as the target backup progress of the operator node corresponding to the at least one group of offsets.

[0075] In some example embodiments, the way of acquiring the state backup progress sent by each of the at least one operator node can comprise: sending a state backup progress acquisition request to each of the at least one operator node according to a set inquiry period; and receiving the state backup progress returned by each of the at least one operator node according to the state backup progress acquisition request.

[0076] In this embodiment, the data source node in the stream computing system can send the stream data in the message queue to the data conversion node for processing; the at least one operator node in the data conversion node can asynchronously backup the state and send the state backup progress to the data source node. When the stream processing system is recovered, the data source node can determine the target backup progress for recovering the system state from the received state backup progress, and re-send the stream data corresponding to the target backup progress to the data conversion node. Thus, the operator node in the data conversion node can resume the processing operation according to the stream data corresponding to the target backup progress. In this implementation, the operator node in the stream computing system can asynchronously backup the state, which reduces the blocking of the processing process of the stream data, thereby improving the stream processing efficiency of the data.

[0077] Figure 3 The flowchart of the data processing method provided by another example embodiment of the present application is shown in FIG. 4. Figure 3As shown, the method is executed at the data conversion node side in the stream processing system, and mainly includes:

[0078] In step 301, during processing of the stream data, the current state is backed up according to a set state backup rule.

[0079] In step 302, a state backup progress is sent to a data source node in the stream processing system, the state backup progress including a queue identifier and an offset of a message queue in which stream data processed and completed at a backup state.

[0080] In step 303, after the stream processing system is recovered abnormally, stream data corresponding to a target backup progress for recovering a system state sent by the data source node is received, the target backup progress being less than or equal to the state backup progress.

[0081] In step 304, the state backed up is recovered according to the target backup progress, and the stream data corresponding to the target backup progress is processed according to the recovered state.

[0082] In some example embodiments, during processing of the stream data, a manner of backing up the current state according to the set state backup rule can include: when a specified time interval after last backup arrives, backing up the current state to a remote database; or when a state amount in a local cache is greater than a set threshold, backing up the current state to the remote database; or in response to a backup instruction sent by the data source node, backing up the current state to the remote database.

[0083] In some example embodiments, a manner of sending the state backup progress to the data source node in the stream processing system can include: if stream data processed and completed at the backup state is split into multiple stream data during processing, selecting multiple non-integer offsets between the offset of the stream data processed and completed at the backup state and the offset of next stream data as offsets of the multiple stream data obtained by splitting.

[0084] In some example embodiments, backing up the current state includes: backing up the current state to a remote database, and backing up a queue identifier and an offset of stream data processed and completed at a backup state to the remote database.

[0085] In some example embodiments, the way of resuming the backup state of the operator node according to the target backup progress and resuming the processing operation of the stream data corresponding to the target backup progress according to the resumed state can include: reading a larger state backup progress with the same queue identifier as the target backup progress from the remote database; performing deduplication processing on the stream data corresponding to the target backup progress according to the offset in the larger state backup progress to obtain stream data that has not been processed completely before the abnormality of the stream processing system; and performing processing operation on the stream data that has not been processed completely according to the state corresponding to the larger backup progress.

[0086] In some example embodiments, the way of performing processing operation on the stream data that has not been processed completely according to the state corresponding to the larger backup progress can include: reading the state corresponding to the larger backup progress from the remote database and performing processing operation on the stream data that has not been processed completely according to the read state; and synchronously downloading the state corresponding to the larger backup progress to the local storage space of the operator node; after the downloading of the state corresponding to the larger backup progress is completed, performing processing operation on the stream data that has not been processed completely according to the state corresponding to the larger backup progress stored in the local storage space.

[0087] In some example embodiments, the method further includes: when the current state is backed up to the remote database, using an asynchronous thread to process the stream data received in real time in real time.

[0088] In the embodiment, the data source node in the stream computing system can send the stream data in the message queue to the data conversion node for processing; at least one operator node in the data conversion node can backup the state asynchronously and send the state backup progress to the data source node. When the stream processing system is recovered abnormally, the data source node can determine the target backup progress for recovering the system state from the received state backup progress and resend the stream data corresponding to the target backup progress to the data conversion node. Thus, the operator node in the data conversion node can resume the processing operation according to the stream data corresponding to the target backup progress. In this implementation, the operator node in the stream computing system can backup the state of each node asynchronously, which reduces the blocking of the processing process of the stream data and improves the efficiency of the stream processing of the data.

[0089] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 301 to 304 can be device a; for another example, the execution subject of steps 301 and 302 can be device a, and the execution subject of step 303 can be device b; and so on.

[0090] In addition, in some of the processes described in the foregoing embodiments and accompanying drawings, a plurality of operations are included in a specific order, but it should be clear that these operations can be executed in the order they appear in this document or in parallel, and the serial numbers of the operations, such as 301, 302, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel.

[0091] It should be noted that the "first", "second", and the like described herein are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do "first" and "second" represent different types.

[0092] Figure 4 is a structural schematic diagram of a server provided by an exemplary embodiment of the present application, which can be used to execute the data processing method described in the foregoing embodiments. As shown in Figure 4 The server includes a memory 401 and a processor 402.

[0093] The memory 401 is used to store computer programs and can be configured to store other various data to support operations on the server. Examples of these data include instructions for any application or method operating on the server, contact data, phonebook data, messages, pictures, videos, etc.

[0094] The memory 401 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0095] In some embodiments, the processor 402 is coupled with the memory 401 and is used to execute the computer programs in the memory 401 to implement the data processing method.

[0096] In some embodiments, Figure 4The schematic server can be used to deploy a data source node in a stream processing system, and the processor 402 is configured to execute a stream processing program corresponding to the function of the data source node in the memory 401, so as to: send stream data in a message queue to a data conversion node for processing; the data conversion node includes at least one operator node, and the at least one operator node forms a topology structure; obtain state backup progress respectively sent by the at least one operator node, and the state backup progress of any operator node includes: queue identifier and offset of a message queue in which stream data processed completely at the time of backup state; after abnormal recovery of the stream processing system, determine a target backup progress for restoring the system state according to the respective state backup progress of the at least one operator node; and resend stream data corresponding to the target backup progress to the data conversion node, so that the at least one operator node resumes processing operation according to the stream data corresponding to the target backup progress.

[0097] Further optionally, when determining the target backup progress for restoring the system state according to the respective state backup progress of the at least one operator node, the processor 402 is specifically configured to: determine at least one group of offsets from the offsets in the respective state backup progress of the at least one operator node, any group of offsets corresponding to the same queue identifier; select a smaller offset from the any group of offsets, and take the smaller offset and the queue identifier corresponding to the any group of offsets as the target backup progress of the operator node corresponding to the any group of offsets.

[0098] Further optionally, when obtaining the state backup progress respectively sent by the at least one operator node, the processor 402 is specifically configured to: send a state backup progress obtaining request to the at least one operator node respectively according to a set inquiry period; and receive the state backup progress returned by the at least one operator node according to the state backup progress obtaining request.

[0099] In this embodiment, the data source node in the stream computing system can send stream data in a message queue to a data conversion node for processing, so that the data conversion node can backup state asynchronously and send state backup progress to the data source node. When the stream processing system is abnormally recovered, the data source node can determine a target backup progress for restoring the system state from the received state backup progress, and resend stream data corresponding to the target backup progress to the data conversion node, so that the data conversion node resumes processing operation according to the stream data corresponding to the target backup progress. In this embodiment, the operator nodes in the stream computing system can backup state asynchronously, which reduces the blocking of the processing process of stream data, thereby improving the stream processing efficiency of data.

[0100] In some embodiments,Figure 4 The schematic server can be used to deploy any operator node in the data conversion node in the stream data processing system, and the processor 402 is configured to execute a stream data processing program corresponding to the function of the operator node in the memory 401, so as to: back up the current state according to a set state backup rule in the process of processing the stream data; send a state backup progress to a data source node in the stream processing system, the state backup progress including: a queue identifier and an offset of a message queue in which stream data processed and completed at the time of backing up the state; after the stream processing system is abnormally recovered, receive stream data corresponding to a target backup progress for restoring the system state sent by the data source node, the target backup progress being less than or equal to the state backup progress; restore the backed up state of the operator node according to the target backup progress, and perform recovery processing operation on the stream data corresponding to the target backup progress according to the restored state.

[0101] Further optionally, when the processor 402 backs up the current state according to the set state backup rule in the process of processing the stream data, the processor 402 is specifically configured to: back up the current state to a remote database when a specified time interval after the last backup arrives; or back up the current state to the remote database when the amount of state cached locally is greater than a set threshold; or back up the current state to the remote database in response to a backup instruction sent by the data source node.

[0102] Further optionally, when the processor 402 sends the state backup progress to the data source node in the stream processing system, the processor 402 is specifically configured to: if the stream data processed and completed at the time of backing up the state is split into multiple stream data in the processing process, select multiple non-integer offsets between the offset of the stream data processed and completed at the time of backing up the state and the offset of the next stream data as the offsets of the multiple stream data obtained by splitting.

[0103] Further optionally, when the processor 402 backs up the current state, the processor 402 is specifically configured to: back up the current state to a remote database, and back up the queue identifier and the offset of the stream data processed and completed at the time of backing up the state to the remote database.

[0104] Further optionally, the processor 402 is specifically configured to: read, from the remote database, a larger state backup progress with a same queue identifier as the target backup progress; perform deduplication processing on the stream data corresponding to the target backup progress according to an offset in the larger state backup progress, to obtain stream data that has not been processed completely before the stream processing system abnormally; and perform processing operation on the stream data that has not been processed completely according to a state corresponding to the larger backup progress.

[0105] Further optionally, the processor 402 is specifically configured to: read, from the remote database, a larger state backup progress with a same queue identifier as the target backup progress; perform deduplication processing on the stream data corresponding to the target backup progress according to an offset in the larger state backup progress, to obtain stream data that has not been processed completely before the stream processing system abnormally; and perform processing operation on the stream data that has not been processed completely according to a state corresponding to the larger backup progress.

[0106] Further optionally, the processor 402 is further configured to: when backing up the current state to the remote database, adopt an asynchronous thread to process the stream data received in real time in real time.

[0107] In this implementation, the data source node in the stream computing system can send the stream data in the message queue to the data conversion node for processing; at least one operator node in the data conversion node can asynchronously backup the state and send the state backup progress to the data source node. When the stream processing system is recovered abnormally, the data source node can determine the target backup progress for recovering the system state from the received state backup progress, and resend the stream data corresponding to the target backup progress to the data conversion node. Thus, the operator node in the data conversion node can recover the processing operation according to the stream data corresponding to the target backup progress. In this implementation, the operator node in the stream computing system can asynchronously backup the respective state, which reduces the blocking formed by the processing process of the stream data, thereby improving the stream processing efficiency of the data.

[0108] Further, as shown in Figure 4 the server further includes a communication component 403 and a power supply component 404 and other components. Figure 4 Some components are only schematically shown in the server, and it does not mean that the server only includes the components shown in Figure 4 .

[0109] The communication component 403 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, or 5G, or a combination thereof. In an example embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component can be implemented based on a near field communication (NFC) technology, a radio frequency identification (RFID) technology, an infrared data association (IrDA) technology, an ultra wideband (UWB) technology, a Bluetooth (BT) technology, and other technologies.

[0110] The power supply component 404 provides power to various components of the device where the power supply component is located. The power supply component can include a power supply management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply component is located.

[0111] Accordingly, the embodiments of the present application also provide a computer readable storage medium storing a computer program, which, when executed, can implement the steps performed by the server in the method embodiments.

[0112] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code.

[0113] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and a combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce the functions specified in the flowchart and / or block diagram flow or flows and / or block or blocks.

[0114] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart or flowsheets and / or block or blocks of the block diagrams.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheets and / or block or blocks of the block diagrams.

[0116] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0117] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the processor can execute instructions. The memory can also include non-volatile memory, such as read only memory (ROM), electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), flash memory, or other memory technologies, about which the processor can execute instructions. The memory is an example of computer readable media.

[0118] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassettes, magnetic disks storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0119] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0120] The above description is merely illustrative of the application, and not restrictive. Various modifications and changes can become apparent to those skilled in the art. Incorporating any modification, equivalent substitution, improvement, etc. within the spirit and principle of the application, shall be included in the scope of the claims of the application.

Claims

1. A data processing method, characterized in that, include: Send the streaming data in the message queue to the data conversion node for processing; The data conversion node includes at least one operator node, which corresponds to a backup of its current state and records the queue identifier and offset of the message queue where the processed streaming data is located when backing up the current state. Obtain the queue identifier and offset of the message queue containing the processed streaming data when the backup status is sent by the at least one operator node; After the streaming processing system recovers from an anomaly, based on the queue identifier and offset of the message queue containing the processed streaming data in the backup state of each of the at least one operator node, the target queue identifier and target offset of the message queue containing the processed streaming data in the backup state are determined for restoring the system state. For any operator node, the state backup progress represented by the target queue identifier and the target offset is less than or equal to the state backup progress represented by the queue identifier and offset sent by any operator node. The target queue identifier and the streaming data corresponding to the target offset are resent to the data conversion node so that the at least one operator node can resume processing operations based on the target queue identifier and the streaming data corresponding to the target offset.

2. The method according to claim 1, characterized in that, Based on the queue identifier and offset of the message queue containing the processed streaming data during the backup state of each of the at least one operator node, determine the target queue identifier and target offset of the message queue containing the processed streaming data during the backup state used to restore the system state, including: From the offsets of each of the at least one operator node, at least one set of offsets is determined, and the queue identifiers corresponding to any set of offsets are the same; Select the smaller offset from any set of offsets, and use the smaller offset and the queue identifier corresponding to any set of offsets as the target queue identifier and target offset of the operator node corresponding to any set of offsets.

3. The method according to any one of claims 1 or 2, characterized in that, Obtain the queue identifier and offset of the message queue containing the processed streaming data when the backup status is sent by the at least one operator node, including: According to the set query cycle, a status backup progress acquisition request is sent to each of the at least one operator node; When the at least one operator node receives the backup status returned by the request based on the status backup progress, it receives the queue identifier and offset of the message queue where the processed streaming data is located.

4. A data processing method, characterized in that, include: During the processing of streaming data, the current state is backed up according to the set state backup rules. Corresponding to the backup of its own current state, the queue identifier and offset of the message queue where the processed streaming data is located when the current state is backed up are recorded. When sending backup status to the data source node in the streaming processing system, the queue identifier and offset of the message queue containing the processed streaming data are displayed. After the streaming processing system recovers from an anomaly, the system receives the queue identifier and offset of the message queue containing the processed streaming data according to the backup status of at least one operator node, and sends the target queue identifier and target offset of the message queue containing the processed streaming data according to the backup status for restoring the system state. The at least one operator node performs state backup asynchronously, and the state backup progress represented by the target queue identifier and the target offset is less than or equal to the state backup progress represented by the queue identifier and offset sent to the data source node. The backed-up state is restored based on the target queue identifier and the target offset, and the streaming data corresponding to the target queue identifier and the target offset is restored based on the restored state.

5. The method according to claim 4, characterized in that, During the processing of streaming data, the current state is backed up according to the set state backup rules, including: When the specified time interval since the last backup is reached, back up the current state to the remote database; or, When the amount of state cached locally exceeds a set threshold, the current state is backed up to a remote database; or, In response to the backup command sent by the data source node, the current state is backed up to the remote database.

6. The method according to claim 4, characterized in that, The message queue identifier and offset of the message queue containing the processed streaming data when sending the backup status to the data source node in the streaming processing system, including: If the streaming data that has been processed in the backup state is split into multiple streaming data during the processing, then multiple non-integer offsets are selected between the offset of the completed streaming data in the backup state and the offset of the next streaming data as the offsets of the multiple streaming data obtained from the split.

7. The method according to claim 4, characterized in that, Back up the current state, including: The current state is backed up to the remote database, and the queue identifiers and offsets of the streaming data that have been processed at the time of backup are also backed up to the remote database.

8. The method according to claim 7, characterized in that, The process includes restoring the backed-up state based on the target queue identifier and the target offset, and performing streaming data recovery processing operations based on the restored state for the target queue identifier and the target offset, including: Read the larger state backup progress that is the same as the target queue identifier and the queue identifier in the target offset from the remote database; Based on the offset in the larger state backup progress, the target queue identifier and the streaming data corresponding to the target offset are deduplicated to obtain the streaming data that has not been processed before the streaming processing system malfunctions. Based on the status corresponding to the larger state backup progress, processing operations are performed on the streaming data that has not yet been processed.

9. The method according to claim 8, characterized in that, Based on the status corresponding to the larger state backup progress, processing operations are performed on the unprocessed streaming data, including: The system reads the status corresponding to the larger state backup progress from the remote database and performs processing operations on the unprocessed streaming data based on the read status; and simultaneously downloads the status corresponding to the larger state backup progress to the local storage space. After the state corresponding to the larger state backup progress is downloaded, the unprocessed streaming data is processed according to the state corresponding to the larger state backup progress stored in the local storage space.

10. The method according to any one of claims 4-9, characterized in that, Also includes: When backing up the current state to a remote database, asynchronous threads are used to process the received streaming data in real time.

11. A server, characterized in that, include: Memory and processor; The memory is used to store one or more computer instructions; The processor is configured to execute one or more computer instructions for: performing the steps in the method according to any one of claims 1-4 or the steps in the method according to any one of claims 5-10.

12. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it is able to perform the steps of the method described in any one of claims 1-4 or any one of claims 5-10.

Images