Task synchronization method based on stream processing pipeline and related apparatus

Abstract

The application provides a task synchronization method based on a stream processing pipeline and a related device, and belongs to the technical field of data processing. The method comprises the following steps: obtaining a to-be-synchronized task object, the to-be-synchronized task object being used to represent a completed subtask of a to-be-processed task in a current processing stage, and the to-be-synchronized task object comprising task synchronization information and a subtask identifier; determining a task distribution key according to the task synchronization information, the task distribution key being used to associate multiple subtasks of the to-be-processed task in the current processing stage to the same barrier synchronization process, and the barrier synchronization process being used to perform stage synchronization on the multiple subtasks; recording the subtask identifier in a key list state corresponding to the task distribution key in a stream processing framework according to the task distribution key; and controlling the to-be-processed task to flow from the current processing stage to a downstream processing stage according to the key list state. The dependence on external storage polling coordination can be reduced, and the timeliness of stage synchronization in the stream processing pipeline can be improved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, specifically relating to a task synchronization method and related apparatus based on a streaming processing pipeline. Background Technology

[0002] In a data processing system based on a streaming processing framework, a complete data processing process typically includes multiple interconnected processing steps. Different processing steps can be implemented by different operators, and the same operator can also process different data ranges through multiple parallel instances in a distributed environment.

[0003] In some business scenarios, the complete processing result of the previous processing step needs to be used as the input for the next processing step. For example, when a processing step is broken down into multiple parallel processing processes, the system can only continue to execute the next processing step after all the parallel processing processes have been completed. If the next processing step cannot accurately know the completion status of each parallel processing process in the previous processing step, problems such as incomplete processing results and inaccurate stage transitions are likely to occur.

[0004] In existing approaches, state consistency can be ensured through the snapshot mechanism of the streaming processing framework itself, or the processing state can be recorded in external storage and queried by subsequent stages. However, snapshot mechanisms are typically used for system state preservation and are difficult to directly express the completion dependencies between business processing stages, while external storage querying increases system coupling and storage access pressure. Summary of the Invention

[0005] This application provides a task synchronization method and related apparatus based on a streaming pipeline to reduce reliance on external storage polling coordination and improve the timeliness of inter-stage synchronization in the streaming pipeline.

[0006] This application provides a task synchronization method based on a streaming pipeline, including: Obtain the task object to be synchronized. The task object to be synchronized is used to represent a completed subtask of the task to be processed in the current processing stage. The task object to be synchronized includes task synchronization information and subtask identifier. A task distribution key is determined based on the task synchronization information. The task distribution key is used to associate multiple subtasks of the task to be processed in the current processing stage with the same barrier synchronization process. The barrier synchronization process is used to perform stage synchronization of the multiple subtasks. Based on the task dispatch key, the subtask identifier is recorded in the keyed list state corresponding to the task dispatch key in the streaming processing framework to update the subtask completion status of the task to be processed in the current processing stage; Based on the key list status, the task to be processed is controlled to flow from the current processing stage to the downstream processing stage.

[0007] According to the task synchronization method based on a streaming processing pipeline provided in this application, the step of recording the subtask identifier into the keying list state corresponding to the task distribution key in the streaming processing framework to update the subtask completion status of the task to be processed in the current processing stage includes: obtaining the keying list state corresponding to the task distribution key in the streaming processing framework; adding the subtask identifier to the keying list state to update the subtask completion status of the task to be processed in the current processing stage, wherein the subtasks corresponding to the subtask identifiers included in the keying list state are completed subtasks; the step of controlling the flow of the task to be processed from the current processing stage to the downstream processing stage according to the keying list state includes: when the number of different subtask identifiers recorded in the keying list state is consistent with the total number of subtasks in the task synchronization information, all subtasks of the task to be processed in the current processing stage have been completed; controlling the flow of the task to be processed from the current processing stage to the downstream processing stage.

[0008] According to the task synchronization method based on streaming processing pipeline provided in this application, controlling the flow of the task to be processed from the current processing stage to the downstream processing stage includes: acquiring intermediate data generated by each subtask during execution; merging the intermediate data of each subtask to obtain aggregated intermediate data; updating the task object corresponding to the task to be processed according to the aggregated intermediate data; and sending the updated task object to the downstream processing stage.

[0009] According to the task synchronization method based on streaming processing pipeline provided in this application, after controlling the flow of the task to be processed from the current processing stage to the downstream processing stage, the method further includes at least one of the following: clearing the key list state corresponding to the task dispatch key; resetting the task mode flag in the task object, the task mode flag being used to indicate the current task stage of the task object.

[0010] According to the task synchronization method based on streaming processing pipeline provided in this application, before obtaining the task object to be synchronized, the method further includes: obtaining the data processing range and estimated data volume corresponding to the task to be processed; determining the subtask division method corresponding to the data processing range according to the estimated data volume and the preset data volume threshold; and dividing the task to be processed into at least one subtask according to the subtask division method and the data processing range.

[0011] According to the task synchronization method based on streaming processing pipeline provided in this application, the step of determining the subtask partitioning method corresponding to the data processing range based on the estimated data volume and the preset data volume threshold includes: when the estimated data volume is not greater than the preset data volume threshold, determining the subtask partitioning method as a first partitioning method, wherein the first partitioning method indicates that at least one subtask is generated based on the processing time corresponding to the data processing range; when the estimated data volume is greater than the preset data volume threshold, determining the subtask partitioning method as a second partitioning method; wherein the second partitioning method indicates that the processing time corresponding to the data processing range is divided into multiple time buckets, each time bucket includes multiple interval-distributed processing periods, different time buckets correspond to different processing periods, and each subtask corresponds to one time bucket.

[0012] According to the task synchronization method based on streaming processing pipeline provided in this application, determining the subtask partitioning method corresponding to the data processing range based on the estimated data volume and the preset data volume threshold includes: determining the time span corresponding to the data processing range when the estimated data volume is greater than the preset data volume threshold; determining the subtask partitioning method as a third partitioning method when the time span is greater than the preset time span; wherein the third partitioning method aggregates the data processing range to obtain multiple processing range groups, the data in each processing range group belongs to the same continuous processing time range, and at least one subtask is generated according to each processing range group.

[0013] According to the task synchronization method based on streaming processing pipeline provided in this application, dividing the task to be processed into at least one subtask according to the subtask division method and the data processing range includes: dividing the data processing range into at least one sub-processing range according to the subtask division method; generating task version information corresponding to the task to be processed according to the current system timestamp, wherein the task version information is used to indicate the processing round of the task to be processed; and generating at least one subtask according to the at least one sub-processing range and the task version information.

[0014] According to the task synchronization method based on a streaming pipeline provided in this application, the downstream processing stage includes a waiting processing stage. The step of controlling the flow of the pending task from the current processing stage to the downstream processing stage based on the keying list state includes: when the keying list state indicates that all subtasks of the pending task in the current processing stage have been completed, controlling the pending task to flow from the current processing stage to the waiting processing stage and triggering a delay timer; when the delay timer is triggered, acquiring external status information corresponding to the pending task, the external status information indicating whether the external processing operation corresponding to the waiting processing stage has been completed; if the external status information indicates that the external processing operation has been completed, controlling the pending task to flow to the downstream processing stage of the waiting processing stage; if the external status information indicates that the external processing operation has not been completed, re-registering the delay timer, and acquiring the external status information again when the delayed timer is triggered again, until the external status information indicates that the external processing operation has been completed.

[0015] This application also provides a task synchronization device based on a streaming pipeline, including: The acquisition unit is used to acquire a task object to be synchronized. The task object to be synchronized is used to represent a completed subtask of the task to be processed in the current processing stage. The task object to be synchronized includes task synchronization information and subtask identifier. The determining unit is configured to determine a task distribution key based on the task synchronization information. The task distribution key is used to associate multiple subtasks of the task to be processed in the current processing stage with the same barrier synchronization process. The barrier synchronization process is used to perform stage synchronization of the multiple subtasks. The recording unit is used to record the subtask identifier into the keying list state corresponding to the task distribution key in the streaming processing framework according to the task distribution key, so as to update the completion status of the subtask corresponding to the task to be processed in the current processing stage. The control unit is used to control the flow of the task to be processed from the current processing stage to the downstream processing stage according to the status of the key list.

[0016] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the task synchronization method based on the streaming pipeline as described above.

[0017] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the task synchronization method based on a streaming pipeline as described above.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the task synchronization method based on a streaming pipeline as described above.

[0019] The task synchronization method and related apparatus based on streaming pipeline provided in this application associate multiple subtasks of the same task to be processed in the current processing stage with the same barrier synchronization process through task distribution keys, and use the key list state record in the streaming framework to record the subtask identifiers of each completed subtask, so that the system can determine the completion status of the subtasks in the current processing stage based on the key list state, and control the task to be processed to flow to the downstream processing stage after the stage synchronization conditions are met, thereby reducing the dependence on external storage polling coordination and improving the timeliness of inter-stage synchronization in the streaming pipeline. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a task synchronization method based on a streaming pipeline provided in this application.

[0022] Figure 2 This is a schematic diagram of the business process provided in this application.

[0023] Figure 3 This is a schematic diagram of the system architecture provided in this application.

[0024] Figure 4 This is a block diagram of the functional units of a task synchronization device based on a streaming processing pipeline provided in this application.

[0025] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0028] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0029] Currently, in multi-stage business processing based on streaming data processing systems, multiple parallel processing results in the same processing stage typically need to be completed before entering subsequent processing stages. Existing checkpoint barrier mechanisms are mainly used for snapshot consistency and are difficult to express the dependencies between business processing stages. On the other hand, periodic queries through external storage increase system coupling and external storage pressure.

[0030] To address the aforementioned issues, this application provides a task synchronization method and related apparatus based on a streaming pipeline. The embodiments of this application will be explained in detail below with reference to the accompanying drawings.

[0031] The streaming pipeline in this embodiment can be an Apache Flink (Flink)-based data processing pipeline, or other processing pipelines with streaming processing and state management capabilities. The streaming framework can run multiple operators, which can be connected via data streams. A processing stage can be implemented by one or more operators, and the same operator can have multiple parallel instances in a distributed environment. Tasks to be processed can flow between different processing stages and can be broken down into multiple subtasks for parallel execution within a specific processing stage.

[0032] The method provided in this application can be executed by an electronic device. The electronic device can be a server running a streaming processing framework, a node in a server cluster, a cloud computing device, a container node, or a device containing operators used for synchronous task processing in a streaming processing pipeline. For ease of explanation, the following description uses an electronic device executing the method as an example.

[0033] Please see Figure 1, Figure 1 This is a flowchart illustrating a task synchronization method based on a streaming pipeline provided in an embodiment of this application. The method may include the following steps.

[0034] S101, obtain the task object to be synchronized.

[0035] The task object to be synchronized represents a completed subtask of the task to be processed in the current processing stage. The task object includes task synchronization information and a subtask identifier. The current processing stage can be the current business processing stage of the task to be processed, such as a discount calculation stage, a remaining discount calculation stage, a red-ink reversal processing stage, or a claims processing stage. A subtask can be a parallel processing unit split from the task to be processed in the current processing stage. The task synchronization information can be information used for task synchronization processing, including information for identifying the task to be processed, the current processing stage, or the processing round, and information for supporting synchronization judgments in subsequent stages. The subtask identifier can be used to distinguish different subtasks in the current processing stage; for example, the subtask identifier can be an hour index, a date index, a processing range index, or other data identifiers that can distinguish subtasks.

[0036] S102, determine the task distribution key based on the task synchronization information.

[0037] The task dispatch key is used to associate multiple subtasks of the task to be processed in the current processing stage with the same barrier synchronization process. The barrier synchronization process is used to perform stage synchronization of the multiple subtasks. In other words, in the same current processing stage of the same task to be processed, although different subtasks can be executed by different parallel operators, these subtasks can be associated with the same synchronization process based on the same task dispatch key after completion. This allows the electronic device to determine the completion status of multiple subtasks from both task and stage dimensions.

[0038] In practice, the task distribution key can be determined based on at least some information from the task synchronization information, including the task identifier, current processing stage identifier, synchronization batch identifier, total number of subtasks, and task version information. For example, it can be generated based on a combination of the task ID, current processing stage identifier, synchronization batch identifier, total number of subtasks, and version number. This task distribution key can serve as the basis for state access and data routing in the streaming processing framework, allowing multiple subtasks of the same task in the current processing stage to enter the same state management scope.

[0039] S103, according to the task distribution key, record the subtask identifier in the keying list state corresponding to the task distribution key in the streaming processing framework, so as to update the subtask completion status of the task to be processed in the current processing stage.

[0040] The keyed list state can be a list-type state corresponding to the task dispatch key in a streaming processing framework. For example, in Flink, this keyed list state can correspond to a keyed list state. The keyed list state can be used to record the identifiers of subtasks that have reached the synchronization position. Whenever a completed subtask's corresponding synchronized task object arrives, the electronic device can access the corresponding keyed list state based on the task dispatch key and record the subtask identifier of that subtask in the keyed list state. The keyed list state corresponding to the same task dispatch key can store the identifiers of completed subtasks within the same pending task and the same current processing stage. Thus, by updating the keyed list state, the electronic device can obtain the subtask completion status of the pending task in the current processing stage.

[0041] S104, based on the key list status, control the task to be processed to flow from the current processing stage to the downstream processing stage.

[0042] The downstream processing stage can be the processing stage following the current processing stage. The electronic device can determine whether a task meets the conditions for moving from the current processing stage to the downstream processing stage based on the subtask completion status reflected in the keying list. When the transition conditions are met, the electronic device can release the task, allowing it to continue into subsequent operators or processing stages. Therefore, a task will not directly enter the downstream stage simply because a single subtask is completed; instead, the stage transition occurs only after the completion status of multiple subtasks related to the current processing stage meets the requirements.

[0043] As can be seen, this scheme associates multiple subtasks of the same pending task in the current processing stage with the same barrier synchronization process through task distribution keys, and uses the keyed list state in the streaming processing framework to record the subtask identifiers of each completed subtask. This enables the system to determine the completion status of subtasks in the current processing stage based on the keyed list state, and to control the pending task to flow to the downstream processing stage after the stage synchronization conditions are met. This reduces the dependence on external storage polling coordination and improves the timeliness of inter-stage synchronization in the streaming processing pipeline.

[0044] In one possible embodiment, recording the subtask identifier in the keying list state corresponding to the task distribution key in the streaming processing framework according to the task distribution key to update the subtask completion status of the task to be processed in the current processing stage includes: obtaining the keying list state corresponding to the task distribution key in the streaming processing framework; adding the subtask identifier to the keying list state to update the subtask completion status of the task to be processed in the current processing stage, wherein the subtasks corresponding to the subtask identifiers included in the keying list state are completed subtasks; controlling the flow of the task to be processed from the current processing stage to the downstream processing stage according to the keying list state includes: if the number of different subtask identifiers recorded in the keying list state is consistent with the total number of subtasks in the task synchronization information, all subtasks of the task to be processed in the current processing stage have been completed; controlling the flow of the task to be processed from the current processing stage to the downstream processing stage.

[0045] The total number of subtasks indicates the number of subtasks that need to be synchronized in the current processing stage of the task to be processed, while the number of different subtask identifiers indicates the number of different subtasks that have been completed and reached the synchronization position. Since the same subtask may arrive multiple times due to retries, repeated transmissions, or other reasons, the electronic device can make a judgment based on the number of different subtask identifiers, rather than simply based on the number of records.

[0046] In practical implementation, the current processing stage can be understood as a stage requiring multiple subtasks to complete together. When the first subtask is completed, the electronic device adds the subtask identifier of that subtask to the keying list state. At this time, if the total number of subtasks in the task synchronization information is 3, but there is only 1 different subtask identifier in the keying list state, it means that the current processing stage is not yet complete. When the second subtask is completed, the electronic device continues to add the second subtask identifier, while still waiting for the third subtask. Only when the third subtask is completed and arrives, and the number of different subtask identifiers recorded in the keying list state matches the total number of subtasks in the task synchronization information, does the electronic device determine that all subtasks in the current processing stage have been completed. The electronic device can also associate the intermediate data carried by the task object to be synchronized with the subtask identifier and record it in the keying list state, and control the task to be processed to flow to the downstream processing stage.

[0047] As can be seen, by using the total number of subtasks as the completion criterion and comparing the number of different subtask identifiers in the key list state with the total number of subtasks, the implementation of this application can reduce the impact of out-of-order or repeated arrival of subtasks on the stage flow judgment.

[0048] In one possible embodiment, controlling the flow of the task to be processed from the current processing stage to the downstream processing stage includes: acquiring intermediate data generated by each subtask during execution; merging the intermediate data of each subtask to obtain aggregated intermediate data; updating the task object corresponding to the task to be processed according to the aggregated intermediate data; and sending the updated task object to the downstream processing stage.

[0049] Intermediate data can be data generated during the execution of subtasks and needs to be aggregated before the stage transition. For example, intermediate data may include sets of other cost type codes, cross-cost type sets, intermediate state values, calculation result sets, or other business data that needs to be formed by multiple subtasks. The electronic device can obtain intermediate data associated with each subtask identifier from the keyed list status. Aggregated intermediate data can be data obtained after merging, deduplicating, summarizing, or concatenating intermediate data corresponding to multiple subtasks. Since each subtask processes different data ranges, the intermediate data obtained may also be different. For example, in a logistics waybill discount calculation scenario, different subtasks can process data from different dates or different time buckets and obtain sets of cost types related to that processing range. If the downstream processing stage needs to continue calculation based on the complete set of cost types, the electronic device, after confirming that all subtasks have been completed, can obtain the intermediate data generated by each subtask and merge these intermediate data to obtain aggregated intermediate data. Subsequently, the electronic device can write the aggregated intermediate data into the task object corresponding to the task to be processed and then send the updated task object to the downstream processing stage.

[0050] As can be seen, the embodiments of this application merge the intermediate data generated by each subtask before releasing the task object, which can enable the task object received by the downstream processing stage to carry the complete processing result of the current processing stage, reducing the processing steps of the downstream processing stage to query or summarize the results of multiple subtasks again.

[0051] In one possible embodiment, after controlling the flow of the task to be processed from the current processing stage to the downstream processing stage, the method further includes at least one of the following: clearing the key list state corresponding to the task dispatch key; resetting the task mode flag in the task object, the task mode flag being used to indicate the current task stage of the task object.

[0052] Clearing the key list status corresponding to the task dispatch key can either delete the subtask identifiers already recorded under that task dispatch key, or reset the key list status under that task dispatch key to empty. The task mode flag can be a status field in the task object, used to indicate the current task stage of the task object. For example, the task mode flag can correspond to different status identifiers of the task object in the subtask processing stage, the stage synchronization stage, or the downstream processing stage.

[0053] In practice, once multiple subtasks of a pending task have completed in the current processing stage, and the task object has been sent to the downstream processing stage, the current barrier synchronization process is complete. At this point, if the keying list state corresponding to the task dispatch key is retained, subsequent processing of the same or similar tasks may be affected by historical states, and this would also consume state resources of the streaming processing framework. Therefore, the electronic device can clear the keying list state corresponding to the task dispatch key, allowing the state corresponding to this synchronization process to be released promptly. Simultaneously, the electronic device can reset the task mode flag in the task object, ensuring that the task object has a stage flag matching the downstream processing logic after entering the downstream processing stage, preventing the downstream processing stage from continuing to interpret the task object as a subtask object.

[0054] As can be seen, by clearing the corresponding key list state and resetting the task mode flag in the task object after the task flow, the embodiments of this application can ensure that a stage synchronization process is completed in a timely manner, which not only reduces the impact of historical state on subsequent processing, but also makes the task object have a clearer stage state when entering the downstream processing stage.

[0055] In one possible embodiment, before obtaining the task object to be synchronized, the method further includes: obtaining the data processing range and estimated data volume corresponding to the task to be processed; determining the subtask division method corresponding to the data processing range based on the estimated data volume and a preset data volume threshold; and dividing the task to be processed into at least one subtask based on the subtask division method and the data processing range.

[0056] The data processing scope refers to the range of data that the task needs to process. For example, the data processing scope could include the start and end dates of the data, the billing period, the waybill range, the fee type range, or other business processing ranges. The estimated data volume can be used to represent the expected scale of data to be processed within the data processing scope, and the preset data volume threshold can be a pre-configured threshold for judging data volume. Before a task enters its current processing stage, the electronic device can first understand the range of data that the task needs to process and the expected amount of data. If the task involves relatively little data, excessive splitting will incur additional scheduling and state management overhead. If the task involves a large amount of data, overly coarse splitting may not fully utilize parallel processing capabilities. Therefore, the electronic device can first obtain the data processing scope and the estimated data volume, and then determine the subtask division method based on the relationship between the estimated data volume and the preset data volume threshold.

[0057] As can be seen, by determining the subtask division method by combining the data processing scope and the estimated data volume before the subtask is generated, the granularity of the subtask can be adapted to the actual data scale, reducing the resource waste or insufficient parallel capability caused by the fixed division method in different data volume scenarios.

[0058] In one possible embodiment, determining the subtask partitioning method corresponding to the data processing range based on the estimated data volume and the preset data volume threshold includes: when the estimated data volume is not greater than the preset data volume threshold, determining the subtask partitioning method as a first partitioning method, wherein the first partitioning method indicates that at least one subtask is generated based on the processing time corresponding to the data processing range; and when the estimated data volume is greater than the preset data volume threshold, determining the subtask partitioning method as a second partitioning method; wherein the second partitioning method indicates that the processing time corresponding to the data processing range is divided into multiple time buckets, each time bucket includes multiple interval-distributed processing periods, different time buckets correspond to different processing periods, and each subtask corresponds to one time bucket.

[0059] The first partitioning method is suitable for scenarios with small data volumes. In this scenario, the electronic device can directly generate a subtask based on the data processing range, and this subtask processes the corresponding data range. The second partitioning method is suitable for scenarios with large data volumes. In this scenario, the electronic device can divide the processing time corresponding to the data processing range into multiple time buckets. Each time bucket includes multiple interval-distributed processing periods, and different time buckets correspond to different processing periods, thus allowing different subtasks to process data within different processing periods.

[0060] In practice, a large time range can be imagined as a task that needs to be processed by multiple processors. If the amount of data within that time range is small, there is no need to break it down into smaller parts. The electronic device can generate a subtask according to the first division method, and let the subtask complete the data processing within the corresponding range.

[0061] If the amount of data within the time frame is large, the electronic device can divide the processing time into multiple time buckets according to the second partitioning method. For example, in a scenario where a day includes 24 hours, the processing time can be divided into three time buckets. One time bucket can include processing periods corresponding to 0:00, 3:00, 6:00, 9:00, 12:00, 15:00, 18:00, and 21:00; another time bucket can include processing periods corresponding to 1:00, 4:00, 7:00, 10:00, 13:00, 16:00, 19:00, and 22:00; and yet another time bucket can include processing periods corresponding to 2:00, 5:00, 8:00, 11:00, 14:00, 17:00, 20:00, and 23:00. In this way, each subtask processes multiple intervals of processing periods within a time bucket. Different subtasks process different ranges of data, allowing the current processing stage to proceed in parallel.

[0062] As can be seen, the embodiments of this application use fewer subtasks when the estimated data volume is no greater than the threshold, and generate multiple subtasks according to multiple time buckets when the estimated data volume is greater than the threshold. This allows the parallel processing granularity to be adjusted according to changes in data volume, thus maintaining a good balance between processing efficiency and state overhead.

[0063] In one possible embodiment, determining the subtask partitioning method corresponding to the data processing range based on the estimated data volume and the preset data volume threshold includes: determining the time span corresponding to the data processing range when the estimated data volume is greater than the preset data volume threshold; and determining the subtask partitioning method as a third partitioning method when the time span is greater than the preset time span. The third partitioning method aggregates the data processing range to obtain multiple processing range groups, where the data in each processing range group belongs to the same continuous processing time range, and at least one subtask is generated based on each processing range group.

[0064] The time span can be used to represent the length of the data processing range in the time dimension. The preset time span can be a pre-configured time length threshold, and the processing range group can be a continuous processing time range obtained by aggregating the data processing range. For example, when the data processing range covers a large number of dates, multiple consecutive dates can be aggregated into a processing range group. The data in each processing range group belongs to the same continuous processing time range, and at least one sub-task can be generated based on each processing range group.

[0065] In practical implementation, if the data is split solely based on its size, too many subtasks may be generated when the data processing scope covers a long time span. For example, a task to be processed may have both a large estimated data volume and cover a long date range. Simply generating independent subtasks for each day and each time bucket, while improving parallelism, would also rapidly increase the number of subtasks, thus increasing the pressure on state management and scheduling. Therefore, when the estimated data volume exceeds a preset data volume threshold, the electronic device can further determine the time span corresponding to the data processing scope. When the time span exceeds a preset time span, the electronic device uses a third partitioning method, first aggregating the data processing scope into multiple processing range groups corresponding to consecutive processing time ranges. For example, when the time range exceeds 15 days, the dates are aggregated into groups of 5 days each. In this way, the task is not infinitely subdivided, but the number of subtasks is controlled while maintaining a certain level of parallelism.

[0066] As can be seen, by introducing processing range groups when the data volume is large and the time span is long, the embodiments of this application can avoid the excessive expansion of the number of subtasks as the time span increases, and keep the number of subtask identifiers that need to be maintained in the subsequent keyed list state within a controllable range.

[0067] In one possible embodiment, dividing the task to be processed into at least one subtask according to the subtask division method and the data processing range includes: dividing the data processing range into at least one sub-processing range according to the subtask division method; generating task version information corresponding to the task to be processed based on the current system timestamp, wherein the task version information is used to indicate the processing round of the task to be processed; and generating at least one subtask according to the at least one sub-processing range and the task version information.

[0068] A sub-processing scope can be a processing scope divided from the data processing scope, and different sub-processing scopes can correspond to different sub-tasks. Task version information can be used to indicate the processing round of the task to be processed. For example, task version information can be generated based on the current system timestamp, or it can be generated based on the current system timestamp and the task identifier. After determining the sub-task division method, the electronic device can divide the data processing scope corresponding to the task to be processed into at least one sub-processing scope. For example, if the first division method is used, the entire data processing scope can be considered as a sub-processing scope; if the second division method is used, multiple time buckets can be used as multiple sub-processing scopes; if the third division method is used, multiple processing scope groups can be used as sub-processing scopes, or sub-processing scopes can be further formed within processing scope groups. Subsequently, the electronic device generates task version information based on the current system timestamp and writes the task version information into the generated sub-tasks.

[0069] In this way, multiple subtasks generated in the same processing round can carry the task version information corresponding to the same processing round. When the task to be processed generates subtasks again due to rerun, compensation processing, or recalculation, the new round of subtasks can carry new task version information, thus distinguishing them from the previous round of subtasks.

[0070] As can be seen, by introducing task version information when generating subtasks, the embodiments of this application can provide an identification basis for the processing round to which the subtask belongs, so that subsequent synchronous processing can be judged around the subtasks in the same processing round, reducing the possibility of subtasks in different processing rounds being mixed up and ensuring the idempotency of subtask reruns.

[0071] In one possible embodiment, the downstream processing stage includes a waiting processing stage. Controlling the flow of the task to be processed from the current processing stage to the downstream processing stage based on the key list state includes: when the key list state indicates that all subtasks of the task to be processed in the current processing stage have been completed, controlling the task to be processed to flow from the current processing stage to the waiting processing stage and triggering a delay timer; when the delay timer is triggered, acquiring external status information corresponding to the task to be processed, the external status information indicating whether the external processing operation corresponding to the waiting processing stage has been completed; if the external status information indicates that the external processing operation has been completed, controlling the task to be processed to flow to the downstream processing stage of the waiting processing stage; if the external status information indicates that the external processing operation has not been completed, re-registering the delay timer, and acquiring the external status information again when the delayed timer is triggered again, until the external status information indicates that the external processing operation has been completed.

[0072] The waiting-to-process stage can be a stage following the current processing stage and requiring external processing operations to complete before continuing the process. External processing operations can include search engine index refresh, data volume statistics, task status updates, or other external processing operations related to the task being processed. External status information can be status information recorded in external storage or external services, used to indicate whether the external processing operations corresponding to the waiting-to-process stage have been completed.

[0073] In practice, once the electronic device determines that all subtasks in the current processing stage have been completed based on the keying list status, it can transfer the pending task to the waiting processing stage and register a delay timer. The delay duration of this timer can be configured according to the business scenario, for example, it can be 30 seconds. When the delay timer is triggered, the electronic device can query the external status information corresponding to the pending task. If the external status information indicates that the external processing operation has been completed, the electronic device can control the pending task to continue to the downstream processing stage after the waiting processing stage. If the external status information indicates that the external processing operation has not been completed, the electronic device can re-trigger the delay timer and query the external status information again when the re-triggered delay timer is triggered, until the external processing operation is completed.

[0074] As can be seen, the embodiments of this application can transform the inter-stage waiting process into a periodically triggered state check process by triggering a delay timer during the waiting process and checking external state information when the timer is triggered, thereby reducing the periodic access pressure on external storage or external services caused by continuous polling.

[0075] It should be noted that, in other embodiments, the streaming processing framework is not limited to Flink, but can also be other streaming processing frameworks with state management capabilities; the number of time buckets, the number of processing periods included in each time bucket, and the distribution of processing periods can be configured according to the amount of business data and the parallelism of operators; the waiting processing stage is not limited to being implemented through a delay timer. In scenarios where an event notification can be generated when an external processing operation is completed, an external processing completion notification can also be received through a message queue, and the pending task can be controlled to flow to the downstream processing stage based on the external processing completion notification.

[0076] The following is combined with Figure 2 and Figure 3 Taking the logistics waybill discount calculation pipeline as an example, the overall business process of this application embodiment is illustrated.

[0077] For example Figure 2As shown, in the scenario of calculating logistics waybill discounts, the streaming pipeline can first read the tasks to be processed from the database. Specifically, the electronic device can poll the task table in the relational database to read discount calculation task records, and can use an optimistic locking mechanism to ensure that the same task is only obtained by one pipeline instance. The task to be processed can include fields such as billing period ID, settlement card code, fee type code, and data start and end dates. After reading the task to be processed, the task object enters the pre-processing stage along the streaming pipeline. The pipeline can first delete historical calculation results, that is, delete the historical calculation results corresponding to the task from columnar storage and key-value cache, in preparation for recalculation in this round. Subsequently, the pipeline can populate single-ticket discount tags, for example, by querying whether there is a single-ticket discount record for this waybill, and populating the found tags into the task object. After that, the pipeline can initialize full-amount claim data, for example, by scanning full-amount claim waybills at a fixed time every day through a scheduled task mechanism, and storing the relevant data in the key-value cache for use in subsequent processing stages. After completing the above preparations, the pipeline can continue to perform promotion matching and calculation, such as querying promotion configurations that match pending tasks, obtaining promotion data, and calculating promotion discounts.

[0078] After promotion matching and calculation are completed, the pipeline begins to consider whether the task needs to be split into multiple subtasks during the discount calculation phase. At this point, the system can determine whether the data volume exceeds a threshold. If the data volume exceeds the threshold, it means that the data volume corresponding to the task to be processed is large, and it can be split into 3 time bucket subtasks. For example, each date can be further split into multiple time buckets, so that different time buckets correspond to different subtasks. If the data volume does not exceed the threshold, time bucket splitting can be omitted, and each date corresponds to 1 subtask.

[0079] After generating discount calculation subtasks, the pipeline can execute discount calculations in parallel. Different subtasks can query matching discount configurations and calculate the discounted amount for their respective data ranges. Although these subtasks execute in parallel, the complete result of the current processing stage can only be released downstream after all subtasks are completed. Therefore, the completion of each subtask does not directly push the entire pending task into the subsequent stage; instead, it first enters the discount calculation barrier synchronization. After each subtask is completed, a task object to be synchronized is formed and reaches the barrier synchronization position. The barrier synchronization process determines the task dispatch key based on the task synchronization information and records the subtask identifier of that subtask in the keyed list state corresponding to the task dispatch key.

[0080] As the discount calculation subtasks arrive, the pipeline can determine if all subtasks have arrived. If the result is no, it means that some subtasks are still incomplete in the current processing stage, and the pipeline continues to wait for subsequent subtasks at the discount calculation barrier synchronization point. If the result is yes, it means that the number of different subtask identifiers recorded in the keyed list status is consistent with the total number of subtasks in the task synchronization information, and all subtasks of the pending task in the discount calculation stage have been completed. At this point, the pipeline can merge the intermediate data set and release it downstream. This intermediate data set may include other cost type sets carried by each subtask, intermediate status values, or other data that needs to be aggregated across subtasks. The pipeline can merge this intermediate data to obtain aggregated intermediate data, update the task object corresponding to the pending task based on the aggregated intermediate data, and then send the updated task object to the downstream processing stage.

[0081] After the discount calculation phase is completed and the task object is released, it can proceed to the protocol matching and calculation phase. Protocol matching and calculation can be used to further process the waybill discount calculation results according to the business protocol configuration. Subsequently, the task object can enter the monthly plan calculation phase to execute calculation logic related to the monthly plan. After the monthly plan calculation, the task object can enter the remaining discount calculation phase. For example, for scenarios like light cargo with no threshold, a dynamic time partitioning strategy is used again to generate subtasks for parallel processing. Similar to the aforementioned discount calculation, after the remaining discount calculation is completed, it can also enter the remaining discount barrier synchronization phase. It waits for all subtasks in this phase to arrive using the task distribution key and keyed list status before releasing the task object to subsequent phases.

[0082] After the remaining discount barrier is synchronized, the pipeline can continue calculating the fuel surcharge for the waybill. Once the fuel surcharge calculation is complete, the task object can enter a timer-delayed waiting phase for Elasticsearch (ES) refresh. Here, ES can be a search engine index, such as an Elasticsearch index. Since some downstream processing may rely on the latest data in the search engine index, if the index refresh is not yet complete, premature entry of the task object into the downstream processing phase may result in the downstream not being able to read complete or up-to-date data. Therefore, after the task reaches the waiting operator, a delay timer can be triggered. When the delay timer is triggered, the system checks the ES refresh status and determines whether the ES refresh is complete. If the result is negative, the delay timer is re-triggered and the waiting continues. If the result is positive, it indicates that the external processing operation has been completed, and the task object can be released to the subsequent processing phase. This process corresponds to the waiting processing phase, delay timer, and external status information check process in this application.

[0083] After Elasticsearch refreshes, the task object can enter the two-stage cross-period red-ink reversal processing. This processing can include two stages: the first stage reads and processes regular red-ink reversal waybills from the key-value cache partition set; the second stage, after synchronizing through barriers in the previous stage, queries and processes special red-ink reversal waybills from the search engine. Since cross-period red-ink reversal processing may also involve multiple parallel subtasks, barriers can be used again within each stage to synchronize and wait for all subtasks to complete.

[0084] After the red alert barrier synchronization is complete, the task object can enter the multi-billing period rebate processing stage to handle rebate calculations across multiple billing periods. After multi-billing period rebate processing, the pipeline can enter the timer delay waiting stage for data volume statistics and status updates. When the timer is triggered, external status information is checked. If the status update is not yet complete, it continues to wait; if the status update is complete, the task object is released and enters the subsequent processing stage.

[0085] After the status update wait is complete, the task object can enter the partial claims processing stage. Partial claims processing can generate partial claims subtasks for parallel processing. Since this stage may also have multiple parallel subtasks, after partial claims processing is complete, it can enter the claims barrier synchronization stage. Claims barrier synchronization can wait for all claims-related subtasks to complete, and after the keyed list status indicates that all subtasks have arrived, it releases the task object to the final output location. Finally, the pipeline can write the final state, writing the final processing result of the pending task to the result receiver, status table, or other result storage.

[0086] pass Figure 2 It can be seen that the task object can utilize parallel computing capabilities between each processing stage, and can also ensure that the subtask result of the current stage has been completed or the external state has met the requirements before entering the downstream processing stage.

[0087] For example Figure 3 As shown, from a system architecture perspective, this logistics waybill discount calculation pipeline can include a data source layer, a streaming processing pipeline, and a coordination mechanism. The data source layer can include a task database, a distributed cache, and a search engine index. The task database can be used to store tasks to be processed, which are then read by the task readers in the streaming processing pipeline. The distributed cache can be used to store data that needs to be reused during task processing, such as full claim data, tagged data, red-inking partition sets, or other intermediate state data. The search engine index can be used to store data required for subsequent queries, such as the data required by the second-stage red-inking processor when querying special red-inking waybills, or it can correspond to the aforementioned external indexes that need to be refreshed.

[0088] A streaming pipeline can be divided into multiple stages according to the business processing sequence. Stage 1 is data preparation, which may include a task reader, a result deleter, and a tag filler. The task reader can read tasks to be processed from the task database; the result deleter can delete historical calculation results; and the tag filler can combine distributed cache or other business data to fill the task object with information such as single-ticket discount tags. After this stage is completed, the task object has the basic information required to enter the discount calculation stage.

[0089] Phase 2 involves discount calculation, which may include a dynamic partition generator, a parallel discount calculation operator, and a barrier synchronization operator. The dynamic partition generator can generate discount calculation subtasks based on the data processing range and estimated data volume; for example, it can generate multiple time-bucket subtasks when the data volume exceeds a threshold, and fewer subtasks when the data volume does not exceed the threshold. The parallel discount calculation operator can perform the data calculations corresponding to each subtask. The barrier synchronization operator can collect subtask identifiers after each subtask is completed and determine whether all subtasks have been reached based on the keyed list status. Only after all subtasks have been reached does the barrier synchronization operator release the aggregated task object downstream.

[0090] Phase 3 involves protocol and remaining computation, which may include a protocol matching operator, a monthly planning operator, a remaining discount operator, and a barrier synchronization operator. The protocol matching operator performs protocol matching and computation, the monthly planning operator performs monthly planning computation, and the remaining discount operator performs remaining discount computation. When the remaining discount computation generates multiple parallel subtasks again, the barrier synchronization operator in Phase 3 can continue to synchronize based on the task distribution key and the keying list state, ensuring that the remaining discount-related subtasks are completed before continuing the process.

[0091] Phase 4, focusing on red-ink rebates and rebates, includes a Phase 1 red-ink rebate processor, a Phase 2 red-ink rebate processor, and a multi-term rebate operator. The Phase 1 red-ink rebate processor handles regular red-ink rebate orders read from a distributed cache partition set; the Phase 2 red-ink rebate processor queries and processes special red-ink rebate orders from the search engine index after the previous phase is completed; and the multi-term rebate operator handles rebate calculations across multiple billing periods. In this phase, the data access relationships between the Phase 1 and Phase 2 red-ink rebate processors and the search engine index can be corresponding to... Figure 3 The arrow relationship from the data source layer to the red-ink processing stage.

[0092] Phase 5 is the claims processing and output phase, which may include partial claims operators and a status writer. Partial claims operators can be used to generate and process some claims-related subtasks, while the status writer can be used to write the final status after task processing is complete. This phase can serve as the output phase of the logistics waybill discount calculation pipeline, enabling the tasks to be processed to complete their final status storage or result writing after going through the aforementioned multiple processing phases.

[0093] Figure 3 The coordination mechanism can include keying state management and timer delay waiting. Keying state management provides state support for the barrier synchronization operator in Phase 2, the barrier synchronization operator in Phase 3, and other synchronization nodes such as red alerts and claims that need to wait for the completion of multiple subtasks. Specifically, keying state management can maintain the state of the keying list corresponding to the task distribution key, enabling the barrier synchronization operator to record the subtask identifiers of completed subtasks and determine whether the current processing phase is complete based on the number of different subtask identifiers. Timer delay waiting can provide delayed triggering capabilities for processing nodes that need to wait for the completion of external processing operations, for example... Figure 2 The system includes timers that delay waiting for Elasticsearch (ES) refreshes and timers that delay waiting for status updates. Once a task object enters these waiting nodes, it can first register a delayed timer and check the external status information when the timer triggers. If the external status is not yet complete, it continues to wait; if the external status is complete, it continues to be released downstream.

[0094] pass Figure 3 The data source layer, streaming pipeline, and coordination mechanism work together to provide input data and external state for business processing. The task database, distributed cache, and search engine index provide input data and external state for business processing. The streaming pipeline organizes business processing logic in the following order: Phase 1 data preparation, Phase 2 discount calculation, Phase 3 protocol and remaining calculation, Phase 4 redemption and rebate, and Phase 5 claims processing and output. The coordination mechanism provides keyed state management and timer delay waiting capabilities at points where synchronization and waiting are required. Therefore, the logistics waybill discount calculation task can achieve data volume splitting, parallel computing, phase synchronization, intermediate data aggregation, external state waiting, and final state writing throughout the complete business chain.

[0095] The following describes a task synchronization device based on a streaming pipeline provided in this application. The task synchronization device based on a streaming pipeline described below corresponds to the task synchronization method based on a streaming pipeline described above.

[0096] Please see Figure 4The task synchronization device 400 based on a streaming processing pipeline includes: an acquisition unit 401, used to acquire a task object to be synchronized, the task object being used to represent a completed subtask of the task to be processed in the current processing stage, the task object including task synchronization information and a subtask identifier; a determination unit 402, used to determine a task distribution key according to the task synchronization information, the task distribution key being used to associate multiple subtasks of the task to be processed in the current processing stage to the same barrier synchronization process, the barrier synchronization process being used to perform stage synchronization of the multiple subtasks; a recording unit 403, used to record the subtask identifier in the keying list state corresponding to the task distribution key in the streaming processing framework according to the task distribution key, so as to update the completion status of the subtask corresponding to the task to be processed in the current processing stage; and a control unit 404, used to control the task to be processed to flow from the current processing stage to the downstream processing stage according to the keying list state.

[0097] In one possible embodiment, regarding the step of recording the subtask identifier to the keying list state corresponding to the task distribution key in the streaming processing framework according to the task distribution key, so as to update the subtask completion status of the task to be processed in the current processing stage, the recording unit 403 is specifically used for: obtaining the keying list state corresponding to the task distribution key in the streaming processing framework; adding the subtask identifier to the keying list state to update the subtask completion status of the task to be processed in the current processing stage, wherein the subtasks corresponding to the subtask identifiers included in the keying list state are completed subtasks; and controlling the task to be processed to flow from the current processing stage to the downstream processing stage according to the keying list state, including: when the number of different subtask identifiers recorded in the keying list state is consistent with the total number of subtasks in the task synchronization information, all subtasks of the task to be processed in the current processing stage have been completed; and controlling the task to be processed to flow from the current processing stage to the downstream processing stage.

[0098] In one possible embodiment, in controlling the flow of the task to be processed from the current processing stage to the downstream processing stage, the control unit 404 is specifically configured to: acquire intermediate data generated by each subtask during execution; merge the intermediate data of each subtask to obtain aggregated intermediate data; update the task object corresponding to the task to be processed according to the aggregated intermediate data; and send the updated task object to the downstream processing stage.

[0099] In one possible embodiment, after controlling the task to be processed to flow from the current processing stage to the downstream processing stage, the control unit 404 is specifically used for at least one of the following: clearing the key list state corresponding to the task dispatch key; resetting the task mode flag in the task object, the task mode flag being used to indicate the current task stage of the task object.

[0100] In one possible embodiment, before acquiring the task object to be synchronized, the acquisition unit 401 is specifically used to: acquire the data processing range and estimated data volume corresponding to the task to be processed; determine the subtask division method corresponding to the data processing range according to the estimated data volume and the preset data volume threshold; and divide the task to be processed into at least one subtask according to the subtask division method and the data processing range.

[0101] In one possible embodiment, regarding the determination of the subtask partitioning method corresponding to the data processing range based on the estimated data volume and the preset data volume threshold, the acquisition unit 401 is specifically configured to: determine the subtask partitioning method as a first partitioning method when the estimated data volume is not greater than the preset data volume threshold, wherein the first partitioning method indicates that at least one subtask is generated based on the processing time corresponding to the data processing range; and determine the subtask partitioning method as a second partitioning method when the estimated data volume is greater than the preset data volume threshold; wherein the second partitioning method indicates that the processing time corresponding to the data processing range is divided into multiple time buckets, each time bucket includes multiple interval-distributed processing periods, different processing periods correspond to different time buckets, and each subtask corresponds to one time bucket.

[0102] In one possible embodiment, regarding the determination of the subtask division method corresponding to the data processing range based on the estimated data volume and the preset data volume threshold, the acquisition unit 401 is specifically used to: determine the time span corresponding to the data processing range when the estimated data volume is greater than the preset data volume threshold; and determine the subtask division method as a third division method when the time span is greater than the preset time span; wherein the third division method aggregates the data processing range to obtain multiple processing range groups, the data in each processing range group belongs to the same continuous processing time range, and generates at least one subtask based on each processing range group.

[0103] In one possible embodiment, regarding the division of the task to be processed into at least one subtask according to the subtask division method and the data processing range, the acquisition unit 401 is specifically configured to: divide the data processing range into at least one sub-processing range according to the subtask division method; generate task version information corresponding to the task to be processed based on the current system timestamp, wherein the task version information is used to indicate the processing round of the task to be processed; and generate at least one subtask according to the at least one sub-processing range and the task version information.

[0104] In one possible embodiment, the downstream processing stage includes a waiting processing stage. Regarding controlling the flow of the task to be processed from the current processing stage to the downstream processing stage based on the key list state, the control unit 404 is specifically configured to: when the key list state indicates that all subtasks of the task to be processed in the current processing stage have been completed, control the task to be processed to flow from the current processing stage to the waiting processing stage and trigger a delay timer; when the delay timer is triggered, acquire external status information corresponding to the task to be processed, the external status information indicating whether the external processing operation corresponding to the waiting processing stage has been completed; if the external status information indicates that the external processing operation has been completed, control the task to be processed to flow to the downstream processing stage of the waiting processing stage; if the external status information indicates that the external processing operation has not been completed, re-register the delay timer, and when the delayed timer is triggered again, acquire the external status information again until the external status information indicates that the external processing operation has been completed.

[0105] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. For example... Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a task synchronization method based on a streaming processing pipeline. The method includes: obtaining a task object to be synchronized, the task object being used to represent a completed subtask of the task to be processed in the current processing stage, the task object including task synchronization information and a subtask identifier; determining a task distribution key based on the task synchronization information, the task distribution key being used to associate multiple subtasks of the task to be processed in the current processing stage with the same barrier synchronization process, the barrier synchronization process being used to perform stage synchronization of the multiple subtasks; recording the subtask identifier in the keying list state corresponding to the task distribution key in the streaming processing framework according to the task distribution key, to update the completion status of the subtask corresponding to the task to be processed in the current processing stage; and controlling the task to be processed to flow from the current processing stage to the downstream processing stage according to the keying list state.

[0106] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] On the other hand, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a task synchronization method based on a streaming processing pipeline provided by the above methods. The method includes: obtaining a task object to be synchronized, the task object being used to represent a completed subtask of the task to be processed in the current processing stage, the task object including task synchronization information and a subtask identifier; determining a task distribution key based on the task synchronization information, the task distribution key being used to associate multiple subtasks of the task to be processed in the current processing stage with the same barrier synchronization process, the barrier synchronization process being used to perform stage synchronization of the multiple subtasks; recording the subtask identifier in a keying list state corresponding to the task distribution key in the streaming processing framework, to update the completion status of the subtask corresponding to the task to be processed in the current processing stage; and controlling the task to be processed to flow from the current processing stage to a downstream processing stage based on the keying list state.

[0108] In another aspect, this application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described task synchronization methods based on a streaming processing pipeline. The method includes: obtaining a task object to be synchronized, the task object representing a completed subtask of the task to be processed in the current processing stage, the task object including task synchronization information and a subtask identifier; determining a task distribution key based on the task synchronization information, the task distribution key associating multiple subtasks of the task to be processed in the current processing stage to the same barrier synchronization process, the barrier synchronization process being used for stage synchronization of the multiple subtasks; recording the subtask identifier in a keying list state corresponding to the task distribution key in the streaming processing framework, to update the completion status of the subtask corresponding to the task to be processed in the current processing stage; and controlling the task to be processed to flow from the current processing stage to a downstream processing stage based on the keying list state.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A task synchronization method based on a streaming pipeline, characterized in that, include: Obtain the task object to be synchronized. The task object to be synchronized is used to represent a completed subtask of the task to be processed in the current processing stage. The task object to be synchronized includes task synchronization information and subtask identifier. A task distribution key is determined based on the task synchronization information. The task distribution key is used to associate multiple subtasks of the task to be processed in the current processing stage with the same barrier synchronization process. The barrier synchronization process is used to perform stage synchronization of the multiple subtasks. Based on the task dispatch key, the subtask identifier is recorded in the keyed list state corresponding to the task dispatch key in the streaming processing framework to update the subtask completion status of the task to be processed in the current processing stage; Based on the key list status, the task to be processed is controlled to flow from the current processing stage to the downstream processing stage.

2. The method according to claim 1, characterized in that, The step of recording the subtask identifier into the keyed list state corresponding to the task distribution key in the streaming processing framework according to the task distribution key, so as to update the completion status of the subtask corresponding to the task to be processed in the current processing stage, includes: Obtain the key list status corresponding to the task distribution key in the streaming processing framework; The subtask identifier is added to the keying list state to update the subtask completion status of the task to be processed in the current processing stage. The subtasks corresponding to the subtask identifiers included in the keying list state are completed subtasks. The step of controlling the flow of the task to be processed from the current processing stage to the downstream processing stage according to the key list status includes: If the number of different subtask identifiers recorded in the keying list state is consistent with the total number of subtasks in the task synchronization information, all subtasks of the task to be processed in the current processing stage have been completed; Control the flow of the task to be processed from the current processing stage to the downstream processing stage.

3. The method according to claim 2, characterized in that, The control of the task to be processed from the current processing stage to the downstream processing stage includes: Obtain the intermediate data generated during the execution of each of the subtasks; Merge the intermediate data from each of the subtasks to obtain aggregated intermediate data; Update the task object corresponding to the task to be processed based on the aggregated intermediate data; The updated task object is sent to the downstream processing stage.

4. The method according to claim 3, characterized in that, After controlling the transfer of the task to be processed from the current processing stage to the downstream processing stage, the method further includes at least one of the following: Clear the key list status corresponding to the task distribution key; Reset the task mode flag in the task object, which is used to indicate the current task stage of the task object.

5. The method according to any one of claims 1-4, characterized in that, Before obtaining the task object to be synchronized, the method further includes: Obtain the data processing range and estimated data volume corresponding to the task to be processed; Based on the estimated data volume and the preset data volume threshold, determine the sub-task division method corresponding to the data processing range; Based on the subtask division method and the data processing scope, the task to be processed is divided into at least one subtask.

6. The method according to claim 5, characterized in that, The step of determining the subtask division method corresponding to the data processing range based on the estimated data volume and the preset data volume threshold includes: If the estimated data volume is not greater than the preset data volume threshold, the subtask division method is determined to be the first division method, and the first division method indicates that at least one subtask is generated based on the processing time corresponding to the data processing range. If the estimated data volume is greater than the preset data volume threshold, the subtask division method is determined to be the second division method; The second division method indicates that the processing time corresponding to the data processing range is divided into multiple time buckets. Each time bucket includes multiple processing periods distributed at intervals. Different time buckets correspond to different processing periods, and each subtask corresponds to one time bucket.

7. The method according to claim 5, characterized in that, The step of determining the subtask division method corresponding to the data processing range based on the estimated data volume and the preset data volume threshold includes: If the estimated data volume is greater than the preset data volume threshold, determine the time span corresponding to the data processing range; If the time span is greater than the preset time span, the subtask division method is determined to be the third division method; The third division method aggregates the data processing range to obtain multiple processing range groups. The data in each processing range group belongs to the same continuous processing time range, and at least one sub-task is generated according to each processing range group.

8. The method according to claim 5, characterized in that, The step of dividing the task to be processed into at least one subtask according to the subtask division method and the data processing scope includes: Based on the subtask division method, the data processing scope is divided into at least one sub-processing scope; The task version information corresponding to the task to be processed is generated based on the current system timestamp, and the task version information is used to indicate the processing round of the task to be processed; At least one subtask is generated based on the at least one subprocessing scope and the task version information.

9. The method according to claim 2, characterized in that, The downstream processing stage includes a waiting processing stage. The step of controlling the flow of the pending task from the current processing stage to the downstream processing stage based on the key list status includes: When the key list status indicates that all subtasks of the task to be processed have been completed in the current processing stage, the task to be processed is controlled to flow from the current processing stage to the waiting processing stage, and a delay timer is triggered. When the delay timer is triggered, the external status information corresponding to the task to be processed is obtained. The external status information is used to indicate whether the external processing operation corresponding to the waiting processing stage has been completed. If the external status information indicates that the external processing operation has been completed, then the task to be processed is controlled to flow to the downstream processing stage of the waiting processing stage; If the external status information indicates that the external processing operation has not been completed, the delay timer is re-registered, and the external status information is obtained again when the delay timer is triggered again, until the external status information indicates that the external processing operation has been completed.

10. A task synchronization device based on a streaming pipeline, characterized in that, include: The acquisition unit is used to acquire a task object to be synchronized. The task object to be synchronized is used to represent a completed subtask of the task to be processed in the current processing stage. The task object to be synchronized includes task synchronization information and subtask identifier. The determining unit is configured to determine a task distribution key based on the task synchronization information. The task distribution key is used to associate multiple subtasks of the task to be processed in the current processing stage with the same barrier synchronization process. The barrier synchronization process is used to perform stage synchronization of the multiple subtasks. The recording unit is used to record the subtask identifier into the keying list state corresponding to the task distribution key in the streaming processing framework according to the task distribution key, so as to update the completion status of the subtask corresponding to the task to be processed in the current processing stage. The control unit is used to control the flow of the task to be processed from the current processing stage to the downstream processing stage according to the status of the key list.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the task synchronization method based on a streaming pipeline as described in any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the task synchronization method based on a streaming pipeline as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the task synchronization method based on a streaming pipeline as described in any one of claims 1 to 9.

Images