Task processing method, task processing apparatus, storage medium, and electronic device

Abstract

This disclosure relates to a task processing method, a task processing apparatus, a storage medium, and an electronic device, and pertains to the field of data processing technology. The method includes: running a first task to generate and output first task data; starting a second task; wherein the first task and the second task are tasks under the same business; determining a first timestamp for switching tasks; starting to run the second task based on a second timestamp earlier than the first timestamp; or starting to run the second task before determining the first timestamp; switching the output of the first task data to the output of the second task data based on the first timestamp; wherein the second task data is data generated by running the second task. This disclosure enables smooth and accurate switching of task data.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of data processing technology, and more specifically, the embodiments of this disclosure relate to a task processing method, a task processing apparatus, a computer-readable storage medium, and an electronic device. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure set forth in the claims, and the description herein is not acknowledged as prior art simply because it is included in this section.

[0003] Currently, in various real-time business scenarios, such as live streaming data and advertising recommendations, service interruptions or changes in business logic can impact the accuracy of services, typically requiring task restarts. However, restarts often fail to load the previous state, leading to data loss and other anomalies. Existing technologies address restart issues by restarting computing tasks within specific time periods or by implementing task switching through fixed methods to reduce restart time. However, these methods only reduce restart time to a certain extent and are insufficient to effectively solve problems such as data interruptions or performance disruptions caused by task switching. Task switching still has significant limitations and cannot achieve seamless switching. Summary of the Invention

[0004] However, the accuracy and effectiveness of task processing need to be improved when facing data restarts.

[0005] Therefore, there is a great need for a task processing method that can improve the accuracy and effectiveness of task processing and enable seamless task switching.

[0006] In this context, embodiments of the present disclosure are intended to provide a task processing method, a task processing apparatus, a computer-readable storage medium, and an electronic device.

[0007] According to a first aspect of this disclosure, a task processing method is provided, comprising: running a first task to generate and output first task data; starting a second task; wherein the first task and the second task are tasks under the same business; determining a first timestamp for switching tasks; starting to run the second task according to a second timestamp earlier than the first timestamp; or starting to run the second task before determining the first timestamp; switching the output of the first task data to the output of the second task data according to the first timestamp; wherein the second task data is data generated by running the second task.

[0008] In one implementation, switching the output of first task data to output of second task data based on the first timestamp includes: outputting second task data after the first timestamp, and not outputting first task data after the first timestamp.

[0009] In one implementation, the first task data following the first timestamp includes: first task data generated by processing input data later than the first timestamp through the first task; the second task data following the first timestamp includes: second task data generated by processing input data later than the first timestamp through the second task.

[0010] In one embodiment, the method further includes at least one of the following steps: if input data later than the first timestamp is read through the first task, the input data is discarded; if input data earlier than the first timestamp is read through the second task, the input data is discarded.

[0011] In one implementation, the timestamp of the input data includes: the time when the upstream provider of the input data stored the input data, or the time when the first task or the second task received the input data, or the event time corresponding to the input data.

[0012] In one implementation, the first task data following the first timestamp includes: first task data whose data processing time is later than the first timestamp; the second task data following the first timestamp includes: second task data whose data processing time is later than the first timestamp.

[0013] In one implementation, after switching the output of the first task data to the output of the second task data, the method further includes: continuing to run the first task; if the watermark time corresponding to the first task is reached, and the first task data before the first timestamp has been completely output, then stopping the operation of the first task.

[0014] In one implementation, determining the first timestamp for switching tasks includes: if the first task and the second task use time windows to process input data, determining the first timestamp based on the first time window used by the first task and the second time window used by the second task.

[0015] In one implementation, determining the first timestamp based on the first time window used by the first task and the second time window used by the second task includes: determining the common boundary time of the first time window and the second time window, wherein the common boundary time is both the end time of the first time window and the start time of the second time window; and determining the first timestamp based on the common boundary time.

[0016] In one implementation, starting the second task based on a second timestamp earlier than the first timestamp includes: reading input data later than the second timestamp through the second task.

[0017] In one embodiment, the method further includes: upon determining the first timestamp, sending a first switching instruction to the first task and a second switching instruction to the second task, wherein both the first switching instruction and the second switching instruction include the first timestamp; in response to receiving first consent information from the first task and second consent information from the second task, sending a third switching instruction to the first task and a fourth switching instruction to the second task; wherein the third switching instruction instructs the first task to switch from outputting first task data to not outputting first task data according to the first timestamp; and wherein the fourth switching instruction instructs the second task to switch from not outputting second task data to outputting second task data according to the first timestamp.

[0018] According to a second aspect of this disclosure, a task processing apparatus is provided, comprising: a first task running module for running a first task, generating and outputting first task data; a second task initiation module for initiating a second task; wherein the first task and the second task are tasks under the same business; a first timestamp determination module for determining a first timestamp for switching tasks; a second task running module for starting to run the second task according to a second timestamp earlier than the first timestamp; or starting to run the second task before determining the first timestamp; and a task data switching module for switching the output of first task data to the output of second task data according to the first timestamp; wherein the second task data is data generated by running the second task.

[0019] According to a third aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the task processing method of the first aspect described above and its possible implementations.

[0020] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the task processing method of the first aspect and possible implementations thereof by executing the executable instructions.

[0021] In this disclosed solution, on the one hand, this exemplary embodiment provides a new task processing method that can switch the output first task data to second task data based on a determined first timestamp. Compared with the related technologies that switch tasks by reducing restart time, this exemplary embodiment is not limited by the length of restart time and relies on the first timestamp to switch task data, avoiding significant interruptions caused by restarts due to differences in task states. The switching of task data is more accurate and smooth. On the other hand, compared with the related technologies that can only restart within a specific time period, this exemplary embodiment can switch task data after determining the first timestamp, without being limited by application scenarios and time periods. It has a wide range of business scenarios and strong applicability. Attached Figure Description

[0022] Figure 1 (a)- Figure 1 (b) A schematic diagram of a task processing method in the related art is shown;

[0023] Figure 2 A schematic diagram of a system architecture in this exemplary embodiment is shown;

[0024] Figure 3 A flowchart illustrating a task processing method in this exemplary embodiment is shown;

[0025] Figure 4 This example illustrates the task processing diagrams for the first and second tasks in this exemplary embodiment.

[0026] Figure 5 A schematic diagram of a scrolling time window is shown;

[0027] Figure 6 A schematic diagram of the first and second tasks in this exemplary embodiment is shown;

[0028] Figure 7 A schematic diagram of a sliding time window is shown;

[0029] Figure 8 This diagram illustrates the architecture of a task processing environment in this exemplary embodiment.

[0030] Figure 9 This illustration shows an interactive flowchart of a task processing method in this exemplary embodiment;

[0031] Figure 10 A flowchart illustrating another task processing method in this exemplary embodiment is shown;

[0032] Figure 11 This diagram illustrates the structure of a task processing method according to this exemplary embodiment.

[0033] Figure 12 This diagram illustrates the structure of a task processing apparatus according to this exemplary embodiment.

[0034] Figure 13 A schematic diagram of the structure of an electronic device in this exemplary embodiment is shown.

[0035] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0036] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0037] The embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0038] The principles and spirit of this disclosure are explained in detail below with reference to several representative embodiments. Invention Overview

[0040] The inventors have discovered that the accuracy and effectiveness of current task processing methods need improvement. Specifically, in related technologies, a common approach to addressing restart issues is to restart during specific time periods, such as off-peak hours when business is less affected. However, this method is only applicable to specific business scenarios and has poor adaptability. Alternatively, startup time can be optimized through certain technical means, such as... Figure 1 (a)- Figure 1 (b) shows a schematic diagram of a task processing method in the related art. Figure 1 (a) illustrates that during a normal upgrade, switching from Task A to Task B requires running Task A, stopping Task A, starting Task B, and then running Task B. Optimizing the startup time through certain technical means can achieve the following... Figure 1As shown in (b), task A and task B are started simultaneously. After starting, task A is stopped and its state is stored once, saving the state to remote storage. Then, task B loads the state data synchronously. The interruption time depends on the size of the task state. Small states can be loaded quickly through certain technical means, but larger states may take longer. Therefore, in some scenarios, although this method can reduce restart time, there will still be a relatively obvious interruption.

[0041] In view of the foregoing, this disclosure provides a task processing method, a task processing apparatus, a computer-readable storage medium, and an electronic device. On one hand, this exemplary embodiment provides a novel task processing method capable of switching output first task data to second task data based on a determined first timestamp. Compared to related technologies that switch tasks by reducing restart time, this exemplary embodiment is not limited by the length of the restart time, relying on the first timestamp to achieve task data switching, thus avoiding significant interruptions due to differences in task states. The task data switching is more accurate and smoother. On the other hand, compared to related technologies that can only restart within a specific time period, this exemplary embodiment can achieve task data switching after determining the first timestamp, without being limited by application scenarios or time periods, making it widely applicable and highly adaptable to various business scenarios.

[0042] The various non-limiting embodiments of this disclosure are described in detail below.

[0043] Application Scenarios Overview

[0044] It should be noted that the following application scenarios are shown only to facilitate understanding of the spirit and principles of this disclosure, and the implementation of this disclosure is not limited in any way. On the contrary, the implementation of this disclosure can be applied to any applicable scenario.

[0045] The implementation methods disclosed herein can be applied to scenarios involving the switching of Flink tasks or real-time computing tasks. They enable smooth and accurate task switching. The application scenarios are described in detail below in conjunction with the system architecture.

[0046] Figure 2A schematic diagram of a system architecture for a task processing method is shown. This system architecture includes an upstream unit 210, a task processing system 220, and a downstream unit 230. The task processing system 220 can obtain input data from the upstream unit 210, run a first task 221 to generate first task data, run a second task 222 to generate second task data, switch the output first task data to output second task data according to a first timestamp used for task switching, and send the first task data and / or the second task data to the downstream unit 230. Specific application scenarios, task quantity / type, and architectural details can be customized or configured; this disclosure does not impose specific limitations in these areas.

[0047] Exemplary methods

[0048] Exemplary embodiments of this disclosure provide a task processing method. (See reference...) Figure 3 As shown, the method may include steps S310 to S350. The following describes... Figure 3 Each step in the process will be explained in detail.

[0049] refer to Figure 3 In step S310, the first task is run, and the first task data is generated and output.

[0050] The first task can be any task in any business scenario, such as data statistics tasks like live stream popularity, dynamic music chart calculation tasks in music programs, media information recommendation tasks, etc. The first task data refers to data related to the first task, such as data to be statistically analyzed, data to be calculated, or data to be analyzed.

[0051] In this exemplary embodiment, before switching tasks, the first task can be run normally to generate and output first task data. Running the first task can involve reading input data, processing the input data, and obtaining the first task data. This processing can be performing specific calculations based on the input data, such as performing statistical calculations on the input data. The result of the statistical calculations is considered as the first task data, which can then be sent downstream. Processing the input data can also involve simply forwarding the input data. For example, after reading the input data, simple cleaning or no processing can be performed on the input data, and the input data can be sent downstream as the first task data. In this exemplary embodiment, the first task can be started before running it.

[0052] In step S320, the second task is started; the first task and the second task are tasks under the same business.

[0053] The second task and the first task can be different scenarios within the same business. The same business could include data statistics such as live stream popularity, dynamic ranking calculations, media information recommendation, etc. Different tasks can have different implementation logic, such as different code. For example, both the first and second tasks might be tasks related to calculating monetary amounts. However, the first task might use method one, while the second task might use method two. Although their business objectives are the same and they belong to the same business, the different methods of calculation lead to differences in their code logic. In related technologies, when developers complete the second task and deploy it, it signifies a logic switch, requiring the first task to be stopped; otherwise, real-time monetary calculations may result in errors.

[0054] In this exemplary embodiment, starting the second task refers to the preliminary work required to run the second task. Starting the task occurs at the beginning of the task lifecycle; only after task initiation is complete can the task meet the conditions for execution. Running the task refers to the process of continuously performing work according to preset logic and steps after the task begins execution. For example, starting a task can be starting a new task or a process. Running a task can be reading data obtained from upstream. The data read during task execution can be processed, or it can be left unprocessed and processed later. This exemplary embodiment can start the second task while running the first task, or it can start the second task within a preset time after running the first task.

[0055] In step S330, a first timestamp for switching tasks is determined.

[0056] The first timestamp refers to the node timestamp used to determine the task data switching. In this exemplary embodiment, switching tasks or switching task data essentially means determining which input data can be processed by running the first task to obtain the first task data, and which data can be processed by running the second task to obtain the second task data. The first task and the second task can determine whether to process the currently acquired input data or process it and send it downstream based on the first timestamp.

[0057] The first timestamp can be a specific moment in time. The first or second task can use this timestamp to determine whether it should process the received input data. The types of first timestamps can include various types, such as processing time, event time, or recording time. This exemplary embodiment can determine the processing method for different data based on the first timestamp. For example, different input data or task data are generated at different times. This exemplary embodiment can associate the timestamp of the data with the data itself, and subsequently determine which data to output based on the chronological relationship between the first timestamp and the data timestamps, such as determining which task data to send downstream.

[0058] In step S340, the second task is started based on a second timestamp that is earlier than the first timestamp; or the second task is started before the first timestamp is determined.

[0059] The second timestamp refers to the timestamp at which the second task is run. In this exemplary embodiment, the second timestamp is located before the first timestamp. The specific time difference can be set according to actual needs, and this disclosure does not impose specific limitations on it. This exemplary embodiment can start running the second task at the second timestamp. For example, the first and second tasks can be run simultaneously before the second timestamp, or the first task can be run first, then the second task can be started, and then the second task can be run before the second timestamp, to ensure that both the first and second tasks are running before the first timestamp. Since the time order of the acquired input data may be out of order, starting the second task before the first timestamp and reading the input data can achieve preheating and seamless switching, avoiding situations such as missing input data. This exemplary embodiment can also start running the second task before determining the first timestamp. For example, the first and second tasks can be run simultaneously before determining the first timestamp, or the first task can be run first, then the second task can be started, and then the second task can be run before determining the first timestamp, etc. Figure 4 The diagram illustrates a task processing schematic of a first task and a second task in this exemplary embodiment. The first task can be run first, and the second task can be started simultaneously, or the second task can be started after a short interval. Then, a first timestamp 410 is determined, and the second task starts running at a second timestamp 420 earlier than the first timestamp 410. When the window of the first task is completed or the traffic ends, the first task stops at the corresponding third timestamp 430.

[0060] In step S350, the output of the first task data is switched to the output of the second task data according to the first timestamp; the second task data is the data generated by running the second task.

[0061] After determining the first timestamp, the output of the first task data can be switched to the output of the second task data based on the first timestamp. In this exemplary embodiment, the switching from outputting the first task data to outputting the second task data can be a variable switch, that is, it is not that after switching from outputting the first task data to outputting the second task data, the first task data cannot be output again. Specifically, whether to output the first task data or the second task data depends on the processing of the input data by the first and second tasks based on the first timestamp. For example, after reading the input data, the first and second tasks can determine which input data they should process based on the first timestamp, generating the first task data and the second task data accordingly, and then outputting the first task data and the second task data downstream. For example, the first task can process input data with timestamps before the first timestamp and generate the first task data to be output downstream, while the second task can process input data with timestamps after the first timestamp and output it downstream. If the first task processes all input data before the first timestamp and outputs this part of the first task data completely downstream, then only the second task will output the second task data downstream. If, during the intersection of the running time of the first and second tasks, the first task data generated by the first task that is earlier than the first timestamp has not been completely output downstream, but the second task generates the second task data that is later than the first timestamp, then both the first task data and the second task data will be sent downstream. This exemplary embodiment enables the first and second tasks to clearly identify the input data they are processing through the aforementioned first timestamp, switch tasks, and output the processed task data downstream. Using the timestamp of the data to be processed itself and the first timestamp as the judgment condition, it is not constrained by the scene time, can be applied to a variety of application scenarios, and can achieve seamless switching and comprehensive processing of task data.

[0062] In an exemplary embodiment, the above-mentioned switching from outputting first task data to outputting second task data based on a first timestamp may include:

[0063] Output the data for the second task after the first timestamp, but do not output the data for the first task after the first timestamp.

[0064] In this embodiment, the first task data can be task data generated by running the first task, and the second task data can be task data generated by running the second task. In this exemplary embodiment, due to the continuous running of the first task or the running of the second task before the first timestamp, first task data later than the first timestamp or second task data earlier than the first timestamp may be generated. Based on this, this exemplary embodiment can determine the order of the first timestamp and the timestamp of the task data according to the first timestamp, and output the second task data later than the first timestamp, instead of outputting the first task data after the first timestamp, so as to achieve accurate output of task data.

[0065] It should be noted that the switching of task data in this exemplary embodiment is not a switching in the conventional sense. Essentially, it determines which data to process or which data to process based on the first timestamp to generate corresponding task data. This method of distinguishing input data or task data based on the first timestamp and determining task processing or task data output can determine whether data is processed, which task processes it, or whether data is output from the dimension of the data itself, without being affected by restart events.

[0066] In one exemplary embodiment, the first task data following the first timestamp includes: first task data generated by processing input data later than the first timestamp through the first task; the second task data following the first timestamp includes: second task data generated by processing input data later than the first timestamp through the second task.

[0067] Here, input data refers to data obtained or transmitted from upstream. In this exemplary embodiment, both the first task and the second task can read the input data. For example, the first task and the second task can read all the data in the same input data table and determine the data that they should process based on the first timestamp. For example, the first task reads all the input data but only processes the input data earlier than the first timestamp to generate the first task data, and the second task reads all the input data but only processes the input data later than the first timestamp to generate the second task data.

[0068] This exemplary embodiment defines the first task data and the second task data based on the timestamp of the input data at the input terminal. The first task data generated by processing the input data later than the first timestamp is regarded as the first task data after the first timestamp; the second task data generated by processing the input data later than the first timestamp is regarded as the second task data after the first timestamp.

[0069] In an exemplary embodiment, the above-described task processing method may further include at least one of the following steps:

[0070] If input data later than the first timestamp is read through the first task, then the input data is discarded;

[0071] If input data earlier than the first timestamp is read through the second task, then the input data is discarded.

[0072] In this exemplary embodiment, for different tasks, based on the determination of the first timestamp, it can be determined which input data it can process. If the first task reads input data earlier than the first timestamp, it can process it to obtain the corresponding first task data. If it reads input data later than the first timestamp, it discards the input data. If the second task reads input data earlier than the first timestamp, it discards the input data. If it reads input data later than the first timestamp, it can process it to obtain the corresponding second task data.

[0073] In one exemplary embodiment, the timestamp of the input data includes: the time when the upstream storage of the input data provides the input data, or the time when the first task or the second task receives the input data, or the event time corresponding to the input data.

[0074] In this exemplary embodiment, the timestamp type of the input data may include the time when the upstream storage of the input data provides the input data, i.e., the data recording time; the time when the first task or the second task receives the input data, i.e., the sub-processing time; or the event time corresponding to the input data, etc.

[0075] In one exemplary embodiment, the first task data following the first timestamp includes: first task data whose data processing time is later than the first timestamp; the second task data following the first timestamp includes: second task data whose data processing time is later than the first timestamp.

[0076] This exemplary embodiment defines the first task data and the second task data based on the processing timestamp of the task data at the output end. The first task data whose data processing time is later than the first timestamp is regarded as the first task data after the first timestamp; the second task data whose data processing time is later than the first timestamp is regarded as the second task data after the first timestamp.

[0077] In an exemplary embodiment, after switching from outputting first task data to outputting second task data, the task processing method may further include:

[0078] Continue running the first task;

[0079] If the watermark time corresponding to the first task is reached, and the data for the first task before the first timestamp has been completely output, then the first task will stop running.

[0080] To ensure that the first task can fully process data that meets its timestamp requirements, or to ensure that it obtains sufficient task data that meets the timestamp requirements (e.g., due to data reading delays, the first task may not have fully read or processed all data earlier than the first timestamp), stopping the first task too early would result in missing some data. Therefore, this exemplary embodiment can continue running the first task after switching the output of the first task data to the output of the second task data. When the watermark time corresponding to the first task is reached, and the first task data before the first timestamp has been completely output, the first task stops running. Here, the watermark time can be a configured watermark time, which in stream computing refers to a time indicator used to process event time. This exemplary embodiment can set the task switching condition to continuously run the first task until the configured watermark time expires and the output flow is cleared to 0. It can determine whether the first task meets the condition of all data being received, for example, through external and internal monitoring. When the configured watermark time expires and the output flow is cleared to 0, it can be determined that all the first task data has been received, and then the first task can be stopped to ensure that the first task is fully running.

[0081] In one exemplary embodiment, determining the first timestamp for switching tasks may include:

[0082] If the first task and the second task use time windows to process the input data, the first timestamp is determined based on the first time window used by the first task and the second time window used by the second task.

[0083] A time window is a mechanism for grouping and processing data streams. It can divide the data stream into finite, fixed-size, or dynamically sized segments according to the time dimension. The first time window corresponds to the input table of the first task, and the second time window corresponds to the input table of the second task. This exemplary embodiment can determine the first timestamp based on the first time window used by the first task and the second time window used by the second task. This exemplary embodiment can obtain the specific flow table input and the window time for subsequent use of the flow table through Flink's syntax parsing, and then perform different logical processing, specifically including segmented processing time windows (using the current running machine time as the window interval) and event time (using a field in the record as the time field for window interval).

[0084] In an exemplary embodiment, determining the first timestamp based on the first time window used by the first task and the second time window used by the second task may include:

[0085] Determine the common boundary time between the first time window and the second time window. The common boundary time is both the end time of the first time window and the start time of the second time window.

[0086] The first timestamp is determined based on the public boundary time.

[0087] The common boundary time between the first and second time windows refers to the time at the same boundary between them. For example, if the range of the first time window is 15 seconds and the range of the second time window is 10 seconds, and if the first and second time windows are rolling time windows, then from 0 seconds to 30 seconds, the first and second time windows share a common boundary time of 30 seconds. After 30 seconds, the first and second time windows can continue to acquire input data using 30 seconds as the starting time. This exemplary embodiment can determine the first timestamp based on the common boundary time, for example, by using the common boundary time as the first timestamp.

[0088] In this exemplary embodiment, time windows can be divided into scrolling time windows and sliding time windows based on their behavior type.

[0089] Using Flink tasks as an example, window splitting in Flink tasks typically starts at the top of the hour. Figure 5 A schematic diagram of a rolling time window is shown. For the rolling time window, this exemplary embodiment can first obtain the first time window and the second time window involved in the input table of the first task and the second task, and use the common boundary time of the first time window and the second time window closest to the current time as the first timestamp, such as the least common multiple of the times closest to the current time. Figure 6The diagram illustrates a first task and a second task in this exemplary embodiment. The first task 610 includes input 1 611, corresponding to a first time window of 10 seconds 612, and input 2 613, corresponding to a first time window of 20 seconds 614. When the first task 610 undergoes a logical change, the new second task 620 adds input 3, and the window time of the original inputs of the first task 610 changes. The second task 620 includes input 1 621, corresponding to a second time window of 30 seconds 622, input 2 623, corresponding to a second time window of 30 seconds 624, and input 3 625, corresponding to a second time window of 100 seconds 626. Inputs 1 611 and 1 621 can read input data from the same table, and inputs 2 613 and 2 623 can read input data from the same table, such as a click-through rate table. In practice, these correspond to the same Kafka topic, and the first and second tasks can subscribe to this topic using different groups. Assuming the task start-up and switching occurs at 10:00:00, calculating the actual switching time requires using the least common multiple of the time windows involved in the selected area 640: 10s, 20s, 30s, and 30s (60s). Adding this to the current time, we can calculate the switching time to be 10:01:00. At this time, the old first task 610 has just output all the old window data, and the new second task 620 has just finished accumulating and discarding the data from the previous 30s. Therefore, we only need to send a command to the first task 610 to stop input at 10:01, and to allow the second task 620 to start processing at 10:01. That is, data after 10:05 will be output by the second task. In the above process, 10:00 can be considered the request time, i.e., the time when the second task starts and completes, while the actual logical switch is at 10:01.

[0090] It's important to note that although the second task adds input 3, since the only common inputs for both tasks are 1 and 2, the time window for input 3 is not included in the calculation of the least common multiple (LCM). Only the common time windows for inputs 1 and 2 are used for LCM calculation. For example, when counting real-time visitors, the logic of the first task is to calculate based on the user's page stay time (5-minute window), while the logic of the second task is to calculate based on 10 minutes. Both tasks read the page exposure table, so the LCM, i.e., 10 minutes, can be used as the first timestamp. However, if the logic of the first task is based on the user's page stay time (5 minutes, exposure table), and the logic of the second task is based on the user's page stay time (10 minutes, exposure table) plus one screen tap (click event table), then only the common multiple of the exposure tables needs to be calculated. This is because the first task does not use the click event table for logical operations and does not need to calculate the LCM. The calculation of the LCM is determined by the common tables requested by both tasks and the windows in their calculation logic.

[0091] Figure 7 A schematic diagram of a sliding time window is shown. Although it differs from a scrolling time window in the way the window moves, it is essentially the same. Therefore, the first timestamp can also be calculated based on the end of the previous window.

[0092] In this exemplary embodiment, the window computation time can include two methods: processing time and recording time. In Flink's logical semantics, processing time represents the runtime of the running machine. However, due to slight differences in runtime between different machines and varying read rates, the first and second tasks may actually experience duplicate or lost data. Two processing schemes can be adopted here:

[0093] First, the semantics of processing time in Flink may not be strictly consistent, which may lead to data loss. So, if we do not process it directly and rely on the timestamps of different machines for processing, the current switching test data loss rate is less than 1 / 10,000.

[0094] Secondly, if strict execution is required, the upstream storage time can be used as the processing time. For example, in Kafka, Kafka has a record time (the time when the Kafka message queue receives the message). Then, replace the built-in ProcessTime method in Flink and use this time as the processing time. Since the upstream system is the same, it can ensure that the data read and discarded by the two tasks are completely matched.

[0095] In one exemplary embodiment, if there is no time window, it can be understood as starting two tasks, such as starting two Flink SQL tasks, which can be executed simultaneously:

[0096] Task A: insertinto table_b from select*from table_a where timestamp<'2025-01-01'

[0097] Task B: insert into table_b from select*from table_a where timestamp>'2025-01-01'

[0098] A concrete understanding is that, without a window, two tasks run simultaneously. Suppose the first task reads data with timestamps before January 1st, discarding any other data it reads. The second task reads and processes data after January 1st, discarding the rest. When January 1st arrives, it's equivalent to the instant the first task stops reading, and the second task immediately resumes execution. There is no interval between these two points because the second task was also reading data, but since it didn't meet the condition of being greater than the first timestamp "2025-01-01", it only discarded the data without writing it down.

[0099] In the absence of a time window, the first timestamp can be determined directly by technical personnel or calculated by the system based on experience or data. This disclosure does not impose any specific limitations on this.

[0100] In this exemplary embodiment, data can be filtered at the input or output end. When data filtering is performed at the reading / input end, input data that does not meet the timestamp requirements can be discarded. When data filtering is performed at the output end, the following situation can be considered: actual processing time (not recording time) + no window. In this case, when performing data filtering at the output end, the difference in processing time of each machine needs to be taken into account. Control filtering at the output end can reduce the error to a low level.

[0101] In one exemplary embodiment, starting the second task based on a second timestamp earlier than the first timestamp may include:

[0102] The second task reads input data that is later than the second timestamp.

[0103] In this exemplary embodiment, running the second task refers to starting the process of reading input data. Specifically, the data reading can be achieved by reading input data that is later than the second timestamp through the second task. Reading input data can include two scenarios: reading without processing (e.g., discarding the data after reading), and reading and processing the data (e.g., performing calculations on the read input data). For the task data obtained from reading and processing the data, the output to downstream can be determined based on the timestamp. For example, second task data generated earlier than the first timestamp is not output downstream, while second task data generated later than the first timestamp is output downstream.

[0104] In one exemplary embodiment, the task processing method described above may further include:

[0105] Given a first timestamp, a first switching instruction is sent to the first task and a second switching instruction is sent to the second task. Both the first switching instruction and the second switching instruction include the first timestamp.

[0106] In response to receiving the first consent information from the first task and the second consent information from the second task, a third switching instruction is sent to the first task and a fourth switching instruction is sent to the second task. The third switching instruction is used to instruct the first task to switch from outputting first task data to not outputting first task data according to the first timestamp. The fourth switching instruction is used to instruct the second task to switch from not outputting second task data to outputting second task data according to the first timestamp.

[0107] The first and second switching instructions may include a first timestamp. In this exemplary embodiment, after the first and second tasks are prepared, the server can send a first switching instruction to the first task and a second switching instruction to the second task, so that the first and second tasks receive information with the determined first timestamp. After receiving the switching instructions, the first and second tasks can return information to the server indicating whether they agree or disagree, depending on their own situation. When the server receives the first agreement information from the first task and the second agreement information from the second task, it can continue to execute the steps of sending a third switching instruction to the first task and a fourth switching instruction to the second task. After receiving the third switching instruction, the first task can switch from outputting first task data to not outputting first task data based on the first timestamp. After receiving the fourth switching instruction, the second task can switch from not outputting second task data to outputting second task data based on the first timestamp. The switching of task data between the first and second tasks is achieved in this way.

[0108] Figure 8This illustration shows an architectural diagram of a task processing environment in this exemplary embodiment, which may include: a task processing platform 810, such as a server for task processing, capable of controlling the lifecycle of a first task 820 and a second task 830. It can seamlessly switch between tasks, launch a new second task, manage task status, and issue various time-operation instruction sets. Currently running task is the first task 820. After receiving a seamless switching instruction, the server launches a new second task 830. Simultaneously, the launched first and second tasks send running data, including running topology, concurrency, and running status, to the server. While waiting for instructions from the server, the first task 820 can listen to server information through the first listening module 840, and the second task 830 can also listen to server information through the second listening module 850 when it starts. The first listening module 840 and the second listening module 850 can connect to the task processing platform 810 through a heartbeat. Listening can be implemented through components such as ZooKeeper, HTTP, and Kafka. In this exemplary embodiment, listening can be performed through HTTP. The first task 820 can process the information listened to through the first event processing module 860, such as a pause event, and the second task 830 can process the information listened to through the second event processing module 870, such as a start event.

[0109] In this exemplary embodiment, once the first task A and the second task B are ready, the server P can begin issuing relevant instructions. Figure 9 The diagram illustrates an interactive flowchart of a task processing method in this exemplary embodiment, which may specifically include the following steps:

[0110] Step S902: Run the first task;

[0111] Step S904: If the first task has already been run, start the second task;

[0112] Step S906: The server sends a first timestamp to the first task and a first timestamp to the second task.

[0113] In step S908, the server receives the return data from the first task and the second task. The return data needs to include the first timestamp agreed upon by both the first task and the second task. The main criteria for judgment here can be whether the filtering can be completed within the specified time, such as whether the time exceeds the current time, whether the event time can be accurately extracted and converted, etc.

[0114] Step S910: After the server determines that the first task and the second task agree based on the returned data, it sends a confirmation notification to the first task and the second task.

[0115] The first and second tasks are now waiting for the first timestamp;

[0116] Step S912, the first task is to prepare to filter data that exceeds the first timestamp;

[0117] Step S914, the second task is to go from completely discarding data to starting preparations to read data beyond the first timestamp;

[0118] Step S916: When the first timestamp is reached or exceeded, the first task and the second task return information to the server regarding whether the switch was successful or whether there was any abnormality.

[0119] In step S918, the server uses the information returned by the first task and the second task, as well as the monitoring information of external traffic, to comprehensively determine the stopping time of the first task, and stops the first task according to the stopping time; in addition, the first task can be converted into the second task.

[0120] Figure 10 A flowchart of another task processing method in this exemplary embodiment is shown, which can achieve a smooth transition from a first task to a second task, and may specifically include the following steps:

[0121] Step S1002: Run the first task;

[0122] Step S1004: Start the second task;

[0123] If the first timestamp is determined, step S1006 is executed to send a first switching instruction to the first task; the first switching instruction includes the first timestamp and prepares for switching based on the first timestamp.

[0124] Step S1008: Send a second switching instruction to the second task; the second switching instruction includes a first timestamp, and prepare for switching based on the first timestamp;

[0125] Step S1010: The first task waits for a switch based on the first timestamp;

[0126] Step S1012: The second task waits for a switch based on the first timestamp;

[0127] Step S1014: If input data later than the first timestamp is read through the first task, then discard the input data;

[0128] Step S1016: If input data earlier than the first timestamp is read through the second task, then the input data is discarded;

[0129] Step S1018: The first task continues to run until the configured watermark time expires and the output flow is cleared to 0, then the flow stops.

[0130] Step S1020: Switch timing sequence and traffic;

[0131] Step S1022: Stop the first task;

[0132] Step S1024: Run the second task normally.

[0133] It should be noted that if the first task fails to detect a switch while preparing for or waiting for a switch, it can return to step S1002 and continue running the first task. If the second task fails to detect a switch while preparing for or waiting for a switch, or fails to switch during traffic switching after the timing sequence, it will return a failure message. Additionally, in step S1024, after the second task runs normally, if there is a further need, the second task can be used as the first task, and the execution can return to step S1002 to perform a switch for a new task.

[0134] In this exemplary embodiment, a single Flink task (either the first or second task) is used as an example to illustrate how data filtering is implemented. Flink can receive instructions from the task processing platform through polling threads and broadcast them to all downstream operators via its Broatcast capability. Upon receiving the broadcast, operators can modify static variables to control the flow and perform filtering operations, such as... Figure 11 As shown in the exemplary embodiment, this model can add a BroadCast operator, such as BroadCastSource 1110, to periodically poll the interface of the task processing platform 1120 and broadcast information such as the "start read timestamp" and "stop read timestamp" required for the current task to downstream operators, such as DeserializerProcessor 1130. The task processing platform and BroadCastSource can interact via the HTTP protocol, and BroadCastSource can broadcast information to DeserializerProcessor via the Broadcast capability. Additionally, this exemplary embodiment can use KafkaSource 1140 as the source for reading data from Kafka, retrieving data from Kafka and passing it to DeserializerProcessor 1130 in Record form.

[0135] Specifically, after receiving the "stop reading timestamp," the first task can return a confirmation message to the task processing platform and stop sending data after consuming the corresponding timestamp. After receiving the "start reading timestamp," the second task can return a confirmation message to the task processing platform and begin sending data after consuming the corresponding timestamp. This can include the following steps: operator information reporting; when initializing the DeserializationFunction operator, requesting the task processing platform to report the name of the current operator, concurrency information, etc., as a reference number for the task processing platform to obtain subsequent submission requests; operator subTask registration, open... When the DeserializationFunction operator is executed, it requests the platform to register the current operator's subTask. The task processing platform can determine whether the current task has started normally based on the registration information of all subTasks. During the first-stage switch, BroadCastSource periodically polls the platform. After obtaining the first-stage switch information, it broadcasts it to the downstream DeserializerProcessor. At this time, the subTasks of each DeserializerProcessor are in the PRE_START / PRE_STOP (pre-start / pre-stop) state, although the startTimestamp and e are modified. The `ndTimestamp` (end timestamp) is used initially, but it is not effective. In the second phase of the switchover, `BroadCastSource` periodically polls the platform. After receiving the switchover information from the first phase, it broadcasts it to the downstream `DeserializerProcessor`. At this point, the subtasks of each `DeserializerProcessor` are in the `COMMIT` state, and `startTimestamp` and `endTimestamp` become effective. The switchover can be aborted or abandoned if problems arise. Upon completion, each subtask reports a successful switchover. Finally, the server controls and manages the overall switchover logic, such as whether to stop streaming, rollback, and whether the switchover was successful. During this process, if an exception occurs, the program will send a recovery command to the first task and stop the second task from running.

[0136] Exemplary device

[0137] Exemplary embodiments of this disclosure also provide a task processing apparatus. (See reference...) Figure 12As shown, the task processing device 1200 may include the following program modules: a first task running module 1210, used to run a first task, generate and output first task data; a second task starting module 1220, used to start a second task; the first task and the second task are tasks under the same business; a first timestamp determination module 1230, used to determine a first timestamp for switching tasks; a second task running module 1240, used to start running the second task according to a second timestamp earlier than the first timestamp; or start running the second task before determining the first timestamp; a task data switching module 1250, used to switch the output of first task data to the output of second task data according to the first timestamp; the second task data is the data generated by running the second task.

[0138] In one embodiment, the task data switching module 1250 includes: a task processing data unit, used to output second task data after the first timestamp, and not to output first task data after the first timestamp.

[0139] In one implementation, the first task data following the first timestamp includes: first task data generated by processing input data later than the first timestamp through the first task; the second task data following the first timestamp includes: second task data generated by processing input data later than the first timestamp through the second task.

[0140] In one embodiment, the task processing device is further configured with at least one of the following modules: a first data discarding module, configured to discard input data if input data later than a first timestamp is read through a first task; and a second data discarding module, configured to discard input data if input data earlier than the first timestamp is read through a second task.

[0141] In one implementation, the timestamp of the input data includes: the time when the upstream storage of the input data provides the input data, or the time when the first task or the second task receives the input data, or the event time corresponding to the input data.

[0142] In one implementation, the first task data following the first timestamp includes: first task data whose data processing time is later than the first timestamp; the second task data following the first timestamp includes: second task data whose data processing time is later than the first timestamp.

[0143] In one embodiment, after switching the output of the first task data to the output of the second task data, the task processing device further includes: a task running module for continuing to run the first task; and a task stopping module for stopping the operation of the first task if the watermark time corresponding to the first task is reached and the first task data before the first timestamp has been completely output.

[0144] In one embodiment, the first timestamp determination module 1230 includes: a timestamp determination unit, configured to determine a first timestamp based on the first time window used by the first task and the second time window used by the second task if the first task and the second task use time windows to process input data.

[0145] In one embodiment, the timestamp determination unit includes: a window calculation subunit, used to determine the common boundary time of a first time window and a second time window, wherein the common boundary time is both the end time of a first time window and the start time of a second time window; and to determine the first timestamp based on the common boundary time.

[0146] In one embodiment, the second task execution module 1240 includes: a data reading unit, used to read input data later than the second timestamp through the second task.

[0147] In one embodiment, the task processing device further includes: an instruction sending module, configured to send a first switching instruction to a first task and a second switching instruction to a second task, wherein both the first switching instruction and the second switching instruction include the first timestamp; and an instruction receiving module, configured to send a third switching instruction to the first task and a fourth switching instruction to the second task in response to receiving first consent information from the first task and second consent information from the second task; the third switching instruction instructs the first task to switch from outputting first task data to not outputting first task data according to the first timestamp; and the fourth switching instruction instructs the second task to switch from not outputting second task data to outputting second task data according to the first timestamp.

[0148] Furthermore, other specific details of the embodiments of this disclosure have been described in detail in the embodiments of the above methods, and will not be repeated here.

[0149] Exemplary storage media

[0150] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the methods described above. The methods can be implemented by a program product, such as a portable compact disc read-only memory (CD-ROM) including program code, and can run on a device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0151] The program product may employ any combination of one or more readable media. A readable media may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0152] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0153] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RE, etc., or any suitable combination thereof.

[0154] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0155] Exemplary electronic devices

[0156] Exemplary embodiments of this disclosure also provide an electronic device, which may be Figure 1 Any device in the present disclosure. The electronic device includes a processor and a memory for storing executable instructions of the processor. The processor is configured to perform the methods described above by executing the executable instructions.

[0157] refer to Figure 13 An electronic device according to an exemplary embodiment of the present disclosure will be described. Figure 13 The electronic device 1300 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0158] like Figure 13 As shown, the electronic device 1300 is presented in the form of a general-purpose computing device. The components of the electronic device 1300 may include, but are not limited to: at least one processing unit 1310, at least one storage unit 1320, and a bus 1330 connecting different system components (including storage unit 1320 and processing unit 1310).

[0159] The storage unit stores program code, which can be executed by the processing unit 1310 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1310 can perform actions such as... Figure 2 The methods and steps shown are as follows.

[0160] Storage unit 1320 may include volatile storage units, such as random access memory (RAM) unit 1321 and / or cache memory unit 1322, and may further include read-only memory unit (ROM) unit 1323.

[0161] Storage unit 1320 may also include a program / utility 1324 having a set (at least one) program module 1325, such program module 1325 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0162] Bus 1330 may include a data bus, an address bus, and a control bus.

[0163] Electronic device 1300 can also communicate with one or more external devices 1400 (e.g., keyboard, pointing device, Bluetooth device, etc.) via input / output (I / O) interface 1340. Electronic device 1300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 1350. As shown, network adapter 1350 communicates with other modules of electronic device 1300 via bus 1330. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1300, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0164] It should be noted that although several modules or sub-modules of the apparatus have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0165] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0166] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A task processing method, characterized in that, include: Run the first task, generate and output the first task data; Initiate the second task; The first task and the second task are tasks under the same business; Determine the first timestamp used for switching tasks; The second task may be started based on a second timestamp earlier than the first timestamp; or the second task may be started before the first timestamp is determined. Based on the first timestamp, switch the output of the first task data to the output of the second task data; The second task data is the data generated by running the second task.

2. The method according to claim 1, characterized in that, The step of switching the output of the first task data to the output of the second task data based on the first timestamp includes: Output the data of the second task after the first timestamp, but do not output the data of the first task after the first timestamp.

3. The method according to claim 2, characterized in that, The first task data following the first timestamp includes: first task data generated by processing input data later than the first timestamp through the first task; the second task data following the first timestamp includes: second task data generated by processing input data later than the first timestamp through the second task.

4. The method according to claim 3, characterized in that, The method further includes at least one of the following steps: If input data later than the first timestamp is read through the first task, then the input data is discarded; If input data earlier than the first timestamp is read through the second task, then the input data is discarded.

5. The method according to claim 3 or 4, characterized in that, The timestamp of the input data includes: the time when the upstream provider of the input data stored the input data, or the time when the first task or the second task received the input data, or the event time corresponding to the input data.

6. The method according to claim 2, characterized in that, The first task data following the first timestamp includes: first task data whose data processing time is later than the first timestamp; the second task data following the first timestamp includes: second task data whose data processing time is later than the first timestamp.

7. The method according to claim 2, characterized in that, After switching the output of the first task data to the output of the second task data, the method further includes: Continue running the first task; If the watermark time corresponding to the first task is reached, and the data of the first task before the first timestamp has been completely output, then the first task will stop running.

8. A task processing device, characterized in that, include: The first task execution module is used to run the first task, generate and output the first task data; The second task startup module is used to start the second task. The first task and the second task are tasks under the same business; The first timestamp determination module is used to determine the first timestamp used for switching tasks. The second task execution module is used to start running the second task based on a second timestamp that is earlier than the first timestamp; Alternatively, the second task may begin running before the first timestamp is determined; The task data switching module is used to switch the output of the first task data to the output of the second task data according to the first timestamp; The second task data is the data generated by running the second task.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1 to 7 by executing the executable instructions.

Images