Data flow control methods, devices, electronic equipment and storage media

Abstract

This disclosure relates to a data flow control method, apparatus, electronic device, and storage medium. The method includes: acquiring a current data processing scenario; the data processing scenario includes at least one of the following: a first scenario characterized by a data input volume exceeding the data processing volume of a concurrent system; a second scenario characterized by inconsistent data processing progress among multiple concurrent tasks in a concurrent system; and a third scenario characterized by historical data accumulation in the data source of the concurrent system; and determining a flow control strategy corresponding to the current data processing scenario. This disclosure allows for the adoption of corresponding flow control strategies for different data processing scenarios. Based on the refined data source control technology in each flow control strategy, it effectively solves the problem of unsatisfactory data flow control effects in concurrent systems, ensures task flexibility and availability, guarantees system stability in different scenarios, and further expands the application scenarios of concurrent systems.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data flow control method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, when processing data through concurrent systems such as the Flink real-time computing engine, the system becomes extremely unstable when faced with a large influx of data, such as a sudden increase in data volume or a backlog of data consumption.

[0003] Traditional methods typically rely on their own backpressure mechanisms to cope with a surge in data. If the downstream cannot process the data sent from the upstream in a timely manner, it will block the upstream from continuing to send data. However, due to the inconsistent speed at which each concurrent task consumes data from the data source, data accumulation may still occur under this mechanism, leading to instability such as the concurrent system freezing.

[0004] Therefore, some related technologies cannot effectively solve the problem of unsatisfactory data flow control in concurrent systems. Summary of the Invention

[0005] This disclosure provides a data flow control method, apparatus, electronic device, and storage medium to at least solve the problem in related technologies where data flow control in concurrent systems is not effectively implemented and results in unsatisfactory performance. The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a data flow control method is provided, comprising:

[0007] Obtain the current data processing scenario; the data processing scenario includes at least one of the following: a first scenario characterized by a data input volume greater than the data processing volume of the concurrent system, a second scenario characterized by inconsistent data processing progress of multiple concurrent tasks in the concurrent system, and a third scenario characterized by historical data accumulation in the data source of the concurrent system;

[0008] Based on the current data processing scenario, determine the traffic control strategy corresponding to the current data processing scenario.

[0009] In one possible implementation, the first scenario corresponds to a first control strategy, the second scenario corresponds to a second control strategy, and the third scenario corresponds to a third control strategy. The step of determining the traffic control strategy corresponding to the current data processing scenario includes:

[0010] If the current data processing scenario is the first scenario, the first control strategy is used as the flow control strategy corresponding to the current data processing scenario. When the flow control strategy is the first control strategy, the target value of the data processing volume per unit time is determined based on the amount of data currently processed by each concurrent task, and the data input volume of each concurrent task is adjusted based on the target value of the data processing volume per unit time.

[0011] If the current data processing scenario is the second scenario, the second control strategy is used as the traffic control strategy corresponding to the current data processing scenario. When the traffic control strategy is the second control strategy, the data processing progress of each concurrent task in the next cycle is controlled based on the current cycle data processing progress of each concurrent task.

[0012] If the current data processing scenario is the third scenario, the third control strategy is used as the traffic control strategy corresponding to the current data processing scenario. When the traffic control strategy is the third control strategy, each concurrent task is controlled to process the historical data of the corresponding data source according to the preset data consumption ratio, and the preset data consumption ratio is greater than 1.

[0013] In one possible implementation, when the flow control strategy is a first control strategy, determining a target value for the amount of data processed per unit time based on the amount of data currently being processed by each concurrent task, and adjusting the data input amount of each concurrent task based on the target value for the amount of data processed per unit time, includes:

[0014] Based on the total amount of data processed per unit time of the concurrent system and the total number of partitions contained in the data source corresponding to the concurrent system, the amount of data processed per unit time for a single partition is determined; the partition is used to temporarily store data and input data to the corresponding concurrent task.

[0015] Based on the amount of data processed per unit time corresponding to a single partition and the number of partitions corresponding to each concurrent task, determine the target value of the amount of data processed per unit time corresponding to each concurrent task;

[0016] According to the target value of data processing volume per unit time for each concurrent task, control the data input flow from the partition corresponding to each concurrent task to the concurrent task.

[0017] In one possible implementation, after the step of determining the target value of the data processing volume per unit time for each concurrent task based on the data processing volume per unit time for the single partition and the number of partitions for each concurrent task, the method further includes:

[0018] When an update is detected in the total number of partitions and / or the number of partitions corresponding to each concurrent task, the target value of the data processing volume per unit time corresponding to each concurrent task is re-determined based on the updated total number of partitions and / or the updated number of partitions, and the data input flow from the partitions corresponding to each concurrent task to the concurrent task is controlled according to the re-determined target value of the data processing volume per unit time.

[0019] In one possible implementation, when the flow control strategy is the second control strategy, controlling the data processing progress of each concurrent task in the next cycle based on the data processing progress of each concurrent task in the current cycle includes:

[0020] Based on the data processing progress of the multiple concurrent tasks in the current cycle, determine the target data processing progress for the next cycle;

[0021] In the next cycle, if it is detected that the data processing progress of a concurrent task exceeds the target data processing progress, the input of data traffic to the concurrent task will be suspended.

[0022] In one possible implementation, the data input to each concurrent task corresponds to a timestamp; determining the target data processing progress for the next period based on the data processing progress of the multiple concurrent tasks in the current period includes:

[0023] Get the maximum timestamp corresponding to the data processed by each concurrent task in the current period;

[0024] Based on the minimum value among the maximum timestamps corresponding to the multiple concurrent tasks, and the preset allowable time difference for processing progress, the target timestamp for the next cycle is determined as the target data processing progress.

[0025] The step of pausing data flow to a concurrent task when the data processing progress of a concurrent task is detected to exceed the target data processing progress includes:

[0026] When it is detected that the timestamp corresponding to the data being processed by a concurrent task exceeds the target timestamp, the input of data traffic to that concurrent task is paused.

[0027] In one possible implementation, the data input to each concurrent task corresponds to a timestamp; when the flow control strategy is a third control strategy, each concurrent task is controlled to process the historical data of the corresponding data source according to a preset data consumption ratio, including:

[0028] Obtain the actual data consumption ratio of each concurrent task for the historical data of the corresponding data source; the actual data consumption ratio represents the maximum time span between the timestamps corresponding to the historical data processed by the concurrent task in one unit of time.

[0029] When the actual data consumption rate of a concurrent task is greater than the preset data consumption rate, the data inflow corresponding to that concurrent task is reduced.

[0030] In one possible implementation, obtaining the actual data consumption ratio of each concurrent task to the historical data of the corresponding data source includes:

[0031] For each concurrent task, obtain the current first system time and the second system time when the concurrent system starts, and obtain the data processing time of the concurrent system based on the first system time and the second system time;

[0032] Obtain the first timestamp of the historical data to be processed at present, and the second timestamp of the historical data to be processed when the concurrent system starts. Based on the first timestamp and the second timestamp, obtain the time span corresponding to the historical data processed within the data processing time.

[0033] Based on the data processing duration and the time span, the actual data consumption rate of the concurrent task is determined.

[0034] According to a second aspect of the present disclosure, a data flow control device is provided, comprising:

[0035] The data processing scenario acquisition unit is configured to acquire the current data processing scenario; the data processing scenario includes at least one of the following: a first scenario representing that the data input volume is greater than the data processing volume of the concurrent system, a second scenario representing that the data processing progress of multiple concurrent tasks in the concurrent system is inconsistent, and a third scenario representing that the data source of the concurrent system has accumulated historical data.

[0036] The flow control strategy determination unit is configured to determine a flow control strategy corresponding to the current data processing scenario based on the current data processing scenario.

[0037] In one possible implementation, the first scenario corresponds to a first control strategy, the second scenario corresponds to a second control strategy, and the third scenario corresponds to a third control strategy. The flow control strategy determination unit is specifically configured to: if the current data processing scenario is the first scenario, use the first control strategy as the flow control strategy corresponding to the current data processing scenario; when the flow control strategy is the first control strategy, determine a target value for the data processing volume per unit time based on the current data volume processed by each concurrent task, and adjust the data input volume of each concurrent task based on the target value for the data processing volume per unit time; if the current data processing scenario is the first control strategy, determine a target value for the data processing volume per unit time based on the current data volume processed by each concurrent task; if the current data processing scenario is the first control strategy, determine a target value for the data processing volume per unit time based on the current data processing volume per concurrent task. If the scenario is the second scenario, the second control strategy is used as the traffic control strategy corresponding to the current data processing scenario. When the traffic control strategy is the second control strategy, the data processing progress of each concurrent task in the next cycle is controlled based on the data processing progress of each concurrent task in the current cycle. If the current data processing scenario is the third scenario, the third control strategy is used as the traffic control strategy corresponding to the current data processing scenario. When the traffic control strategy is the third control strategy, each concurrent task is controlled to process the historical data of the corresponding data source according to a preset data consumption ratio, where the preset data consumption ratio is greater than 1.

[0038] In one possible implementation, the flow control strategy determination unit is specifically configured to: determine the amount of data processed per unit time for a single partition based on the total amount of data processed per unit time of the concurrent system and the total number of partitions contained in the data source corresponding to the concurrent system; the partition is used to temporarily store data and input data to the corresponding concurrent task; based on the amount of data processed per unit time for a single partition and the number of partitions corresponding to each concurrent task, determine the target value of the amount of data processed per unit time for each concurrent task; and control the data input flow from the partitions corresponding to each concurrent task to the concurrent task according to the target value of the amount of data processed per unit time for each concurrent task.

[0039] In one possible implementation, the device further includes:

[0040] The re-determination unit is specifically configured to, when an update is detected in the total number of partitions and / or the number of partitions corresponding to each concurrent task, re-determine the target value of the data processing volume per unit time for each concurrent task based on the updated total number of partitions and / or the updated number of partitions, and control the data input flow from the partitions corresponding to each concurrent task to the concurrent task according to the re-determined target value of the data processing volume per unit time.

[0041] In one possible implementation, the flow control strategy determination unit is specifically configured to determine the target data processing progress for the next period based on the data processing progress of the multiple concurrent tasks in the current period; and in the next period, when it is detected that the data processing progress of a certain concurrent task exceeds the target data processing progress, to suspend the input of data flow to the certain concurrent task.

[0042] In one possible implementation, the data input to each concurrent task corresponds to a timestamp. The flow control strategy determination unit is specifically configured to: obtain the maximum timestamp corresponding to the data processed by each concurrent task in the current period; determine the target timestamp for the next period based on the minimum value among the maximum timestamps corresponding to the multiple concurrent tasks and a preset allowable time difference for processing progress, as the target data processing progress; and suspend the input of data flow to the concurrent task when it is detected that the data processing progress of a certain concurrent task exceeds the target data processing progress, including: suspending the input of data flow to the concurrent task when it is detected that the timestamp corresponding to the data processed by a certain concurrent task exceeds the target timestamp.

[0043] In one possible implementation, the data input to each concurrent task corresponds to a timestamp. The flow control strategy determination unit is specifically configured to obtain the actual data consumption ratio of each concurrent task for the historical data of the corresponding data source. The actual data consumption ratio represents the maximum time span between the timestamps corresponding to the historical data processed by the concurrent task within one unit of time. When the actual data consumption ratio of a certain concurrent task is greater than the preset data consumption ratio, the data inflow corresponding to that concurrent task is reduced.

[0044] In one possible implementation, the flow control strategy determination unit is specifically configured to perform the following for each concurrent task: obtain the current first system time of the concurrent system and the second system time when the concurrent system starts; obtain the data processing duration of the concurrent system based on the first system time and the second system time; obtain the first timestamp of the historical data to be processed currently and the second timestamp of the historical data to be processed when the concurrent system starts; obtain the time span corresponding to the historical data processed within the data processing duration based on the first timestamp and the second timestamp; and determine the actual data consumption rate of the concurrent task based on the data processing duration and the time span.

[0045] According to a third aspect of the present disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the data flow control method as described in the first aspect or any possible implementation thereof.

[0046] According to a fourth aspect of the present disclosure, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the data flow control method as described in the first aspect or any possible implementation thereof.

[0047] According to a fifth aspect of the present disclosure, a computer program product is provided, the program product including a computer program stored in a readable storage medium, wherein at least one processor of a device reads from the readable storage medium and executes the computer program, causing the device to perform the data flow control method described in any embodiment of the first aspect.

[0048] The technical solution provided by the embodiments of this disclosure brings at least the following beneficial effects: By acquiring the current data processing scenario, which includes at least one of the following: a first scenario representing a data input volume greater than the data processing volume of the concurrent system; a second scenario representing inconsistent data processing progress among multiple concurrent tasks in the concurrent system; and a third scenario representing historical data accumulation in the data source of the concurrent system, a flow control strategy corresponding to the current data processing scenario is determined. In this way, corresponding flow control strategies can be adopted for different data processing scenarios. Based on the refined data source control technology in each flow control strategy, the problem of unsatisfactory data flow control effect in concurrent systems is effectively solved, ensuring task flexibility and availability, guaranteeing system stability in different scenarios, and further expanding the application scenarios of concurrent systems.

[0049] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0051] Figure 1 This is a flowchart illustrating a data flow control method according to an exemplary embodiment.

[0052] Figure 2a This is a schematic diagram illustrating a processing flow in a first scenario according to an exemplary embodiment.

[0053] Figure 2b This is a schematic diagram illustrating a processing flow in a second scenario according to an exemplary embodiment.

[0054] Figure 2c This is a schematic diagram illustrating a processing flow in a third scenario according to an exemplary embodiment.

[0055] Figure 3 This is a flowchart illustrating another data flow control method according to an exemplary embodiment.

[0056] Figure 4 This is a block diagram illustrating a data flow control device according to an exemplary embodiment.

[0057] Figure 5 This is an internal structural diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0058] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0059] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this disclosure.

[0060] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0061] Figure 1 This is a flowchart illustrating a data flow control method according to an exemplary embodiment. This method can be used in computer devices such as servers. Figure 1 As shown, the method includes the following steps.

[0062] In step S110, the current data processing scenario is obtained; the data processing scenario includes at least one of the following: a first scenario that represents the amount of data input being greater than the amount of data processing in the concurrent system, a second scenario that represents the inconsistent data processing progress of multiple concurrent tasks in the concurrent system, and a third scenario that represents the accumulation of historical data in the data source of the concurrent system.

[0063] The concurrent system can be the Flink real-time computing engine, which can perform computation jobs on data input from multiple data sources. A concurrent system can contain multiple concurrent tasks, each representing a concurrent operation within a job. For example, in a distributed scenario, multiple tasks can process data concurrently. The data processing scenario characterizes the data flow of the concurrent system during job execution.

[0064] In practical applications, when using the Flink real-time computing engine to perform jobs, by obtaining the current data processing scenario, the corresponding traffic control strategy can be selected based on the current data processing scenario, so as to further control the data traffic for different data processing scenarios based on the corresponding traffic control strategy.

[0065] Specifically, data processing scenarios can include a first scenario, where the amount of data input exceeds the amount of data processed by the concurrent system, such as a high-volume scenario with a sudden surge in data; a second scenario, where the data processing progress of multiple concurrent tasks in the concurrent system is inconsistent; and a third scenario, where historical data accumulates in the data source of the concurrent system, such as a scenario with historical data accumulating to be consumed.

[0066] In step S120, a flow control strategy corresponding to the current data processing scenario is determined based on the current data processing scenario.

[0067] After obtaining the current data processing scenario, the corresponding traffic control strategy can be determined from multiple preset traffic control strategies. For example, corresponding traffic control strategies can be preset for different data processing scenarios, and then the corresponding traffic control strategy can be selected according to the specific data processing scenario.

[0068] In one example, the first scenario can correspond to the first control strategy, the second scenario can correspond to the second control strategy, and the third scenario can correspond to the third control strategy.

[0069] The aforementioned data flow control method identifies the current data processing scenario, which includes at least one of the following: a first scenario indicating that the input data volume exceeds the processing volume of the concurrent system; a second scenario indicating that the processing progress of multiple concurrent tasks in the concurrent system is inconsistent; and a third scenario indicating that historical data is accumulating in the data source of the concurrent system. Based on this scenario, a flow control strategy corresponding to the current data processing scenario is determined. This allows for the application of corresponding flow control strategies for different data processing scenarios. Based on the refined data source control technology within each flow control strategy, the method effectively solves the problem of unsatisfactory data flow control performance in concurrent systems, ensuring task flexibility and availability, guaranteeing system stability in different scenarios, and further expanding the application scenarios of concurrent systems.

[0070] In an exemplary embodiment, a first scenario corresponds to a first control strategy, a second scenario corresponds to a second control strategy, and a third scenario corresponds to a third control strategy. Determining a flow control strategy corresponding to the current data processing scenario includes: if the current data processing scenario is the first scenario, using the first control strategy as the flow control strategy corresponding to the current data processing scenario; when the flow control strategy is the first control strategy, determining a target value for the amount of data processed per unit time based on the amount of data currently processed by each concurrent task, and adjusting the amount of data input for each concurrent task based on the target value for the amount of data processed per unit time.

[0071] If the current data processing scenario is the second scenario, the second control strategy will be used as the flow control strategy corresponding to the current data processing scenario. When the flow control strategy is the second control strategy, the data processing progress of each concurrent task in the next cycle will be controlled based on the current cycle data processing progress of each concurrent task.

[0072] If the current data processing scenario is the third scenario, the third control strategy will be used as the traffic control strategy corresponding to the current data processing scenario. When the traffic control strategy is the third control strategy, each concurrent task will be controlled to process the historical data of the corresponding data source according to the preset data consumption ratio, which is greater than 1.

[0073] In specific implementation, for the first scenario, a first control strategy can be adopted, which determines the target value of data processing volume per unit time based on the current data volume processed by each concurrent task, and then adjusts the data input volume of each concurrent task based on the target value of data processing volume per unit time. For the second scenario, a second control strategy can be adopted, which controls the data processing progress of each concurrent task in the next cycle based on the data processing progress of each concurrent task in the current cycle. For the third scenario, a third control strategy can be adopted, which controls each concurrent task to process the historical data of the corresponding data source according to a preset data consumption ratio greater than 1, thereby improving the availability of the cluster, ensuring the flexible availability of tasks in the scenario of holding large-scale events, and achieving a smooth and controllable effect when consuming accumulated historical data in complex large-scale operations.

[0074] The technical solution of this embodiment, by using a first control strategy as the flow control strategy corresponding to the current data processing scenario (first scenario), determines the target value of data processing volume per unit time based on the current data volume processed by each concurrent task, and adjusts the data input volume of each concurrent task based on the target value of data processing volume per unit time. If the current data processing scenario is a second scenario, the second control strategy is used as the flow control strategy corresponding to the current data processing scenario. When the flow control strategy is the second control strategy, the data processing progress of each concurrent task in the next cycle is controlled based on the current cycle data processing progress of each concurrent task. If the current data processing scenario is a third scenario, the third control strategy is used as the flow control strategy corresponding to the current data processing scenario. When the flow control strategy is the third control strategy, each concurrent task is controlled to process the historical data of the corresponding data source according to a preset data consumption ratio. Based on the refined data source control technology in each flow control strategy, the problem of unsatisfactory data flow control effect in concurrent systems can be effectively solved, and the stability of the system can be guaranteed in different scenarios.

[0075] In an exemplary embodiment, when the flow control strategy is the first control strategy, a target value for the data processing volume per unit time is determined based on the amount of data currently being processed by each concurrent task. The data input volume of each concurrent task is then adjusted based on this target value, including: determining the data processing volume per unit time for a single partition based on the total data processing volume per unit time of the concurrent system and the total number of partitions contained in the data source corresponding to the concurrent system; the partition is used to temporarily store data and input data to the corresponding concurrent task; a target value for the data processing volume per unit time for each concurrent task is determined based on the data processing volume per unit time for a single partition and the number of partitions corresponding to each concurrent task; and the data input flow from the partitions corresponding to each concurrent task to the concurrent task is controlled according to the target value for the data processing volume per unit time for each concurrent task.

[0076] The total amount of data processed per unit time in a concurrent system can be determined based on the maximum load of concurrent system jobs, such as the total QPS of Flink system jobs; the target value of the amount of data processed per unit time for concurrent tasks can be the QPS value (Queries per second), which can characterize the data flow processed within a certain period of time.

[0077] As an example, the data source for a concurrent system can have multiple partitions, which can be used to temporarily store data and input data to the corresponding concurrent tasks. For example, the data middleware Kafka can be the input data source for Flink system jobs. Kafka can contain multiple partitions, and one or more partitions can transmit data to the corresponding concurrent tasks.

[0078] In practical implementation, the amount of data processed per unit time for a single partition can be determined based on the total amount of data processed per unit time of the concurrent system and the total number of partitions contained in the data source corresponding to the concurrent system. Since a single concurrent task can receive data input from one or more partitions, the target value of the amount of data processed per unit time for each concurrent task can be calculated based on the number of partitions it corresponds to and the amount of data processed per unit time for a single partition. Then, the data input flow from the partitions corresponding to each concurrent task to the concurrent task can be controlled according to the target value of the amount of data processed per unit time for each concurrent task.

[0079] In one example, the target value for the amount of data processed per unit time for each concurrent task can be calculated as follows:

[0080] Task QPS = Total QPS / Total number of partitions * Number of partitions consumed by the current task

[0081] Here, Task is a single concurrent task, Task QPS is the target value of data processing per unit time for the concurrent task, the number of partitions consumed by the current Task is the number of partitions corresponding to the concurrent task, the total QPS is the total amount of data processed per unit time for the concurrent system, and the total number of partitions is the total number of partitions contained in the data source corresponding to the concurrent system.

[0082] In yet another example, such as Figure 2a As shown, the input data source can include multiple partitions, such as partition 0 and partition 1. The concurrent system can include multiple concurrent tasks, such as concurrent task 0 and concurrent task 1. The central control node can be used to coordinate the various concurrent task nodes. Figure 2a The left side indicates that the influx of massive amounts of data puts enormous pressure on the concurrent system, leading to extreme instability and poor health, and posing a risk of paralyzing the entire real-time system. Figure 2a The right side represents how the QPS of input data allows for precise control of the amount of data transmitted per second from partitions to concurrent tasks based on the token bucket algorithm. The intelligent and uniform rate limiting method based on the token bucket can effectively control the amount of data input, thereby making the concurrent system operation healthier and more stable.

[0083] Compared to traditional methods that use a backpressure mechanism, the passive blocking behavior of which can cause job crashes such as avalanche when a large amount of data enters the Flink system job, this embodiment uses the token bucket algorithm for data flow control. It can automatically adjust the input data flow of each concurrent task according to the actual running situation of the job, so that the amount of data entering the Flink system job becomes uniform and matches the computing power of the Flink system job.

[0084] The technical solution of this embodiment determines the amount of data processed per unit time for a single partition based on the total amount of data processed per unit time of the concurrent system and the total number of partitions contained in the data source corresponding to the concurrent system. Then, based on the amount of data processed per unit time for a single partition and the number of partitions corresponding to each concurrent task, the target value of the amount of data processed per unit time for each concurrent task is determined. Then, according to the target value of the amount of data processed per unit time for each concurrent task, the data input flow from the partition corresponding to each concurrent task to the concurrent task is controlled. The data input flow of each concurrent task can be automatically adjusted according to the actual operation of the job, ensuring the health and stability of the concurrent system operation.

[0085] In an exemplary embodiment, after determining the target value of the data processing volume per unit time for each concurrent task based on the data processing volume per unit time for a single partition and the number of partitions for each concurrent task, the method further includes: when an update is detected in the total number of partitions and / or the number of partitions for each concurrent task, re-determining the target value of the data processing volume per unit time for each concurrent task based on the updated total number of partitions and / or the updated number of partitions, and controlling the data input flow from the partitions for each concurrent task to the concurrent task according to the re-determined target value of the data processing volume per unit time.

[0086] In practical implementation, since the target value of data processing volume per unit time for the current concurrent task can be automatically adjusted based on changes in the number of partitions, the total number of partitions in the data source corresponding to the concurrent system and the number of partitions corresponding to each concurrent task can be periodically checked by setting a preset time threshold to determine whether there have been changes. Then, when the total number of partitions and / or the number of partitions are updated, the target value of data processing volume per unit time for the concurrent task can be recalculated and applied.

[0087] In one example, such as Figure 2a As shown, when a new partition 2 is added, the target value of the amount of data processed per unit time for concurrent task 1 can be recalculated. In the concurrent system, it is possible to dynamically add upstream data source processing partitions, and it is also possible to delete processing partitions.

[0088] The technical solution of this embodiment, when an update is detected in the total number of partitions and / or the number of partitions corresponding to each concurrent task, redetermines the target value of the data processing volume per unit time for each concurrent task based on the updated total number of partitions and / or the updated number of partitions, and controls the data input flow from the partitions corresponding to each concurrent task to the concurrent task according to the redetermined target value of the data processing volume per unit time. It can automatically adjust the target value of the data processing volume per unit time for the current concurrent task based on the change in the total number of partitions and / or the number of partitions, and can effectively control the data input flow.

[0089] In an exemplary embodiment, when the target flow control strategy is the second control strategy, the data processing progress of each concurrent task in the next cycle is controlled based on the data processing progress of each concurrent task in the current cycle, including: determining the target data processing progress for the next cycle based on the data processing progress of multiple concurrent tasks in the current cycle; and pausing the input of data flow to the concurrent task when the data processing progress of a certain concurrent task is detected to exceed the target data processing progress in the next cycle.

[0090] In practical implementation, in order to ensure that the timestamps of the data processed by each concurrent task are similar during concurrent system operations, and to avoid problems such as out-of-order data loss and job stagnation, the target data processing progress for the next period can be determined based on the data processing progress of multiple concurrent tasks in the current period. Then, in the next period, for each concurrent task, if it is detected that the data processing progress of the concurrent task exceeds the target data processing progress, the concurrent task can be paused to continue data processing, thereby ensuring that the data processing progress of each concurrent task is consistent.

[0091] The technical solution of this embodiment determines the target data processing progress for the next period based on the data processing progress of multiple concurrent tasks in the current period. Then, in the next period, when it is detected that the data processing progress of a certain concurrent task exceeds the target data processing progress, the input of data traffic to that concurrent task is suspended, which can ensure that the data processing progress of each concurrent task is consistent.

[0092] In an exemplary embodiment, the data input to each concurrent task corresponds to a timestamp; determining the target data processing progress for the next period based on the data processing progress of multiple concurrent tasks in the current period includes: obtaining the maximum timestamp corresponding to the data processed by each concurrent task in the current period; determining the target timestamp for the next period as the target data processing progress based on the minimum value among the maximum timestamps corresponding to multiple concurrent tasks and a preset allowable time difference for processing progress; and pausing the input of data traffic to a concurrent task when it is detected that the data processing progress of a certain concurrent task exceeds the target data processing progress, including: pausing the input of data traffic to a concurrent task when it is detected that the timestamp corresponding to the data processed by a certain concurrent task exceeds the target timestamp.

[0093] In practical applications, the maximum timestamp corresponding to the data processed by each concurrent task in the current cycle can be obtained. This timestamp represents the maximum data time received by the concurrent task in the current processing data. The minimum value can be determined from the maximum timestamps corresponding to multiple concurrent tasks. This minimum value represents the slowest data time received by multiple concurrent tasks. Then, based on the obtained minimum value and the preset allowable time difference for processing progress, the target timestamp for the next cycle can be calculated as the target data processing progress.

[0094] After obtaining the target data processing progress, since the data input to each concurrent task can correspond to a timestamp, when it is detected that the timestamp corresponding to the data processed by the concurrent task exceeds the target timestamp, the data input flow to the concurrent task can be paused.

[0095] Specifically, such as Figure 2b As shown, the central control node can be used to coordinate the data processing progress of each concurrent task. It receives status information reported by each concurrent task, such as status information 0, status information 1, etc. The status information can include a time watermark, which can represent the current data processing progress of the concurrent task. The time watermark can carry a timestamp. Then, based on the data processing progress of each concurrent task in the current period, the target watermark for the next period can be calculated. The target watermark for the next period can then be sent to each concurrent task. In the next period, each concurrent task can control its data processing progress based on the target watermark.

[0096] Compared to traditional processing methods, when there is a large difference in the data processing progress of multiple input data sources, the data source with faster data processing progress fills up the operators, while the data source with slower data processing progress cannot enter the operators, which can lead to instability problems such as freezing. This embodiment controls the data processing progress of each concurrent task through the master control mode, making the data processing progress more proactive and controllable.

[0097] For example, if a concurrent task's data processing progress exceeds the target watermark in the next cycle, data processing will be paused to wait for a concurrent task with slower data processing progress (which is generally farther from the target watermark). By having the concurrent task with faster data processing progress wait for the concurrent task with slower data processing progress, the data processing progress gap between concurrent tasks can be narrowed.

[0098] In one example, the data input to each concurrent task can be associated with a timestamp, which indicates that data with timestamps less than that timestamp has arrived. By using the target timestamp carried in the target watermark, the processing progress of each concurrent task on the input data can be controlled based on the target timestamp. That is, when it is detected that the timestamp corresponding to the data being processed by the concurrent task exceeds the target timestamp, the flow of input data to the concurrent task can be paused.

[0099] The target watermark can be calculated using the following method:

[0100] Target watermark = slowest time watermark + Δt / 2

[0101] The slowest time watermark can be the minimum of the maximum timestamps corresponding to multiple concurrent tasks, and Δt is the maximum allowed time difference (i.e., the preset allowed time difference of processing progress), which can be preset to 10 minutes. Users can configure the maximum time difference that each concurrent task can tolerate.

[0102] Within a concurrent system, the central control node, acting as the master node, receives the data time of each concurrent task. It can calculate the predicted time value (i.e., the target data processing progress) for the next cycle based on factors such as the concurrent task with the slowest data processing progress and the maximum time difference. If a job exceeds the predicted time value in the next cycle, it will be restricted, thereby ensuring that the data processing progress of each concurrent task is consistent.

[0103] The technical solution of this embodiment obtains the maximum timestamp corresponding to the data processed by each concurrent task in the current period, and then determines the target timestamp of the next period based on the minimum value of the maximum timestamps corresponding to multiple concurrent tasks and the preset allowable time difference of processing progress. This target timestamp is used as the target data processing progress. When it is detected that the timestamp corresponding to the data processed by a certain concurrent task exceeds the target timestamp, the input of data traffic to that concurrent task is paused. The target timestamp of the next period can be obtained based on factors such as the concurrent task with the slowest data processing progress and the maximum time difference, ensuring that the data processing progress of each concurrent task is consistent.

[0104] In an exemplary embodiment, the data input to each concurrent task corresponds to a timestamp. When the flow control strategy is the third control strategy, each concurrent task is controlled to process the historical data of the corresponding data source according to a preset data consumption ratio. This includes: obtaining the actual data consumption ratio of each concurrent task for the historical data of the corresponding data source; the actual data consumption ratio represents the maximum time span between the timestamps corresponding to the historical data processed by the concurrent task within one unit of time; when the actual data consumption ratio of a certain concurrent task is greater than the preset data consumption ratio, the data inflow of the corresponding concurrent task is reduced.

[0105] The preset data consumption ratio can be configured by the user. For example, the data consumption ratio N means that N units of historical data are consumed in one unit of time.

[0106] In practical implementation, to process accumulated historical data, the actual data consumption rate of each concurrent task to the corresponding data source can be obtained. Then, when the actual data consumption rate of a concurrent task is greater than the preset data consumption rate, the data inflow of a concurrent task can be reduced.

[0107] For example, through precise data playback technology, users can configure a preset data consumption multiplier of 5. The concurrent system can control each concurrent task to consume 5 minutes of historical data after 1 minute of system time. If a concurrent task consumes too quickly, it will be limited, thus ensuring traffic control and avoiding inconsistencies in the progress of historical data consumption.

[0108] In one example, such as Figure 2cAs shown, when the preset data consumption multiplier is 2, in the stage of processing accumulated historical data, time t in the concurrent system can correspond to the processing of historical data of time 2t in each partition of the input data source. Through processing the historical data for a period of time, in the real-time data processing stage, time t in the concurrent system can correspond to the processing of real-time data of time t in each partition of the input data source, effectively controlling the consumption of historical data.

[0109] For example, if a Flink system job starts at 8 o'clock and needs to process historical data from 7 o'clock in Kafka, the data consumption multiplier can be preset to 2, meaning the Flink system job can consume two minutes' worth of data from Kafka in one minute. Then, at 9 o'clock, it can switch to real-time consumption of real-time data from Kafka at 9 o'clock, meaning the Flink system job can process real-time data from Kafka at the current time.

[0110] The technical solution of this embodiment obtains the actual data consumption rate of each concurrent task to the historical data of the corresponding data source. Then, when the actual data consumption rate of a concurrent task is greater than the preset data consumption rate, the data inflow of the concurrent task is reduced. This can limit the consumption of a concurrent task when it is too fast, and ensure that the progress of historical data consumption is consistent.

[0111] In an exemplary embodiment, obtaining the actual data consumption ratio of each concurrent task to the historical data of the corresponding data source includes: for each concurrent task, obtaining the current first system time and the second system time when the concurrent system starts, and obtaining the data processing duration of the concurrent system based on the first system time and the second system time; obtaining the first timestamp of the historical data to be processed now, and the second timestamp of the historical data to be processed when the concurrent system starts, and obtaining the time span corresponding to the historical data processed within the data processing duration based on the first timestamp and the second timestamp; and determining the actual data consumption ratio of the concurrent task based on the data processing duration and the time span.

[0112] In practical implementation, the data input to each concurrent task can correspond to a timestamp. Based on this timestamp, the maximum time span (i.e., the actual data consumption rate) between the timestamps corresponding to the historical data processed by the concurrent task within one unit of time can be calculated in the following way:

[0113]

[0114] Wherein, currentKafkaTime is the first timestamp of the historical data to be processed, that is, the timestamp of the data input to each concurrent task, such as the timestamp of the Kafka data currently consumed; baseKafkaTime is the second timestamp of the historical data to be processed when the concurrent system starts, such as the timestamp of the Kafka data consumed when the Flink system job starts; currentSystemTime is the first system time, such as the current time of the Flink system job; and baseSystemTime is the second system time when the concurrent system starts, such as the time when the Flink system job starts.

[0115] The technical solution of this embodiment obtains the current first system time and the second system time when the concurrent system starts for each concurrent task. Based on the first system time and the second system time, the data processing duration of the concurrent system is obtained. Then, the first timestamp of the historical data to be processed and the second timestamp of the historical data to be processed when the concurrent system starts are obtained. Based on the first timestamp and the second timestamp, the time span corresponding to the historical data processed within the data processing duration is obtained. Then, based on the data processing duration and the time span, the actual data consumption ratio of the concurrent task is determined. By determining the actual data consumption ratio of each concurrent task, effective control can be further performed according to the preset data consumption ratio.

[0116] Figure 3 This is a flowchart illustrating another data flow control method according to an exemplary embodiment, such as... Figure 3 As shown, this method is used in computer devices such as servers and includes the following steps.

[0117] In step S310, the current data processing scenario is obtained. The data processing scenario includes at least one of the following: a first scenario indicating that the data input volume is greater than the data processing volume of the concurrent system; a second scenario indicating that the data processing progress of multiple concurrent tasks in the concurrent system is inconsistent; and a third scenario indicating that the data source of the concurrent system has historical data accumulation. In step S320a, the first scenario corresponds to a first control strategy. If the current data processing scenario is the first scenario, the first control strategy is used as the flow control strategy corresponding to the current data processing scenario. In step S3201, based on the total data processing volume per unit time of the concurrent system and the total number of partitions contained in the data source corresponding to the concurrent system, the data processing volume per unit time corresponding to a single partition is determined. The partition is used to temporarily store data and input data to the corresponding concurrent task. In step S3202, based on the data processing volume per unit time corresponding to the single partition and the number of partitions corresponding to each concurrent task, the target value of the data processing volume per unit time corresponding to each concurrent task is determined. In step S3203, according to the target value of the data processing volume per unit time corresponding to each concurrent task, the data input flow from the partition corresponding to each concurrent task to the concurrent task is controlled. In step S320b, the second scenario corresponds to the second control strategy. If the current data processing scenario is the second scenario, the second control strategy is used as the traffic control strategy corresponding to the current data processing scenario. In step S3211, the target data processing progress for the next period is determined based on the data processing progress of the multiple concurrent tasks in the current period. In step S3212, in the next period, when it is detected that the data processing progress of a certain concurrent task exceeds the target data processing progress, the input of data traffic to that concurrent task is paused. In step S320c, the third scenario corresponds to the third control strategy. If the current data processing scenario is the third scenario, the third control strategy is used as the traffic control strategy corresponding to the current data processing scenario. In step S3221, the actual data consumption ratio of each concurrent task to the historical data of the corresponding data source is obtained; the actual data consumption ratio represents the maximum time span between the timestamps corresponding to the historical data processed by the concurrent task within one unit of time. In step S3222, when the actual data consumption rate of a concurrent task is greater than the preset data consumption rate, the data inflow corresponding to that concurrent task is reduced. It should be noted that the specific limitations of the above steps can be found in the specific limitations of a data flow control method described above, and will not be repeated here.

[0118] It should be understood that, although Figure 1 and Figure 3The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 and Figure 3 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0119] Figure 4 This is a block diagram illustrating a data flow control device according to an exemplary embodiment. (Refer to...) Figure 4 The device includes:

[0120] The data processing scenario acquisition unit 401 is configured to acquire the current data processing scenario; the data processing scenario includes at least one of the following: a first scenario representing that the data input volume is greater than the data processing volume of the concurrent system, a second scenario representing that the data processing progress of multiple concurrent tasks in the concurrent system is inconsistent, and a third scenario representing that the data source of the concurrent system has accumulated historical data.

[0121] The flow control strategy determination unit 402 is configured to determine a flow control strategy corresponding to the current data processing scenario based on the current data processing scenario.

[0122] In one possible implementation, the first scenario corresponds to a first control strategy, the second scenario corresponds to a second control strategy, and the third scenario corresponds to a third control strategy. The flow control strategy determination unit 402 is specifically configured to: if the current data processing scenario is the first scenario, use the first control strategy as the flow control strategy corresponding to the current data processing scenario; when the flow control strategy is the first control strategy, determine a target value for the data processing volume per unit time based on the current data volume processed by each concurrent task, and adjust the data input volume of each concurrent task based on the target value for the data processing volume per unit time; if the current data... If the processing scenario is the second scenario, the second control strategy is used as the traffic control strategy corresponding to the current data processing scenario. When the traffic control strategy is the second control strategy, the data processing progress of each concurrent task in the next cycle is controlled based on the data processing progress of each concurrent task in the current cycle. If the current data processing scenario is the third scenario, the third control strategy is used as the traffic control strategy corresponding to the current data processing scenario. When the traffic control strategy is the third control strategy, each concurrent task is controlled to process the historical data of the corresponding data source according to a preset data consumption ratio, where the preset data consumption ratio is greater than 1.

[0123] In one possible implementation, the flow control strategy determination unit 402 is specifically configured to: determine the amount of data processed per unit time for a single partition based on the total amount of data processed per unit time of the concurrent system and the total number of partitions contained in the data source corresponding to the concurrent system; the partition is used to temporarily store data and input data to the corresponding concurrent task; based on the amount of data processed per unit time for a single partition and the number of partitions corresponding to each concurrent task, determine the target value of the amount of data processed per unit time for each concurrent task; and control the data input flow from the partitions corresponding to each concurrent task to the concurrent task according to the target value of the amount of data processed per unit time for each concurrent task.

[0124] In one possible implementation, the data flow control device further includes:

[0125] The re-determination unit is specifically configured to, when an update is detected in the total number of partitions and / or the number of partitions corresponding to each concurrent task, re-determine the target value of the data processing volume per unit time for each concurrent task based on the updated total number of partitions and / or the updated number of partitions, and control the data input flow from the partitions corresponding to each concurrent task to the concurrent task according to the re-determined target value of the data processing volume per unit time.

[0126] In one possible implementation, the flow control strategy determination unit 402 is specifically configured to determine the target data processing progress for the next period based on the data processing progress of the multiple concurrent tasks in the current period; and in the next period, when it is detected that the data processing progress of a certain concurrent task exceeds the target data processing progress, to suspend the input of data flow to the certain concurrent task.

[0127] In one possible implementation, the data input to each concurrent task corresponds to a timestamp; the flow control strategy determination unit 402 is specifically configured to obtain the maximum timestamp corresponding to the data processed by each concurrent task in the current period; based on the minimum value among the maximum timestamps corresponding to the multiple concurrent tasks and a preset allowable time difference for processing progress, determine the target timestamp for the next period as the target data processing progress; the step of pausing the input of data flow to the concurrent task when it is detected that the data processing progress of a certain concurrent task exceeds the target data processing progress includes: pausing the input of data flow to the concurrent task when it is detected that the timestamp corresponding to the data processed by a certain concurrent task exceeds the target timestamp.

[0128] In one possible implementation, the data input to each concurrent task corresponds to a timestamp; the flow control strategy determination unit 402 is specifically configured to execute the acquisition of the actual data consumption ratio of each concurrent task to the historical data of the corresponding data source; the actual data consumption ratio represents the maximum time span between the timestamps corresponding to the historical data processed by the concurrent task in one unit of time; when the actual data consumption ratio of a certain concurrent task is greater than the preset data consumption ratio, the data inflow corresponding to the certain concurrent task is reduced.

[0129] In one possible implementation, the flow control strategy determination unit 402 is specifically configured to perform the following for each concurrent task: obtain the current first system time of the concurrent system and the second system time when the concurrent system starts; obtain the data processing duration of the concurrent system based on the first system time and the second system time; obtain the first timestamp of the historical data to be processed currently and the second timestamp of the historical data to be processed when the concurrent system starts; obtain the time span corresponding to the historical data processed within the data processing duration based on the first timestamp and the second timestamp; and determine the actual data consumption rate of the concurrent task based on the data processing duration and the time span.

[0130] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0131] Figure 5This is a block diagram illustrating an electronic device 500 for data flow control according to an exemplary embodiment. For example, the electronic device 500 may be a server. (Refer to...) Figure 5 The electronic device 500 includes a processing component 520, which further includes one or more processors, and memory resources represented by memory 522 for storing instructions, such as application programs, that can be executed by the processing component 520. The application programs stored in memory 522 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 520 is configured to execute instructions to perform the methods described above.

[0132] Electronic device 500 may further include: a power supply component 524 configured to perform power management of electronic device 500, a wired or wireless network interface 526 configured to connect electronic device 500 to a network, and an input / output (I / O) interface 528. Electronic device 500 may operate on an operating system stored in memory 522, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or similar.

[0133] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 522 including instructions, which can be executed by a processor of an electronic device 500 to perform the above-described method. The storage medium may be a computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0134] In an exemplary embodiment, a computer program product is also provided, the computer program product including instructions that can be executed by a processor of an electronic device 500 to perform the above-described method.

[0135] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0136] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0137] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data flow control method, characterized in that, The method includes: Obtain the current data processing scenario; the data processing scenario includes at least one of the following: a first scenario characterized by a data input volume greater than the data processing volume of the concurrent system, a second scenario characterized by inconsistent data processing progress of multiple concurrent tasks in the concurrent system, and a third scenario characterized by historical data accumulation in the data source of the concurrent system; Based on the current data processing scenario, a traffic control strategy corresponding to the current data processing scenario is determined; the first scenario corresponds to a first control strategy, the second scenario corresponds to a second control strategy, and the third scenario corresponds to a third control strategy; including: when the traffic control strategy is the first control strategy, determining a target value for data processing volume per unit time based on the current data volume processed by each concurrent task, and adjusting the data input volume of each concurrent task based on the target value for data processing volume per unit time; when the traffic control strategy is the second control strategy, controlling the data processing progress of each concurrent task in the next cycle based on the data processing progress of each concurrent task in the current cycle; when the traffic control strategy is the third control strategy, controlling each concurrent task to process historical data from the corresponding data source according to a preset data consumption ratio, wherein the preset data consumption ratio is greater than 1.

2. The method according to claim 1, characterized in that, When the flow control strategy is the first control strategy, a target value for the amount of data processed per unit time is determined based on the amount of data currently being processed by each concurrent task. Based on this target value, the data input amount for each concurrent task is adjusted, including: Based on the total amount of data processed per unit time of the concurrent system and the total number of partitions contained in the data source corresponding to the concurrent system, the amount of data processed per unit time for a single partition is determined; the partition is used to temporarily store data and input data to the corresponding concurrent task. Based on the amount of data processed per unit time corresponding to a single partition and the number of partitions corresponding to each concurrent task, determine the target value of the amount of data processed per unit time corresponding to each concurrent task; According to the target value of data processing volume per unit time for each concurrent task, control the data input flow from the partition corresponding to each concurrent task to the concurrent task.

3. The method according to claim 2, characterized in that, After the step of determining the target value of the data processing volume per unit time for each concurrent task based on the data processing volume per unit time for the single partition and the number of partitions for each concurrent task, the method further includes: When an update is detected in the total number of partitions and / or the number of partitions corresponding to each concurrent task, the target value of the data processing volume per unit time corresponding to each concurrent task is re-determined based on the updated total number of partitions and / or the updated number of partitions, and the data input flow from the partitions corresponding to each concurrent task to the concurrent task is controlled according to the re-determined target value of the data processing volume per unit time.

4. The method according to claim 1, characterized in that, When the flow control strategy is the second control strategy, based on the data processing progress of each concurrent task in the current cycle, the data processing progress of each concurrent task in the next cycle is controlled, including: Based on the data processing progress of the multiple concurrent tasks in the current period, determine the target data processing progress for the next period; In the next cycle, if it is detected that the data processing progress of a concurrent task exceeds the target data processing progress, the input of data traffic to the concurrent task will be suspended.

5. The method according to claim 4, characterized in that, The data input to each concurrent task corresponds to a timestamp; determining the target data processing progress for the next period based on the data processing progress of the multiple concurrent tasks in the current period includes: Get the maximum timestamp corresponding to the data processed by each concurrent task in the current period; Based on the minimum value among the maximum timestamps corresponding to the multiple concurrent tasks, and the preset allowable time difference for processing progress, the target timestamp for the next cycle is determined as the target data processing progress. The step of pausing data flow to a concurrent task when the data processing progress of a concurrent task is detected to exceed the target data processing progress includes: When it is detected that the timestamp corresponding to the data being processed by a concurrent task exceeds the target timestamp, the input of data traffic to that concurrent task is paused.

6. The method according to claim 1, characterized in that, The data input to each concurrent task has a corresponding timestamp; when the flow control strategy is the third control strategy, each concurrent task is controlled to process the historical data of the corresponding data source according to a preset data consumption ratio, including: Obtain the actual data consumption ratio of each concurrent task for the historical data of the corresponding data source; the actual data consumption ratio represents the maximum time span between the timestamps corresponding to the historical data processed by the concurrent task in one unit of time. When the actual data consumption rate of a concurrent task is greater than the preset data consumption rate, the data inflow corresponding to that concurrent task is reduced.

7. The method according to claim 6, characterized in that, The step of obtaining the actual data consumption ratio of each concurrent task to the historical data of the corresponding data source includes: For each concurrent task, obtain the current first system time and the second system time when the concurrent system starts, and obtain the data processing time of the concurrent system based on the first system time and the second system time; Obtain the first timestamp of the historical data to be processed at present, and the second timestamp of the historical data to be processed when the concurrent system starts. Based on the first timestamp and the second timestamp, obtain the time span corresponding to the historical data processed within the data processing time. Based on the data processing duration and the time span, the actual data consumption rate of the concurrent task is determined.

8. A data flow control device, characterized in that, include: The data processing scenario acquisition unit is configured to acquire the current data processing scenario. The data processing scenarios include at least one of the following: a first scenario characterized by a data input volume greater than the data processing volume of the concurrent system; a second scenario characterized by inconsistent data processing progress of multiple concurrent tasks in the concurrent system; and a third scenario characterized by historical data accumulation in the data source of the concurrent system. The flow control strategy determination unit is configured to determine a flow control strategy corresponding to the current data processing scenario based on the current data processing scenario. The first scenario corresponds to a first control strategy, the second scenario corresponds to a second control strategy, and the third scenario corresponds to a third control strategy; including: when the flow control strategy is the first control strategy, determining a target value for data processing volume per unit time based on the current data volume processed by each concurrent task, and adjusting the data input volume of each concurrent task based on the target value for data processing volume per unit time; when the flow control strategy is the second control strategy, controlling the data processing progress of each concurrent task in the next cycle based on the data processing progress of each concurrent task in the current cycle; when the flow control strategy is the third control strategy, controlling each concurrent task to process historical data from the corresponding data source according to a preset data consumption ratio, wherein the preset data consumption ratio is greater than 1.

9. The apparatus according to claim 8, characterized in that, The flow control strategy determination unit is specifically configured to determine the amount of data processed per unit time for a single partition based on the total amount of data processed per unit time of the concurrent system and the total number of partitions contained in the data source corresponding to the concurrent system. The partition is used to temporarily store data and input data to the corresponding concurrent tasks; based on the data processing volume per unit time corresponding to the single partition and the number of partitions corresponding to each concurrent task, the target value of the data processing volume per unit time corresponding to each concurrent task is determined; According to the target value of data processing volume per unit time for each concurrent task, control the data input flow from the partition corresponding to each concurrent task to the concurrent task.

10. The apparatus according to claim 9, characterized in that, The device further includes: The re-determination unit is specifically configured to, when an update is detected in the total number of partitions and / or the number of partitions corresponding to each concurrent task, re-determine the target value of the data processing volume per unit time for each concurrent task based on the updated total number of partitions and / or the updated number of partitions, and control the data input flow from the partitions corresponding to each concurrent task to the concurrent task according to the re-determined target value of the data processing volume per unit time.

11. The apparatus according to claim 8, characterized in that, The flow control strategy determination unit is specifically configured to determine the target data processing progress for the next period based on the data processing progress of the multiple concurrent tasks in the current period; and in the next period, when it is detected that the data processing progress of a certain concurrent task exceeds the target data processing progress, to suspend the input of data flow to the certain concurrent task.

12. The apparatus according to claim 11, characterized in that, The data input to each concurrent task corresponds to a timestamp. The flow control strategy determination unit is specifically configured to obtain the maximum timestamp corresponding to the data processed by each concurrent task in the current period. Based on the minimum value among the maximum timestamps corresponding to the multiple concurrent tasks, and the preset allowable time difference for processing progress, the target timestamp for the next cycle is determined as the target data processing progress. The step of pausing the input of data traffic to a concurrent task when the data processing progress of a concurrent task is detected to exceed the target data processing progress includes: pausing the input of data traffic to a concurrent task when the timestamp corresponding to the data processed by a concurrent task is detected to exceed the target timestamp.

13. The apparatus according to claim 8, characterized in that, The data input to each concurrent task corresponds to a timestamp. The flow control strategy determination unit is specifically configured to execute the acquisition of the actual data consumption ratio of each concurrent task to the historical data of the corresponding data source. The actual data consumption ratio represents the maximum time span between the timestamps corresponding to the historical data processed by the concurrent task in one unit of time. When the actual data consumption rate of a concurrent task is greater than the preset data consumption rate, the data inflow corresponding to that concurrent task is reduced.

14. The apparatus according to claim 13, characterized in that, The flow control strategy determination unit is specifically configured to perform the following for each concurrent task: obtain the current first system time of the concurrent system and the second system time when the concurrent system starts; and obtain the data processing duration of the concurrent system based on the first system time and the second system time. Obtain the first timestamp of the historical data to be processed at present, and the second timestamp of the historical data to be processed when the concurrent system starts. Based on the first timestamp and the second timestamp, obtain the time span corresponding to the historical data processed within the data processing time. Based on the data processing duration and the time span, the actual data consumption rate of the concurrent task is determined.

15. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the data flow control method as described in any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the data flow control method as described in any one of claims 1 to 7.

17. A computer program product, the computer program product comprising instructions, characterized in that, When the instruction is executed by the processor of the electronic device, the electronic device is able to perform the data flow control method as described in any one of claims 1 to 7.

Images