Method and apparatus for dynamic control of data processing

Abstract

The application discloses a dynamic control method and device for data processing. The method comprises the following steps: in response to a task processing request of a target data stream task to be processed, obtaining task configuration information of the target data stream task and task strategy configuration information of the data processing device, wherein the task configuration information of the target data stream task contains a data volume of the target data stream task, and the task strategy configuration information contains resource configuration information of a unit task of the data processing device; determining resource configuration information for processing the target data stream task based on the task configuration information of the target data stream task, a task strategy priority of the target data stream task and the task strategy information; and allocating resources for the target data stream task based on the resource configuration information for processing the target data stream task, so as to process the target data stream task.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a dynamic control method and device for data processing. BACKGROUND

[0002] At present, stream data processing is mainly realized based on Storm, Spark Streaming or Flink, and the data acquisition source end is usually in the form of a message queue such as Kafka, and the data output end is usually various storage systems. In this process, the acquisition, processing and output of data are performed by different ends, which may lead to inconsistent processing capabilities of different ends.

[0003] For example, when data acquisition is too fast, the data processing and output capabilities may not keep up, which may cause data processing delay, and greater risk may cause excessive pressure on the data processing server and the data output end, thereby causing instability. At the same time, in this process, some ends may be under excessive pressure, and some ends may be relatively idle, which may also cause waste of computing or storage resources, especially when facing a sudden large-scale data volume peak, this phenomenon is particularly obvious.

[0004] Therefore, how to cope with the problem of inconsistent processing capabilities of different ends leading to instability of the entire stream data processing still needs to provide further solutions. SUMMARY

[0005] Embodiments of the present application provide a dynamic control method, device and equipment for data processing to solve the problem of inconsistent processing capabilities of different ends leading to instability of the entire stream data processing in the prior art.

[0006] Embodiments of the present application adopt the following technical solutions:

[0007] A dynamic control method for data processing, comprising:

[0008] In response to a task processing request of a target data stream task to be processed, acquiring task configuration information of the target data stream task and task policy configuration information of the data processing equipment, the task configuration information of the target data stream task containing a data volume of the target data stream task, the task policy configuration information containing resource configuration information of a unit task of the data processing equipment, the resource configuration information of the unit task containing configuration information for indicating input flow, output flow and resource occupation information of the unit task;

[0009] Determining a task policy priority of the target data stream task;

[0010] determine resource configuration information for processing the target data stream task based on the task configuration information of the target data stream task, the task policy priority of the target data stream task, and the task policy information;

[0011] allocate resources for the target data stream task based on the resource configuration information for processing the target data stream task.

[0012] A data processing device comprises:

[0013] An information obtaining module is configured to obtain task configuration information of a target data stream task and task policy configuration information of the data processing device in response to a task processing request of the target data stream task to be processed, wherein the task configuration information of the target data stream task comprises a data volume of the target data stream task, and the task policy configuration information comprises processing traffic of the data processing device, resource configuration information of a unit task, and configuration information for indicating input traffic, output traffic, and resource occupation information of the unit task.

[0014] A priority determining module is configured to determine a task policy priority of the target data stream task.

[0015] An information determining module is configured to determine resource configuration information for processing the target data stream task based on the task configuration information of the target data stream task, the task policy priority of the target data stream task, and the task policy information.

[0016] A resource allocating module is configured to allocate resources for the target data stream task based on the resource configuration information for processing the target data stream task, so as to process the target data stream task.

[0017] An electronic device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is executed by the processor to implement the steps of the data processing dynamic control method.

[0018] A computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the data processing dynamic control method.

[0019] The above at least one technical solution adopted by the embodiments of the present application can achieve the following beneficial effects:

[0020] In the present application, the task configuration information of the target data stream task and the task strategy configuration information of the data processing device are acquired in response to a task processing request of the target data stream task to be processed, the task strategy priority of the target data stream task is determined, the resource configuration information for processing the target data stream task is determined based on the task configuration information of the target data stream task, the task strategy priority of the target data stream task and the task strategy information, and finally, the target data stream task is allocated with resources based on the resource configuration information for processing the target data stream task to process the target data stream task. Before each data stream task processing, the data stream task is configured with resources according to the data volume of the data stream task, the processing flow of the data processing device, and the preset resource configuration information of the unit task, the resource configuration information of the unit task contains configuration information for indicating the input flow, the output flow and the resource occupation information of the unit task, so that the task maximum processing capacity, the task occupied resources and the task output flow in the data processing device can be dynamically controlled according to the actual processing capacity of the device regardless of the data volume of the data stream task and whether the processing capacities of different ends are the same, thereby effectively ensuring the stability of each data stream task in the data processing device in the whole data stream processing process. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate the illustrative embodiments of the present application and their description serve to explain the present application, but do not constitute improper limitations on the present application. In the drawings:

[0022] Figure 1 A specific implementation flowchart of a data processing dynamic control method provided for the embodiments of the present application;

[0023] Figure 2 A structure schematic diagram of a system to which the data processing dynamic control method provided for the embodiments of the present application is applied;

[0024] Figure 3 A structure schematic diagram of a data processing device provided for the embodiments of the present application;

[0025] Figure 4 A structure schematic diagram of an electronic device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0026] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0027] The technical solutions provided by the embodiments of the present application will be described in detail below in combination with the drawings.

[0028] As described in the background, the difference in processing capacity of different ends can cause the problem of instability of the entire stream data processing, especially when facing a sudden peak of large-scale data volume, the elastic jitter is particularly obvious. In view of this, the existing data processing scheme generally adopts time window control or sets the data volume of each, which can alleviate the stability of the system to a certain extent, but cannot solve the elastic jitter under burst traffic, such as low processing delay, low submission frequency and low utilization of computing resources under small data volume. Under a large amount of burst data, the processing delay is high, the submission frequency is high, the computing is overloaded, and the pressure on the storage end is also large.

[0029] In order to solve the problem of instability of the entire stream data processing caused by the difference in processing capacity of different ends in the prior art, the embodiments of the present application propose a dynamic control method for data processing, which can respond to the task processing request of the target data stream task to be processed, obtain the task configuration information of the data stream task and the task strategy configuration information of the data processing device, determine the task strategy priority of the target data stream task, and based on the task configuration information of the target data stream task, the task strategy priority of the target data stream task and the task strategy information, determine the resource configuration information for processing the target data stream task. Finally, based on the resource configuration information for processing the target data stream task, the target data stream task is allocated resources to process the target data stream task.

[0030] Since the data stream task can be configured with resources according to the data volume of the data stream task, the processing capacity of the data processing device, the preset resource configuration information of the unit task before each data stream task processing, the resource configuration information of the unit task contains the configuration information for indicating the input flow, output flow and resource occupation information of the unit task, so that no matter how large the data volume of the data stream task is, and no matter whether the processing capacity of different ends is the same, the maximum processing capacity of the task in the data processing device, the resource occupied by the task and the output flow of the task can be dynamically controlled according to the actual processing capacity of the device, so as to effectively guarantee the stability of each data stream task in the data processing device in the entire data stream processing process.

[0031] The technical solutions provided by the embodiments of the present application will be described in detail below in combination with the drawings.Figure 1 The diagram illustrates the implementation process of the dynamic control method for data processing, and provides a detailed explanation of the method's implementation. Figure 1 The methods shown may include:

[0032] Step 110: In response to the task processing request of the target data stream task to be processed, obtain the task configuration information of the target data stream task and the task policy configuration information of the data processing device.

[0033] The target data stream task configuration information includes the data volume of the target data stream task, the task strategy configuration information includes the maximum processing flow of the data processing device and the resource configuration information of the unit task, and the resource configuration information of the unit task includes configuration information used to indicate the input flow, output flow and resource usage information of the unit task.

[0034] The task policy configuration information for the data processing device is pre-configured and deployed to the data processing device. This configuration information includes the policy name, the maximum processing capacity of the data processing device (i.e., the maximum throughput the device can handle per unit time), the policy mode (including policy modes corresponding to different policy priorities), the maximum parallelism limit (i.e., the maximum parallelism the data processing device can support), and the resource configuration information for the unit task. The policy priority can include a first priority indicating parallelism priority, a second priority indicating throughput priority, and a third priority indicating throughput priority and enabling elastic scaling.

[0035] In some exemplary embodiments, the resource configuration information of a unit task includes at least the parallelism of the unit task, the memory usage of the unit task, the CPU usage of the unit task, the length of the time window of the unit task, the amount of data in a time window of the unit task, and the submission frequency of the unit task. Here, a unit task is the smallest unit of processing, meaning the target data stream task can be divided into multiple unit tasks for parallel processing. The length of the time window of a unit task is the duration of one time window of the unit task, and the amount of data in a time window of the unit task is the amount of data that can be read from the unit task within that time window. The parallelism of a unit task indicates the number of unit tasks that can be processed simultaneously, i.e., the number of unit tasks executed in parallel.

[0036] The CPU usage of a task = CPU usage of a unit task × parallelism of the unit task; the memory usage of a task = memory usage of a unit task × parallelism of the unit task.

[0037] For data stream tasks, a reasonable time window can be configured so that the amount of data read within the configured time window can be processed within that time window.

[0038] The processing capability of a unit task per second = the data volume of the unit task in a time window × 1 ÷ the length of the time window of the unit task, and then the processing capability of each task per second = the parallelism of the unit task × the data volume of the unit task in a time window × 1 ÷ the length of the time window of the unit task. The output submission frequency of the task = the parallelism of the unit task × (the length of the time window of the unit task ÷ the submission frequency of the unit task ÷ 60).

[0039] In step 120, the task strategy priority of the target data stream task is determined.

[0040] In some exemplary embodiments, the task strategy priority can be parsed according to the task processing requirement of the target data stream task, that is, according to the actual processing requirement of the task to match the task strategy priority that can meet the task processing requirement of the task, or, to improve the determination efficiency of the task strategy priority and simplify the priority confirmation process, the priority identifier indicating the task strategy priority can also be configured in the task configuration information. Specifically, the task strategy priority of the target data stream task is determined, including:

[0041] The task strategy priority of the target data stream task is determined based on the data processing requirement of the target data stream task contained in the task configuration information of the target data stream task; or,

[0042] The task strategy priority of the target data stream task is determined based on the priority identifier of the target data stream task contained in the task configuration information of the target data stream task.

[0043] When the task strategy priority of the target data stream task is determined based on the data processing requirement of the target data stream task contained in the task configuration information of the target data stream task, a mapping relationship table between the keywords of the task processing requirement and the task strategy priority can be maintained in the data processing device, such as that the time efficiency high can be associated with the strategy priority for indicating the traffic priority, such as that the time efficiency low can be associated with the strategy priority for indicating the parallel priority, such as that the low time delay can be associated with the strategy priority for indicating the traffic priority and starting the elastic scaling.

[0044] When the priority identifier of the target data stream task contained in the task configuration information of the target data stream task is based, a mapping relationship table between the priority identifier and the task strategy priority can be maintained in the data processing device, such as that the priority identifier 0 can be associated with the strategy priority for indicating the parallel priority, such as that the priority identifier 1 can be associated with the strategy priority for indicating the traffic priority, such as that the priority identifier 2 can be associated with the strategy priority for indicating the traffic priority and starting the elastic scaling.

[0045] At step 130, based on the task configuration information of the target data stream task, the task policy priority of the target data stream task and the task policy information, the resource configuration information for processing the target data stream task is determined.

[0046] Different policy priorities can correspond to different resource allocation modes. Hereinafter, taking the priority identifier 0 indicating the parallel priority policy priority (hereinafter referred to as the first priority), the priority identifier 1 indicating the traffic priority policy priority (hereinafter referred to as the second priority), and the priority identifier 2 indicating the traffic priority policy priority (hereinafter referred to as the third priority) as an example.

[0047] The resource allocation mode corresponding to the first priority is: performing resource configuration according to the task policy information, and performing resource configuration according to the setting information of CPU, memory and parallelism in the task policy information; the resource allocation mode corresponding to the second priority is: re-determining the parallelism according to the maximum processing traffic of the data processing device, wherein the parallelism = maximum processing traffic ÷ processing capacity per second of the unit task, and then configuring the CPU, memory, processing capacity per second, submission frequency of output traffic and other information of the target data stream task according to the re-determined parallelism; the resource allocation mode corresponding to the third priority is: determining the initial parallelism = maximum processing traffic ÷ data volume of the unit task, and the initial parallelism is less than or equal to the upper limit of parallelism, and the parallelism is dynamically adjusted according to the data processing capacity during the running process, that is, the number of parallel processing of the unit task is adjusted, so as to guarantee the data processing capacity of the data processing device, and the adjusted parallelism does not exceed the upper limit of parallelism. Specifically, the task policy priority includes the first priority, the second priority and the third priority, based on the task configuration information and the task policy information of the target data stream task, the resource configuration information for processing the target data stream task is determined, including:

[0048] When the task policy priority of the target data stream task is the first priority, based on the data volume of the target data stream task in the task configuration information of the target data stream task and the resource configuration information of the unit task in the task policy configuration information, the resource configuration information of the target data stream task is determined;

[0049] When the task policy priority of the target data stream task is the second priority, based on the maximum processing traffic of the data processing device and the resource configuration information of the unit task, the parallelism of the target data stream task is determined, and based on the parallelism of the target data stream task and the data volume of the target data stream task, the resource configuration information of the target data stream task is determined;

[0050] When the task policy priority of the target data stream task is the third priority, the resource occupation information of the data processing device is determined, and based on the resource occupation information of the data processing device, the resource configuration information of the unit task and the data volume of the target data stream task, the resource configuration information of the target data stream task is determined.

[0051] For example, the maximum processing flow of the data processing device configured in the task strategy configuration information of the data processing device is 1800000, the upper limit of parallelism is 80, the CPU occupancy of each unit task is set to 2, the memory occupancy of each unit task is 4G, the length of the time window of each unit task is 3s, the data volume of each unit task in a time window is 90000, the parallelism of the unit task is 30, and the submission frequency of each unit task is 3. Then, the processing capacity of each unit task per second = 90000 x 1 ÷ 3 = 30000 / second.

[0052] When the task strategy priority of the target data stream task is the first priority, the target data stream task can be configured with resources according to the task strategy configuration information of the data processing device, at this time, the parallelism of the target data stream task = the parallelism of the unit task is 30, the processing capacity of the target data stream task per second = 30 x 30000 = 900000 / second, the CPU occupancy of the target data stream task = 30 x 2 = 60 cores, the memory occupancy of the target data stream task = 30 x 4 = 120G, and the submission pressure of the output end, i.e., the output submission frequency of the target data stream task = 30 ÷ 3 = 10 times, i.e., batch submission is performed 10 times per second.

[0053] In some exemplary embodiments, when the task strategy priority of the target data stream task is the second priority, the parallelism of the target data stream task is determined based on the maximum processing flow of the data processing device and the resource configuration information of the unit task, and the resource configuration information of the target data stream task is determined based on the parallelism of the target data stream task and the data volume of the target data stream task, including:

[0054] When the task strategy priority of the target data stream task is the second priority, the processing flow of the unit task in a unit time period is determined based on the length of the time window of the unit task and the data volume of the unit task in a time window.

[0055] The parallelism of the data processing device is determined based on the ratio between the maximum processing flow of the data processing device and the processing flow of the unit task in a unit time period.

[0056] The resource configuration information of the target data stream task is determined based on the parallelism of the data processing device, the memory occupancy of the unit task, the CPU occupancy of the unit task, and the submission frequency of the unit task.

[0057] Continuing with the example of the task strategy configuration information of the data processing device, when the task strategy priority of the target data stream task is the second priority, the parallelism can be adjusted according to the maximum processing flow and the task strategy, and resource configuration is performed according to the adjusted parallelism to ensure maximum flow output. At this time, the parallelism of the target data stream task = 1800000 / 30000 = 60, the processing capacity of the target data stream task per second = 60 x 30000 = 1800000 / second, the CPU occupancy of the target data stream task = 60 x 2 = 120 cores, the memory occupancy of the target data stream task = 60 x 4 = 240G, and the submission pressure to the output, i.e., the output submission frequency of the target data stream task = 60 ÷ 3 = 20 times, i.e., batch submission is performed 20 times per second.

[0058] When the task strategy priority of the target data stream task is the third priority, the resource occupancy information of the data processing device can be determined, and based on the resource occupancy information of the data processing device, the resource configuration information of the unit task, and the data volume of the target data stream task, the resource configuration information of the target data stream task is determined to elastically expand the parallelism according to the processing capacity. At this time, the initial parallelism of the target data stream task = 1800000 / 30000 = 60, and since the upper limit of the parallelism of the data processing device is 80, the expansion range of the parallelism of the target data stream task is [60, 80]. The maximum processing capacity of the target data stream task per second = 60 x 30000 = 1800000 / second, the CPU occupancy of the target data stream task = 60 x 2 = 120 cores, the memory occupancy of the target data stream task = 60 x 4 = 240G, and the submission pressure to the output, i.e., the output submission frequency of the target data stream task = 60 ÷ 3 = 20 times, i.e., batch submission is performed 20 times per second. Moreover, when it is determined that the actual data flow processing capacity exceeds a certain proportion of the maximum processing capacity of the target data stream task per second, the parallelism can be expanded to a maximum of 80, at which time the CPU occupancy of the target data stream task = 80 x 2 = 160 cores, and the memory occupancy of the target data stream task = 80 x 4 = 320G.

[0059] In step 140, resources are allocated to the target data stream task based on the resource configuration information for processing the target data stream task, so as to process the target data stream task.

[0060] In step 140, resources are allocated to the target data stream task based on the resource configuration information for processing the target data stream task, so as to process the target data stream task.

[0061] In some example embodiments, to achieve unified management of multiple tasks in a data processing device and resource scheduling, so that the resources configured for each task can adapt to the current processing capability of the data processing device, the embodiments of the present application can maintain a task statistics list, and when a new task is added, the new task can be added to the task statistics list, so as to obtain task index data of each task according to a certain data acquisition frequency, and adjust the resource configuration of some tasks based on the task index data of each task. Specifically, the method provided by the embodiments of the present application further comprises:

[0062] adding the target data stream task to the task statistics list;

[0063] acquiring task index data of multiple data stream tasks in the task statistics list every preset time period;

[0064] determining an occupation ratio of the multiple data stream tasks based on the task index data of the multiple data stream tasks in the task statistics list;

[0065] when the resource occupation ratio of the multiple data stream tasks reaches a preset resource occupation ratio, dynamically adjusting a parallel task quantity of a data stream task with a specified priority in task strategy priority in the multiple data stream tasks.

[0066] In some example embodiments, the task strategy priority of each task can indicate whether the task supports dynamic adjustment of resource configuration. Specifically, when the resource occupation ratio of the multiple data stream tasks reaches a preset resource occupation ratio, the parallel task quantity of a data stream task with a specified priority in task strategy priority in the multiple data stream tasks is dynamically adjusted, comprising:

[0067] when the resource occupation ratio of the multiple data stream tasks reaches a preset resource occupation ratio, determining a to-be-adjusted data stream task with a specified priority in task strategy priority from the multiple data stream tasks;

[0068] adjusting a parallel degree of the to-be-adjusted data stream task based on a parallel degree adjustment range of the to-be-adjusted data stream task;

[0069] adjusting the parallel task quantity of the to-be-adjusted data stream task based on the adjusted parallel degree of the to-be-adjusted data stream task.

[0070] Figure 2 A structural schematic diagram of a system to which the dynamic control method for data processing provided by the embodiments of the present application is applied. In Figure 2In the middle, the task processing system can include a flow task strategy module, a task control module, a task statistics module, a resource scheduling module, an index monitoring module and a Job execution module. Among them, the flow task strategy module is used for the configuration management of the flow control strategy of the data flow task and the calculation of the data flow task resource configuration information under different strategy priorities. The flow task strategy module can configure the resource configuration information of the flow, parallelism and output submission frequency of each data flow task.

[0071] The task control module is used for the submission and interactive perception of the data flow task. The data flow task can submit a specific data flow task to the data flow engine through the task control module, and after submitting the data flow task, the data task is added to the task statistics list, and the task statistics module is triggered to collect task index data. The task control module supports the submission control and dynamic adjustment of the resource configuration of the data flow task. For example, the task index obtained by the task statistics module can be combined with the flow task strategy module to reconfigure the resources.

[0072] The task statistics module is mainly responsible for obtaining, counting and outputting some task index data of the data flow task. The task index data includes the parallelism, resource occupation (including memory and CPU), processing rate, data processing amount in the last few time windows, data processing delay in the last few time windows, data processing segmentation reading, processing and output index of the data flow task. The task statistics module can provide the user with the overall situation of the task, and provide data support for the task control module to trigger dynamic adjustment of the resource configuration.

[0073] The resource scheduling module is used for resource allocation and scheduling. The module can be implemented based on the open source Yarn framework, can manage the available resources of the entire data processing device cluster, the available resources are CPU and memory, realizes the unified management and allocation of these available resources, and supports linear expansion and scaling of the cluster. The index monitoring module is used for collecting statistical information of the engine, including Job (i.e. data flow task), resource, processing capacity related information. The Job execution module itself includes a data reading module, a data processing module and a data output module.

[0074] Figure 2 The data processing process shown can include:

[0075] ① Strategy loading

[0076] This step includes configuring the flow task strategy file, loading the strategy configuration file, calculating the resource settings of the task according to the configuration, and finally outputting the resource settings to the task control module.

[0077] ② Task submission

[0078] The step includes firstly, the task control module combines the task program and the resource setting into a data stream task running command; then, submits the data stream task to the data stream task engine and returns data stream task information; again, acquires the running state of the data stream task according to the data stream task information; and finally, sends the data stream task information to the task statistical module, which periodically collects and statistically processes the task index data of the data stream task.

[0079] ③Resource request and allocation

[0080] The step includes firstly, the data stream engine produces a running topology structure according to the resource setting, determines how many unit tasks the data stream task can be split into and the configuration information of each unit task, and sends a resource configuration request to the resource scheduling module, which allocates resources for the data stream task according to the task topology structure and the resource configuration request.

[0081] ④Task running

[0082] Specifically, it can include 1) the data reading module realizes connection with the data source end of the data stream task through the data pipeline, and each unit task establishes a separate data pipeline; 2) the data output module can establish a database connection pool between the target end of the data stream task; 3) in the data processing process, the length of the time window is slid and the data in each time window is acquired, and each unit task maintains its own position point; 4) the data processing module processes the data, such as conversion and statistics; 5) the database connection pool is connected and the data is written into the database connection pool; 6) the operations of 3) to 5) are repeated until the data stream task processing is completed.

[0083] ⑤Information collection

[0084] Specifically, the index monitoring module can continuously collect the index of the task.

[0085] ⑥Index statistics

[0086] Specifically, the task statistical module can acquire the task index data from the index monitoring module of the data stream engine and statistically process the task index data.

[0087] ⑦Calling

[0088] Specifically, it can include 1) determining whether the strategy priority of the data stream task is a specified priority, if yes, then entering the subsequent step, otherwise the task remains running; 2) the task control module periodically calls the task statistical information to acquire the processing flow and the delay of the task; 3) when the processing capacity exceeds a preset proportion (such as 30%), the parallel degree of the elastic scaling data stream task is triggered.

[0089] ⑧Dynamic adjustment

[0090] Specifically, the method can comprise: 1) obtaining a value of task parallelism that can be increased according to a policy configured according to task strategy configuration information; 2) sending a parallelism adjustment request to a task control module by the data flow engine, so that the task control module adjusts the value of parallelism of the data flow task; and 3) increasing the number of unit tasks according to the adjusted value of parallelism, and improving the processing capability of the data processing device.

[0091] The dynamic control method for data processing provided in the embodiments of the present application can respond to a task processing request of a target data flow task to be processed, obtain task configuration information of the data flow task and task strategy configuration information of a data processing device, determine a task strategy priority of the target data flow task, determine resource configuration information for processing the target data flow task based on the task configuration information of the target data flow task, the task strategy priority of the target data flow task, and task strategy information, and finally allocate resources for the target data flow task based on the resource configuration information for processing the target data flow task, to process the target data flow task. Because the data flow task can be configured with resources according to the data volume of the data flow task, the processing flow of the data processing device, and preset resource configuration information of a unit task before processing of each data flow task, the resource configuration information of the unit task contains configuration information for indicating input flow, output flow, and resource occupation information of the unit task, so that the maximum processing capability of a task in the data processing device, resource occupation of the task, and output flow of the task can be dynamically controlled according to the actual processing capability of the device, regardless of the data volume of the data flow task or whether the processing capabilities of different ends are the same, thereby effectively ensuring the stability of each data flow task in the data processing device in the entire data flow processing process.

[0092] The embodiments of the present application also provide a data processing device 300, as shown in Figure 3 The data processing device 300 comprises:

[0093] An information obtaining module 301 is configured to respond to a task processing request of a target data flow task to be processed, obtain task configuration information of the target data flow task and task strategy configuration information of the data processing device, the task configuration information of the target data flow task containing a data volume of the target data flow task, the task strategy configuration information containing a processing flow of the data processing device and resource configuration information of a unit task, and the resource configuration information of the unit task containing configuration information for indicating input flow, output flow, and resource occupation information of the unit task.

[0094] A priority determining module 302 is configured to determine a task strategy priority of the target data flow task.

[0095] The information determining module 303 is configured to determine resource configuration information for processing the target data stream task based on the task configuration information of the target data stream task, the task policy priority of the target data stream task, and the task policy information.

[0096] The resource allocating module 304 is configured to allocate resources for the target data stream task based on the resource configuration information for processing the target data stream task, so as to process the target data stream task.

[0097] In some exemplary embodiments, the apparatus further comprises:

[0098] The task joining module is configured to add the target data stream task to a task statistical list.

[0099] The data obtaining module is configured to obtain task index data of a plurality of data stream tasks in the task statistical list every preset time period.

[0100] The proportion determining module is configured to determine an occupation proportion of the plurality of data stream tasks based on the task index data of the plurality of data stream tasks in the task statistical list.

[0101] The dynamic adjusting module is configured to dynamically adjust a parallel task quantity of a data stream task with a specified priority in task policy priority among the plurality of data stream tasks when a resource occupation proportion of the plurality of data stream tasks reaches a preset resource occupation proportion.

[0102] In some exemplary embodiments, when the dynamic adjusting module dynamically adjusts the parallel task quantity of the data stream task with the specified priority in task policy priority among the plurality of data stream tasks when the resource occupation proportion of the plurality of data stream tasks reaches the preset resource occupation proportion, the dynamic adjusting module is specifically configured to:

[0103] When the resource occupation proportion of the plurality of data stream tasks reaches the preset resource occupation proportion, determine a to-be-adjusted data stream task with the specified priority in task policy priority from the plurality of data stream tasks;

[0104] Adjust a parallel degree of the to-be-adjusted data stream task based on a parallel degree adjustment range of the to-be-adjusted data stream task;

[0105] Adjust the parallel task quantity of the to-be-adjusted data stream task based on the adjusted parallel degree of the to-be-adjusted data stream task.

[0106] In some example embodiments, the task strategy priority includes a first priority, a second priority and a third priority, and the information determination module 303, when determining the resource configuration information for processing the target data stream task based on the task configuration information of the target data stream task and the task strategy information, is specifically configured to:

[0107] when the task strategy priority of the target data stream task is the first priority, determining the resource configuration information of the target data stream task based on the data volume of the target data stream task in the task configuration information of the target data stream task and the resource configuration information of the unit task in the task strategy configuration information;

[0108] when the task strategy priority of the target data stream task is the second priority, determining the parallelism of the target data stream task based on the maximum processing flow of the data processing device and the resource configuration information of the unit task, and determining the resource configuration information of the target data stream task based on the parallelism of the target data stream task and the data volume of the target data stream task;

[0109] when the task strategy priority of the target data stream task is the third priority, determining the resource occupation information of the data processing device, and determining the resource configuration information of the target data stream task based on the resource occupation information of the data processing device, the resource configuration information of the unit task and the data volume of the target data stream task.

[0110] In some example embodiments, the resource configuration information of the unit task at least includes the parallelism of the unit task, the memory occupation volume of the unit task, the CPU occupation volume of the unit task, the length of the time window of the unit task, the data volume of the unit task in a time window and the submission frequency of the unit task.

[0111] In some example embodiments, when the information determination module 303, when the task strategy priority of the target data stream task is the second priority, determines the parallelism of the target data stream task based on the maximum processing flow of the data processing device and the resource configuration information of the unit task, and determines the resource configuration information of the target data stream task based on the parallelism of the target data stream task and the data volume of the target data stream task, is specifically configured to:

[0112] when the task strategy priority of the target data stream task is the second priority, determining the processing flow of the unit task in a unit time period based on the length of the time window of the unit task and the data volume of the unit task in a time window;

[0113] determine parallelism of the data processing device based on a ratio between a maximum processing flow of the data processing device and a processing flow of the unit task in a unit time period;

[0114] determine resource configuration information of the target data flow task based on the parallelism of the data processing device, memory occupancy of the unit task, CPU occupancy of the unit task and submission frequency of the unit task.

[0115] In some exemplary embodiments, when the priority determination module 302 determines the task strategy priority of the target data flow task, specifically for:

[0116] determine the task strategy priority of the target data flow task based on the data processing requirement of the target data flow task contained in the task configuration information of the target data flow task; or,

[0117] determine the task strategy priority of the target data flow task based on the priority identifier of the target data flow task contained in the task configuration information of the target data flow task.

[0118] The data processing apparatus 300 provided by the embodiments of the present application can implement Figures 1-2 The method embodiment of the dynamic control method of data processing is implemented in each process of the method embodiment, to avoid repetition, which will not be described here.

[0119] Figure 4 is a structural schematic diagram of an electronic device of an embodiment of the present application. Please refer to Figure 4 At the hardware level, the electronic device includes a processor, and optionally further includes an internal bus, a network interface and a memory. The memory can include a memory such as a high-speed random access memory (RAM), and can also include a non-volatile memory such as at least one disk memory. Of course, the electronic device can also include other hardware required by the business.

[0120] The processor, the network interface and the memory can be connected to each other through the internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4Only one bidirectional arrow is used to represent multiple buses or multiple types of buses.

[0121] The memory is configured to store a program. Specifically, the program can include program code including computer operation instructions. The memory can include an internal memory and a non-volatile memory, and provide instructions and data for the processor.

[0122] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs, and forms a data processing apparatus at a logical level. The processor executes the program stored in the memory, and is specifically configured to perform the following operations:

[0123] In response to a task processing request of a target data stream task to be processed, task configuration information of the target data stream task and task policy configuration information of the data processing device are acquired, the task configuration information of the target data stream task contains a data volume of the target data stream task, the task policy configuration information contains a processing flow of the data processing device, resource configuration information of a unit task, and the resource configuration information of the unit task contains configuration information for indicating input flow, output flow and resource occupation information of the unit task;

[0124] A task policy priority of the target data stream task is determined.

[0125] Based on the task configuration information of the target data stream task, the task policy priority of the target data stream task and the task policy information, resource configuration information for processing the target data stream task is determined.

[0126] Based on the resource configuration information for processing the target data stream task, resources are allocated for the target data stream task to process the target data stream task.

[0127] The above as described in the present application Figures 1-2The method performed by the data processing apparatus disclosed in the embodiment shown can be applied in a processor or implemented by the processor. The processor can be an integrated circuit chip with signal processing capability. In the implementation, each step of the above method can be completed by integrated logic circuits in the hardware of the processor or instructions in the form of software. The processor mentioned above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; or a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. Each method, step and logic block disclosed in the embodiment of the present application can be implemented or executed. The general processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the embodiment of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory, and the processor reads the information in the memory and combines the hardware to complete the steps of the above method.

[0128] The electronic device can also perform the method of Figures 1-2 and implement the function of the data processing apparatus in Figures 1-2 the embodiment shown. The embodiment of the present application will not be described here.

[0129] The embodiment of the present application also proposes a computer readable storage medium storing one or more programs, the one or more programs including instructions capable of causing the portable electronic device including a plurality of application programs to perform the method of Figures 1-2 the embodiment shown, and specifically for performing the following operations:

[0130] In response to a target exploration request, the target exploration request is parsed to obtain data processing parameters corresponding to the target exploration request, the data processing parameters including parameters for indicating a target data source corresponding to the target data processing request;

[0131] The data processing parameter corresponding to the target exploration request is filled into a preset data processing task template based on a correspondence between a data processing parameter name and a data processing parameter value, so as to establish a target data processing task corresponding to the target data processing request; and the target data processing task is associated with the target data source according to a parameter in the data processing parameter used for indicating the target data source;

[0132] The target data processing task is executed to perform data processing on the target data source, so as to obtain a data processing result corresponding to the target data processing task.

[0133] Of course, in addition to the software implementation, the electronic device of the present application does not exclude other implementation manners, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0134] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0135] In summary, the above only describes preferred embodiments of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0136] The system, device, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or entity, or by a product with certain functions. A typical implementation device is a computer. Specifically, the computer may, for example, be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0137] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0138] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or inherent to such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0139] The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

Claims

1. A dynamic control method for data processing, characterized in that, The method is applied to a data processing device, and the method includes: In response to a task processing request from a target data stream task to be processed, the task configuration information of the target data stream task and the task policy configuration information of the data processing device are obtained. The task configuration information of the target data stream task includes the data volume of the target data stream task, and the task policy configuration information includes the maximum processing flow of the data processing device and the resource configuration information of the unit task. The resource configuration information of the unit task includes configuration information for indicating the input flow, output flow and resource usage information of the unit task. Determine the task policy priority for the target data stream task; Based on the task configuration information of the target data stream task, the task policy priority of the target data stream task, and the task policy configuration information, the resource configuration information for processing the target data stream task is determined; Based on the resource configuration information for processing the target data stream task, resources in the data processing device are allocated to the target data stream task to process the target data stream task; When the task policy priorities are different, the resource configuration information for processing the target data stream task is determined in different ways based on the task configuration information and the task policy configuration information.

2. The method as described in claim 1, characterized in that, The method further includes: Add the target data stream task to the task statistics list; At preset time intervals, acquire task indicator data for multiple data stream tasks in the task statistics list; Based on the task indicator data of multiple data stream tasks in the task statistics list, determine the occupancy ratio of the multiple data stream tasks; When the resource occupancy ratio of the multiple data stream tasks reaches the preset resource occupancy ratio, the number of parallel tasks of the data stream tasks with the task policy priority of the multiple data stream tasks is dynamically adjusted.

3. The method as described in claim 2, characterized in that, When the resource occupancy ratio of the multiple data stream tasks reaches a preset resource occupancy ratio, the number of parallel tasks of the data stream tasks with a specified priority in the task policy priority is dynamically adjusted, including: When the resource usage ratio of the multiple data stream tasks reaches the preset resource usage ratio, the data stream task to be adjusted with the specified priority is determined from the multiple data stream tasks. Based on the parallelism adjustment range of the data stream task to be adjusted, adjust the parallelism of the data stream task to be adjusted. Based on the adjusted parallelism of the data stream task to be adjusted, the number of parallel tasks of the data stream task to be adjusted is adjusted.

4. The method as described in claim 1, characterized in that, The task policy priority includes a first priority, a second priority, and a third priority. The step of determining the resource configuration information for processing the target data stream task based on the task configuration information of the target data stream task, the task policy priority of the target data stream task, and the task policy configuration information includes: When the task policy priority of the target data stream task is the first priority, the resource configuration information of the target data stream task is determined based on the data volume of the target data stream task in the task configuration information of the target data stream task and the resource configuration information of the unit task in the task policy configuration information. When the task policy priority of the target data stream task is the second priority, the parallelism of the target data stream task is determined based on the maximum processing flow of the data processing device and the resource configuration information of the unit task, and the resource configuration information of the target data stream task is determined based on the parallelism of the target data stream task and the data volume of the target data stream task. When the task policy priority of the target data stream task is the third priority, the resource occupancy information of the data processing device is determined, and the resource configuration information of the target data stream task is determined based on the resource occupancy information of the data processing device, the resource configuration information of the unit task, and the data volume of the target data stream task.

5. The method as described in claim 4, characterized in that, The resource configuration information of the unit task includes at least the parallelism of the unit task, the memory usage of the unit task, the CPU usage of the unit task, the length of the time window of the unit task, the amount of data of the unit task within a time window, and the submission frequency of the unit task.

6. The method as described in claim 5, characterized in that, When the task policy priority of the target data stream task is the second priority, the parallelism of the target data stream task is determined based on the maximum processing traffic of the data processing device and the resource configuration information of the unit task. The resource configuration information of the target data stream task is also determined based on the parallelism and the data volume of the target data stream task, including: When the task policy priority of the target data stream task is the second priority, the processing traffic of the unit task within a unit time period is determined based on the length of the time window of the unit task and the amount of data of the unit task within a time window. The parallelism of the data processing device is determined based on the ratio between the maximum processing throughput of the data processing device and the processing throughput of the unit task within a unit time period. Based on the parallelism of the data processing device, the memory usage of the unit task, the CPU usage of the unit task, and the submission frequency of the unit task, the resource configuration information of the target data stream task is determined.

7. The method as described in claim 1, characterized in that, The process of determining the task strategy priority for the target data stream task includes: Based on the data processing requirements of the target data stream task contained in its task configuration information, the task strategy priority of the target data stream task is determined; or... The task policy priority of the target data stream task is determined based on the priority identifier of the target data stream task contained in the task configuration information of the target data stream task.

8. A data processing device, characterized in that, include: The information acquisition module is used to respond to the task processing request of the target data stream task to be processed, and to acquire the task configuration information of the target data stream task and the task strategy configuration information of the data processing device. The task configuration information of the target data stream task includes the data volume of the target data stream task, and the task strategy configuration information includes the maximum processing flow of the data processing device and the resource configuration information of the unit task. The resource configuration information of the unit task includes configuration information for indicating the input flow, output flow and resource usage information of the unit task. The priority determination module is used to determine the task strategy priority of the target data stream task; The information determination module is used to determine the resource configuration information for processing the target data stream task based on the task configuration information of the target data stream task, the task policy priority of the target data stream task, and the task policy configuration information. The resource allocation module is used to allocate resources in the data processing device to the target data stream task based on the resource configuration information for processing the target data stream task, so as to process the target data stream task; The information determination module is specifically used to: determine the resource configuration information for processing the target data stream task in different ways based on the task configuration information and the task strategy configuration information when the task strategy priorities are different.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images