Data processing method and device, electronic equipment, storage medium and chip

Abstract

According to the data processing method and device, the electronic equipment, the storage medium and the chip provided by the invention, the current resource consumption information of the target task is acquired under the condition of determining that the target task starts the preset adjustment strategy; wherein the preset adjustment strategy is used for indicating the task concurrency and the task memory of the target task to be dynamically adjusted; estimating the load of the target task according to the current resource consumption information; and when it is determined that the load accords with a preset adjustment threshold in the preset adjustment strategy, dynamically adjusting the task concurrency and / or the task memory of the target task according to the preset adjustment strategy. Compared with the prior art, the method has the advantages that the current resource consumption information of the target task is acquired in real time, the load of the target task is estimated according to the current resource consumption information, and the resource of the target task is dynamically adjusted according to the load, so that the resource consumption condition of the target task can be immediately responded and adjusted; and the hysteresis of resource adjustment is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular, to a method and apparatus for processing data, an electronic device, a storage medium, and a chip. Background Art

[0002] A data processing task refers to immediately processing and analyzing data after it is generated or arrives at the system to obtain real-time or near-real-time results. This data processing method aims to quickly respond to and adapt to data changes and is usually used in applications and systems that require immediate insights and decisions.

[0003] Resource dynamic prediction and elastic adjustment is the behavior of dynamically scaling the resources consumed by a program during real-time data processing. It involves real-time monitoring, evaluation, and adjustment of the resources consumed by the program to ensure that the system can operate efficiently in different scenarios. The existing resource dynamic prediction and elastic adjustment method is to predict the metrics of the next cycle based on the running conditions of each program in the current cycle. This adjustment method has a certain lag. Summary of the Invention

[0004] The present disclosure provides a method and apparatus for processing data, an electronic device, a storage medium, and a chip to solve the problems in the related art. By obtaining the current resource consumption information of the target task in real time, estimating the load of the target task based on the current resource consumption information, and dynamically adjusting the resources of the target task according to the load, it is possible to immediately respond to and adjust the resource consumption situation of the target task and reduce the lag of resource adjustment.

[0005] A first aspect embodiment of the present disclosure provides a method for processing data, the method including:

[0006] When it is determined that the target task activates a preset adjustment strategy, obtaining the current resource consumption information of the target task; wherein, the preset adjustment strategy is used to indicate dynamic adjustment of the task concurrency and task memory of the target task;

[0007] Estimating the load of the target task based on the current resource consumption information;

[0008] When it is determined that the load meets a preset adjustment threshold in the preset adjustment strategy, dynamically adjusting the task concurrency and / or task memory of the target task according to the preset adjustment strategy.

[0009] In some embodiments of the present disclosure, the dynamically adjusting the task concurrency and / or task memory of the target task according to the preset adjustment strategy includes:

[0010] Dynamically adjust the task concurrency and / or task memory of the target task by using a first preset adjustment strategy;

[0011] Or, dynamically adjust the task concurrency and / or task memory of the target task by using a second preset adjustment strategy, where the priority of the first preset adjustment strategy is higher than that of the second preset adjustment strategy.

[0012] In some embodiments of the present disclosure, obtaining the current resource consumption information of the target task includes:

[0013] Obtain the partition information of the processing data source corresponding to the target task;

[0014] Determine the average traffic information of each partition, the data processing backlog delay information of each partition, the task memory usage information, and the task processing performance of each partition; the current resource consumption information at least includes the average traffic information of each partition, the data processing backlog delay information of each partition, the task memory usage information, and the task processing performance of each partition;

[0015] Record the data processing backlog delay information of each partition in sequence.

[0016] In some embodiments of the present disclosure, dynamically adjusting the task concurrency and / or task memory of the target task by using a first preset adjustment strategy includes:

[0017] Judge whether the data processing backlog delay information exceeds a first concurrency adjustment threshold; the preset adjustment threshold includes a first concurrency adjustment threshold, a second concurrency adjustment threshold, a first memory adjustment threshold, and a second memory adjustment threshold;

[0018] When it is determined that the data processing backlog delay information exceeds the first concurrency adjustment threshold, estimate a first target concurrency according to the number of partitions, the average traffic information of each partition, and the task processing performance of each partition, where the first target concurrency is not higher than the smaller value between the user-defined concurrency upper bound and the number of partitions, and the number of partitions is determined according to the partition information of the processing data source;

[0019] Judge whether the task memory usage information is higher than the first memory adjustment threshold, or the task memory usage information is lower than the second memory adjustment threshold;

[0020] Calculate the target task memory according to the average memory usage and the maximum memory configuration.

[0021] In some embodiments of the present disclosure, the method further includes:

[0022] When it is determined that the data processing backlog delay information does not exceed the first concurrency adjustment threshold, determine the backlog change trend in the backlog delay information queue, where the data processing backlog delay information of each partition is recorded in the backlog delay information queue;

[0023] When it is determined that the backlog change trend is continuously rising, adjust to the second target concurrency according to the first preset step size, and the second target concurrency does not exceed the smaller value of the custom concurrency upper bound and the number of partitions.

[0024] In some embodiments of the present disclosure, the method further includes:

[0025] When it is determined that the data processing backlog delay information does not exceed the first concurrency adjustment threshold and the backlog change trend is not continuously rising, determine whether the data processing backlog delay information is less than the second concurrency adjustment threshold; the second concurrency adjustment threshold is less than the first concurrency adjustment threshold;

[0026] When it is determined that the data processing backlog delay information is less than the second concurrency adjustment threshold and the first target concurrency is greater than or equal to the preset concurrency threshold, adjust the lower bound of the concurrency of the target task down to the third target concurrency according to the second preset step size, and the third target concurrency is not lower than the custom concurrency lower bound, and the preset concurrency threshold is determined according to the number of partitions.

[0027] In some embodiments of the present disclosure, dynamically adjusting the task concurrency and / or task memory of the target task by using the second preset adjustment strategy includes:

[0028] Call a preset concurrency algorithm to calculate the fourth target concurrency, and determine the smaller value from the fourth target concurrency and the third concurrency;

[0029] Determine the concurrency corresponding to the smaller value as the concurrency of the target task.

[0030] A second aspect embodiment of the present disclosure provides a data processing device, and the device includes:

[0031] An acquisition unit, configured to acquire the current resource consumption information of the target task when it is determined that the target task starts a preset adjustment strategy; wherein, the preset adjustment strategy is used to indicate dynamic adjustment of the task concurrency and task memory of the target task;

[0032] An estimation unit, configured to estimate the load of the target task according to the current resource consumption information;

[0033] An adjustment unit, configured to dynamically adjust the task concurrency and / or task memory of the target task according to a preset adjustment policy when it is determined that the load meets a preset adjustment threshold in the preset adjustment policy.

[0034] In some embodiments of the present disclosure, the adjustment unit includes:

[0035] A first adjustment module, configured to dynamically adjust the task concurrency and / or task memory of the target task by using a first preset adjustment policy;

[0036] A second adjustment module, configured to dynamically adjust the task concurrency and / or task memory of the target task by using a second preset adjustment policy, where the priority of the first preset adjustment policy is higher than that of the second preset adjustment policy.

[0037] In some embodiments of the present disclosure, the acquisition unit includes:

[0038] An acquisition module, configured to acquire partition information of a processing data source corresponding to the target task;

[0039] A determination module, configured to determine the average traffic information of each partition, the data processing backlog delay information of each partition, the task memory usage information, and the task processing performance of each partition; the current resource consumption information at least includes the average traffic information of each partition, the data processing backlog delay information of each partition, the task memory usage information, and the task processing performance of each partition;

[0040] A recording module, configured to sequentially record the data processing backlog delay information of each partition.

[0041] In some embodiments of the present disclosure, the first adjustment module is further configured to:

[0042] Determine whether the data processing backlog delay information exceeds a first concurrency adjustment threshold; the preset adjustment threshold includes a first concurrency adjustment threshold, a second concurrency adjustment threshold, a first memory adjustment threshold, and a second memory adjustment threshold;

[0043] In the case where it is determined that the data processing backlog delay information exceeds the first concurrency adjustment threshold, estimate a first target concurrency according to the number of partitions, the average traffic information of each partition, and the task processing performance of each partition, where the first target concurrency is not higher than the smaller value between the custom concurrency upper bound and the number of partitions, and the number of partitions is determined according to the partition information of the processing data source;

[0044] Determine whether the task memory usage information is higher than a first memory adjustment threshold, or the task memory usage information is lower than a second memory adjustment threshold;

[0045] Calculate the target task memory according to the average memory usage and the maximum memory configuration.

[0046] In some embodiments of the present disclosure, the first adjustment module is further configured to:

[0047] When it is determined that the data processing backlog delay information does not exceed the first concurrency adjustment threshold, determine the backlog change trend in the backlog delay information queue, where the data processing backlog delay information of each partition is recorded in the backlog delay information queue;

[0048] When it is determined that the backlog change trend is continuously rising, adjust it to the second target concurrency according to the first preset step amount, and the second target concurrency does not exceed the smaller value of the custom concurrency upper bound and the number of partitions.

[0049] In some embodiments of the present disclosure, the first adjustment module is further configured to:

[0050] When it is determined that the data processing backlog delay information does not exceed the first concurrency adjustment threshold and the backlog change trend is not continuously rising, determine whether the data processing backlog delay information is less than the second concurrency adjustment threshold; the second concurrency adjustment threshold is less than the first concurrency adjustment threshold;

[0051] When it is determined that the data processing backlog delay information is less than the second concurrency adjustment threshold and the first target concurrency is greater than or equal to the preset concurrency threshold, adjust the lower bound of the concurrency of the target task down to the third target concurrency according to the second preset step amount, and the third target concurrency is not lower than the custom concurrency lower bound, and the preset concurrency threshold is determined according to the number of partitions.

[0052] In some embodiments of the present disclosure, the second adjustment module is further configured to:

[0053] Call a preset concurrency algorithm to calculate the fourth target concurrency and determine the smaller value from the fourth target concurrency and the third concurrency;

[0054] Determine the concurrency corresponding to the smaller value as the concurrency of the target task.

[0055] An embodiment of the third aspect of the present disclosure provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the first aspect embodiment of the present disclosure.

[0056] A fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0057] A fifth aspect embodiment of the present disclosure proposes a chip, which includes one or more interfaces and one or more processors; the interfaces are used to receive signals from the memory of an electronic device and send signals to the processors, and the signals include computer instructions stored in the memory. When the processors execute the computer instructions, the electronic device is caused to execute the method described in the first aspect embodiment of the present disclosure.

[0058] In summary, according to the data processing method, apparatus, electronic device, storage medium and chip proposed by the present disclosure, when it is determined that a preset adjustment strategy is started for a target task, the current resource consumption information of the target task is obtained; wherein, the preset adjustment strategy is used to indicate dynamic adjustment of the task concurrency and task memory of the target task; the load of the target task is estimated according to the current resource consumption information; when it is determined that the load meets a preset adjustment threshold in the preset adjustment strategy, the task concurrency and / or task memory of the target task are dynamically adjusted according to the preset adjustment strategy. Compared with the related art, the present disclosure can immediately respond to and adjust the resource consumption situation of the target task by obtaining the current resource consumption information of the target task in real time, estimating the load of the target task according to the current resource consumption information, and dynamically adjusting the resources of the target task according to the load, reducing the lag of resource adjustment.

[0059] It should be understood that the above general description and subsequent detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure and do not constitute an improper limitation of the present disclosure.

[0061] Figure 1 It is a flowchart of a data processing method provided by an embodiment of the present disclosure;

[0062] Figure 2 It is a schematic flowchart of dynamically adjusting a target task by using a first preset adjustment strategy provided by an embodiment of the present disclosure;

[0063] Figure 3 It is another schematic flowchart of dynamically adjusting a target task by using a first preset adjustment strategy provided by an embodiment of the present disclosure;

[0064] Figure 4Another schematic flowchart for dynamically adjusting a target task using a first preset adjustment strategy provided by an embodiment of the present disclosure;

[0065] Figure 5 A schematic flowchart for dynamically adjusting a target task using a second preset adjustment strategy provided by an embodiment of the present disclosure;

[0066] Figure 6 An overall framework diagram for data processing provided by an embodiment of the present disclosure;

[0067] Figure 7 A schematic structural diagram of a data processing device provided by an embodiment of the present disclosure;

[0068] Figure 8 Another schematic structural diagram of a data processing device provided by an embodiment of the present disclosure;

[0069] Figure 9 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure;

[0070] Figure 10 A schematic structural diagram of a chip provided by an embodiment of the present disclosure. Detailed implementation manners

[0071] The embodiments of the present disclosure are described in detail below. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present disclosure, and should not be construed as a limitation of the present disclosure.

[0072] A data processing task refers to immediately processing and analyzing the data after the data is generated or arrives at the system to obtain real-time or near-real-time results. This data processing method aims to quickly respond to and adapt to data changes and is usually used in application programs and systems that require immediate insights and decisions.

[0073] Resource dynamic prediction and elastic adjustment is the behavior of dynamically scaling the resources consumed by program operation during the real-time data processing process, which involves real-time monitoring, evaluation, and adjustment of the resources consumed by program operation to ensure that the system can operate efficiently in different scenarios. The existing resource dynamic prediction and elastic adjustment method is to predict the indicators of the next cycle according to the running conditions of each program in the current cycle. This adjustment method has a certain lag.

[0074] Therefore, to solve the problems existing in the related art, the present disclosure provides a method and apparatus for processing data, an electronic device, a storage medium, and a chip. When it is determined that a preset adjustment strategy is started for a target task, the current resource consumption information of the target task is obtained; wherein the preset adjustment strategy is used to indicate dynamic adjustment of the task concurrency and task memory of the target task; the load of the target task is estimated according to the current resource consumption information; when it is determined that the load meets a preset adjustment threshold in the preset adjustment strategy, the task concurrency and / or task memory of the target task is dynamically adjusted according to the preset adjustment strategy.

[0075] This solution can obtain the current resource consumption information of the target task in real time, estimate the load of the target task according to the current resource consumption information, and dynamically adjust the resources of the target task according to the load, so as to immediately respond to and adjust the resource consumption situation of the target task, and reduce the lag of resource adjustment.

[0076] The embodiments of the present disclosure are not exhaustive, but only illustrate some embodiments, and do not specifically limit the protection scope of the present disclosure. Without contradiction, each step in an embodiment can be implemented as an independent embodiment, and the steps can be combined arbitrarily. For example, the solution after removing some steps in an embodiment can also be implemented as an independent embodiment, and the order of the steps in an embodiment can be arbitrarily exchanged. In addition, the optional implementation manners in an embodiment can be combined arbitrarily; furthermore, the embodiments can be combined arbitrarily. For example, some or all of the steps of different embodiments can be combined arbitrarily, and an embodiment can be arbitrarily combined with the optional implementation manners of other embodiments.

[0077] In each embodiment of the present disclosure, unless otherwise specified and there is no logical conflict, the terms and / or descriptions between the embodiments are consistent and can be cited from each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0078] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure.

[0079] In the embodiments of the present disclosure, unless otherwise specified, elements expressed in the singular form, such as "one", "a kind of", "the", "above-mentioned", "said", "aforementioned", "this", etc., may mean "one and only one", or may also mean "one or more", "at least one", etc. For example, in the case of using articles such as "a", "an", "the" in English in the translation, the noun after the article can be understood as a singular expression form or a plural expression form.

[0080] In some embodiments, terms such as "in response to...", "in response to determining...", "in the case of...", "when...", "while...", "if...", "if...", etc. may be used interchangeably.

[0081] In some embodiments, terms such as "greater than", "greater than or equal to", "not less than", "more than", "more than or equal to", "not less than", "higher than", "higher than or equal to", "not lower than", "above", etc. may be used interchangeably, and terms such as "less than", "less than or equal to", "not greater than", "less than", "less than or equal to", "not more than", "lower than", "lower than or equal to", "not higher than", "below", etc. may be used interchangeably.

[0082] Prefix words such as "first", "second", etc. in the embodiments of the present disclosure are only used to distinguish different described objects, and do not limit the position, order, priority, quantity, content, etc. of the described objects. The description of the described objects refers to the description in the context of the claims or embodiments, and should not constitute redundant limitations due to the use of prefix words.

[0083] In the embodiments of the present disclosure, "a plurality of" means two or more.

[0084] In the embodiments of the present disclosure, terms such as "import", "input", "read in", etc. may be used interchangeably.

[0085] In some embodiments, a device, etc. may be interpreted as physical or virtual, and its name is not limited to the name recorded in the embodiments. Terms such as "device", "equipment", "device", "circuit", "network element", "node", "function", "unit", "section", "system", "network", "chip", "chip system", "entity", "subject", etc. may be used interchangeably.

[0086] In some embodiments, terms such as "terminal", "terminal device", "user equipment (UE)", "user terminal", "mobile station (MS)", "mobile terminal (MT)", subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, etc. may be used interchangeably.

[0087] Figure 1 The flowchart of a data processing method provided by an embodiment of the present disclosure is shown. This method can be applied to application scenarios such as big data real-time data stream processing. For example, when processing massive data and performing complex computing tasks, a stable process for supporting real-time data stream processing is required. When it is impossible to clearly set how many concurrences and how much resources are provided for each concurrence, the process of dynamically adjusting resource data, or a Flink stream processing job, etc., the present disclosure is not limited. As Figure 1 shown, the data processing method includes:

[0088] Step 101, when it is determined that the target task starts a preset adjustment strategy, obtain the current resource consumption information of the target task; wherein, the preset adjustment strategy is used to indicate dynamic adjustment of the task concurrency and task memory of the target task.

[0089] In the embodiment of the present disclosure, it is determined whether to enable dynamic prediction and elastic adjustment of resource management according to the target task, as well as the time interval of elastic adjustment. When it is necessary to enable dynamic prediction and elastic adjustment of resource management, a timed task will be created to trigger resource dynamic prediction detection at regular intervals and start the preset adjustment strategy of the target task.

[0090] Among them, whether to enable dynamic prediction and elastic adjustment of resource management is determined customarily. The time interval of the elastic adjustment is a customarily set time interval, such as: 5 minutes, 10 minutes, etc. After creating a scheduled task, dynamic prediction and elastic adjustment of resource management for the target task are performed at the time interval of the elastic adjustment. Specifically, the embodiments of the present disclosure do not limit this.

[0091] The resource consumption information is the buried point information of the target task, which refers to the key information captured and recorded during the execution of the target task, including but not limited to: partition information of the real-time data processing data source, average traffic information of each partition, data processing backlog delay information of each partition, task memory usage, etc. Specifically, the embodiments of the present disclosure do not limit this.

[0092] Step 102, predict the load of the target task according to the current resource consumption information.

[0093] In the embodiments of the present disclosure, the load of the target task refers to the workload or resource consumption that the system, service, or application needs to bear when executing the task. The load can include various forms, specifically depending on the type of the task and the execution environment. For example: network load, memory load, storage load, concurrent load, etc. Among them, the memory load is the task memory of the target task, indicating the memory requirements of the target task, including but not limited to: data storage, caching, and memory usage during the execution process, etc. The concurrent load is the task concurrency of the target task, indicating the processing ability of the target task in a multi-user or high-concurrency environment, including but not limited to: the ability to process multiple requests simultaneously, etc. Specifically, the embodiments of the present disclosure do not limit this.

[0094] Step 103, when it is determined that the load meets the preset adjustment threshold in the preset adjustment policy, dynamically adjust the task concurrency and / or task memory of the target task according to the preset adjustment policy.

[0095] In the embodiments of the present disclosure, the preset adjustment threshold is a customarily set threshold, including multiple concurrency adjustment thresholds and multiple memory adjustment thresholds. Specifically, the embodiments of the present disclosure do not limit this.

[0096] Among them, dynamic adjustment refers to performing expansion processing or contraction processing on the task concurrency and / or task memory of the target task. If the predicted task concurrency of the target task is compared with the current task concurrency of the target task, if the predicted task concurrency is greater than the current task concurrency, an expansion processing is triggered; if the predicted task concurrency is less than or equal to the current task concurrency, a contraction processing is triggered.

[0097] In summary, according to the data processing method proposed in the present disclosure, when it is determined that the target task starts a preset adjustment strategy, the current resource consumption information of the target task is obtained; wherein, the preset adjustment strategy is used to indicate dynamic adjustment of the task concurrency and task memory of the target task; the load of the target task is estimated according to the current resource consumption information; when it is determined that the load meets the preset adjustment threshold in the preset adjustment strategy, the task concurrency and / or task memory of the target task are dynamically adjusted according to the preset adjustment strategy. Compared with the related art, the present disclosure can immediately respond to and adjust the resource consumption situation of the target task by obtaining the current resource consumption information of the target task in real time, estimating the load of the target task according to the current resource consumption information, and dynamically adjusting the resources of the target task according to the load, thereby reducing the lag of resource adjustment.

[0098] In an implementable manner of the embodiment of the present disclosure, it supports users to customize resource dynamic adjustment algorithms. At the same time, the service itself also provides a set of basic algorithms to provide basic resource dynamic adjustment solutions. Users can customize and access algorithms to overwrite the basic algorithms, with the user-configured algorithm taking precedence. When there is no user-configured algorithm, it is routed to the basic algorithm for adjustment. When dynamically adjusting according to the preset adjustment strategy, different adjustment strategies can be divided according to different situations. Specifically, it includes but is not limited to the following ways: dynamically adjusting the task concurrency and / or task memory of the target task by adopting a first preset adjustment strategy; or, dynamically adjusting the task concurrency and / or task memory of the target task by adopting a second preset adjustment strategy, and the priority of the first preset adjustment strategy is higher than that of the second preset adjustment strategy.

[0099] In an implementable manner of the embodiment of the present disclosure, in order to be able to immediately respond to the traffic change situation during the processing of the target task, it is necessary to obtain the current resource consumption information of the target task in real time. Therefore, in order to dynamically adjust the target task in a timely manner, it can also be implemented by but not limited to the following ways: obtaining the partition information of the processing data source corresponding to the target task; determining the average traffic information of each partition, the data processing backlog delay information of each partition, the task memory usage information, and the task processing performance of each partition; the current resource consumption information at least includes the average traffic information of each partition, the data processing backlog delay information of each partition, the task memory usage information, and the task processing performance of each partition; and sequentially recording the data processing backlog delay information of each partition.

[0100] In the embodiments of the present disclosure, when recording the data processing backlog delay information, the data processing backlog delay information of the most recent several times (for example, 3 times, 4 times, etc.) of the target task is recorded in a queue. When the queue is full and new data processing backlog delay information is obtained, the earliest data processing backlog delay information in the queue is removed to ensure that the upper limit of the queue size is a controllable value, thereby improving the stability during the task execution process.

[0101] In an implementable manner of the embodiments of the present disclosure, to facilitate the understanding of the dynamic adjustment process of the target task, the following is provided Figure 2 for illustration. Figure 2 FIG. is a schematic flowchart of dynamically adjusting a target task by using a first preset adjustment strategy provided by the embodiments of the present disclosure, as Figure 2 shown:

[0102] Step 201, determining whether the data processing backlog delay information exceeds a first concurrency adjustment threshold; the preset adjustment threshold includes a first concurrency adjustment threshold, a second concurrency adjustment threshold, a first memory adjustment threshold, and a second memory adjustment threshold.

[0103] In the embodiments of the present disclosure, the first concurrency adjustment threshold is a user-defined threshold, which is an acceptable backlog delay threshold pre-configured by the user when creating the target task. If the data processing backlog delay information exceeds the first concurrency adjustment threshold, it indicates that the load of the target task is relatively large, and it is necessary to estimate the task concurrency of the new target task and perform capacity expansion processing on the target task. If the data processing backlog delay information does not exceed the first concurrency adjustment threshold, it indicates that the target task does not need to be directly expanded, and further dynamic estimation of the target task is required to determine whether to perform capacity expansion processing or capacity reduction processing on the target task.

[0104] Step 202, in the case where it is determined that the data processing backlog delay information exceeds the first concurrency adjustment threshold, estimating a first target concurrency according to the number of partitions, the average traffic information of each partition, and the task processing performance of each partition, where the first target concurrency is not higher than the smaller value between the user-defined concurrency upper bound and the number of partitions, and the number of partitions is determined according to the partition information of the processing data source.

[0105] In the embodiments of the present disclosure, it is necessary to perform capacity expansion processing on the target task according to the first target concurrency, and at the same time set an upper limit value for the first target concurrency, that is, the smaller value between the user-defined concurrency upper bound and the number of partitions, to ensure that the maximum resource upper limit is restricted when performing capacity expansion processing on the target task. When the first target concurrency is too large, an alarm can also be issued to notify the user to check the rationality of the target task, rather than unrestrictedly increasing the resources, which may affect the overall resources of the cluster.

[0106] Among them, the upper bound of the custom concurrency degree is the upper bound of the resource elastic adjustment pre-customized and configured by the user.

[0107] Step 203, determine whether the task memory usage information is higher than the first memory adjustment threshold or lower than the second memory adjustment threshold.

[0108] In the embodiment of the present disclosure, the task memory usage information includes the average memory usage and the maximum memory configuration. The first memory adjustment threshold and the second memory adjustment threshold are custom-set thresholds. The first memory adjustment threshold is greater than the second memory adjustment threshold. When the ratio of the average memory usage to the maximum memory configuration is higher than the first memory adjustment threshold, or the ratio of the average memory usage to the maximum memory configuration is lower than the second memory adjustment threshold, it is determined that the task memory of the target task needs to be adjusted. For example: set the first memory adjustment threshold to 0.8, set the second memory adjustment threshold to 0.4, the average memory usage is M, and the maximum memory configuration is N. Then when M / N > 0.8 or M / N < 0.4, the adjustment of the task memory of the target task is triggered.

[0109] Step 204, calculate the target task memory according to the average memory usage and the maximum memory configuration.

[0110] In the embodiment of the present disclosure, when calculating the target task memory, the calculation method can be custom-configured. For example: (((M + 256) * 2) / 0.75) + 256, etc., where the average memory usage is M, and 0.75, 256, and 2 are custom-configured parameter constants.

[0111] In an implementable manner of the embodiment of the present disclosure, when the data processing backlog delay information does not exceed the first concurrency adjustment threshold, the backlog change trend can be considered to trigger resource scaling in advance, and timely resource adjustment countermeasures can be made to reduce the delay problem of scaling. Therefore, in order to make timely resource adjustment countermeasures, the present disclosure further provides a flow schematic diagram for dynamically adjusting the target task by using a first preset adjustment strategy, as Figure 3 shown, including:

[0112] Step 301, in the case of determining that the data processing backlog delay information does not exceed the first concurrency adjustment threshold, determine the backlog change trend in the backlog delay information queue, and the data processing backlog delay information of each partition is recorded in the backlog delay information queue.

[0113] In the embodiments of the present disclosure, the backlog delay information refers to the phenomenon that the data processing task is delayed in completion during the execution of the target task due to certain reasons. These delays may be caused by reasons such as insufficient resources, improper task priorities, system failures, complex data processing processes, or excessive data processing volumes. The backlog change trend is the change trend of the delay in the completion of the data processing task. For example, if the time for the data processing task to be delayed in completion becomes longer, it indicates that the backlog change trend is upward; if the time for the data processing task to be delayed in completion becomes shorter, it indicates that the backlog change trend is downward.

[0114] Step 302, when it is determined that the backlog change trend is continuously upward, adjust to the second target concurrency degree according to the first preset step size, and the second target concurrency degree does not exceed the smaller value of the custom concurrency degree upper bound and the number of partitions.

[0115] In the embodiments of the present disclosure, the first preset step size is a numerically value set by the user. For example: 2, 3, etc. During the process of adjusting the current concurrency degree of the target task to the second target concurrency degree, it is necessary to incrementally adjust the current concurrency degree of the target task by a multiple according to the first preset step size to obtain the second target concurrency degree. For example: if the first preset step size is 2, then the second target concurrency degree is twice the current concurrency degree of the target task, and if it is still determined that the backlog change trend is continuously upward subsequently, continue to expand the capacity in an exponential growth manner with a base of 2.

[0116] At the same time, an upper limit value also needs to be set for the second target concurrency degree, that is, the smaller value of the custom concurrency degree upper bound and the number of partitions, and the expansion method is to adjust by a multiple of the current first preset step size. Although there is a certain lag problem in the face of sudden traffic increase, each expansion grows exponentially with the first preset step size, which can quickly reach the expected value and cope with the traffic growth problem. In the face of frequent fluctuations, it will not cause waste due to excessive resources adjusted at one time.

[0117] In an implementable manner of the embodiments of the present disclosure, when the data processing backlog delay information does not exceed the first concurrency degree adjustment threshold and the backlog change trend is not continuously upward, it indicates that the current backlog delay situation is controllable. At this time, it can be judged according to other conditions whether it is necessary to lower the lower bound of the concurrency degree to avoid waste of resources adjusted each time due to a relatively high lower bound of the concurrency degree. Specifically, a flow diagram for dynamically adjusting the target task by adopting the first preset adjustment strategy is further provided, as Figure 4 shown, including:

[0118] Step 401: When it is determined that the data processing backlog delay information does not exceed the first concurrency adjustment threshold and the backlog change trend is not continuously rising, determine whether the data processing backlog delay information is less than the second concurrency adjustment threshold; the second concurrency adjustment threshold is less than the first concurrency adjustment threshold.

[0119] In the embodiments of the present disclosure, the second concurrency adjustment threshold is a custom - set threshold. If the data processing backlog delay information is less than the second concurrency adjustment threshold, further evaluation is required to determine whether the lower bound of concurrency needs to be reduced. If the data processing backlog delay information is greater than or equal to the second concurrency adjustment threshold, mark this dynamic detection rule as no operation required and directly exit the dynamic adjustment.

[0120] Step 402: When it is determined that the data processing backlog delay information is less than the second concurrency adjustment threshold and the current concurrency of the target task is greater than or equal to the preset concurrency threshold, reduce the lower bound of the concurrency of the target task by a second preset step size to a third target concurrency. The third target concurrency is not lower than the custom lower bound of concurrency, and the preset concurrency threshold is determined according to the number of partitions.

[0121] In the embodiments of the present disclosure, the preset concurrency threshold is a value calculated according to the number of partitions. For example: the number of partitions * 0.8, the number of partitions * 0.9, etc.

[0122] Among them, the second preset step size is a custom - set value. For example: 1, 2, 3, etc. When reducing the lower bound of concurrency to the third target concurrency, the lower bound of the concurrency of the target task needs to be decreased step by step according to the second preset step size to obtain the third target concurrency. For example: if the second preset step size is 1, the third target concurrency is the lower bound of the concurrency of the target task minus 1, and the third target concurrency is used as the new lower bound of concurrency. Subsequently, the lower bound of concurrency will be tried to be increased according to the actual situation to avoid a relatively low default lower bound of concurrency, which may lead to frequent capacity expansion in the future, but also to avoid a relatively high lower bound of concurrency after adjustment, resulting in waste of resources for each adjustment, so the lower bound of concurrency will be tried to be reduced.

[0123] At the same time, when releasing resources, the lower bound of scaling will be reset, and the lower bound will gradually decrease. When the traffic is small and resources need to be released, it can effectively alleviate the problem that frequent elastic scaling will be triggered in the case of frequent traffic fluctuations, affecting the running efficiency of the program.

[0124] In an implementable manner of the embodiments of the present disclosure, for the convenience of understanding the dynamic adjustment process of the target task, Figure 5 an illustration is provided. Figure 5A flowchart for dynamically adjusting a target task using a second preset adjustment strategy provided by an embodiment of the present disclosure is as follows Figure 5 as shown:

[0125] Step 501: Invoke a preset concurrency algorithm to calculate a fourth target concurrency, and determine the smaller value between the fourth target concurrency and the third concurrency.

[0126] In an embodiment of the present disclosure, the preset concurrency algorithm is a custom-configured algorithm, such as: dynamic load balancing algorithm, adaptive scheduling algorithm, Ansible, Puppet, Chef, etc. Specifically, the embodiment of the present disclosure does not limit it.

[0127] Step 502: Determine the concurrency corresponding to the smaller value as the concurrency of the target task.

[0128] In an embodiment of the present disclosure, taking the smaller value between the fourth target concurrency and the third concurrency as the new dynamic adjustment lower bound can appropriately adjust the lower bound range, and the scaling down can be relatively smooth instead of large fluctuations.

[0129] In summary, the present disclosure obtains the current traffic situation of the target task in real time, combines the processing performance of the target task, and estimates the load of the target task. When it is found that there is a backlog in the processing of the target task, the latest resource configuration is calculated and a request is sent to the resource manager to allocate more resources to handle peak traffic and improve the stability of the target task; when it is found that there is no backlog in the target task and the traffic is small, an attempt is made to release some resources to improve resource utilization and reduce costs; when there are fluctuations in the task backlog, relevant calculations are performed according to the fluctuations to determine whether resource adjustment is required, and it has an immediate response to the traffic change situation.

[0130] At the same time, to facilitate understanding of the overall framework of the embodiment of the present disclosure, an overall framework diagram of data processing is provided, as shown in Figure 6 wherein, the example of processing data in a Flink stream is used for illustration. If a certain operator or a certain type of operator for real-time data processing consumes a relatively large amount of computing resources and belongs to a CPU-intensive operator, or consumes a relatively large amount of memory resources and has a relatively large state quantity and belongs to a memory-intensive task, then due to the lack of support for "fine-grained resource management", all operators can only use the same CPU and memory resources. According to the "barrel principle", the overall performance of the Flink job depends on the performance of the slowest operator in the job. Therefore, in order to enable the operator with the largest resource consumption to also meet the performance requirements, the user has to configure resources according to the largest resource consumption, thus implicitly causing waste of resources.

[0131] The method provided by the present disclosure can achieve the goal of resource conservation without affecting stability. It can predict the recommended resource allocation for a period of time in the future through a resource dynamic prediction algorithm, and achieve the self-start of tasks through elastic adjustment. Without the user's awareness, it can complete the dynamic scaling of resources, and realize the operation guarantee and resource conservation of tasks.

[0132] Therefore, the present solution has the following beneficial effects:

[0133] 1. The present disclosure can immediately respond to and adjust the resource consumption of the target task by obtaining the current resource consumption information of the target task in real time, estimating the load of the target task based on the current resource consumption information, and dynamically adjusting the resources of the target task according to the load, reducing the lag of resource adjustment.

[0134] 2. The present disclosure supports users to customize the upper and lower bounds of the concurrency of resource elastic adjustment, flexibly control the resource adjustment range, supports users to customize the resource dynamic adjustment algorithm, and the service itself also provides a set of basic algorithms to provide basic resource dynamic adjustment solutions. Users can customize and access algorithms to override the basic algorithms, with user-configured algorithms taking precedence. When users do not configure algorithms, they are routed to the basic algorithms for adjustment.

[0135] 3. The present disclosure uses the historical running rate of the target task as the processing performance index of the target task, without the need for basic training data, and can be commonly used for the same type of jobs, reducing the problem that the algorithm cannot be migrated due to the lack of basic data in the application process.

[0136] 4. The present disclosure can estimate the initial resource configuration based on the historical traffic data of the target task, solve the cold start problem, and avoid job failure caused by improper resource configuration when the task is started for the first time.

[0137] 5. The present disclosure estimates the load of the target task by obtaining the current traffic situation of the target task in real time and combining the processing performance of the target task. When it is found that there is a backlog in the processing of the target task, the latest resource configuration is calculated and a request is made to the resource manager to allocate more resources to cope with the peak traffic and improve the stability of the target task; when it is found that there is no backlog and the traffic is small for the target task, an attempt is made to release some resources to improve resource utilization and reduce costs; when there are fluctuations in the task backlog, relevant calculations are performed according to the fluctuations to determine whether resource adjustment is required, and it has an immediate response to traffic changes.

[0138] 6. The present disclosure sets an upper limit value for the concurrency, ensuring that when the target task is expanded, the maximum resource limit is restricted. When the concurrency is too large, an alarm can be issued to notify the user to check the rationality of the target task, rather than unrestrictedly increasing resources, which affects the overall resources of the cluster.

[0139] Corresponding to the above data processing method, the present invention also provides a data processing device. Since the device embodiment of the present invention corresponds to the above method embodiment, for the details not disclosed in the device embodiment, reference may be made to the above method embodiment, and no further elaboration will be made in the present invention.

[0140] Figure 7 As shown in the structure diagram of a data processing device provided by an embodiment of the present disclosure, Figure 7 as shown, the device includes:

[0141] An acquisition unit 71, configured to acquire the current resource consumption information of the target task when it is determined that the target task starts a preset adjustment strategy; wherein, the preset adjustment strategy is used to indicate dynamic adjustment of the task concurrency and task memory of the target task;

[0142] An estimation unit 72, configured to estimate the load of the target task according to the current resource consumption information;

[0143] An adjustment unit 73, configured to dynamically adjust the task concurrency and / or task memory of the target task according to the preset adjustment strategy when it is determined that the load meets the preset adjustment threshold in the preset adjustment strategy.

[0144] In summary, according to the data processing device proposed by the present disclosure, by acquiring multiple input images of multiple channels; marking the multiple input images to determine multiple marked images; determining a superimposed image based on the multiple marked images; and determining an output image based on the superimposed image and the multiple marked images. This solution marks multiple input images to determine a superimposed image and multiple marked images, so that a set of data processing engines can be used to uniformly process multi-layer images, reducing the hardware cost and device power consumption. At the same time, by setting different enhancement effects and enhancement levels, the effect superimposition of multiple layers can be adaptively completed, improving the data processing efficiency.

[0145] Further, in a possible implementation manner of an embodiment of the present disclosure, Figure 8 as shown, the adjustment unit 73 includes:

[0146] A first adjustment module 731, configured to dynamically adjust the task concurrency and / or task memory of the target task by using a first preset adjustment strategy;

[0147] A second adjustment module 732, configured to dynamically adjust the task concurrency and / or task memory of the target task by using a second preset adjustment strategy, and the priority of the first preset adjustment strategy is higher than that of the second preset adjustment strategy.

[0148] Further, in a possible implementation manner of the embodiments of the present disclosure, as Figure 8 shown, the obtaining unit 71 includes:

[0149] An obtaining module 711, configured to obtain partition information of a processing data source corresponding to the target task;

[0150] A determining module 712, configured to determine average traffic information of each partition, data processing backlog delay information of each partition, task memory usage information, and task processing performance of each partition; the current resource consumption information at least includes the average traffic information of each partition, the data processing backlog delay information of each partition, the task memory usage information, and the task processing performance of each partition;

[0151] A recording module 713, configured to record the data processing backlog delay information of each partition in sequence.

[0152] Further, in a possible implementation manner of the embodiments of the present disclosure, the first adjustment module 731 is further configured to:

[0153] Determine whether the data processing backlog delay information exceeds a first concurrency adjustment threshold; the preset adjustment thresholds include a first concurrency adjustment threshold, a second concurrency adjustment threshold, a first memory adjustment threshold, and a second memory adjustment threshold;

[0154] When it is determined that the data processing backlog delay information exceeds the first concurrency adjustment threshold, estimate a first target concurrency according to the number of partitions, the average traffic information of each partition, and the task processing performance of each partition, where the first target concurrency is not higher than the smaller value of the custom concurrency upper bound and the number of partitions, and the number of partitions is determined according to the partition information of the processing data source;

[0155] Determine whether the task memory usage information is higher than the first memory adjustment threshold or the task memory usage information is lower than the second memory adjustment threshold;

[0156] Calculate the target task memory according to the average memory usage and the maximum memory configuration.

[0157] Further, in a possible implementation manner of the embodiments of the present disclosure, the first adjustment module 731 is further configured to:

[0158] When it is determined that the data processing backlog delay information does not exceed the first concurrency adjustment threshold, determine the backlog change trend in the backlog delay information queue, where the data processing backlog delay information of each partition is recorded in the backlog delay information queue;

[0159] When it is determined that the backlog change trend is continuously rising, adjust it to the second target concurrency degree according to the first preset step size, and the second target concurrency degree does not exceed the smaller value of the custom concurrency degree upper bound and the number of partitions.

[0160] Further, in a possible implementation manner of the embodiments of the present disclosure, the first adjustment module 731 is further configured to:

[0161] When it is determined that the data processing backlog delay information does not exceed the first concurrency degree adjustment threshold and the backlog change trend is not continuously rising, determine whether the data processing backlog delay information is less than the second concurrency degree adjustment threshold; the second concurrency degree adjustment threshold is less than the first concurrency degree adjustment threshold;

[0162] When it is determined that the data processing backlog delay information is less than the second concurrency degree adjustment threshold and the first target concurrency degree is greater than or equal to the preset concurrency degree threshold, adjust the lower bound of the concurrency degree of the target task down to the third target concurrency degree according to the second preset step size, and the third target concurrency degree is not lower than the custom concurrency degree lower bound, and the preset concurrency degree threshold is determined according to the number of partitions.

[0163] Further, in a possible implementation manner of the embodiments of the present disclosure, the second adjustment module 732 is further configured to:

[0164] Call a preset concurrency degree algorithm to calculate the fourth target concurrency degree, and determine the smaller value from the fourth target concurrency degree and the third concurrency degree;

[0165] Determine the concurrency degree corresponding to the smaller value as the concurrency degree of the target task.

[0166] In the above embodiments provided by the present application, the methods and devices provided by the embodiments of the present application are introduced. To implement each function in the methods provided by the embodiments of the present application, an electronic device may include a hardware structure and software modules, and implement the above functions in the form of a hardware structure, software modules, or a combination of a hardware structure and software modules. A certain function among the above functions may be executed in the form of a hardware structure, software module, or a combination of a hardware structure and software module.

[0167] Figure 9 It is a block diagram of an electronic device 900 for implementing the above data processing method shown according to an exemplary embodiment. For example, the electronic device 900 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0168] Refer to Figure 9, the electronic device 900 may include one or more of the following components: a processing component 902, a memory 904, a power component 906, a multimedia component 908, an audio component 910, an input / output (I / O) interface 912, a sensor component 914, and a communication component 916.

[0169] The processing component 902 generally controls the overall operation of the electronic device 900, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above methods. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0170] The memory 904 is configured to store various types of data to support the operation of the electronic device 900. Examples of these data include instructions for any application or method operating on the electronic device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disk.

[0171] The power component 906 provides power to the various components of the electronic device 900. The power component 906 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the electronic device 900.

[0172] The multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0173] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 900 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 904 or transmitted via the communication component 916. In some embodiments, the audio component 910 further includes a speaker for outputting audio signals.

[0174] The I / O interface 912 provides an interface between the processing component 902 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to: a home button, a volume button, a power button, and a lock button.

[0175] The sensor component 914 includes one or more sensors for providing an assessment of the various aspects of the status of the electronic device 900. For example, the sensor component 914 can detect the on / off state of the electronic device 900, the relative positioning of components, such as the display and keypad of the electronic device 900. The sensor component 914 can also detect a change in the position of the electronic device 900 or a component of the electronic device 900, the presence or absence of user contact with the electronic device 900, the orientation or acceleration / deceleration of the electronic device 900, and the temperature change of the electronic device 900. The sensor component 914 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 914 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 914 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0176] The communication component 916 is configured to facilitate communication between the electronic device 900 and other devices in a wired or wireless manner. The electronic device 900 can access a communication standard-based wireless network, such as WiFi, 2G or 3G, 4G LTE, 5G NR (New Radio), or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0177] In an exemplary embodiment, the electronic device 900 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0178] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, and the above instructions can be executed by a processor 920 of the electronic device 900 to complete the above method by processing data. For example, the non-transitory computer-readable storage medium can be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0179] Embodiments of the present disclosure also propose a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method described in the above embodiments of the present disclosure.

[0180] For the case where the electronic device can be a chip or a chip system, reference can be made to Figure 10 the structural schematic diagram of the chip shown. Figure 10 The chip shown includes a processor 1001 and an interface 1002. Among them, the number of processors 1001 can be one or more, and the number of interfaces 1002 can be multiple.

[0181] Optionally, the chip further includes a memory 1003, and the memory 1003 is used to store necessary computer programs and data.

[0182] Those skilled in the art can also understand that the various illustrative logical blocks and steps listed in the embodiments of the present application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functions are implemented by hardware or software depends on the specific application and the design requirements of the entire system. For each specific application, those skilled in the art can use various methods to implement the functions, but such implementation should not be construed as exceeding the scope protected by the embodiments of the present application.

[0183] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0184] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples", etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic descriptions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0185] Any process or method description in a flowchart or described in other ways herein can be understood to represent a module, segment, or part of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations, where the functions can be executed in a manner other than shown or discussed, including in a substantially simultaneous manner according to the functions involved or in a reverse order, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0186] The logic and / or steps represented in the flowchart or otherwise described herein can, for example, be considered as a defined sequence list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in conjunction with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection portion having one or more wirings (control method), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable medium on which a program can be printed, as the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpretation, or otherwise processing as appropriate, and then storing it in a computer memory.

[0187] It should be understood that each part of the embodiments of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0188] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the methods of the above embodiments can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0189] In addition, each functional unit in various embodiments of the present invention may be integrated into a processing module, may exist physically alone for each unit, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, or the like.

[0190] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for processing data, characterized in that, The method includes: When it is determined that a preset adjustment strategy is to be started for the target task, obtaining the current resource consumption information of the target task; wherein, the preset adjustment strategy is used to indicate dynamic adjustment of the task concurrency degree and task memory of the target task; Estimating the load of the target task according to the current resource consumption information; When it is determined that the load meets the preset adjustment threshold in the preset adjustment strategy, dynamically adjusting the task concurrency degree and / or task memory of the target task according to the preset adjustment strategy.

2. The method according to claim 1, wherein The dynamically adjusting the task concurrency degree and / or task memory of the target task according to the preset adjustment strategy includes: Dynamically adjusting the task concurrency degree and / or task memory of the target task by using a first preset adjustment strategy; Or, dynamically adjusting the task concurrency degree and / or task memory of the target task by using a second preset adjustment strategy, and the priority of the first preset adjustment strategy is higher than that of the second preset adjustment strategy.

3. The method according to claim 2, wherein The obtaining the current resource consumption information of the target task includes: Obtaining the partition information of the processing data source corresponding to the target task; Determining the average traffic information of each partition, the data processing backlog delay information of each partition, the task memory usage information, and the task processing performance of each partition; the current resource consumption information at least includes the average traffic information of each partition, the data processing backlog delay information of each partition, the task memory usage information, and the task processing performance of each partition; Sequentially recording the data processing backlog delay information of each partition.

4. The method according to claim 3, characterized in that The dynamically adjusting the task concurrency degree and / or task memory of the target task by using a first preset adjustment strategy includes: Judging whether the data processing backlog delay information exceeds a first concurrency degree adjustment threshold; the preset adjustment threshold includes a first concurrency degree adjustment threshold, a second concurrency degree adjustment threshold, a first memory adjustment threshold, and a second memory adjustment threshold; When it is determined that the data processing backlog delay information exceeds the first concurrency degree adjustment threshold, estimating a first target concurrency degree according to the number of partitions, the average traffic information of each partition, and the task processing performance of each partition, and the first target concurrency degree is not higher than the smaller value of the custom concurrency degree upper bound and the number of partitions, and the number of partitions is determined according to the partition information of the processing data source; Judging whether the task memory usage information is higher than the first memory adjustment threshold, or the task memory usage information is lower than the second memory adjustment threshold; Calculating the target task memory according to the average memory usage and the maximum memory configuration.

5. The method according to claim 4, wherein The method further includes: When it is determined that the data processing backlog delay information does not exceed the first concurrency degree adjustment threshold, determining the backlog change trend in the backlog delay information queue, and the data processing backlog delay information of each partition is recorded in the backlog delay information queue; When it is determined that the backlog change trend is continuously rising, adjusting to a second target concurrency degree according to a first preset step size, and the second target concurrency degree does not exceed the smaller value of the custom concurrency degree upper bound and the number of partitions.

6. The method according to claim 4, wherein The method further includes: When it is determined that the data processing backlog delay information does not exceed the first concurrency adjustment threshold and the backlog change trend is not continuously increasing, determining whether the data processing backlog delay information is less than a second concurrency adjustment threshold; the second concurrency adjustment threshold is less than the first concurrency adjustment threshold; When it is determined that the data processing backlog delay information is less than the second concurrency adjustment threshold and the current concurrency of the target task is greater than or equal to a preset concurrency threshold, reducing the lower bound of the concurrency of the target task by a second preset step size to a third target concurrency, the third target concurrency is not lower than a custom lower bound of concurrency, and the preset concurrency threshold is determined according to the number of partitions.

7. The method according to claim 2, characterized in that, Dynamically adjusting the task concurrency and / or task memory of the target task by using a second preset adjustment strategy includes: Invoking a preset concurrency algorithm to calculate a fourth target concurrency, and determining the smaller value from the fourth target concurrency and the third concurrency; Determining the concurrency corresponding to the smaller value as the concurrency of the target task.

8. A data processing device, characterized in that, The apparatus includes: An acquisition unit, configured to acquire the current resource consumption information of the target task when it is determined that the target task starts a preset adjustment strategy; wherein, the preset adjustment strategy is used to indicate dynamic adjustment of the task concurrency and task memory of the target task; An estimation unit, configured to estimate the load of the target task according to the current resource consumption information; An adjustment unit, configured to, when it is determined that the load meets a preset adjustment threshold in the preset adjustment strategy, dynamically adjust the task concurrency and / or task memory of the target task according to the preset adjustment strategy.

9. An electronic device, characterized in that, Includes: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

11. A chip, characterized in that, Includes one or more interfaces and one or more processors; the interfaces are configured to receive signals from the memory of the electronic device and send the signals to the processors, the signals include computer instructions stored in the memory, and when the processors execute the computer instructions, the electronic device executes the method according to any one of claims 1-7.

Images