Method and device for elastically adjusting task processing parallelism

Abstract

This specification provides a method and device for elastically adjusting the parallelism of task processing. The method includes: obtaining delay information fed back by the task executor during the process of processing the shard storage data of the data source; comparing the delay information with a preset normal delay threshold range; and adjusting the parallelism of the task executor for processing the shard storage data when the comparison result shows that the delay of the task executor does not fall within the normal delay threshold range.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of real-time computing technology, and more particularly, to a method and device for elastically adjusting task processing parallelism. Background Art

[0002] Real-time computing in the field of cloud computing is a general service with a large number of customers, complex usage scenarios, and a wide variety of problems to solve, resulting in large differences in the computational complexity of different jobs. Among them, Flink, KafkaStreaming, and Rocketmq Streaming are the most widely used traditional real-time computing systems.

[0003] In the related art, when providing real-time computing services based on the above-mentioned traditional real-time computing system, it is necessary to estimate the data throughput in advance, determine the number of shards when storing data shards based on the estimated data throughput, and bind the number of shards to the parallelism of task processing in the task executor. However, when the actual data volume far exceeds the estimated throughput, the data cannot be processed in time due to the limitation of parallelism, resulting in data processing delays. When the actual data volume is lower than the estimated throughput, although there is no delay, it will inevitably cause a waste of task executor resources. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a method and device for flexibly adjusting task processing parallelism.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a method for elastically adjusting task processing parallelism is provided, the method comprising:

[0007] In the process of processing the shard storage data of the data source by the task executor, obtaining the delay information fed back by the task executor;

[0008] Comparing the delay information with a preset normal delay threshold range;

[0009] When the comparison result shows that the delay of the task executor does not fall within the normal delay threshold range, the parallelism of the task executor in processing the sharded storage data is adjusted.

[0010] According to a second aspect of one or more embodiments of this specification, a device for elastically adjusting task processing parallelism is provided, the device comprising:

[0011] An acquisition unit, used for acquiring delay information fed back by the task executor during the process of the task executor processing the shard storage data of the data source;

[0012] A comparison unit, configured to compare the delay information with a preset normal delay threshold range;

[0013] The adjustment unit adjusts the parallelism of the task executor in processing the sharded storage data when the comparison result shows that the delay of the task executor does not fall within the normal delay threshold range.

[0014] According to a third aspect of one or more embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0015] According to a fourth aspect of one or more embodiments of this specification, an electronic device is proposed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification, and for ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0017] Figure 1 is a system architecture diagram of a real-time computing system provided by an exemplary embodiment of this specification;

[0018] Figure 2 is a flow chart of a method for elastically adjusting task processing parallelism provided by an exemplary embodiment of this specification;

[0019] Figure 3 It is a schematic diagram of a real-time computing system scheduling task processing parallelism provided by an exemplary embodiment of this specification;

[0020] Figure 4 is a schematic structural diagram of an electronic device provided by an exemplary embodiment of this specification;

[0021] Figure 5 It is a block diagram of a device for elastically adjusting task processing parallelism provided by an exemplary embodiment of this specification. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0023] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0024] Real-time computing in the field of cloud computing is a general service with a large number of customers, complex usage scenarios, and a wide variety of problems to solve, resulting in large differences in the computational complexity of different jobs. Among them, Flink, KafkaStreaming, and Rocketmq Streaming are widely used traditional real-time computing systems. In related technologies, when providing real-time computing services based on the above-mentioned traditional real-time computing systems, it is necessary to estimate the data throughput in advance, determine the number of shards when storing data shards based on the estimated data throughput, and bind the number of shards to the parallelism of task processing in the task executor. In other words, the parallelism of the task executor in the real-time computing system is consistent with the size of the number of shards. When the actual data volume far exceeds the estimated throughput, the data cannot be processed in time due to the limitation of parallelism, resulting in data processing delays. When the actual data volume is lower than the estimated throughput, although the data can be processed in time, it is inevitable that the resources of the task executor will be wasted.

[0025] In view of this, this specification provides a method for flexibly adjusting the parallelism of task processing, which flexibly adjusts the parallelism within the task executor based on the delay information fed back by the task executor, thereby ensuring the timeliness of data processing while also ensuring the rationality of resource allocation within the real-time computing system.

[0026] Figure 1 FIG. 1 is a schematic diagram of the architecture of a real-time computing system provided by an exemplary embodiment of this specification. Figure 1 As shown, the system architecture may include a client 101 , a task coordinator 102 , a task executor 103 and a shard storage 104 .

[0027] The client 101 can interact with the task coordinator 102, and then configure the task execution logic used by the task executor 103 through the task coordinator 102, so that the task executor 103 can process the shard storage data provided by the shard storage 104 based on the actual needs of the user. Among them, the client 101 can be deployed on any type of electronic device used by the user, for example, the electronic device can include a tablet device, a laptop computer, a PDA (Personal Digital Assistants), a wearable device (such as smart glasses, smart watches, etc.), etc., and this specification does not limit this.

[0028] The task coordinator 102, the task executor 103 and the shard storage 104 can be deployed on the same subject, or any two of them can be deployed on the same subject, or the three can be deployed on different subjects, and this specification does not limit this. The above subjects deployed by the task coordinator 102, the task executor 103 and the shard storage 104 can include a physical server of an independent host, or a virtual server (such as a cloud server) hosted by a host cluster, and this specification does not limit this.

[0029] Among them, the shard storage 104 is used to store the data from the data source in shards to obtain the shard storage data. The task executor 103 is used to process the above-mentioned shard storage data stored in the shard storage 104. The task coordinator 102 is used to coordinate the above-mentioned data processing process: on the one hand, the task coordinator 102 can interact with the client 101 as described above, and configure the task execution logic adopted by the task executor 103 based on the needs of the client 101; on the other hand, the task coordinator 102 can obtain the delay information fed back by the task executor 103, and flexibly adjust the parallelism of the task processing in the task executor 103 accordingly. When the delay is greater than the upper limit of the normal delay threshold range, the parallelism of the task processing is quickly increased so that the task executor 103 speeds up the data processing speed and ensures that the delay of the task processing of the task executor 103 is within the control time limit; at the same time, when the delay is less than the lower limit of the normal delay threshold range, the parallelism of the task processing is slowly reduced, and the system resources occupied by the task executor 103 are released on the premise of ensuring the delay, so as to ensure the reasonable allocation of the resources of the task executor 103. In other words, it ensures the timeliness of data processing while also ensuring the rationality of resource allocation in the real-time computing system.

[0030] Based on the above real-time computing system architecture, it is possible to achieve flexible adjustment of the parallelism of task processing. When the delay is greater than the upper limit of the normal delay threshold range, the parallelism of task processing is increased to enable the task executor to speed up data processing and ensure that the delay of task processing by the task executor is within the control time limit; at the same time, when the delay is less than the lower limit of the normal delay threshold range, the parallelism of task processing is reduced. On the premise of ensuring the delay, the over-occupied system resources are released to ensure the reasonable allocation of real-time computing system resources.

[0031] Figure 2 The present invention is a flowchart of a method for elastically adjusting the parallelism of task processing provided by an exemplary embodiment, which is applied to a task scheduler of a real-time computing system. The method comprises the following steps:

[0032] Step 201: During the process of processing the sharded storage data of the data source by the task executor, delay information fed back by the task executor is obtained.

[0033] The task executor can process the shard storage data according to the pre-configured task execution logic. The task execution logic can be the default logic or configured by the user according to actual needs. For example Figure 3 As shown, the user can provide the written DSL (Domain-Specific Language) file to the task coordinator 102 through the client 101, so that the task coordinator 102 generates a corresponding executable program based on the content of the DSL file. It can be understood by those skilled in the art that: although described as the client 101 in the above text, the technical solution of this specification is not limited to the C / S (Client / Server) architecture, and the user can also interact with the task coordinator 102 based on the browser in the B / S (Brower / Server) architecture.

[0034] In one embodiment, after obtaining the DSL file sent by the client 101, the task coordinator 102 can construct the original AST (Abstract Syntax Tree) through the Antlr4 (Another Tool for Language Recognition) tool. In the process of constructing the AST, the content of the DSL file can be subjected to lexical analysis and semantic verification, such as whether the parameter types of the function meet the requirements, whether the number of functions meets the requirements, etc. Then, the task coordinator 102 converts the AST into a logical execution plan, and further optimizes the logical execution plan through the optimizer, thereby improving the execution efficiency. Finally, the task coordinator 102 translates the logical execution plan into program code and packages it into an executable program, which includes the task execution logic described above. At this point, the task coordinator 102 can configure the executable program to Figure 3 The task executors 203-204 etc. shown can ensure that these task executors 203-204 etc. process the shard storage data according to the user's needs.

[0035] Each task executor can further contain one or more shard executors. The parallelism referred to in this specification can be regarded as the total number of shard executors, for example Figure 3 As shown, there are two shard executors in task executor 203 and task executor 204 respectively, and the parallelism at this time can be considered to be 4. Accordingly, adjusting the parallelism of task processing can be regarded as adjusting the number of shard executors. Among them, the task executor can be a computer process, and the shard executors contained in the task executor can be regarded as computer threads contained in the corresponding process.

[0036] The adjustment of parallelism can be achieved by the task scheduler. The task scheduler can be configured in Figure 3 In the task coordinator 102 shown in FIG. 102, it is assumed that at the beginning: there are only Figure 3The task executor 203 shown in the figure includes a shard executor 1. The task scheduler can adjust the number of shard executors in the task executor 203. For example, if the parallelism needs to be increased to 2, the task scheduler can add shard executor 2 to the task executor 203. At this point, because the task executor 203 includes shard executor 1 and shard executor 2, the parallelism is increased and the magnitude of the parallelism is 2. When the task scheduler wants to increase the parallelism by adding shard executors to the task executor 203, but if the remaining threads of the task executor 203 are insufficient, the task scheduler can increase the parallelism by adding a new task executor 204, thereby increasing the number of shard executors in the task executor 204. Correspondingly, when the task scheduler wants to reduce the parallelism, it can reduce the number of shard executors of any task executor to achieve the purpose of reducing the parallelism. The number of shard executors reduced in the task executor is consistent with the magnitude of the reduced parallelism. Of course, if the shard executors in any task executor do not exist for the purpose of reducing parallelism, the task executor can also be deleted in time to reduce the consumption of system resources and ensure the reasonable allocation of system resources.

[0037] For example, when the number of shards for data source shard storage is 2, Figure 3 Shard0 and shard1 shown in the figure have a current parallelism of 2, that is, the number of shard executors in task executor 203 is 2, namely, shard executor 1 and shard executor 2. If the parallelism needs to be increased, but task executor 203 cannot add the required number of shard executors, task executor 204 can be added, and corresponding shard executors, namely, shard executor 3 and shard executor 4, can be configured in task executor 204, so that the parallelism of task processing by task executor is increased to 4. Task executors can obtain executable programs issued by task schedulers so that shard executors can process shard storage data based on the executable programs. Correspondingly, if the current parallelism 4 needs to be reduced to 2, the task scheduler can delete shard executor 3 and shard executor 4 in task executor 204. At this time, all shard executors in task executor 204 have been deleted. Then task executor 204 can be further deleted, and only task executor 203 exists, and shard executor 1 and shard executor 2 exist in task executor 203. At this point, the reduction of parallelism is completed and the reduction of parallelism to 2 is achieved.

[0038] The sharded data can be obtained by sharding the data provided by the data source. The real-time computing system can estimate the data throughput in advance, determine the number of shards based on the data throughput, and then use the number of shards to shard the data from the data source. For example Figure 3As shown, when the number of shards determined by the real-time computing system is 2, the shard storage 104 stores the data from the data source as shard0 and shard1, a total of two shard storage data. Further, each shard storage data can include a number of data segments, or in other words, the data in the shard is stored in the form of data segments, which can be, for example, Figure 3 The segment shown. There may be multiple conditions for the division of data segments, for example: the condition may be whether the amount of data provided by the current data source reaches the preset amount of data, for example, assuming the preset amount of data is 128M, then whenever the amount of data received reaches 128M, a new segment may be generated, without considering the reception time of the 128M data; another example: the condition may be whether the current storage time reaches the preset time, for example, assuming the preset time is 1 second, then a new segment may be generated every 1 second, without considering the amount of data received in the 1 second. For another example, the above-mentioned conditions of data volume and time may be combined, and a new data segment may be generated as long as any one of them is met, for example, assuming the preset data volume is 128M and the preset time is 1 second, if the amount of data received at 0.5 seconds has reached 128M, then it may be divided into a segment accordingly, without having to continue waiting; if the amount of data received within 1 second does not reach 128M, then it may also be divided into a segment accordingly.

[0039] As mentioned above, the task scheduler can adjust the parallelism of the task executor so that the adjusted parallelism matches the number of shards of the sharded data storage, rather than setting the values ​​of parallelism and the number of shards uniformly based on the estimated data throughput. Therefore, the values ​​of parallelism and the number of shards are not necessarily equal, and there may often be differences in the values ​​of the two. Then, when scheduling the task executor, the task scheduler can determine the scheduling granularity based on the values ​​of parallelism and the number of shards. For example, when the degree of parallelism of the task executor in processing the sharded storage data is not greater than the number of shards of the sharded storage data of the data source, the task scheduler schedules the task executor to process the sharded storage data of the data source in units of shards, so that all data of the same shard is processed by the same shard executor, avoiding the splitting of data segments within the same shard, and reducing the complexity of scheduling; when the degree of parallelism of the task executor in processing the sharded storage data is greater than the number of shards of the sharded storage data of the data source, the task executor is scheduled to process the sharded storage data of the data source in units of data segments, and the data segment is the data storage unit contained in each shard storage data, so that parallel processing can be achieved for the data segments within the same shard, so that the degree of parallelism breaks through the limit of the number of shards, so that the degree of parallelism can be greater than the number of shards, further improving the speed of data processing, and making the delay of task processing fall within the normal delay threshold as soon as possible.

[0040] When the task scheduler adjusts the parallelism of the task executor, the adjustment is based on the delay information generated by the task executor when processing the shard storage data. Figure 3 As shown, task executors 203-204 respectively feed back delay information to the task scheduler, and the delay information includes the delays corresponding to slice executors 1 and 2 fed back by task executor 203, and the delays corresponding to slice executors 3 and 4 fed back by task executor 204. The delay information fed back by each task executor can be the delay of each slice executor contained in the task executor, such as the real-time delay of the corresponding slice executor, or the average delay of the corresponding slice executor in the most recent feedback cycle, or other types of delay information, which is not limited in this specification; or, when a task executor contains multiple slice executors, the delay information can be represented as the average delay of multiple slice executors, or can be represented as the maximum delay of multiple slice executors, etc., which is not limited in this specification.

[0041] In one embodiment, the delay information can be periodically fed back by the task executor according to a preset feedback duration, and if the task scheduler does not receive the delay information within a preset abnormal duration, it is determined that the task executor is abnormal, and the preset abnormal duration is greater than the preset feedback duration. For example, the task executor feeds back the delay information to the task scheduler every 1 millisecond, and assuming that the task scheduler does not receive the delay information within a preset abnormal duration of 2 milliseconds, the task scheduler determines that the task executor is abnormal, and an abnormal alarm message can be generated at this time to notify personnel to troubleshoot. By periodically feeding back the delay information, the delay information can be regarded as a heartbeat credential, thereby realizing health monitoring of the task executor, thereby ensuring the normal operation of the task executor or promptly discovering the abnormality of the task executor.

[0042] Step 202: Compare the delay information with a preset normal delay threshold range.

[0043] The delay information indicates the delay of the current task executor in processing data, and compares the delay information with the normal delay threshold range. Of course, the normal delay threshold range can be flexibly adjusted according to functional requirements. For example, assuming that a faster response to data is required in the current scenario, the normal delay threshold range can be set to 1 millisecond to 3 milliseconds; assuming that a faster response to data is not required in the current scenario, the normal delay threshold range can be set to 10 milliseconds to 20 milliseconds. The preset normal delay threshold range may also only include the upper threshold limit or only include the lower threshold limit. For example, the normal delay threshold range is set to less than 3 milliseconds, or the normal delay threshold range is set to greater than 3 milliseconds, etc. This specification does not limit this.

[0044] Step 203: When the comparison result indicates that the delay of the task executor does not fall within the normal delay threshold range, adjust the parallelism of the task executor in processing the sharded storage data.

[0045] As mentioned above, the normal delay threshold range can be flexibly adjusted according to functional requirements, and when the comparison results show that the delay of the task executor does not fall within the normal delay threshold, the parallelism of the task executor for processing the sharded storage data can be adjusted. By adjusting the parallelism, the delay of the task executor can fall within the normal delay threshold, thereby ensuring the timeliness of data processing while also ensuring the rationality of resource allocation of the real-time computing system.

[0046] In one embodiment, when the comparison result shows that the delay of the task executor is greater than the upper threshold limit of the preset normal delay threshold range, the parallelism is increased at a relatively large ratio; when the comparison result shows that the delay of the task executor is less than the lower threshold limit of the preset normal delay threshold range, the parallelism is reduced at a relatively small ratio. For example, if the comparison result shows that the delay of the task executor is 5 milliseconds, then the delay is greater than the upper threshold limit of the above-mentioned normal delay threshold range (1 millisecond to 3 milliseconds), and the parallelism can be increased at a relatively large ratio. For another example, if the comparison result shows that the delay of the task executor is 0.5 milliseconds, then the delay is less than the upper threshold limit of the above-mentioned normal delay threshold range (1 millisecond to 3 milliseconds), and the parallelism can be reduced at a relatively small ratio. Increasing the parallelism by a relatively large ratio and reducing the parallelism by a relatively small ratio, that is, quickly increasing and slowly reducing the parallelism. The reason for adopting the method of quickly increasing the degree of parallelism and slowly decreasing the degree of parallelism is that the actual data throughput fluctuates. If the actual data throughput increases, it needs to be increased quickly to process the data in time, shorten the time for adjusting the degree of parallelism, and reduce the impact on data processing. However, when the data throughput decreases, it is impossible to determine whether the subsequent data throughput will still increase. For example, after reducing the degree of parallelism now, the actual data throughput may increase again, then the degree of parallelism needs to be increased. Therefore, if the degree of parallelism is reduced quickly, then when the subsequent data throughput increases, the degree of parallelism needs to be increased immediately, resulting in repeated changes in the degree of parallelism and frequent scheduling. Therefore, the method of quickly increasing the degree of parallelism and slowly decreasing the degree of parallelism can effectively avoid the above-mentioned situation of repeatedly adjusting the degree of parallelism, so that the degree of parallelism falls within the normal threshold range as soon as possible.

[0047] In one embodiment, when the comparison result shows that the delay of the task executor does not fall within the normal delay threshold range, the size relationship between the current value of the parallelism and the number of shards storing the data is determined, the adjustment ratio for the parallelism is determined based on the size relationship, and the parallelism is adjusted according to the determined ratio. In other words, when increasing or decreasing the parallelism, the size of the number of shards can be considered at the same time, and the adjustment ratio is determined based on the number of shards, so that the parallelism is adjusted according to the determined adjustment ratio. As mentioned above, the need for adjustment of the parallelism can be divided into two situations: the first is that the delay of the task executor is greater than the upper threshold limit of the preset normal delay threshold range, and the second is that the delay of the task executor is less than the lower threshold limit of the preset normal delay threshold range. The following describes how to determine the adjustment ratio in these two cases, thereby determining the size of the parallelism.

[0048] When the delay of the task executor is greater than the upper threshold limit of the preset normal delay threshold range, determine the size relationship between the current value of the parallelism and the number of shards of the shard storage data. If the current value of the parallelism is less than the number of shards, determine the estimated value of the parallelism after increasing according to the first preset ratio: when the estimated value exceeds the number of shards, determine the adjustment ratio to increase the value of the parallelism to the number of shards. When the estimated value does not exceed the number of shards, determine the adjustment ratio as the first preset ratio. For example: if the number of shards of the shard storage data of the data source is 4 (i.e., shard1, shard2, shard3 and shard4 respectively), and the current value of the parallelism is 3, the first preset ratio is 2, then the estimated value after increasing according to the first preset ratio is 6, but at this time the size of the estimated value 6 has exceeded the number of shards 4, then the adjustment ratio can be changed to 4 / 3, and the parallelism obtained according to the adjustment ratio is 4, so based on the adjustment ratio, the parallelism of the task executor for processing the shard storage data can be adjusted to 4. As another example: if the number of shards of the sharded storage data of the data source is 4 (i.e., shard1, shard2, shard3, and shard4), and the current value of the parallelism is 2, and the first preset ratio is 2, then because the estimated value obtained according to the first preset ratio is 4, which does not exceed the number of shards, the adjustment ratio can be determined as the first preset ratio, and the parallelism of the task executor for processing the sharded storage data can be adjusted to 4. If the current value of the parallelism is not less than the number of shards of the sharded storage data of the data source, the adjustment ratio is determined to be the first preset ratio. Exemplary: the parallelism of the current task processing is 4, and the number of shards is 3. Because the current value of the parallelism is not less than the number of shards of the sharded storage data of the data source, the adjustment ratio can be directly determined as the first preset ratio of 2, and the parallelism is adjusted to 8.

[0049] Of course, the above adjustment ratio can also be a constant value and does not change dynamically according to the number of shards. In the case where the delay of the task executor is greater than the upper threshold limit of the preset normal delay threshold range, if the current value of the parallelism is less than the number of shards, the adjustment ratio is determined to be constant as the preset value. When the calculated value exceeds the number of shards, the parallelism is increased to the number of shards. When the calculated value does not exceed the number of shards, the number of shards is increased according to the fixed adjustment ratio. For example: the adjustment ratio is constant at 2. In the case where the delay of the task executor is greater than the upper threshold limit of the preset normal delay threshold range, the parallelism needs to be increased. If the current parallelism is less than the number of shards, the current parallelism is expanded by 2 times according to the adjustment ratio. Assuming that the number of shards is 4 and the current parallelism is 2, the parallelism is adjusted to 4; for another example: the adjustment ratio is constant at 2. In the case where the delay of the task executor is greater than the upper threshold limit of the preset normal delay threshold range, the parallelism needs to be increased, but at this time the number of shards is 5 and the current parallelism is 3. Because the parallelism obtained according to the adjustment ratio is 6, which exceeds the number of shards, the parallelism can be directly adjusted to the size of the number of shards, that is, 5. It should be noted that no matter whether the adjustment ratio is dynamically changed or a constant preset value, the purpose of increasing the parallelism can be achieved, so this specification does not limit this.

[0050] When the delay of the task executor is less than the lower threshold limit of the preset normal delay threshold range, determine the size relationship between the current value of the parallelism and the number of shards of the sharded storage data. If the current value of the parallelism is greater than the number of shards, the adjustment ratio is determined to be the second preset ratio; if the current value of the parallelism is not greater than the number of shards of the sharded storage data of the data source, the adjustment ratio is determined to be the third preset ratio. Among them, the third preset ratio is less than the second preset ratio. For example: when the delay of the task executor is less than the lower threshold limit of the preset normal delay threshold range, it is necessary to reduce the parallelism. If the number of shards of the sharded storage data of the data source is 8, the current value of the parallelism is 4, and the size of the second preset ratio is 1 / 2. Because the current value of the parallelism is not greater than the number of shards, the adjustment ratio can be determined to be the second preset ratio, and the size of the parallelism is adjusted to 2. Another example: when the delay of the task executor is less than the lower threshold of the preset normal delay threshold range, it is necessary to reduce the degree of parallelism. If the number of shards of the sharded storage data of the data source is 4, the current value of the degree of parallelism is 8, and the size of the third preset ratio is 3 / 4, because the current value of the degree of parallelism is greater than the number of shards, it can be determined that the adjustment ratio is the third preset ratio, and the size of the parallelism is adjusted to 6. It should be noted that, although in this embodiment, the second preset ratio is 1 / 2 and the third preset ratio is 3 / 4, it can be understood that the values ​​are not fixed, but can be flexibly adjusted according to actual conditions, and this specification does not limit this. By determining the adjustment ratio based on the relationship between the number of shards and the degree of parallelism, the delay can be brought into the normal delay threshold range more quickly, thereby reducing the number of times the degree of parallelism is adjusted and shortening the time spent on adjusting the degree of parallelism.

[0051] It can be seen from the above embodiments that based on the elastic adjustment method of the task processing parallelism, the task scheduler can flexibly adjust the parallelism of task processing in the task executor based on the delay information fed back by the task executor. When the delay is greater than the upper limit of the normal delay threshold range, the parallelism of task processing is quickly increased so that the task executor speeds up data processing and ensures that the delay of task processing by the task executor is within the control time limit; at the same time, when the delay is less than the lower limit of the normal delay threshold range, the parallelism of task processing is slowly reduced. Under the premise of ensuring the delay, the system resources occupied by the task executor are released to ensure the reasonable allocation of resources of the real-time computing system. In other words, the elastic adjustment method of parallelism disclosed in this specification ensures the timeliness of data processing while also ensuring the rationality of resource allocation of the real-time computing system.

[0052] Corresponding to the above method embodiments, this specification also provides device embodiments.

[0053] Figure 4 This is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Please refer to Figure 4At the hardware level, the electronic device includes a processor 401, a network interface 402, a memory 403, a non-volatile memory 404, and an internal bus 405, and may also include hardware required for other functions. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it. Of course, in addition to the software implementation, this specification does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0054] Figure 5 is a block diagram of a device for implementing hot upgrade of a virtual switch provided by an exemplary embodiment. Figure 5 As shown, the device comprises:

[0055] The acquisition unit 501 is used to acquire the delay information fed back by the task executor during the process of the task executor processing the shard storage data of the data source;

[0056] A comparing unit 502, configured to compare the delay information with a preset normal delay threshold range;

[0057] The adjusting unit 503 is used to adjust the parallelism of the task executor for processing the sharded storage data when the comparison result shows that the delay of the task executor does not fall within the normal delay threshold range.

[0058] Optionally, the adjusting unit 503 is specifically configured to:

[0059] When the comparison result shows that the delay of the task executor is greater than the upper threshold limit of the preset normal delay threshold range, the parallelism is increased at a relatively large ratio; when the comparison result shows that the delay of the task executor is less than the lower threshold limit of the preset normal delay threshold range, the parallelism is reduced at a relatively small ratio.

[0060] Optionally, the adjusting unit 503 is specifically configured to:

[0061] When the comparison result shows that the delay of the task executor does not fall within the normal delay threshold range, determine the size relationship between the current value of the parallelism and the number of shards of the sharded data; determine the adjustment ratio for the parallelism based on the size relationship, and adjust the parallelism according to the determined ratio.

[0062] Optionally, the adjusting unit 503 is specifically configured to:

[0063] Determine that the comparison result indicates that the delay of the task executor is greater than the upper threshold limit of the preset normal delay threshold range; if the current value of the parallelism is less than the number of shards, determine the estimated value of the parallelism after increasing by a first preset ratio: when the estimated value exceeds the number of shards, determine the adjustment ratio to increase the value of the parallelism to the number of shards; when the estimated value does not exceed the number of shards, determine the adjustment ratio as the first preset ratio; if the current value of the parallelism is not less than the number of shards, determine the adjustment ratio as the first preset ratio.

[0064] Optionally, the adjusting unit 503 is specifically configured to:

[0065] Determine that the comparison result indicates that the delay of the task executor is less than the lower limit of the preset normal delay threshold range; if the current value of the parallelism is greater than the number of shards, determine the adjustment ratio as the second preset ratio; if the current value of the parallelism is not greater than the number of shards, determine the adjustment ratio as the third preset ratio.

[0066] Optionally, the device further comprises:

[0067] The scheduling unit 504 is used to schedule the task executor to process the sharded storage data of the data source in units of shards when the degree of parallelism of the task executor in processing the sharded storage data is not greater than the number of shards of the sharded storage data of the data source; and to schedule the task executor to process the sharded storage data of the data source in units of data segments when the degree of parallelism of the task executor in processing the sharded storage data is greater than the number of shards of the sharded storage data of the data source, wherein the data segments are data storage units contained in each sharded storage data.

[0068] Optionally, each data segment includes data provided by the data source within a preset time period, and the data contained in each data segment is no more than a preset data volume.

[0069] Optionally, the delay information is periodically fed back by the task executor according to a preset feedback duration.

[0070] Optionally, the device further comprises:

[0071] The determination unit 505 is configured to determine that an abnormality occurs in the task executor if the delay information is not received within a preset abnormality duration; wherein the preset abnormality duration is greater than the preset feedback duration.

[0072] Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the relevant method, and will not be elaborated here.

[0073] It can be seen from the technical solutions provided in the above specification that the method for elastically adjusting the parallelism of task processing described in the specification can enable the task scheduler to flexibly adjust the parallelism of task processing in the task executor based on the delay information fed back by the task executor. When the delay is greater than the upper limit of the normal delay threshold range, the parallelism of task processing is quickly increased so that the task executor can speed up data processing and ensure that the delay of task processing by the task executor is within the control time limit; at the same time, when the delay is less than the lower limit of the normal delay threshold range, the parallelism of task processing is slowly reduced. On the premise of ensuring the delay, the system resources occupied by the task executor are released to ensure the reasonable allocation of resources of the real-time computing system. In other words, the elastic adjustment method of parallelism disclosed in this specification ensures the timeliness of data processing while also ensuring the rationality of resource allocation of the real-time computing system.

[0074] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0075] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0076] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0077] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0078] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0079] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for elastically adjusting task processing parallelism, characterized in that: A task scheduler applied to a real-time computing system, the method comprising: In the process of processing the shard storage data of the data source by the task executor, obtaining the delay information fed back by the task executor; Compare the delay information with a preset normal delay threshold range, and if the comparison result shows that the delay of the task executor is greater than the upper threshold limit of the preset normal delay threshold range, determine the size relationship between the current value of the parallelism and the number of shards of the sharded storage data; Determine an adjustment ratio for the parallelism according to the size relationship, and adjust the parallelism according to the determined ratio; wherein, If the current value of the parallelism is less than the number of shards, determine the estimated value of the parallelism after increasing according to the first preset ratio; when the estimated value exceeds the number of shards, determine the adjustment ratio to increase the value of the parallelism to the number of shards; when the estimated value does not exceed the number of shards, determine the adjustment ratio to be the first preset ratio; If the current value of the parallelism is not less than the number of shards, the adjustment ratio is determined to be the first preset ratio.

2. The method according to claim 1, characterized in that Also includes: When the degree of parallelism of the task executor for processing the sharded storage data is not greater than the number of shards of the sharded storage data of the data source, scheduling the task executor to process the sharded storage data of the data source in units of shards; When the degree of parallelism of the task executor in processing the sharded storage data is greater than the number of shards of the sharded storage data of the data source, the task executor is scheduled to process the sharded storage data of the data source in units of data segments, where the data segments are data storage units contained in each sharded storage data.

3. The method according to claim 2, characterized in that Each data segment includes data provided by the data source within a preset time period, and the data contained in each data segment is no greater than a preset data volume.

4. The method according to claim 1, characterized in that: The delay information is periodically fed back by the task executor according to a preset feedback duration, and the method further includes: If the delay information is not received within a preset abnormal time period, it is determined that an abnormality occurs in the task executor; wherein the preset abnormal time period is greater than the preset feedback time period.

5. A method for elastically adjusting task processing parallelism, characterized in that: A task scheduler applied to a real-time computing system, the method comprising: In the process of processing the shard storage data of the data source by the task executor, obtaining the delay information fed back by the task executor; Compare the delay information with a preset normal delay threshold range, and if the comparison result shows that the delay of the task executor is less than the lower limit of the preset normal delay threshold range, determine the size relationship between the current value of the parallelism and the number of shards of the sharded storage data; Determine an adjustment ratio for the parallelism according to the size relationship, and adjust the parallelism according to the determined ratio; wherein, If the current value of the parallelism is greater than the number of shards, the adjustment ratio is determined to be a second preset ratio; If the current value of the parallelism is not greater than the number of shards, the adjustment ratio is determined to be a third preset ratio; wherein the third preset ratio is smaller than the second preset ratio.

6. The method according to claim 5, characterized in that Also includes: When the degree of parallelism of the task executor for processing the sharded storage data is not greater than the number of shards of the sharded storage data of the data source, scheduling the task executor to process the sharded storage data of the data source in units of shards; When the degree of parallelism of the task executor in processing the sharded storage data is greater than the number of shards of the sharded storage data of the data source, the task executor is scheduled to process the sharded storage data of the data source in units of data segments, where the data segments are data storage units contained in each sharded storage data.

7. The method according to claim 6, characterized in that Each data segment includes data provided by the data source within a preset time period, and the data contained in each data segment is no greater than a preset data volume.

8. The method according to claim 5, characterized in that The delay information is periodically fed back by the task executor according to a preset feedback duration, and the method further includes: If the delay information is not received within a preset abnormal time period, it is determined that an abnormality occurs in the task executor; wherein the preset abnormal time period is greater than the preset feedback time period.

9. A device for elastically adjusting task processing parallelism, characterized in that: A task scheduler applied to a real-time computing system, the device comprising: An acquisition unit, used for acquiring delay information fed back by the task executor during the process of the task executor processing the shard storage data of the data source; A comparison unit, used to compare the delay information with a preset normal delay threshold range, and determine the size relationship between the current value of the parallelism and the number of shards of the sharded storage data when the comparison result shows that the delay of the task executor is greater than the upper threshold limit of the preset normal delay threshold range; An adjustment unit is used to determine an adjustment ratio for the parallelism according to the size relationship, and adjust the parallelism according to the determined ratio; wherein, if the current value of the parallelism is less than the number of shards, the estimated value of the parallelism after increasing by a first preset ratio is determined: when the estimated value exceeds the number of shards, the adjustment ratio is determined to increase the value of the parallelism to the number of shards; when the estimated value does not exceed the number of shards, the adjustment ratio is determined to be the first preset ratio; if the current value of the parallelism is not less than the number of shards, the adjustment ratio is determined to be the first preset ratio.

10. A device for elastically adjusting task processing parallelism, characterized in that: A task scheduler applied to a real-time computing system, the device comprising: An acquisition unit, used for acquiring delay information fed back by the task executor during the process of the task executor processing the shard storage data of the data source; A comparison unit, used to compare the delay information with a preset normal delay threshold range, and determine the size relationship between the current value of the parallelism and the number of shards of the sharded storage data when the comparison result shows that the delay of the task executor is less than the lower limit of the threshold of the preset normal delay threshold range; An adjustment unit is used to determine an adjustment ratio for the parallelism according to the size relationship, and adjust the parallelism according to the determined ratio; wherein, if the current value of the parallelism is greater than the number of shards, the adjustment ratio is determined to be a second preset ratio; if the current value of the parallelism is not greater than the number of shards, the adjustment ratio is determined to be a third preset ratio; wherein the third preset ratio is smaller than the second preset ratio.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images