Resource scheduling method, resource scheduling device, storage medium and electronic equipment

Abstract

The present disclosure provides a resource scheduling method, a resource scheduling device, a computer readable storage medium and an electronic device, and belongs to the technical field of computers. The method comprises: acquiring performance monitoring data of a real-time processing task; analyzing performance data of each operator in the performance monitoring data to determine a target parallelism of each operator in the real-time processing task; and performing capacity expansion or capacity reduction processing on the real-time processing task according to the target parallelism of each operator. The present disclosure can realize elastic resource scaling at the operator level and improve data processing efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a resource scheduling method, a resource scheduling device, a computer-readable storage medium, and an electronic device. Background Technology

[0002] With the development of computer technology, the scale of various application systems and the amount of data generated are constantly increasing. In order to monitor the operation status of networks and application systems in real time and to mine the value of data, it is necessary to process the data generated by application systems in real time.

[0003] However, in real-time data processing, data productivity and other metrics change dynamically with business usage. If data processing resources are set to a fixed value, they may not be able to meet the data processing needs at certain times. For example, when data processing resources exceed the current data productivity resource requirements, the data processing task can be completed, but some processing resources will be wasted. When data processing resources are less than the current data productivity resource requirements, the application system may not be able to process all the data in a timely manner, and it may even cause the entire system to crash.

[0004] Therefore, there is a need to provide a method that can dynamically adjust data processing resources.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] This disclosure provides a resource scheduling method, a resource scheduling device, a computer-readable storage medium, and an electronic device, thereby at least partially improving the problem that the prior art cannot dynamically adjust data processing resources.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0008] According to a first aspect of this disclosure, a resource scheduling method is provided, the method comprising: acquiring performance monitoring data of a real-time processing task; analyzing the performance data of each operator in the performance monitoring data to determine the target parallelism of each operator in the real-time processing task; and scaling up or down the real-time processing task according to the target parallelism of each operator.

[0009] In one exemplary embodiment of this disclosure, the step of analyzing the performance data of each operator in the performance monitoring data to determine the target parallelism of each operator in the real-time processing task includes: determining whether the data processing rate of each operator in the real-time processing task matches the data processing rate of the source operator corresponding to each operator based on the performance monitoring data; when it is determined that the data processing rate of any operator does not match the data processing rate of the source operator corresponding to any operator, calculating the target parallelism of each operator in the real-time processing task based on the node relationship between the any operator and other operators in the real-time processing task and the performance data of the any operator.

[0010] In one exemplary embodiment of this disclosure, calculating the target parallelism of each operator in the real-time processing task based on the node relationship between the any operator and other operators in the real-time processing task and the performance data of the any operator includes: calculating the target processing rate of the any operator based on the performance data of the any operator; calculating the target processing rate of other operators in the real-time processing task based on the node relationship between the any operator and other operators in the real-time processing task using the target processing rate of the any operator; and calculating the target parallelism of each operator based on the target processing rate of each operator.

[0011] In one exemplary embodiment of this disclosure, calculating the target processing rate of any operator based on the performance data of any operator includes: when the data processing rate of any operator is less than the data processing rate of the source operator corresponding to the any operator, determining the data backlog rate of the any operator within a corresponding time period based on the performance data of the any operator, and determining the sum of the data processing rate of the any operator and the data backlog rate as the target processing rate of the any operator; when the data processing rate of any operator is greater than the data processing rate of the source operator corresponding to the any operator, determining the amount of data processed by the any operator within a corresponding time period and the working time of the any operator based on the performance data of the any operator, and determining the ratio of the amount of data processed by the any operator within a corresponding time period to the working time of the any operator as the target processing rate of the any operator.

[0012] In one exemplary embodiment of this disclosure, after determining the target parallelism of each operator in the real-time processing task, the method further includes: searching for historical scheduling records with the same parallelism parameter corresponding to the real-time processing task in historical scheduling data, wherein the parallelism parameter includes parallelism data before and after scaling up or down the real-time processing task; determining the performance optimization of the real-time processing task before and after scaling up or down based on the historical scheduling records, and verifying the availability of the target parallelism of each operator in the real-time processing task based on the performance optimization.

[0013] In one exemplary embodiment of this disclosure, after verifying the availability of the target parallelism of each operator in the real-time processing task based on the performance optimization, the method further includes: when it is determined that the target parallelism of each operator in the real-time processing task is unavailable based on the performance optimization, increasing the target parallelism of each operator, and monitoring the performance optimization of the real-time processing task based on the increased target parallelism of each operator, until the parallelism of each operator reaches a resource threshold.

[0014] In one exemplary embodiment of this disclosure, the method further includes: when it is determined that the data processing rate of any operator does not match the data processing rate of the source operator corresponding to the any operator, determining whether the real-time processing task has been scheduled and completed based on the historical scheduling data; when it is determined that the real-time processing task has been scheduled and completed, determining the performance optimization status of the real-time processing task based on the performance monitoring data, and setting the availability of the corresponding historical scheduling record in the historical scheduling data based on the performance optimization status.

[0015] According to a second aspect of this disclosure, a resource scheduling apparatus is provided, the apparatus comprising: an acquisition module for acquiring performance monitoring data of a real-time processing task; an analysis module for analyzing the performance data of each operator in the performance monitoring data to determine the target parallelism of each operator in the real-time processing task; and a processing module for scaling up or down the real-time processing task according to the target parallelism of each operator.

[0016] In one exemplary embodiment of this disclosure, the analysis module is used to determine whether the data processing rate of each operator in the real-time processing task matches the data processing rate of the source operator corresponding to each operator based on the performance monitoring data. When it is determined that the data processing rate of any operator does not match the data processing rate of the source operator corresponding to any operator, the target parallelism of each operator in the real-time processing task is calculated based on the node relationship between the any operator and other operators in the real-time processing task and the performance data of the any operator.

[0017] In one exemplary embodiment of this disclosure, the analysis module is used to calculate the target processing rate of any operator based on the performance data of any operator, calculate the target processing rate of other operators in the real-time processing task based on the node relationship between any operator and other operators in the real-time processing task using the target processing rate of any operator, and calculate the target parallelism of each operator based on the target processing rate of each operator.

[0018] In one exemplary embodiment of this disclosure, the analysis module is configured to: determine the data backlog rate of any operator within a corresponding time period based on the performance data of any operator when the data processing rate of any operator is less than the data processing rate of the source operator corresponding to any operator; and determine the sum of the data processing rate of any operator and the data backlog rate as the target processing rate of any operator. When the data processing rate of any operator is greater than the data processing rate of the source operator corresponding to any operator, determine the amount of data processed by any operator within a corresponding time period and the working time of any operator based on the performance data of any operator; and determine the ratio of the amount of data processed by any operator within a corresponding time period to the working time of any operator as the target processing rate of any operator.

[0019] In one exemplary embodiment of this disclosure, after determining the target parallelism of each operator in the real-time processing task, the analysis module is further configured to search for historical scheduling records with the same parallelism parameter corresponding to the real-time processing task in historical scheduling data. The parallelism parameter includes parallelism data before and after scaling up or down the real-time processing task. Based on the historical scheduling records, the performance optimization of the real-time processing task before and after scaling up or down is determined, and the availability of the target parallelism of each operator in the real-time processing task is verified based on the performance optimization.

[0020] In one exemplary embodiment of this disclosure, after verifying the availability of the target parallelism of each operator in the real-time processing task based on the performance optimization, the analysis module is further configured to increase the target parallelism of each operator when it is determined that the target parallelism of each operator in the real-time processing task is unavailable based on the performance optimization, and monitor the performance optimization of the real-time processing task based on the increased target parallelism of each operator until the parallelism of each operator reaches the resource threshold.

[0021] In one exemplary embodiment of this disclosure, the analysis module is further configured to determine whether the real-time processing task has been scheduled and completed based on the historical scheduling data when it is determined that the data processing rate of any operator does not match the data processing rate of the source operator corresponding to the any operator; when it is determined that the real-time processing task has been scheduled and completed, the module determines the performance optimization status of the real-time processing task based on the performance monitoring data, and sets the availability of the corresponding historical scheduling record in the historical scheduling data based on the performance optimization status.

[0022] According to a third aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements any of the above-described resource scheduling methods.

[0023] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the above-described resource scheduling methods by executing the executable instructions.

[0024] This disclosure has the following beneficial effects:

[0025] In summary, based on the resource scheduling method, resource scheduling device, computer-readable storage medium, and electronic device in this exemplary embodiment, the performance data of each operator in the acquired performance monitoring data of the real-time processing task can be analyzed to determine the target parallelism of each operator in the real-time processing task, and the real-time processing task can be scaled up or down according to the target parallelism of each operator. By determining the target parallelism of each operator and scaling up or down the real-time processing task according to the target parallelism of each operator, it is possible to monitor the data processing task at the operator level and perform operator-level resource scheduling, meeting the needs of dynamically adjusting resource allocation. Furthermore, it can improve data processing efficiency while maximizing the utilization of operator resources, saving computer resources and cluster operation costs, and thus improving data processing efficiency.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0028] Figure 1This diagram illustrates a resource scheduling method according to an exemplary embodiment.

[0029] Figure 2 A sub-flowchart of a resource scheduling method in this exemplary embodiment is shown;

[0030] Figure 3 A schematic diagram of an operator node is shown in this exemplary embodiment;

[0031] Figure 4 This illustrates a flowchart of calculating the target parallelism of an operator in this exemplary embodiment;

[0032] Figure 5 This illustrates a flowchart of a verification operator parallelism in this exemplary embodiment;

[0033] Figure 6 This diagram illustrates a flowchart of a capacity expansion or reduction process in this exemplary embodiment.

[0034] Figure 7 A flowchart illustrating another resource scheduling method in this exemplary embodiment is shown;

[0035] Figure 8 A schematic diagram illustrating a method for expanding or shrinking capacity in this exemplary embodiment is shown.

[0036] Figure 9 This diagram illustrates a structural block diagram of a resource scheduling device according to this exemplary embodiment;

[0037] Figure 10 This illustration shows a computer-readable storage medium for implementing the above-described method in this exemplary embodiment;

[0038] Figure 11 An electronic device for implementing the above method is shown in this exemplary embodiment. Detailed Implementation

[0039] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0040] One approach in related technologies involves configuring automatically scalable processing nodes for data processing tasks. For example, in a cloud or virtualized cluster environment, resources are dynamically adjusted by monitoring the load of each virtual node and automatically starting or stopping the corresponding processing nodes. However, in a multi-tenant environment, this method makes it impossible for the system to monitor the specific operating status of each processing node, hindering finer-grained measurement and resource adjustment of data processing tasks.

[0041] Based on one or more of the aforementioned problems, the exemplary embodiments of this disclosure first provide a resource scheduling method that can respond in real time to the resource requirements of data processing, expand or shrink the operators that perform real-time processing tasks, and realize elastic resource scaling at the operator level.

[0042] Figure 1 A process of this exemplary embodiment is shown, which may include the following steps S110 to S130:

[0043] Step S110. Obtain performance monitoring data for real-time processing tasks.

[0044] In this exemplary embodiment, an operator is also called a task, and an operator is a process. In real-time processing tasks, an operator can be divided into multiple threads for parallel execution. A thread within an operator process can be called a sub-thread, and the parallelism of an operator is the number of sub-threads of that operator. A real-time processing task can be a data processing task in a stream processing system, which may include reading, analyzing, and processing data files, as well as controlling the interaction of data files with other business systems. Each real-time processing task can consist of one or more processing nodes, and each processing node can run one or more operators. Performance monitoring data may include task performance data of the real-time processing task and performance data of the operators executing the real-time processing task. Task performance data may be the performance data of the entire data processing task, including the memory occupied by the real-time processing task, CPU (central processing unit) utilization data, the number of processing nodes, etc. Operator performance data may include the memory occupied by the operator, CPU utilization data, and the number of threads of the processing node corresponding to the operator.

[0045] Specifically, performance monitoring data for real-time processing tasks can be obtained through specific data interfaces. For example, task performance data for real-time processing tasks can be obtained in real time through the application programming interface provided by the stream processing system, while the performance data of operators can be obtained in real time through the data interface on each processing node.

[0046] The above methods can be used to monitor the running status of real-time processing tasks and each operator within those tasks, enabling more granular monitoring of the business system and providing data support for a thorough understanding of the system's operating status.

[0047] Step S120. Analyze the performance data of each operator in the performance monitoring data to determine the target parallelism of each operator in the real-time processing task.

[0048] The target parallelism of an operator refers to the degree of parallelism that an operator should have in order to meet the data processing requirements of real-time processing tasks.

[0049] After acquiring performance monitoring data for real-time processing tasks, this data can be analyzed to extract performance data for each operator. Based on this data, the target parallelism for each operator can be calculated. For example, the required data processing rate for each operator can be determined based on its performance data, and the target parallelism for each operator can be determined accordingly. When operators run at their corresponding target parallelism, the real-time processing task can meet the performance requirements of data processing while avoiding the waste of idle resources.

[0050] Specifically, in one alternative implementation, refer to Figure 2 As shown, step S120 can be achieved through the following steps S210 to S220:

[0051] Step S210: Based on the performance monitoring data, determine whether the data processing rate of each operator in the real-time processing task matches the data processing rate of the source operator corresponding to each operator.

[0052] In real-time processing tasks, data flows between processing nodes according to a specific direction. For example, refer to... Figure 3As shown, a real-time processing task can be abstracted as a directed acyclic graph. The source operator can extract data from a data source, such as a specific database. The processing operator can transform the data transmitted by the source operator according to user-defined processing logic. The output operator can write the processed data into a specified database or data platform. In this exemplary embodiment, the source operator corresponding to an operator refers to the data source operator of that operator, which can be the preceding operator. Each operator can receive processed data transmitted from its associated data source operator. Furthermore, for any given operator, its preceding and following operators may reside on different processing nodes. Therefore, data between operators can be transmitted via physical lines or network lines. For example, a source operator can transmit acquired data to its associated next operator via a network node. When transmitting data over a network, the data transmission rate between operators can directly reuse the TCP (Transmission Control Protocol) rate control method to achieve data backpressure between operators.

[0053] After acquiring performance monitoring data, the data processing rate of each operator can be calculated using the performance monitoring data. Based on the matching relationship between the data processing rate of each operator and the data processing rate of the corresponding source operator, it can be determined whether resource scheduling is required for real-time processing tasks.

[0054] Specifically, based on whether historical data is used in data processing, operators can be divided into stateful operators and stateless operators. Stateful operators can utilize historical data to process the data currently being processed and then pass the data down to downstream operators. For example... Figure 3 In the operator structure shown, the source operator and the output operator can record information about data processing. Except for the source operator, when other operators are processing data, they can obtain the data to be read from the input buffer data, write the processed data into the output buffer data, and transmit the processed data to the downstream operator through the network.

[0055] When calculating the data processing rate of each operator, the throughput of each operator can be calculated using performance monitoring data. For example, the CPU utilization data of the operator instance of each operator can be determined based on performance monitoring data, and the throughput of any operator can be calculated using the following formula (1):

[0056] f i t =k i ×nf i att (n) (1)

[0057] Where, k iThis represents the operator performance coefficient related to CPU performance and operator processing logic, where n is the number of operator instances, and f is the number of operators. i att (n) is the operator performance decay function.

[0058] At the same time, considering the CPU contention overhead of the operator, f i att (n) can be calculated using the following formula (2):

[0059]

[0060] Where N is the number of working nodes, α CPU CPU contention overhead factor P is the number of operator instances running on worker node i. i total This represents the total number of running threads on the worker node.

[0061] For stateful operators, state-related performance metrics, such as CPU utilization, memory usage data for each operator, and persistent storage I / O performance metrics, can also be collected to calculate the performance degradation function f. i att (n). That is, the performance degradation value f of operator i can be calculated using the following equation (3). i att (n):

[0062] f i att (n)=f i att_CPU (n)+f i att_mem (n)+f i att_io (n) (3)

[0063] Among them, f i att_CPU (n) is the performance degradation function caused by CPU contention. A decay function for performance degradation caused by memory contention. The decay function α is the performance degradation caused by I / O contention. mem For memory contention factor, For memory-related functions, mem used Mem is the total memory usage of the worker nodes. total Total memory for worker nodes α represents the amount of memory used by all instances of operator i on the current working node. io For storage I / O contention coefficient, Functions related to storage I / O usage, io usedThe current storage I / O rate on the worker node, io total Maximum I / O rate for worker nodes. For the current working node, all instances of operator i use the IO rate.

[0064] Step S220: When it is determined that the data processing rate of any operator does not match the data processing rate of the source operator corresponding to that operator, the target parallelism of each operator in the real-time processing task is calculated based on the node relationship between the above-mentioned operator and other operators in the real-time processing task and the performance data of the above-mentioned operator.

[0065] When it is determined that the data processing rate of any operator does not match the data processing rate of the source operator corresponding to that operator, it indicates that the data processing requirements and resource quantity of the real-time processing task at the current moment do not match, and expansion or reduction processing is required. At this time, the target parallelism of each operator in the real-time processing task can be calculated based on the node relationship between operators and the performance data of operators.

[0066] Specifically, in one alternative implementation, refer to Figure 4 As shown, the target parallelism of each operator can be calculated using the following method:

[0067] Step S410: Calculate the target processing rate of any operator based on the performance data of any operator.

[0068] The target processing rate refers to the data processing rate that an operator should possess to meet the current data processing requirements. When determining the target parallelism of each operator, performance data for any of the aforementioned operators can be extracted from performance monitoring data, and the target processing rate can be calculated based on that performance data. For example, the amount of data processed by any operator within a corresponding time period can be calculated based on its performance data to obtain its data processing efficiency. The target processing efficiency can then be calculated by determining the actual working time of the operator.

[0069] In real-time processing tasks, if the data processing efficiency of an operator is lower than its expected efficiency, data backlog will occur; conversely, if the data processing efficiency is higher than its expected efficiency, the operator will remain idle for a period of time. Therefore, in an optional implementation, the target processing rate of any of the above operators can be calculated using the following method:

[0070] When the data processing rate of any operator is less than the data processing rate of its corresponding source operator, the data backlog rate of that operator within the corresponding time period is determined based on its performance data. The sum of the data processing rate and the data backlog rate of that operator is then determined as the target processing rate of that operator. For example, the data backlog rate Tb of that operator at the current moment can be determined based on performance test data, and the target processing efficiency Tt = Tc + Tb of that operator can be calculated, where Tc is the data processing rate of that operator at the current moment.

[0071] When the data processing rate of any operator is greater than the data processing rate of its corresponding source operator, the data volume and working time of any operator within a corresponding time period are determined based on the performance data of that operator. The ratio of the data volume to the working time of any operator within that time period is then determined as the target processing rate of that operator. For example, the data volume Dt processed by any operator within a corresponding time period and the working time t of any operator can be determined based on performance monitoring data, thereby calculating the target processing rate Tt = Tc + Dt / t, where Tc is the data processing rate of any operator at the current moment.

[0072] When calculating the data processing efficiency of any of the above operators at the current moment, due to the existence of the data backpressure mechanism, the idle time (OP) of the waiting output of any of the above operators can be used. i wo and waiting for input idle time OP i wi Determine the time (OP) used by the operator during actual operation. i work This allows us to calculate the data processing rate of each operator. For example, we can utilize the idle time of the current operator, i.e., OP. i wo and OP i wi , and the idle time of the previous operator and Calculate the correction coefficient η of the target processing rate of the previous operator using the target processing rate Tt of any of the above operators at the current time. i-1 The data processing rate of the previous operator is obtained.

[0073] Specifically, the calculation process for the correction factor is as follows:

[0074] when When, the correction coefficient η i-1 =1, meaning the data processing rate of the previous operator is the same as the target processing rate.

[0075] when The correction factor is calculated as follows:

[0076] First, calculate the data processing rate of the current operator. Then, based on the input-output ratio of the current operator data... The actual throughput of the upstream operator was calculated as follows: The correction coefficients of the upstream operator are obtained as follows:

[0077] Step S420: Based on the node relationship between any operator and other operators in the real-time processing task, calculate the target processing rate of other operators in the real-time processing task using the target processing rate of any operator, and calculate the target parallelism of each operator according to the target processing rate of each operator.

[0078] The node relationship between operators can be the data flow relationship between operators. For example, for any operator, if it can receive the processing data passed by the previous operator, then there is a node connection relationship between the previous operator and any operator, and the connection level is 1, that is, direct connection.

[0079] After obtaining the target processing rate of any operator, the target parallelism of each operator can be calculated based on the ratio of data transfer rates between operators. For example, it can be achieved by using... inverse function The calculations are performed to determine the target parallelism that all operators should have.

[0080] Through the above steps S410 to S420, the target processing efficiency of each operator can be obtained by calculating the target processing rate of any operator, and the target parallelism of each operator can be determined.

[0081] To improve the reliability of resource scheduling, after calculating the target parallelism of each operator based on its data processing efficiency, the target parallelism of each operator can be verified. Specifically, in one optional implementation, refer to... Figure 5 As shown, the target parallelism of each operator can be verified by performing the following method:

[0082] Step S510: In the historical scheduling data, find the historical scheduling record with the same parallelism parameter corresponding to the real-time processing task.

[0083] Historical scheduling data refers to records of scaling up or down real-time processing tasks at historical points in time. This data may include the parallelism of operators, performance monitoring data, and availability of recorded data before and after the scaling up or down process. The parallelism parameter may include parallelism data before and after scaling up or down the real-time processing task.

[0084] Step S520: Determine the performance optimization of the real-time processing task before and after scaling up or down based on historical scheduling records, and verify the availability of the target parallelism of each operator in the real-time processing task based on the performance optimization.

[0085] For example, after determining the target parallelism of each operator, the parallelism of each operator in the real-time processing task at the current moment and the target parallelism of each operator can be used as parallelism parameters. Historical scheduling records with the same parallelism parameters can be searched in the historical scheduling data. Then, based on the historical scheduling records, it can be determined whether the performance of the real-time processing task changes before and after scaling up or down. If the performance after scaling up or down is better than the performance before scaling up or down, it indicates that the scheduling rule corresponding to the historical scheduling record has achieved performance optimization of the real-time processing task, and that the target parallelism of each operator determined by the above method is usable. Conversely, if the performance after scaling up or down is not better, it indicates that the scheduling rule corresponding to the historical scheduling record cannot improve the performance of the real-time processing task, and that the target parallelism of each operator is unusable. This method allows for availability verification of the determined target parallelism of each operator based on historical scheduling data, ensuring the effectiveness of subsequent resource scheduling processing.

[0086] Furthermore, in an alternative implementation, after verifying the availability of the target parallelism of each operator in the real-time processing task based on performance optimization, the following method can also be performed:

[0087] When it is determined that the target parallelism of each operator in the real-time processing task is unavailable based on the performance optimization, the target parallelism of each operator is increased, and the performance optimization of the real-time processing task is monitored based on the increased target parallelism of each operator until the parallelism of each operator reaches the resource threshold.

[0088] When it is determined that the target parallelism of each operator is unavailable, it means that the target parallelism of each operator may still not meet the current data processing requirements. The target parallelism of each operator can be increased, and based on the increased target parallelism of each operator, the performance optimization of the real-time processing task can continue to be monitored until the parallelism of each operator reaches the resource limit.

[0089] Furthermore, in an optional implementation, when it is determined that the data processing efficiency of any operator does not match the data processing rate of the source operator corresponding to any operator, the availability of historical scheduling records in the historical scheduling data can be set by the following method:

[0090] Determine whether real-time processing tasks have been completed based on historical scheduling data;

[0091] When it is determined that the real-time processing task scheduling is completed, the performance optimization status of the real-time processing task is determined based on the performance monitoring data, and the availability of the corresponding historical scheduling record in the historical scheduling data is set according to the performance optimization status.

[0092] For example, the parallelism parameter at the time of the last scaling up or down of a real-time processing task can be determined based on historical scheduling data. If this parallelism parameter is the same as the parallelism of each operator and the target parallelism of the real-time processing task at the current moment, as determined by performance monitoring data, it indicates that the real-time processing task has been scheduled successfully. Alternatively, the scheduling time of the last scaling up or down of the real-time processing task can be used. If this scheduling time is within a preset time range, such as 1 minute, it indicates that the real-time processing task has been scheduled successfully. The performance optimization of the real-time processing task can be determined based on performance monitoring data. If the performance of the real-time processing task at the current moment is improved compared to before scheduling, it indicates that the previously recorded historical scheduling record in the historical scheduling data is available; otherwise, it indicates that the previously recorded historical scheduling record is unavailable. This method allows for the verification of real-time processing tasks, avoiding duplicate resource scheduling.

[0093] In one optional implementation, when it is determined that the data processing rate of each operator in the real-time processing task matches the data processing rate of the source operator corresponding to each operator, the parallelism of each operator at the current moment can be stored in historical scheduling data so that when the real-time processing task is expanded or shrunk again, the target parallelism of each operator in the real-time processing task can be determined according to the historical scheduling data.

[0094] Step S130. Expand or shrink the real-time processing task according to the target parallelism of each operator.

[0095] After obtaining the target parallelism of each operator, the operators in the real-time processing task can be expanded or reduced in size. For example, operators whose parallelism at the current time is less than the target parallelism can be expanded, and operators whose parallelism at the current time is greater than the target parallelism can be reduced in size, so that the number of sub-threads of the operators is equal to the target parallelism.

[0096] In one optional implementation, the need for scaling up or down the real-time processing task can be determined based on whether the target parallelism of each operator is significantly greater or less than the target parallelism of the operators in the last scheduling or calculation. For example, if the variance between the target parallelism of each operator and the target parallelism of the operators in the last calculation is greater than a preset variance threshold, it is determined that the real-time processing task needs to be scaled up or down; conversely, if the variance between the target parallelism of each operator and the target parallelism of the operators in the last calculation is not greater than the preset variance threshold, it is determined that the real-time processing task does not need to be scaled up or down. In this way, the real-time processing task can be comprehensively evaluated based on the parallelism of the operators calculated in two consecutive calculations to determine whether scaling up or down is necessary, avoiding frequent resource scheduling and helping to maintain the stability of the resource scheduling system.

[0097] Figure 6 A process for elastic stretching is shown in the figure, which may include the following steps S601 to S612:

[0098] Step S601: Determine the target parallelism of each operator in the real-time processing task.

[0099] Based on the above steps S110 to S120, the target parallelism of each operator at the current time can be calculated. The specific method will not be described in detail here.

[0100] Step S602: Determine whether the real-time processing task needs to perform a scaling task based on the target parallelism of each operator.

[0101] When determining the target parallelism of each operator, the relationship between each operator's parallelism and its previous configuration can be used to determine whether the operator needs to be scaled up at the current moment. For example, the number of operators with a target parallelism greater than the previous configuration can be calculated. If this number is greater than half of the total number of operators, it can be determined that the real-time processing task needs to be scaled up; conversely, if this number is less than half of the total number of operators, it can be determined that the real-time processing task needs to be scaled down. When it is determined that the real-time processing task needs to perform a scaling up task, step S603 can be executed; conversely, when it is determined that the real-time processing task needs to perform a scaling down task, step S604 can be executed.

[0102] Step S603: Obtain historical scheduling data. For example, historical scheduling data for real-time processing tasks can be obtained from the corresponding historical scheduling database.

[0103] Step S604: Determine whether the target parallelism of each operator is significantly less than the target parallelism of each operator calculated in the previous step.

[0104] When the target parallelism of each operator is significantly less than the target parallelism of each operator calculated in the previous calculation, step S612 can be executed to scale down the real-time processing task according to the target parallelism of each operator. Conversely, when the difference between the target parallelism of each operator and the target parallelism of each operator calculated in the previous calculation is very small, step S610 can be executed to stop the current scaling down process.

[0105] Specifically, when determining whether the target parallelism of each operator is significantly less than the target parallelism of the operators calculated in the previous calculation, the variance of the target parallelism of each operator compared to the target parallelism of the operators calculated in the previous calculation can be calculated, and its relationship with a preset variance threshold can be determined. When the calculated variance is greater than the preset variance threshold, it can be considered that the target parallelism of each operator calculated in the current calculation is significantly less than the target parallelism of each operator calculated in the previous calculation; otherwise, it is considered that the difference between the target parallelism of each operator calculated in the current calculation and the target parallelism of each operator calculated in the previous calculation is small.

[0106] Step S605: Determine whether there is a matching historical scheduling record for the target parallelism of each operator based on historical scheduling data.

[0107] Specifically, historical scheduling records with the same parallelism parameters as the current operators can be searched in the historical scheduling data. When a matching historical scheduling record exists, step S606 is executed to determine whether the historical scheduling record is available. When no corresponding historical scheduling record exists, step S612 is executed to expand or shrink the real-time processing task according to the target parallelism of each operator.

[0108] Step S606: Determine whether the historical scheduling record is available.

[0109] When historical scheduling records are available, step S607 is executed to determine whether the parallelism of each operator corresponding to the historical scheduling record has reached the bottleneck parallelism. When historical scheduling records are unavailable, step S608 is executed to determine whether there is a next available historical scheduling record. Specifically, when each operator reaches the bottleneck parallelism, the real-time processing task reaches its resource limit.

[0110] Step S607: Determine whether the parallelism of each operator corresponding to the historical scheduling record has reached the bottleneck parallelism.

[0111] The bottleneck parallelism can be either the maximum parallelism of each operator or the maximum parallelism of all operators.

[0112] When the parallelism of each operator corresponding to the historical scheduling record reaches the bottleneck parallelism, step S608 can be executed to determine whether there is a next available historical scheduling record. Conversely, when the parallelism of each operator corresponding to the historical scheduling record has not reached the bottleneck parallelism, step S612 is executed to expand or shrink the real-time processing task according to the target parallelism corresponding to the historical scheduling record.

[0113] Step S608: Determine if there is a next available historical scheduling record.

[0114] When there is a next available historical scheduling record, execute step S609 to determine whether the parallelism of each operator corresponding to the next available historical scheduling record has reached the bottleneck parallelism. Conversely, when there is no next available historical scheduling record, and the parallelism corresponding to the historical scheduling record has reached the bottleneck parallelism, it is still necessary to expand the real-time processing task. This indicates that the data processing rate of the real-time processing task is very high at the current moment. Therefore, step S611 can be executed to increase the bottleneck parallelism of each operator.

[0115] Step S609: Determine whether the parallelism of each operator corresponding to the next available historical scheduling record has reached the bottleneck parallelism.

[0116] When the parallelism of each operator corresponding to the next available historical scheduling record reaches the bottleneck parallelism, step S610 is executed to stop the current expansion or reduction process; otherwise, step S612 is executed to expand or reduce the real-time processing task according to the parallelism of each operator corresponding to the next available historical scheduling record.

[0117] Step S610: Stop the current expansion or reduction process.

[0118] Step S611: Increase the bottleneck parallelism of each operator.

[0119] Specifically, the bottleneck parallelism of each operator can be increased by a certain amount, or set to N times the current parallelism, where N is greater than 1. After increasing the bottleneck parallelism of each operator, the real-time processing task can be scaled up or down according to the increased bottleneck parallelism of each operator by executing step S612.

[0120] Step S612: Expand or shrink the real-time processing task according to the target parallelism of each operator.

[0121] The above methods can be used to verify the availability of the target parallelism of each operator, thereby improving the accuracy and reliability of resource scheduling for real-time processing tasks. At the same time, the bottleneck parallelism of each operator can be dynamically adjusted according to the data processing requirements of real-time processing tasks, enhancing the applicability and flexibility of the resource scheduling method.

[0122] Figure 7 Another process of this exemplary embodiment is illustrated, as shown in the figure, which may include the following steps S701 to S715:

[0123] Step S701: Obtain performance monitoring data for real-time processing tasks.

[0124] Step S702: Determine whether the data processing efficiency of each operator matches the data processing efficiency of the corresponding source operator based on the performance monitoring data.

[0125] When it is determined that the data processing efficiency of each operator matches the data processing efficiency of the corresponding source operator, step S703 is executed to store the parallelism parameters of each operator at the current time into the historical scheduling data. When it is determined that the data processing efficiency of each operator does not match the data processing efficiency of the corresponding source operator, step S704 is executed to determine whether the real-time processing task has been scheduled and completed.

[0126] Step S703: Store the parallelism parameters of each operator at the current time into the historical scheduling data.

[0127] By storing the parallelism parameters of each operator at the current moment into historical scheduling data, the corresponding historical scheduling record with the same parallelism parameter can be found based on the parallelism parameter in the historical scheduling data during the next resource scheduling. Thus, when the historical scheduling record is determined to be valid, the real-time processing task can be scaled up or down according to the parallelism corresponding to the historical scheduling record.

[0128] Step S704: Determine whether the real-time processing task has been scheduled and completed.

[0129] For example, based on the parallelism parameter of the real-time processing task at the current moment, that is, the calculated target parallelism of each operator and the parallelism at the current moment, it can be determined whether the parallelism parameter is consistent with the parallelism parameter configured last time. When the parallelism parameter of the real-time processing task at the current moment is consistent with the parallelism parameter configured last time, it means that the real-time processing task has completed the scheduling process. Conversely, when the parallelism parameter of the real-time processing task at the current moment is inconsistent with the parallelism parameter configured last time, it means that the real-time processing task has not yet been scheduled.

[0130] When it is determined that the real-time processing task has been completed, step S705 can be executed to determine whether the data processing rate of the source operator corresponding to each operator has changed based on the real-time acquired performance monitoring data. Conversely, when it is determined that the real-time processing task has not been completed, step S706 can be executed to determine whether the data processing efficiency of each operator is greater than the data processing efficiency of the source operator corresponding to each operator.

[0131] Step S705: Determine whether the data processing rate of the source operator corresponding to each operator has changed based on the real-time acquired performance monitoring data.

[0132] When it is determined that the data processing efficiency of the source operator corresponding to any operator has changed, step S706 can be executed to determine whether the data processing efficiency of each operator is greater than the data processing efficiency of the source operator corresponding to each operator. When it is determined that the data processing efficiency of the source operator corresponding to each operator has not changed, step S707 can be executed to determine whether the performance of the real-time processing task at the current moment is better than the performance before the scheduling was completed.

[0133] Step S706: Determine whether the data processing efficiency of each operator is greater than the data processing efficiency of the source operator corresponding to each operator.

[0134] When the data processing efficiency of each operator is greater than the data processing efficiency of the corresponding source operator, it is determined that the real-time processing task needs to be scaled down. Therefore, step S708 can be executed to determine the idle time and working time of each operator. Conversely, when the data processing efficiency of each operator is less than the data processing efficiency of the corresponding source operator, it is determined that the real-time processing task needs to be scaled up. Step S709 can be executed to determine the bottleneck parallelism of each operator and calculate the theoretical parallelism of each operator according to the bottleneck parallelism.

[0135] Step S707: Determine whether the performance of the real-time processing task at the current moment is better than the performance before the scheduling was completed.

[0136] When the performance of the real-time processing task at the current moment is better than the performance before the scheduling was completed, it means that the previous scheduling was effective. Step S712 can be executed to determine whether the previous expansion or contraction process was an unavailable configuration. Conversely, when the performance of the real-time processing task at the current moment is lower than the performance before the scheduling was completed, step S713 can be executed to mark the parallelism of each operator in the real-time processing task at the current moment as an unavailable configuration.

[0137] Step S708: Determine the idle time and working time of each operator.

[0138] Step S709: Determine the bottleneck parallelism of each operator and calculate the theoretical parallelism of each operator according to the bottleneck parallelism.

[0139] When the data processing rate of each operator is less than the data processing efficiency of its corresponding source operator, it is necessary to scale up the real-time processing task. In this case, the theoretical parallelism of each operator can be calculated based on its bottleneck parallelism. For example, the parallelism of other operators can be extrapolated from the bottleneck parallelism of any one operator to obtain the theoretical parallelism of each operator.

[0140] Step S710: Calculate the target parallelism of each operator.

[0141] Specifically, the target processing rate of each operator can be calculated based on the data processing rate of each operator, and the target parallelism of each operator can be determined based on the target processing rate. Alternatively, the theoretical parallelism of each operator at the current moment can be used to calculate whether the target processing rate of each operator is met, and then the parallelism of each operator can be reduced according to the corresponding decreasing rules. Thus, when it is determined that the target processing rate of each operator is met, the parallelism of each operator obtained is determined as the target parallelism.

[0142] Step S711: Expand or shrink the real-time processing task according to the target parallelism of each operator.

[0143] Step S712: Determine whether the previous expansion or reduction process was a usable configuration.

[0144] When it is determined that the parallelism parameters of each operator in the previous expansion or contraction process are available configurations, step S714 can be executed to mark the parallelism of each operator in the current real-time processing task as available configurations; conversely, when it is determined that the parallelism parameters of each operator in the previous expansion or contraction process are unavailable configurations, step S715 can be executed to update the target parallelism of each operator, and to expand or contract the real-time processing task according to the updated target parallelism of each operator.

[0145] Step S713: Mark the parallelism of each operator in the real-time processing task at the current moment as an unavailable configuration.

[0146] Step S714: Mark the parallelism of each operator being processed in real time at the current moment as an available configuration.

[0147] Step S715: Update the target parallelism of each operator, and scale up or down the real-time processing task according to the updated target parallelism of each operator. For example, if the performance of the real-time processing task after the last scaling up or down is lower than the performance before scheduling, the parallelism of each operator can be appropriately increased or decreased, and the real-time processing task can be scaled up or down according to the target parallelism obtained after the increase or decrease.

[0148] The resource scheduling method in this exemplary embodiment will be described below using a specific real-time processing task as an example.

[0149] refer to Figure 8As shown, the real-time processing task runs in a worker cluster with 4 nodes, where the flatmap and output operators are stateless operators. The parallelism of the flatmap operator is 88, indicating that there are 88 instances of the flatmap operator at the current time, and the parallelism of the count operator is 512, indicating that there are 512 instances of the count operator at the current time.

[0150] By collecting the CPU utilization, memory, and I / O usage of the thread corresponding to each operator instance, and also collecting the data backlog value from the data source, the current data backlog value is 71,669,780. The ratio of this backlog value to the current data processing rate of each operator (275,653 rec / s) is 260, indicating that it will take at least 4 minutes to process the backlog data. Therefore, the real-time processing task needs to be scaled up. After scaling up, the data backlog value is again 71,666,780, indicating that the data processing rate of each operator is now comparable to the data processing rate of the corresponding source operator, and no scaling up or down is needed.

[0151] For example, if the data processing rate of the source operator remains basically constant, to complete the processing of the backlog of data within 1 minute, a processing rate of 1470149 rec / s is required. In this case, calculate the actual throughput Tf_i for each operator, excluding the output operator. t The calculations show that the bottleneck operator, `flatmap`, needs a parallelism of 277, while the corresponding parallelism for the `count` operator is 534. The real-time processing tasks are reconfigured according to these operator parallelisms, and task performance is monitored. If the data processing rate requirements of the source operators are met, the expansion is considered successful; otherwise, performance monitoring data is collected again to re-determine the target parallelism for each operator.

[0152] In summary, according to the resource scheduling method in this exemplary embodiment, the performance data of each operator in the acquired performance monitoring data of the real-time processing task can be analyzed to determine the target parallelism of each operator in the real-time processing task, and the real-time processing task can be scaled up or down according to the target parallelism of each operator. By determining the target parallelism of each operator and scaling up or down the real-time processing task according to the target parallelism of each operator, it is possible to monitor the data processing task at the operator level and perform operator-level resource scheduling, meeting the needs of dynamically adjusting resource allocation. Furthermore, it can improve data processing efficiency while maximizing the utilization of operator resources, saving computer resources and cluster operation costs, and thus improving data processing efficiency.

[0153] This exemplary embodiment also provides a resource scheduling device, with reference to Figure 9As shown, the resource scheduling device 900 may include: an acquisition module 910, which can be used to acquire performance monitoring data of real-time processing tasks; an analysis module 920, which can be used to analyze the performance data of each operator in the performance monitoring data and determine the target parallelism of each operator in the real-time processing task; and a processing module 930, which can be used to expand or shrink the real-time processing task according to the target parallelism of each operator.

[0154] In one exemplary embodiment of this disclosure, the analysis module 920 can be used to determine whether the data processing rate of each operator in the real-time processing task matches the data processing rate of the source operator corresponding to each operator based on performance monitoring data. When it is determined that the data processing rate of any operator does not match the data processing rate of the source operator corresponding to any operator, the target parallelism of each operator in the real-time processing task is calculated based on the node relationship between any operator and other operators in the real-time processing task and the performance data of any operator.

[0155] In one exemplary embodiment of this disclosure, the analysis module 920 can be used to calculate the target processing rate of any operator based on the performance data of any operator, calculate the target processing rate of other operators in the real-time processing task based on the node relationship between any operator and other operators in the real-time processing task, and calculate the target parallelism of each operator based on the target processing rate of each operator.

[0156] In one exemplary embodiment of this disclosure, the analysis module 920 can be used to determine the data backlog rate of any operator within a corresponding time period based on the performance data of any operator when the data processing rate of any operator is less than the data processing rate of the source operator corresponding to any operator, and to determine the sum of the data processing rate and the data backlog rate of any operator as the target processing rate of any operator; and when the data processing rate of any operator is greater than the data processing rate of the source operator corresponding to any operator, to determine the amount of data processed by any operator within a corresponding time period and the working time of any operator based on the performance data of any operator, and to determine the ratio of the amount of data processed by any operator within a corresponding time period to the working time of any operator as the target processing rate of any operator.

[0157] In one exemplary embodiment of this disclosure, after determining the target parallelism of each operator in the real-time processing task, the analysis module 920 can also be used to search for historical scheduling records with the same parallelism parameter corresponding to the real-time processing task in historical scheduling data. The parallelism parameter includes the parallelism data before and after the real-time processing task is expanded or shrunk. Based on the historical scheduling records, the performance optimization of the real-time processing task before and after the expansion or shrunk processing is determined, and the availability of the target parallelism of each operator in the real-time processing task is verified based on the performance optimization.

[0158] In one exemplary embodiment of this disclosure, after verifying the availability of the target parallelism of each operator in the real-time processing task based on the performance optimization, the analysis module 920 can also be used to increase the target parallelism of each operator when it is determined that the target parallelism of each operator in the real-time processing task is unavailable based on the performance optimization, and monitor the performance optimization of the real-time processing task based on the increased target parallelism of each operator until the parallelism of each operator reaches the resource threshold.

[0159] In one exemplary embodiment of this disclosure, the analysis module 920 can also be used to determine whether the real-time processing task has been scheduled and completed based on historical scheduling data when it is determined that the data processing rate of any operator does not match the data processing rate of the source operator corresponding to any operator. When it is determined that the real-time processing task has been scheduled and completed, the analysis module 920 can determine the performance optimization status of the real-time processing task based on performance monitoring data, and set the availability of the corresponding historical scheduling record in the historical scheduling data based on the performance optimization status.

[0160] The specific details of each module in the above-mentioned device have been described in detail in the method section of the implementation plan. For details of the undisclosed scheme, please refer to the implementation plan of the method section, and therefore will not be repeated here.

[0161] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0162] Exemplary embodiments of this disclosure also provide a computer-readable storage medium having a program product stored thereon capable of implementing the methods described above in this specification. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0163] refer to Figure 10 As shown, a program product 1000 for implementing the above-described method according to an exemplary embodiment of the present disclosure is described. This product may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0164] The program product 1000 may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0165] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0166] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0167] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0168] Exemplary embodiments of this disclosure also provide an electronic device capable of implementing the above-described method. Referring below... Figure 11 To describe an electronic device 1100 according to such an exemplary embodiment of the present disclosure. Figure 11 The electronic device 1100 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0169] like Figure 11 As shown, the electronic device 1100 can be represented as a general-purpose computing device. The components of the electronic device 1100 may include, but are not limited to: at least one processing unit 1110, at least one storage unit 1120, a bus 1130 connecting different system components (including storage unit 1120 and processing unit 1110), and a display unit 1140.

[0170] The storage unit 1120 stores program code, which can be executed by the processing unit 1110, causing the processing unit 1110 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1110 can perform... Figures 1 to 2 , Figures 4 to 7 The methods and steps shown are as follows.

[0171] Storage unit 1120 may include readable media in the form of volatile storage units, such as random access memory (RAM) 1121 and / or cache memory 1122, and may further include read-only memory (ROM) 1123.

[0172] Storage unit 1120 may also include a program / utility 1124 having a set (at least one) program module 1125, such program module 1125 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0173] Bus 1130 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0174] Electronic device 1100 can also communicate with one or more external devices 1200 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1100, and / or any device that enables electronic device 1100 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1150. Furthermore, electronic device 1100 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1160. As shown, network adapter 1160 communicates with other modules of electronic device 1100 via bus 1130. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0175] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to exemplary embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0176] Furthermore, the above figures are merely illustrative representations of the processes included in the methods according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0177] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the exemplary embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the method according to the exemplary embodiments of this disclosure.

[0178] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A resource scheduling method, characterized in that, The method includes: Obtain performance monitoring data for real-time processing tasks; Analyze the performance data of each operator in the performance monitoring data to determine the target parallelism of each operator in the real-time processing task; In the historical scheduling data, find the historical scheduling record with the same parallelism parameter corresponding to the real-time processing task. The parallelism parameter includes the parallelism data before and after the real-time processing task is expanded or reduced in size. The performance optimization of the real-time processing task before and after scaling up or down is determined based on the historical scheduling records, and the availability of the target parallelism of each operator in the real-time processing task is verified based on the performance optimization. When it is determined that the data processing rate of any operator does not match the data processing rate of the source operator corresponding to that operator, the real-time processing task is determined to be completed based on the historical scheduling data. When it is determined that the real-time processing task scheduling is completed, the performance optimization status of the real-time processing task is determined based on the performance monitoring data, and the availability of the corresponding historical scheduling record in the historical scheduling data is set according to the performance optimization status. The real-time processing task is scaled up or down according to the target parallelism of each operator.

2. The method according to claim 1, characterized in that, The step of analyzing the performance data of each operator in the performance monitoring data to determine the target parallelism of each operator in the real-time processing task includes: Based on the performance monitoring data, determine whether the data processing rate of each operator in the real-time processing task matches the data processing rate of the source operator corresponding to each operator; When it is determined that the data processing rate of any operator does not match the data processing rate of the source operator corresponding to that operator, the target parallelism of each operator in the real-time processing task is calculated based on the node relationship between the operator and other operators in the real-time processing task and the performance data of the operator.

3. The method according to claim 2, characterized in that, The step of calculating the target parallelism of each operator in the real-time processing task based on the node relationship between any operator and other operators in the real-time processing task and the performance data of any operator includes: Calculate the target processing rate of any operator based on the performance data of any operator; Based on the node relationship between any operator and other operators in the real-time processing task, the target processing rate of other operators in the real-time processing task is calculated using the target processing rate of any operator, and the target parallelism of each operator is calculated based on the target processing rate of each operator.

4. The method according to claim 3, characterized in that, The step of calculating the target processing rate of any operator based on the performance data of any operator includes: When the data processing rate of any operator is less than the data processing rate of the source operator corresponding to any operator, the data backlog rate of any operator in the corresponding time period is determined according to the performance data of any operator, and the sum of the data processing rate of any operator and the data backlog rate is determined as the target processing rate of any operator. When the data processing rate of any operator is greater than the data processing rate of the source operator corresponding to any operator, the amount of data processed by any operator in the corresponding time and the working time of any operator are determined according to the performance data of any operator, and the ratio of the amount of data processed by any operator in the corresponding time to the working time of any operator is determined as the target processing rate of any operator.

5. The method according to claim 1, characterized in that, After verifying the availability of the target parallelism of each operator in the real-time processing task based on the performance optimization results, the method further includes: When it is determined that the target parallelism of each operator in the real-time processing task is unavailable based on the performance optimization, the target parallelism of each operator is increased, and the performance optimization of the real-time processing task is monitored based on the increased target parallelism of each operator until the parallelism of each operator reaches the resource threshold.

6. A resource scheduling device, characterized in that, The device includes: The acquisition module is used to acquire performance monitoring data for real-time processing tasks; The analysis module is used to analyze the performance data of each operator in the performance monitoring data to determine the target parallelism of each operator in the real-time processing task; in the historical scheduling data, it searches for historical scheduling records with the same parallelism parameter corresponding to the real-time processing task, the parallelism parameter including the parallelism data before and after scaling up or down the real-time processing task; based on the historical scheduling records, it determines the performance optimization of the real-time processing task before and after scaling up or down, and verifies the availability of the target parallelism of each operator in the real-time processing task based on the performance optimization; when it is determined that the data processing rate of any operator does not match the data processing rate of the source operator corresponding to that operator, it determines whether the real-time processing task has been scheduled successfully based on the historical scheduling data; when it is determined that the real-time processing task has been scheduled successfully, it determines the performance optimization of the real-time processing task based on the performance monitoring data, and sets the availability of the corresponding historical scheduling record in the historical scheduling data based on the performance optimization. The processing module is used to expand or shrink the real-time processing task according to the target parallelism of each operator.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-5.

8. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the method of any one of claims 1-5 by executing the executable instructions.

Images