Heterogeneous computing task mapping method based on reinforcement learning

Abstract

The invention discloses a heterogeneous calculation task mapping method based on reinforcement learning, and particularly relates to the technical field of reinforcement learning, the method detects the CPU utilization rate and the task queue length in real time, dynamically screens hardware actions, calculates a memory bandwidth demand difference value, then implements double filtering, selects high-success-rate low-delay actions during resource conflicts, and finally performs task mapping. And identifying a key path and preferentially distributing low-delay hardware by an operation task dependency chain parameter, dynamically adjusting a weight based on a key path ratio, scaling an action space scale in combination with a load change rate, and finally selecting a highest reward estimated value to complete mapping. According to the method, response delay is remarkably reduced through a multi-stage cooperation mechanism, task accumulation is effectively controlled, key path execution time is compressed, burst load adaptive capacity is enhanced, resource utilization efficiency is improved, and stable and efficient task scheduling is achieved in a heterogeneous computing environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of reinforcement learning, in particular to a heterogeneous computing task mapping method based on reinforcement learning. BACKGROUND

[0002] The technical field based on reinforcement learning technology includes a machine learning method in which an agent optimizes a decision-making strategy by interacting with a dynamic environment. The core is to guide the agent to learn the optimal behavior strategy through the reward mechanism. It is typically applied to game control, robot navigation, resource scheduling, and other fields. This field mainly uses value function iteration, policy gradient optimization, time difference learning, and other technical means to solve sequential decision-making problems. Among them, the heterogeneous computing task mapping method based on reinforcement learning refers to directly optimizing the allocation strategy of computing tasks on heterogeneous hardware units using reinforcement learning algorithms. The specific technical means involve constructing the state space of the heterogeneous computing environment, defining the task mapping action space, designing the reward function to reflect the execution efficiency, and dynamically generating the optimal mapping scheme of tasks to processing units through Q-learning policy gradient or deep reinforcement learning network.

[0003] The prior art uses static threshold to allocate hardware resources, which cannot perceive real-time queue length changes, and high-load scenarios are prone to cause task accumulation. Resource conflict detection relies on fixed capacity thresholds, and there is no linkage screening mechanism between historical success rate and delay. Low-reliability hardware leads to high task retry rate. Task scheduling ignores dependency chain analysis, and the critical path is not allocated to low-delay hardware, resulting in long longest path time consumption. The size of the action space is fixed and does not change, and the decision delay is high when sudden load surges. The reward evaluation only compares the current action value, and lacks a dynamic scaling mechanism driven by the load change rate, resulting in low resource utilization rate. SUMMARY

[0004] The main purpose of the present application is to provide a heterogeneous computing task mapping method based on reinforcement learning, which solves the technical problems of high response delay, frequent resource conflicts, critical path blocking, sudden load overload, and inefficient mapping decision caused by static resource allocation in a heterogeneous computing environment through real-time load detection and dynamic hardware screening, resource conflict double filtering, priority optimization driven by critical path, action space scaling adaptive to load change rate, and reward-driven intelligent decision five-level linkage mechanism.

[0005] To achieve the above purpose, the technical scheme adopted by the present application is: A heterogeneous computing task mapping method based on reinforcement learning, the method comprising: Real-time detection of CPU utilization percentage and task queue length, selection of hardware actions with idle rate higher than the average when the threshold value is exceeded 80%, otherwise retaining the full action space, generating the current action set; Based on the current action set, calculate the memory / bandwidth requirement difference value, filter the actions with a history success rate > 90% and a delay < 10ms when there is a resource conflict, and generate a constraint filtered set; According to the constraint filtered action set, call the predecessor task completion state Boolean value and the data flow delay millisecond value, calculate the dependency chain depth and the critical path length, judge the critical path condition, select the lowest delay hardware action and promote the priority, and generate a priority adjusted action sequence; Based on the priority adjusted action sequence, calculate the critical path and task execution time ratio, adjust the weight when the ratio > 1.2, combine the load change rate to dynamically scale the action space size, and generate a dynamic action space coefficient; Based on the dynamic action space coefficient, call the action reward estimation value, compare the value, select the maximum value, and complete the mapping.

[0006] Preferably, the CPU utilization percentage is calculated by reading the processor time slice occupancy ratio in real time through the system kernel interface, and calculating the non-idle state time length ratio in unit time; the task queue length value is collected by the operating system scheduler interface to count the number of tasks waiting to be executed in the ready queue.

[0007] Preferably, the real-time detection of CPU utilization percentage and task queue length, when the threshold value is greater than 80%, selects the hardware action subset with idle rate higher than the average, otherwise, the full action space is retained, and a current action set is generated, including: According to the current CPU utilization percentage value and the task queue length value, calculate the difference value of the CPU utilization percentage value and the preset high threshold value 80%, judge whether the difference value is greater than zero, calculate the difference value of the task queue length value and the preset upper limit value, judge whether the difference value is greater than zero; if the CPU utilization difference value is greater than zero and / or the task queue length difference value is greater than zero, the load is true; otherwise, the load is false; Call all hardware unit idle rate percentage values, calculate the average value of the idle rate percentage values; if the load state judgment value is true, compare the idle rate percentage value with the average value, select the hardware unit mapping action with the idle rate percentage value higher than the average value, and construct an action subset; if the load state judgment value is false, maintain the full action space; generate an action subset candidate; Based on the action subset candidate, calculate the number of mapping actions in the action subset; if the action subset candidate is the full action space, calculate the total number of mapping actions in the full action space; generate a current action set.

[0008] Preferably, based on the current action set, calculate the memory / bandwidth requirement difference value, filter the actions with a history success rate > 90% and a delay < 10ms when there is a resource conflict, and generate a constraint filtered set, including: Based on the current action set, the hardware memory occupancy in megabytes and the task memory requirement in megabytes are obtained, the memory difference is calculated by subtracting the available memory from the task requirement, the bandwidth usage percentage and the task bandwidth requirement percentage are obtained, the bandwidth difference is calculated by subtracting the available bandwidth from the task requirement, and if the memory difference is greater than zero and / or the bandwidth difference is greater than zero, the logic is set to true, otherwise the logic is set to false; If the logic is true, the historical mapping success rate percentage is obtained, the success rate is compared with the 90% threshold, and the action with a success rate higher than 90% is selected, and if the logic is false, all current actions are retained; regardless of the true or false of the logic, the task priority level is obtained, and it is judged whether the task with a priority greater than 7 is mapped to a unit with a delay less than 10 milliseconds; a preliminary filtered action set is generated. Based on the preliminary filtered action set, the memory difference and the bandwidth difference are obtained, only the actions with a memory difference less than zero and a bandwidth difference less than zero and a delay less than 10 milliseconds are retained, and a constraint filtered action set is generated.

[0009] Preferably, the task priority level is a system preset value, specifically: level ≥ 7: hard real-time task, critical path task, safety-related task; level 4-6: soft real-time task, high throughput demand task; level ≤ 3: batch processing task, background maintenance task.

[0010] Preferably, according to the constraint filtered action set, the predecessor task completion state Boolean value and the data flow delay in milliseconds are called, the dependency chain depth and the critical path length are calculated, the critical path condition is judged, the lowest delay hardware action is selected and the priority is raised, and a priority adjusted action sequence is generated, including: Based on the constraint filtered action set, the predecessor task completion state Boolean value and the data flow delay in milliseconds are obtained, the dependency chain depth value is calculated as the path node count in the task dependency graph, and the critical path length in milliseconds is calculated as the longest path time sum, and it is judged whether the dependency chain depth value is greater than 3 or the critical path length in milliseconds is greater than 50; if the judgment is true, the logic is set to true, otherwise the logic is set to false; If the dependency criticality flag is true, the hardware unit delay in milliseconds is called, the delay in milliseconds of all units is compared, the action of the unit with the smallest delay in milliseconds is selected, the priority weight value of the action is raised to the highest level, and an optimized action candidate is generated. Based on the optimized action candidate, the predecessor task completion state Boolean value is obtained, and if the predecessor task completion state Boolean value is true, the subsequent action order is dynamically rearranged, and if the predecessor task completion state Boolean value is false, the current action order is maintained, and a priority adjusted action sequence is generated.

[0011] Preferably, the priority-based action sequence is adjusted, the critical path and task execution time ratio is calculated, when the ratio>1.2, the weight is adjusted, the action space size is dynamically scaled according to the load change rate, the dynamic action space coefficient is generated, and the method comprises the following steps: Based on the priority-based action sequence, the task execution time value in milliseconds and the critical path length value in milliseconds are called, the ratio value is calculated as the critical path length divided by the task execution time, it is judged whether the ratio value is greater than the reference value 1.2, if yes, the value 1 is set, otherwise the value 0 is set; If the critical path ratio flag is 1, the hardware delay value in milliseconds is called, the delay value in milliseconds of all action mapping units is compared, the action corresponding to the unit with the smallest delay value in milliseconds is selected, the priority weight value of the action is adjusted to decrease by 20% of the original value, and the weight adjustment sequence is generated; Based on the weight adjustment sequence, the load change rate percentage value is obtained, it is judged whether the load change rate percentage value is greater than 5%, if yes, the scaling ratio value is calculated as 1 minus the load change rate percentage value divided by 100, the action space size value is calculated as the current action number multiplied by the scaling ratio value and rounded; otherwise, the original action space size is maintained; and the dynamic action space coefficient is generated.

[0012] Preferably, based on the dynamic action space coefficient, the action reward estimation value is called, the values are compared, the maximum value is selected, and the mapping is completed, comprising: Based on the dynamic action space coefficient, the action reward estimation value is called, all the values are compared, the maximum value is calculated as the highest value, and the maximum reward value is generated; Based on the maximum reward value, the corresponding action, i.e. the action with the reward estimation value equal to the value, is selected, and the mapping is completed.

[0013] Compared with the prior art, the application has the following beneficial effects: by real-time detection of CPU utilization and task queue length and dynamic screening of hardware actions, only the hardware option with the optimal idle rate is retained in a high-load scenario, and invalid calculation overhead is reduced. After calculating the memory bandwidth demand difference, double filtering is implemented, when resource conflicts occur, high success rate and low delay actions are forcibly selected, and task reliability is improved. The critical path is calculated by calling the task dependency chain parameters, when the depth is out of limit or the time is too long, the priority of the lowest delay action is actively improved, and the critical path execution time is compressed. The action space size is dynamically scaled based on the critical path ratio, when the load suddenly changes, the candidate set is proportionally contracted, and resource overload is avoided. The optimal mapping is selected by combining the load change rate and the reward estimation value, and the task execution delay is reduced and the resource utilization is improved. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 It is a schematic diagram of the overall workflow of the application; Figure 2This is a schematic diagram of the process for generating a constraint filter set in some embodiments of the present invention; Figure 3 This is a schematic diagram of the process for generating dynamic motion space coefficients in some embodiments of the present invention. Detailed Implementation

[0015] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the linguistic context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0016] It should be understood that the terms "system," "device," "unit," and / or "module" as used in this specification are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0017] As indicated in this specification and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0018] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0019] The following describes in detail, with reference to the accompanying drawings, the heterogeneous computing task mapping method based on reinforcement learning provided in the embodiments of this specification.

[0020] Figure 1 This is an exemplary flowchart of a reinforcement learning-based heterogeneous computing task mapping method according to some embodiments of this specification. In some embodiments, the reinforcement learning-based heterogeneous computing task mapping method can be executed by processing logic, which may include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (instructions running on a processing device to execute hardware simulations), and any combination thereof. In some embodiments, Figure 1One or more operations in the flowchart of the reinforcement learning-based heterogeneous computing task mapping method shown can be implemented by a processing device and / or a terminal device. For example, the reinforcement learning-based heterogeneous computing task mapping method can be stored in a storage device in the form of a computer program or instructions, and invoked and / or executed by the processing device and / or the terminal device.

[0021] Specifically, refer to Figure 1 The heterogeneous computing task mapping method based on reinforcement learning in this embodiment of the invention may include: steps 1-5.

[0022] Step 1: Real-time detection of CPU utilization percentage and task queue length. When the threshold of 80% is exceeded, select hardware actions with idle rate higher than the average to form a subset; otherwise, retain the entire action space and generate the current action set. The system continuously monitors the CPU utilization percentage, i.e., the processor's busyness level; for example, 70% means 70% of the computing power is occupied. It also monitors the task queue length, i.e., the number of tasks waiting to be processed; for example, fifteen tasks are queued. When either value exceeds the 80% threshold, such as CPU usage reaching 85% or the queue reaching twenty, hardware actions with idle rates higher than the average are selected. For example, if a GPU's current idle rate is 45%, which is greater than the system average of 30%, this is considered a subset. If the limit is not exceeded, all available actions are retained; if no filtering is performed, two hundred hardware mapping options are retained directly. The final output is the current action set, which is a set of candidate hardware execution schemes.

[0023] The CPU utilization percentage is read in real time via the system kernel interface to determine the proportion of processor time slices used, and the non-idle state time is calculated per unit time. The task queue length value is obtained by collecting the count of pending tasks through the operating system scheduler interface, and the number of task entities waiting to be executed in the ready queue is counted. Step 2: Based on the current action set, calculate the difference in memory / bandwidth requirements. When resource conflicts occur, filter actions with a historical success rate >90% and a latency <10ms to generate a constraint filter set. Step 3: Based on the set of constraint filtering actions, call the boolean value of the predecessor task completion status and the millisecond value of the data flow delay, calculate the dependency chain depth and critical path length, determine the critical path conditions, select the lowest latency hardware action and increase its priority, and generate a priority adjustment action sequence. Step 4: Adjust the action sequence based on priority, calculate the ratio of critical path to task execution time, adjust the weight when the ratio is >1.2, and dynamically scale the action space scale in combination with the load change rate to generate dynamic action space coefficients; Step 5: Based on the dynamic action space coefficients, call the estimated action reward value, compare the values, select the maximum value, and complete the mapping.

[0024] In some embodiments of the present invention, the CPU utilization percentage and task queue length are detected in real time. When the CPU utilization exceeds a threshold of 80%, hardware actions with idle rates higher than the average are selected to form a subset; otherwise, the entire action space is retained, and the current action set is generated. Specifically, the following steps are included: Step 101: Based on the current CPU utilization percentage and task queue length, calculate the difference between the CPU utilization percentage and the preset high threshold of 80%, and determine whether the difference is greater than zero. Calculate the difference between the task queue length and the preset upper limit, and determine whether the difference is greater than zero. If the CPU utilization difference is greater than zero and / or the task queue length difference is greater than zero, the logical judgment is true; otherwise, the logical judgment is false. Based on the current CPU utilization percentage and task queue length, the system obtains the current CPU utilization percentage (e.g., 85% in real-time data center monitoring) and the task queue length (e.g., 20 tasks waiting to be processed). A preset high threshold is set to 80% (based on system stability requirements), and a preset upper limit is set to 15 (based on server processing capacity). The difference between the CPU utilization percentage and the preset high threshold is calculated to be 85%–80% = 5%, and the difference is checked to see if it is greater than zero (5% > 0). Similarly, the difference between the task queue length and the preset upper limit is calculated to be 20–15 = 5%, and the difference is checked to see if it is greater than zero (5 > 0). Since either the CPU utilization difference > 0 or the task queue length difference > 0 is true, a logical judgment is generated indicating a system overload state, such as a CPU overload triggering alarm mechanism in a cloud computing environment.

[0025] Step 102: Based on the load status judgment value, call the idle rate percentage values ​​of all hardware units and calculate the average value of the idle rate percentage values; if the load status judgment value is true, compare the idle rate percentage values ​​with the average value, select the hardware units with idle rate percentage values ​​higher than the average value to map actions, and form an action subset; if the load status judgment value is false, maintain the full action space; generate action subset candidates. Based on the load status judgment value, the idle rate percentage values ​​of all hardware units are called. For example, if the idle rates of three server nodes are 30%, 40%, and 50%, the average idle rate percentage value is calculated as (30%+40%+50%) / 3=40%. If the load status judgment value is true (e.g., system overload), the idle rate percentage value of each hardware unit is compared with the average value (e.g., 30%<40%, 40%=40%, 50%>40%). The hardware units with idle rate percentage values ​​higher than the average value are selected to map actions. For example, only the actions corresponding to the nodes with an idle rate of 50% are selected to form an action subset containing one mapping option. If the load status judgment value is false, the full action space is maintained, such as retaining the actions of all three nodes, and generating action subset candidates, such as outputting the selected hardware mapping list in the task scheduler.

[0026] Step 103: Based on the action subset candidates, calculate the number of mapped actions in the action subset; if the action subset candidates are the entire action space, calculate the total number of mapped actions in the entire action space; generate the current action set.

[0027] Based on the action subset candidates, the number of mapped actions in the action subset is calculated. For example, if the action subset candidate contains three mapped actions, the number is 3. If the action subset candidate is the entire action space and contains ten mapped actions, the total number of mapped actions in the entire action space is calculated to be 10. The current action set is generated, for example, the value 3 or 10 is output in the resource allocation system for subsequent decision reference.

[0028] In some embodiments of the present invention, such as Figure 2 As shown, step 2: Based on the current action set, calculate the difference in memory / bandwidth requirements. When resource conflicts occur, filter actions with a historical success rate >90% and a latency <10ms to generate a constraint filter set, specifically including steps 201-203: Step 201: Based on the current action set, obtain the hardware memory usage in megabytes and the task memory requirement in megabytes. Subtract the available memory from the task requirement to calculate the memory difference. Obtain the bandwidth utilization percentage and the task bandwidth requirement percentage. Subtract the available bandwidth from the task requirement to calculate the bandwidth difference. If the memory difference is greater than zero and / or the bandwidth difference is greater than zero, set the logic to true; otherwise, set the logic to false. For example, based on the current action set, the system obtains the hardware memory usage in megabytes (e.g., a server has 8000MB of available memory), the task memory requirement in megabytes (e.g., an image processing task requires 10000MB), and calculates the memory difference as the task requirement minus the available memory (10000MB minus 8000MB equals 2000MB). It also obtains the bandwidth utilization percentage (e.g., the current network bandwidth usage is 60%), the task bandwidth requirement percentage (e.g., the task requires 80%), and calculates the bandwidth difference as the task requirement minus the available bandwidth (80% minus 60% equals 20%). The system then checks if the memory difference is greater than zero (2000MB is greater than zero) and if the bandwidth difference is greater than zero (20% is greater than zero). Since either the memory difference or the bandwidth difference is greater than zero, the logic is set to true to indicate a resource conflict. In edge computing scenarios, tasks cannot be directly mapped due to insufficient memory.

[0029] Step 202: If the logic is true, obtain the historical mapping success rate percentage, compare the success rate with the 90% threshold, and select actions with a success rate higher than 90%. If the logic is false, retain all current actions. Regardless of whether the logic is true or false, obtain the task priority level and determine whether tasks with a priority greater than 7 are mapped to units with a latency value of less than 10 milliseconds. Generate a preliminary filtered action set. It should be noted that the task priority levels are system preset values, specifically: Level ≥ 7: Hard real-time tasks, critical path tasks, security-critical tasks; Level 4-6: Soft real-time tasks, high-throughput tasks; Level ≤ 3: Batch processing tasks, background maintenance tasks. In this embodiment of the invention, based on logical judgment values, if the logic is true (e.g., resource conflict), the historical mapping success rate percentage is obtained, for example, the success rates of three actions are 85%, 92%, and 95%, respectively. The success rate value is compared with a 90% threshold, which is set based on system reliability standards. Actions with a success rate higher than 90% are selected, such as actions with success rates of 92% and 95%. If the logic is false, all current actions are retained, for example, all five actions are retained. Regardless of whether the logic is true or false, the task priority level value is obtained, for example, if the task priority is 8, it is determined whether the priority is greater than 7, which is based on the task importance classification. At the same time, it is determined whether the task is mapped to a unit with a latency value of less than 10 milliseconds, such as a GPU unit with a latency of 9ms. In the autonomous driving system, high-priority perception tasks are forcibly mapped to low-latency hardware, generating a preliminary filtered action set, for example, outputting a list of two actions that meet the conditions.

[0030] Step 203: Based on the initial filtering action set, obtain the memory difference and bandwidth difference, and retain only actions with a memory difference less than zero, a bandwidth difference less than zero, and a latency of less than 10 milliseconds to generate a constraint filtering action set.

[0031] For example, based on the initial set of filtering actions, the memory difference values ​​are obtained, such as action A having a memory difference of -500MB and action B having a memory difference of -300MB; the bandwidth difference values ​​are obtained, such as action A having a bandwidth difference of -10% and action B having a bandwidth difference of -5%; and the latency values ​​in milliseconds are obtained, such as action A having a latency of 8ms and action B having a latency of 12ms. Only actions with memory differences less than zero, bandwidth differences less than zero, and latency values ​​in milliseconds less than 10 are retained. For example, action A with a memory difference of -500MB less than zero, a bandwidth difference of -10% less than zero, and a latency of 8ms less than 10 meet the conditions and are retained. Action B with a latency of 12ms not less than 10 is removed. A set of constraint filtering actions is generated in the cloud computing resource pool, and finally, an optimization action option is output.

[0032] Furthermore, step 3 in this embodiment of the invention: based on the constraint filtering action set, calling the boolean value of the predecessor task completion status and the millisecond value of the data flow delay, calculating the dependency chain depth and critical path length, determining the critical path condition, selecting the lowest latency hardware action and increasing its priority, and generating a priority adjustment action sequence; specifically including steps 301-303: Step 301: Based on the set of constraint filtering actions, obtain the boolean value of the predecessor task completion status and the data flow delay time in milliseconds. The computation dependency chain depth value is the path node count in the task dependency graph, and the computation critical path length in milliseconds is the sum of the longest path time. Determine whether the dependency chain depth value is greater than 3 or the critical path length in milliseconds is greater than 50. If the determination is true, set the logic to true; otherwise, set the logic to false. Based on the constraint filtering action set, the boolean value of the predecessor task completion status is obtained. For example, in a distributed computing system, the task dependency graph shows that the perception task completion status is true. The data flow latency time in milliseconds is obtained. For example, the data transmission time between modules is recorded as 20ms. The computation dependency chain depth is the number of path nodes in the task dependency graph. For example, the number of nodes in the path visual processing → target recognition → decision planning is 4. The computation critical path length in milliseconds is the sum of the longest path time. For example, the sum of the time for path ABC is 10ms for visual processing, 25ms for target recognition, and 20ms for decision planning, which equals 55ms. The dependency chain depth is determined to be greater than 3. This threshold is set to 4 based on system fault tolerance design experience. A value greater than 3 is considered true. The critical path length in milliseconds is determined to be greater than 50. This threshold is derived from the real-time response requirement. A value greater than 50 is considered true. Since any judgment is true, the logic is set to true. In autonomous driving scenarios, path exceeding the limit triggers the optimization mechanism.

[0033] Step 302: If the criticality flag is true, the hardware unit latency millisecond value is called, the latency millisecond values ​​of all units are compared, the unit with the smallest latency millisecond value is selected to map the action, the priority weight of the action is increased to the highest level, and optimization action candidates are generated. Based on the criticality flag being true, the system calls the latency value in milliseconds for each hardware unit. For example, if the latency of three computing units is 12ms for the CPU unit, 5ms for the GPU unit, and 8ms for the NPU unit, it compares the latency values ​​of all units. For example, if 5 is less than 8, which is less than 12, the system selects the unit with the smallest latency value to map the action to the corresponding action for the GPU unit. The priority weight of this action is then increased to the highest level, for example, the weight is increased from the original value of 6 to 10. Optimization action candidates are generated, for example, the GPU action is output as the preferred option, prioritizing the allocation of low-latency hardware in cloud computing resource scheduling.

[0034] Step 303: Based on the optimized action candidates, obtain the Boolean value of the predecessor task completion status, determine if the Boolean value of the predecessor task completion status is true, dynamically rearrange the order of subsequent actions if the determination is true, maintain the current action order, and generate a priority adjustment action sequence.

[0035] Based on the optimized action candidates, the Boolean value of the predecessor task completion status is obtained. If the data preprocessing task status is true, it is determined that the Boolean value of the predecessor task completion status is true, i.e. the status is equal to true. The subsequent action order is dynamically rearranged. For example, the original sequence of action A with weight 5, action B with weight 7, and action C with weight 3 is rearranged into action B, action A, and action C. Priority adjustment action sequence is generated. For example, the new sequence list has action B with weight 7, action A with weight 5, and action C with weight 3. In industrial automation production lines, the task order is optimized in real time based on priority.

[0036] In some embodiments of the present invention, such as Figure 3 As shown, step 4: Adjust the action sequence based on priority, calculate the ratio of critical path to task execution time, adjust the weight when the ratio is >1.2, and dynamically scale the action space scale in combination with the load change rate to generate dynamic action space coefficients; specifically including steps 401-403: Step 401: Adjust the action sequence based on priority, call the task execution time in milliseconds and the critical path length in milliseconds, calculate the ratio value as the critical path length divided by the task execution time, and determine whether the ratio value is greater than the baseline value of 1.2. If it is greater, set the value to 1; otherwise, set the value to 0. The action sequence is adjusted based on priority. The task execution time in milliseconds is called, for example, 120 in a real-time task of an industrial control system. The critical path length in milliseconds is called, for example, 150 in the total time of the dependency chain. The ratio is calculated by dividing the critical path length by the task execution time, i.e., 150 divided by 120 equals 1.25. The baseline value of 1.2 is set based on the system's real-time response optimization standard. This value is derived from historical performance analysis to ensure task smoothness. It is judged whether the ratio value is greater than the baseline value of 1.25. If it is greater than 1.2, it is true. The value is set to 1 to indicate that the ratio exceeds the limit. A critical path ratio flag value is generated.

[0037] Step 402: If the critical path ratio flag is 1, then call the hardware latency millisecond value, compare the latency millisecond values ​​of all action mapping units, select the action corresponding to the unit with the smallest latency millisecond value, adjust the priority weight value of the action by 20% of the original value, and generate a weight adjustment sequence. If the critical path ratio flag is 1, the hardware latency value in milliseconds is retrieved. For example, the latency values ​​of the three computing units are 8, 12, and 15. The latency values ​​of all units in milliseconds are compared. For example, 8 is less than 12 and 15. The unit with the smallest latency value in milliseconds is selected, and the corresponding action is the unit with a latency of 8. The priority weight value of this action is adjusted by reducing the original value by 20%. For example, the original weight of 10 is reduced by 20% to 8, and a weight adjustment sequence is generated.

[0038] Step 403: Based on the weight adjustment sequence, obtain the load change rate percentage value, determine whether the load change rate percentage value is greater than 5%, if it is greater, calculate the scaling ratio value as 1 minus the load change rate percentage value divided by 100, calculate the action space scale value as the current number of actions multiplied by the scaling ratio value and rounded down; otherwise, maintain the original action space scale; generate dynamic action space coefficients.

[0039] Based on the weighted adjustment sequence, the percentage value of the load change rate is obtained. For example, if the CPU utilization changes from 70% to 78%, the change rate is calculated to be 8%. It is then determined whether the percentage value of the load change rate is greater than 5%. This threshold is designed based on system stability, and 8 is considered true if it is greater than 5. The scaling ratio is calculated as 1 minus the percentage value of the load change rate divided by 100, i.e., 1 minus 0.08 equals 0.92. The action space scale is calculated as the current number of actions multiplied by the scaling ratio and rounded down. For example, if the number of actions is 100, multiplying by 0.92 equals 92 and rounding down to 92, dynamic action space coefficients are generated.

[0040] Furthermore, in some embodiments of the present invention, the specific process of step 5 is as follows: based on the dynamic action space coefficient, call the action reward estimate value, compare the size of all values, calculate the maximum value as the highest value, and generate the maximum reward value; based on the maximum reward value, select the corresponding action, that is, the action whose reward estimate value is equal to that value, and complete the mapping.

[0041] For example, based on the dynamic action space coefficient, the dynamic action space coefficient value is obtained. For example, in edge computing task scheduling, a coefficient of 92 indicates that the current action space has 92 mapping options. The action reward estimate value is called. For example, the reward value of each action in the 92 actions is read from the real-time prediction system. The value range is 80 to 100. For example, action A rewards 85, action B rewards 92, and action C rewards 88. The size of all action reward estimates is compared. That is, the 92 values ​​are traversed to find the maximum value. By pairing them one by one, for example, 92 is greater than 85, 92 is greater than 88, the maximum value is calculated and the highest value is determined. The maximum reward value value 92 is generated. Based on the maximum reward value value 92, the corresponding action is selected. That is, all action reward estimates are scanned to find the action that is equal to 92 as action B. The mapping is completed and the task is assigned to the hardware unit specified by action B, such as a GPU node.

[0042] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for mapping heterogeneous computing tasks based on reinforcement learning, characterized in that, The method comprises: Real-time detection of CPU utilization percentage and task queue length, screening hardware actions with idle rate higher than the average to form a subset when the threshold value is 80%, otherwise retaining the full action space, generating a current action set; Based on the current action set, calculate the memory / bandwidth demand difference, filter actions with a history success rate > 90% and a delay < 10ms when there is a resource conflict, generate a constraint filtering set; According to the constraint filtering action set, call the predecessor task completion state Boolean value and the data flow delay millisecond value, calculate the dependency chain depth and the critical path length, judge the critical path condition, select the lowest delay hardware action and promote the priority, generate a priority adjustment action sequence; Based on the priority adjustment action sequence, calculate the critical path and the task execution time ratio, adjust the weight when the ratio > 1.2, combine the load change rate to dynamically scale the action space size, generate a dynamic action space coefficient; Based on the dynamic action space coefficient, call the action reward estimation value, compare the value, select the maximum value, complete the mapping. 2.The method of claim 1, wherein: The CPU utilization percentage is read in real time through the system kernel interface to obtain the processor time slice occupancy ratio, and the non-idle state time length ratio in unit time is calculated; the task queue length value is collected by the operating system scheduler interface to count the number of task entities waiting for execution in the ready queue. 3.The method of claim 2, wherein: The real-time detection of CPU utilization percentage and task queue length, screening hardware actions with idle rate higher than the average to form a subset when the threshold value is 80%, otherwise retaining the full action space, generating a current action set, comprises: According to the current CPU utilization percentage value and the task queue length value, calculate the difference between the CPU utilization percentage value and the preset high threshold value 80%, judge whether the difference is greater than zero, calculate the difference between the task queue length value and the preset upper limit value, judge whether the difference is greater than zero; if the CPU utilization difference is greater than zero and / or the task queue length difference is greater than zero, the load judgment is true; otherwise, the load judgment is false; Call all hardware unit idle rate percentage values, calculate the average value of the idle rate percentage values; if the load state judgment value is true, compare the idle rate percentage value with the average value, select the hardware unit mapping action with the idle rate percentage value higher than the average value to form an action subset; if the load state judgment value is false, maintain the full action space; generate an action subset candidate; Based on the action subset candidate, calculate the number of mapping actions in the action subset; if the action subset candidate is the full action space, calculate the total number of mapping actions in the full action space; generate a current action set. 4.The method of claim 1, wherein: The calculation of the memory / bandwidth demand difference based on the current action set, the filtering of actions with a history success rate > 90% and a delay < 10ms when there is a resource conflict, and the generation of a constraint filtering set, comprise: Based on the current action set, the hardware memory occupation is obtained in megabytes and the task memory requirement is obtained in megabytes, the memory difference is calculated by subtracting the available memory from the task requirement, the bandwidth usage rate is obtained in percentage and the task bandwidth requirement is obtained in percentage, the bandwidth difference is calculated by subtracting the available bandwidth from the task requirement, and if the memory difference is greater than zero and / or the bandwidth difference is greater than zero, the logic is set to true, otherwise the logic is set to false; If the logic is true, the historical mapping success rate is obtained in percentage, the success rate is compared with the 90% threshold, and the action with a success rate higher than 90% is selected, and if the logic is false, all current actions are retained; regardless of the logic, the task priority level is obtained, and it is judged whether the task with a priority greater than 7 is mapped to a unit with a delay less than 10 milliseconds; a preliminary filtered action set is generated; Based on the preliminary filtered action set, the memory difference and the bandwidth difference are obtained, only the actions with a memory difference less than zero and a bandwidth difference less than zero and a delay less than 10 milliseconds are retained, and a constraint filtered action set is generated. 5.The method of claim 4, wherein: The task priority level is a system preset value, specifically: level >= 7: hard real-time task, critical path task, safety-related task; level 4-6: soft real-time task, high throughput demand task; level <= 3: batch processing task, background maintenance task. 6.The method of claim 1, wherein: According to the constraint filtered action set, the predecessor task completion state Boolean value and the data flow delay in milliseconds are called, the dependency chain depth and the critical path length are calculated, the critical path condition is judged, the lowest delay hardware action is selected and the priority is raised, and a priority adjusted action sequence is generated, including: Based on the constraint filtered action set, the predecessor task completion state Boolean value and the data flow delay in milliseconds are obtained, the dependency chain depth value is calculated as the path node count in the task dependency graph, and the critical path length in milliseconds is calculated as the longest path time sum, and it is judged whether the dependency chain depth value is greater than 3 or the critical path length in milliseconds is greater than 50; if true, set the logic to true, otherwise set the logic to false; If the dependency criticality flag is true, the hardware unit delay in milliseconds is called, the delay in milliseconds of all units is compared, the action of the unit with the smallest delay in milliseconds is selected, the priority weight value of the action is raised to the highest level, and an optimized action candidate is generated; Based on the optimized action candidate, the predecessor task completion state Boolean value is obtained, and if the predecessor task completion state Boolean value is true, the subsequent action sequence is dynamically rearranged, and if the predecessor task completion state Boolean value is false, the current action sequence is maintained, and a priority adjusted action sequence is generated. 7.The reinforcement learning based heterogeneous computing task mapping method of claim 1, wherein: Based on the priority adjusted action sequence, the critical path and task execution time ratio is calculated, the ratio is adjusted when the ratio > 1.2, the action space scale is dynamically scaled combined with the load change rate, a dynamic action space coefficient is generated, including: Based on the priority adjusted action sequence, the task execution time in milliseconds and the critical path length in milliseconds are called, the ratio value is calculated as the critical path length divided by the task execution time, and it is judged whether the ratio value is greater than the reference value 1.2, if greater, set the value to 1, otherwise set the value to 0; If the critical path ratio flag is 1, the hardware delay millisecond value is called, the delay millisecond values of all action mapping units are compared, the action corresponding to the unit with the smallest delay millisecond value is selected, the priority weight value of the action is adjusted to reduce the original value by 20%, and the weight adjustment sequence is generated; Based on the weight adjustment sequence, the load change rate percentage value is obtained, and it is judged whether the load change rate percentage value is greater than 5%. If it is greater, the scaling ratio value is calculated as 1 minus the load change rate percentage value divided by 100, the action space size value is calculated as the current action quantity multiplied by the scaling ratio value and rounded, otherwise the original action space size is maintained; and the dynamic action space coefficient is generated. 8.The method of claim 1, wherein: Based on the dynamic action space coefficient, the action reward estimation value is called, the values are compared, the maximum value is selected, and the mapping is completed, including: Based on the dynamic action space coefficient, the action reward estimation value is called, all values are compared, the maximum value is calculated as the highest value, and the maximum reward value is generated. Based on the maximum reward value, the corresponding action, i.e. the action with the reward estimation value equal to the value, is selected, and the mapping is completed.

Images