A task allocation method and device, a storage medium and an electronic device

Abstract

The application discloses a task allocation method and device, a storage medium and an electronic device, relates to the chip technical field, and comprises the following steps: obtaining a plurality of test tasks to be allocated to a test system; predicting memory requirement data and execution time data required for executing the plurality of test tasks in the test system; selecting at least one target test task meeting a target load condition from the plurality of test tasks based on the memory requirement data and the execution time data; and allocating the at least one target test task to at least one different available computing node in the test system in a test task allocation mode corresponding to the target load condition. The application can distinguish between large-memory and long-time-consumption test tasks, and disperse and deploy these test tasks, so that each computing node can equally bear the load of the whole batch of power consumption test tasks, thereby shortening the running period of the whole batch of power consumption test tasks and effectively improving the project research and development delivery speed.

Description

Technical Field

[0001] This application relates to the field of chip technology, and in particular to a task allocation method, apparatus, storage medium and electronic device. Background Technology

[0002] Power consumption analysis in Electronic Design Automation (EDA) is a core tool for performance trade-offs, power supply scheme evaluation, and pre-tape-out power risk identification and optimization. For ultra-large-scale chip design scenarios, limitations such as chip size, computing node memory, and computing power limits prevent direct full-chip integrated power consumption simulation. In engineering, the process begins by dividing the complete chip into multiple independent physical implementation blocks (PD blocks) according to the physical design (PD) flow. Each PD block is then further decomposed to generate a large number of independent, parallel-executable test tasks. After all test tasks corresponding to all PD blocks have been executed, the test data from each PD block are aggregated to obtain the full-chip power consumption assessment result. Therefore, full-chip power consumption analysis generates a massive batch of parallel test tasks at once.

[0003] The industry typically uses Load Sharing Facility (LSF) scheduling systems to handle these batch parallel test tasks. Existing task allocation methods directly reuse the native task allocation strategy built into LSF to match and distribute nodes with test tasks. This native task allocation strategy relies solely on fixed logic for basic dispatch, making it highly susceptible to centrally distributing large-memory, long-duration test tasks to the same computing node. This can lead to memory overflow on these nodes, while the computing power and memory resources of other nodes remain idle and wasted. This significantly lengthens the overall runtime of the entire batch of simulation tasks, delays the progress of the full-chip power consumption assessment, and reduces the overall R&D and delivery efficiency of the chip project. Summary of the Invention

[0004] In view of this, this application provides a task allocation method, apparatus, storage medium and electronic device, the main purpose of which is to solve the technical problem that existing test task allocation strategies are prone to causing node load imbalance, resulting in long test task operation cycles and low R&D delivery efficiency.

[0005] Firstly, this application provides a task allocation method, including: Retrieve multiple test tasks to be assigned to the test system.

[0006] Predict the memory requirements and execution time of multiple test tasks in the test system.

[0007] Based on memory requirement data and execution time data, from multiple test tasks, at least one test task whose memory requirement data is greater than a preset memory threshold is determined as at least one first target test task that meets the first target load condition; and from multiple test tasks, at least one test task whose memory requirement data is less than or equal to a preset memory threshold and whose execution time data is greater than a preset time threshold is determined as at least one second target test task that meets the second target load condition.

[0008] According to the first test task allocation method corresponding to the first target load condition, at most one first target test task is allocated to the available computing nodes in the test system; according to the second test task allocation method corresponding to the second target load condition, at least one second target test task is allocated to each available computing node in the test system.

[0009] Secondly, this application provides a task allocation device, comprising: The acquisition module is configured to acquire multiple test tasks to be assigned to the test system.

[0010] The prediction module is configured to predict the memory requirements and execution time of multiple test tasks in the test system.

[0011] The selection module is configured to, based on memory requirement data and execution duration data, identify at least one test task from multiple test tasks whose memory requirement data is greater than a preset memory threshold as at least one first target test task that meets the first target load condition; and to identify at least one test task from multiple test tasks whose memory requirement data is less than or equal to a preset memory threshold and whose execution duration data is greater than a preset duration threshold as at least one second target test task that meets the second target load condition.

[0012] The allocation module is configured to allocate at most one first target test task to the available computing nodes in the test system according to the first test task allocation method corresponding to the first target load condition; and to allocate at least one second target test task to each available computing node in the test system according to the second test task allocation method corresponding to the second target load condition.

[0013] Thirdly, this application provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the task allocation method of the first aspect described above.

[0014] Fourthly, this application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the task allocation method described in the first aspect.

[0015] By employing the above technical solutions, this application provides a task allocation method, apparatus, storage medium, and electronic device. Compared with existing technologies, the task allocation method provided in this application collects all test tasks to be allocated, predicts the memory requirements and execution time of each test task, and after selecting suitable target test tasks based on memory requirements and execution time, distributes the test tasks to different computing nodes according to load constraints. In this way, this application can distinguish between large-memory and long-duration test tasks and deploy these test tasks in a distributed manner, avoiding memory overflow, idle node computing power, and load imbalance problems caused by similar test tasks being concentrated on the same node. This allows each computing node to evenly bear the load of the entire batch of power consumption test tasks, thereby shortening the running cycle of the entire batch of power consumption test tasks and effectively improving the project development and delivery speed. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a task allocation method provided in an embodiment of this application is shown; Figure 2 The diagram illustrates the execution flow of a task allocation method provided in an embodiment of this application. Figure 3 This paper shows a schematic diagram of the structure of a task allocation device provided in an embodiment of this application; Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] The embodiments of this application will now be described in more detail with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0020] In the integrated circuit EDA R&D testing process, chip power consumption analysis is a core means of performance trade-offs, power supply scheme evaluation, and power consumption risk identification and optimization before tape-out. The total data volume of a complete chip power consumption analysis test task is enormous. To improve overall processing efficiency, the engineering team breaks down the entire test task into a large number of independent parallel test tasks with no execution dependencies. All test tasks are uniformly delivered to the LSF scheduling system (i.e., the test system), and are executed in a distributed parallel manner by multiple Linux computing nodes within the system's computing cluster.

[0021] In actual engineering workflows, R&D personnel compile a list of test tasks to be analyzed and submit these tasks to the LSF scheduling system via local scripts or command-line interaction. The LSF scheduling system provides dedicated commands such as `bhosts`, `lsload`, `bsub`, and `bjobs` to respectively implement real-time load querying of computing nodes, test task distribution, test task queuing, and real-time monitoring of running status. Throughout the entire execution cycle of each test task, the EDA power analysis tool and the LSF scheduling system continuously output standardized operation logs. By parsing the log files, key operational characteristics such as peak memory usage, actual execution time, project group affiliation, and business priority for each test task can be extracted. The system's computing cluster deploys multiple computing nodes with varying hardware memory specifications. Each node continuously carries multiple batches of regular R&D tasks, and the node's idle memory capacity dynamically changes in real-time with the start and stop of existing test tasks. Optimizing cluster scheduling efficiency in the industry typically requires modifying the underlying scheduling kernel of the LSF scheduling system and adjusting global queue permissions, resulting in cumbersome processes, risks of online business interruption, and high implementation costs.

[0022] Therefore, the current industry standard is to directly use the basic test task allocation strategy built into the LSF scheduling system to match and distribute test tasks between nodes. This allocation strategy relies solely on fixed rules to complete the basic dispatch of test tasks and cannot take into account multiple dimensions such as the real-time idle memory of nodes, the memory usage of test tasks, and the execution time of test tasks for comprehensive allocation. This easily leads to the concentrated deployment of large-memory, long-duration test tasks on the same computing node, causing cluster node load imbalance. At the same time, the clustered deployment of long-duration test tasks will lengthen the overall operation cycle of the entire batch of power consumption test tasks, significantly reducing the R&D and delivery efficiency of integrated circuit projects.

[0023] To address the technical problem that existing test task allocation strategies easily lead to node load imbalance, resulting in long test task execution cycles and low R&D delivery efficiency, this application filters target test tasks for allocation based on their memory requirements and execution time data, and then distributes these target test tasks in a distributed manner according to node load constraints. This balances the load on cluster nodes, shortens the overall test task execution time, and improves R&D delivery efficiency.

[0024] This embodiment provides a task allocation method, such as Figure 1 The diagram shows a flowchart of a task allocation method, which includes: S101. Obtain multiple test tasks to be assigned to the test system.

[0025] In one implementation, in response to a scheduling start command triggered manually, by a timed trigger, or by linkage with the R&D process, a pre-compiled list of power consumption analysis test tasks is imported, and test tasks that are independent of each other and have no execution dependency within the list are read in batches.

[0026] For example, R&D personnel pre-sort out all parallel test tasks corresponding to this chip power consumption analysis and generate a standardized power consumption analysis test task list. After the script detects a manually executed command, a timed test task signal, or a scheduling trigger signal pushed by the R&D process, it reads all test tasks without prior or subsequent dependencies from the power consumption analysis test task list file and completes the batch acquisition of test tasks to be assigned.

[0027] S102. Predict the memory requirements and execution time of multiple test tasks in the test system.

[0028] In one implementation, all acquired test tasks are traversed, and the memory requirements and execution time of each test task are predicted based on historical running data.

[0029] For example, a continuously updated task profile database can be pre-built to store historical peak memory usage and execution time data for various test tasks. When a test task with historical execution records is encountered, the metric is predicted by querying the historical records of the corresponding test task in the task profile database.

[0030] In another example, a prediction model can be trained based on a full set of historical test task run samples. After training, the prediction model is input with features such as test task type and analysis scale, and outputs the memory requirements and execution time predictions for the corresponding test task. This prediction model uses massive amounts of historical test task execution logs as training samples, taking the business type, chip analysis scale, and process node of the test task as input features, and using the actual peak memory and actual execution time of the test task as labels to complete model training. For brand-new test tasks with no historical records, only the basic features of the test task need to be entered to quickly output accurate memory and execution time estimates, compensating for the data gap problem when there are no corresponding records in the task profile database.

[0031] In another example, for a brand new test task with no historical records, memory and duration benchmark thresholds can be preset based on industry experience with similar test tasks. These thresholds can be proportionally converted based on the current chip analysis scale to obtain initial estimated data. After the test task is completed, real running indicators are collected and written back to update the task profile database to improve the prediction accuracy of subsequent similar test tasks.

[0032] S103. Based on memory requirement data and execution time data, from multiple test tasks, at least one test task whose memory requirement data is greater than a preset memory threshold is determined as at least one first target test task that meets the first target load condition; and from multiple test tasks, at least one test task whose memory requirement data is less than or equal to a preset memory threshold and whose execution time data is greater than a preset time threshold is determined as at least one second target test task that meets the second target load condition.

[0033] In one implementation, test task load levels are divided based on the predicted memory requirements and execution time, and target test tasks that need to be scattered and scheduled are selected according to the isolation allocation rules for large memory test tasks and long time-consuming test tasks.

[0034] For example, by setting a corresponding threshold as the target load judgment standard, test tasks that require more memory than a preset memory threshold or have an execution time that exceeds a preset time threshold can be judged as high-load test tasks. These high-load test tasks can be identified as target test tasks, and cross-node distributed allocation can be prioritized for target test tasks during subsequent scheduling.

[0035] In another example, if there are test tasks among multiple test tasks that do not meet the target load conditions, these test tasks are classified separately as regular light load test tasks and are treated differently from the target test tasks for subsequent scheduling.

[0036] In one implementation, the target load condition includes a first target load condition and a second target load condition. The first target load condition is used to filter test tasks with large memory requirements, and the second target load condition is used to filter test tasks with normal memory requirements but long execution time.

[0037] In this embodiment, from multiple test tasks, at least one test task whose memory requirement data is greater than a preset memory threshold is determined as at least one first target test task that meets the first target load condition.

[0038] In one implementation, a memory threshold is pre-set based on the cluster node configuration and the memory usage of regular test tasks. Test tasks with large memory requirements that exceed this threshold are classified as the first target test tasks and are used as the primary isolated and dispersed objects to prevent single-node memory overload.

[0039] For example, the preset memory threshold is set to 512G, and test tasks with memory requirements exceeding 512G are identified as the first target test tasks and are preferentially deployed across nodes.

[0040] In this embodiment, from multiple test tasks, at least one test task whose memory requirement data is less than or equal to a preset memory threshold and whose execution time data is greater than a preset time threshold is determined as at least one second target test task that meets the second target load condition.

[0041] In one implementation, a time threshold is pre-set based on the average execution time of regular simulation test tasks, and test tasks with compliant memory usage but long execution time are selected as the second target test tasks to avoid long-running jobs clustering on a single node and consuming computing power, causing overall scheduling blockage.

[0042] For example, the preset duration threshold is set to 8 hours. Test tasks with memory requirements of less than or equal to 512G but execution time exceeding 8 hours are identified as the second target test tasks and are simultaneously included in the high-load distributed scheduling scope, so as to achieve the uniform distribution of long-term test tasks across nodes.

[0043] In one implementation, the real-time remaining memory resources of each available computing node are collected, nodes with remaining memory sufficient to accommodate a single first target test task are prioritized, and first target test tasks are allocated to each node. The number of first target test tasks that a single node can carry is limited to no more than one, thus avoiding memory contention and overflow crashes caused by multiple first target test tasks on the same computing node.

[0044] For example, a list of available compute nodes is obtained through a cluster load query command. The list contains five compute nodes: M1, M2, M3, M4, and M5, sorted sequentially by their remaining memory size. The first target test tasks to be assigned are J01 (530G), J02 (520G), and J03 (512G). The available compute nodes in the list are traversed, and J01 is assigned to M2, J02 to M4, and J03 to M1 in sequence. Each node is assigned no more than one first target test task.

[0045] S104. According to the first test task allocation method corresponding to the first target load condition, allocate at most one first target test task to the available computing nodes in the test system; according to the second test task allocation method corresponding to the second target load condition, allocate at least one second target test task to each available computing node in the test system.

[0046] In one implementation, the real-time remaining memory load of each computing node in the cluster is first collected, and then the target test tasks are deployed in a distributed manner based on the greedy bin packing logic to avoid the concentration of similar high-load test tasks on the same node.

[0047] For example, all available computing nodes are traversed, and the target test tasks (the first target test task and the second target test task) are allocated to different nodes one by one, with the real-time remaining memory of the node as the allocation limit, so as to realize the cross-node dispersion and isolation of large memory and long time-consuming test tasks.

[0048] In another example, after all target test tasks have been assigned, if there are regular light-load test tasks, these light-load test tasks will be filled into the remaining memory resources of each node to maximize the utilization of node resources.

[0049] Compared with existing technologies, this application can distinguish between large memory and long-duration test tasks and deploy these test tasks in a distributed manner, avoiding memory overflow, idle computing power and load imbalance caused by similar test tasks being concentrated on the same node. This allows each computing node to bear the load of the entire batch of power consumption test tasks in a balanced manner, thereby shortening the running cycle of the entire batch of power consumption test tasks and effectively improving the speed of project development and delivery.

[0050] The task allocation method provided in the embodiments of this application will now be described in detail.

[0051] Optionally, this application also provides an embodiment for predicting the memory requirement data and execution time data of a test task. In this embodiment, historical running data of the test system executing historical test tasks are obtained, including historical memory usage data and historical execution time data; the current test task is matched with historical test tasks, and if the same historical test task is matched, the memory requirement data and execution time data of the current test task are determined based on the historical running data corresponding to the same historical test task; if the current test task is not matched with the same historical test task, the memory requirement data and execution time data of the current test task are determined based on a preset prediction template.

[0052] In one implementation, a task profile database is pre-established, continuously collecting and storing full runtime logs of past power consumption analysis test tasks. The log information includes historical runtime data such as peak memory usage, complete execution time, test task type, chip size, process parameters, and version information for each test task. A mapping relationship is established between test task identifiers and historical runtime data for subsequent test task matching and metric prediction. During prediction, the basic identifier and feature information of the current test task are read, and matching searches are performed according to key fields such as test task type, chip module, and analysis type to find identical historical test tasks.

[0053] For example, if the current test task can accurately match the same historical test task, the actual peak memory and actual execution time of that historical test task are retrieved, and a preset safety redundancy coefficient is added to generate the memory requirement data and execution time prediction data used for this scheduling, ensuring sufficient memory reservation and avoiding memory overflow during operation. The preset safety redundancy coefficient is set comprehensively based on chip simulation fluctuations, additional overhead of the background system, and instantaneous peak memory jitter; for example, it can be set to 1.1 to reserve an appropriate amount of memory margin, balancing memory operation safety and overall cluster resource utilization.

[0054] For example, if the current test task is a completely new type and cannot be matched with an existing historical test task profile, a preset prediction template is invoked. The template contains baseline memory and baseline duration parameters for similar analysis scenarios. Based on the current chip design scale and simulation parameters, a linear scaling is performed to generate initial estimated data, ensuring the normal operation of the scheduling process. After the new test task is executed for the first time, its actual execution data is entered into the task profile database to complete the template iteration update and improve the prediction accuracy of subsequent similar test tasks.

[0055] In one embodiment, the available computing nodes in the test system are traversed, and at most one first target test task is assigned to each available computing node according to the first test task allocation method corresponding to the first target load condition.

[0056] In one implementation, a greedy bin packing algorithm adapted to the power consumption analysis cluster scenario is used to execute the allocation logic. With the real-time remaining memory of the node as a hard constraint, the first target test task with larger memory usage is preferentially allocated to the node with sufficient remaining memory. At the same time, it is constrained that a single node can only carry a single large memory test task.

[0057] For example, if the number of first target test tasks is greater than the number of available computing nodes, the first round of traversal allocates one first target test task to each available node in the list. The remaining unallocated first target test tasks are marked as test tasks to be scheduled. After the nodes have existing test tasks running and releasing memory, they are allocated in batches in a second round.

[0058] For example, if the number of first target test tasks is less than the number of available computing nodes, only some nodes in the list whose memory capacity can match the test task requirements are selected for allocation. The remaining nodes in the list that are not allocated first target test tasks are reserved to carry second target test tasks and regular light load test tasks, making full use of the idle computing power of the cluster.

[0059] In one embodiment, at least one second target test task is allocated to each available computing node in the test system according to the second test task allocation method corresponding to the second target load condition.

[0060] In one implementation, after the first target test task allocation is completed and the remaining memory of each node is refreshed, the second target test task with memory ≤512G and execution time >8h is distributed evenly. The round-robin distribution avoids long-duration test tasks from clustering on the same node. At the same time, the remaining memory after the node has allocated large memory test tasks is used to carry long-duration test tasks, thereby improving the overall resource utilization of the machine.

[0061] For example, the available compute nodes are M1, M2, M3, M4, and M5. After the first target test task is allocated, the remaining memory on each node is, for example, 888G, 1170G, 1100G, 1180G, and 1400G. The second target test tasks to be allocated are J04 (460G), J05 (440G), J06 (430G), J07 (418G), J08 (407G), J09 (396G), and J10 (385G). The tasks are allocated in a round-robin fashion: M5→M4→M2→M3→M1, sequentially assigning J04 to M5, J05 to M4, J06 to M2, J07 to M3, J08 to M1, J09 to M5, and J10 to M4. Each node carries at least one second target test task, and long-term test tasks are evenly distributed across the nodes of the cluster.

[0062] Furthermore, the second target test task is assigned to the available computing nodes in a round-robin manner until each second target test task is assigned to the corresponding available computing node.

[0063] In one implementation, allocation is performed in a fixed node traversal order using a round-robin approach. After each successful allocation of a second target test task, the remaining memory of the corresponding node is deducted based on the memory requirement of the test task, and the remaining memory resource information of the node is updated. If the remaining memory of the current node is insufficient to support the next second target test task, the node is skipped and the allocation is carried over to the next node.

[0064] For example, if the number of second target test tasks exceeds the number of available computing nodes, multiple rounds of allocation are performed. In each round, all available nodes are traversed and a single second target test task is allocated. After each allocation, the remaining memory of the node is deducted and the node's memory status is updated. If there are still unallocated second target test tasks at the end of the current round, the next round of allocation is started until all second target test tasks are allocated.

[0065] Furthermore, based on the memory requirement data of the first target test task, the remaining memory resource information of the available computing nodes allocated to the first target test task is updated.

[0066] In one implementation, after all the first target test tasks have been allocated, all available computing nodes carrying the first target test tasks are uniformly traversed, and the remaining memory resource information of each available computing node is refreshed in batches by subtracting the memory requirement data of the corresponding first target test task from the original real-time remaining memory of each node. After the batch update is completed, the updated remaining memory resource information of each available computing node is used as the upper limit constraint of the memory allocation for greedy binning when allocating the second target test tasks and regular light load test tasks.

[0067] Optionally, this application also provides an embodiment for allocating target test tasks. In this embodiment, a node mapping table is generated based on the test task allocation relationship between at least one target test task and at least one different available computing node; the target test task includes a first target test task and / or a second target test task; the node mapping table is input to the test system, so that the test system schedules at least one different available computing node to execute at least one target test task according to the node mapping table.

[0068] In one implementation, before multiple test tasks are sent to the test system, the matching calculation between all test tasks and available computing nodes is completed at the test task submission layer without modifying the underlying scheduling kernel of the test system.

[0069] For example, the first target test task is first allocated to a single node in isolation, and the remaining memory resources of each node are updated. Then, the second target test task is allocated in a round-robin fashion. Finally, the remaining memory of the nodes is used to fill regular light-load test tasks. The one-to-one binding relationship between all test tasks and available computing nodes is summarized, and a node mapping table carrying test task identifiers, node identifiers, memory requirements, execution durations, and business priorities is generated in a structured manner. Subsequently, the bsub batch submission interface function of the test system is called to batch pass the node mapping table into the test system. The test system parses the preset binding rules in the mapping table, skips the native random scheduling logic, and directs each test task to the available computing nodes specified in the mapping table for execution. The entire scheduling chain is built only at the submission layer, without the need to modify the underlying LSF scheduling strategy, cluster queue, and permission configuration, avoiding the risk of online business interruption caused by underlying modifications, and can be seamlessly compatible with the existing integrated circuit power consumption analysis cluster environment.

[0070] Optionally, before generating the node mapping table, this application also provides an embodiment for determining the scheduling order of target test tasks. In this embodiment, multiple target test tasks are sorted from high to low according to their business priority; target test tasks with the same business priority are sorted from large to small according to their memory requirements; and target test tasks with the same memory requirements are sorted from large to small according to their execution time, resulting in a target test task sequence. This target test task sequence is used to determine the scheduling order of the target test tasks.

[0071] This application also provides a specific embodiment of a task allocation method, such as... Figure 2 The diagram illustrates the execution flow of a task allocation method. This flow encompasses the complete process from test task triggering, test task preprocessing, cluster node data collection, hierarchical greedy allocation, batch test task submission, runtime monitoring, to iterative updates of the task profile database. Specifically: S201, script execution triggered.

[0072] S201 is the test task triggering step in the process, supporting three triggering methods: manual execution, system-scheduled test tasks, and R&D process linkage, adapting to the R&D work rhythm of integrated circuit power consumption analysis projects. After the test task is triggered, the full-link automated scheduling process starts, waiting for the import of the list of test tasks (Jobs) to be submitted.

[0073] S202. Import the list of test tasks to be submitted.

[0074] Batch read the power analysis test task list pre-compiled by the user. The test tasks in the list are independent and parallel power test tasks with no execution dependency. After the basic information of all test tasks is collected and uniformly aggregated, execute S203.

[0075] S203, Historical log information judgment.

[0076] Iterate through all test tasks and check whether each test task has historical running logs and task profile database records. Based on the judgment result, the task is routed to branch S204 or S205 to execute the estimated process. If the judgment result is yes, then execute S204; if the judgment result is no, then execute S205.

[0077] S204. Determine the memory requirements, execution time, and priority data of the current test task based on historical running data.

[0078] For the current test task that matches historical logs and has a profile record, retrieve the historical running data stored in the task profile database, and extract the actual peak memory (Mem_Peak), actual execution time (Runtime), and business priority from the historical running data as the memory requirement, execution time, and priority data of the current test task.

[0079] S205. Estimate memory usage and duration based on preset rules.

[0080] For brand-new test tasks with no historical execution records, the system prioritizes matching the historical average of similar test tasks to complete the estimation. If no similar samples are available, it uses an industry-preset prediction template and benchmark threshold for proportional conversion, outputting the memory requirements and execution time data for the test task.

[0081] S206. Label the test tasks with load tags and sort them.

[0082] Each test task is labeled with three load tags: memory requirement, execution time, and business priority. All test tasks are uniformly arranged according to the rules of business priority from high to low, memory requirement from large to small under the same priority, and execution time from large to small under the same memory requirement, so as to ensure that test tasks with high priority, large memory requirement, and long execution time participate in resource allocation first.

[0083] For example, all test tasks are divided into four groups (A, B, C, and D) based on load labels, and the test task data for each group is as follows: Group A consists of test tasks with memory requirements greater than 512G and runtime greater than 8h, including: J01 (530G, priority 2, 18h), J02 (520G, priority 2, 16h), and J03 (512G, priority 1, 15h).

[0084] Group B corresponds to test tasks with memory requirements ≤ 512G and runtime > 8h. These include: J04 (460G, priority 1, 14h), J05 (440G, priority 1, 13h), J06 (430G, priority 1, 12h), J07 (418G, priority 1, 11h), J08 (407G, priority 1, 20h), J09 (396G, priority 1, 19h), and J10 (385G, priority 1, 17h).

[0085] Group C corresponds to test tasks with memory requirements of 140G~250G and runtime ≤10h; including: J11~J25, memory range of 152G-250G, runtime of 1h~10h.

[0086] Group D corresponds to test tasks with a memory requirement of 128G and a runtime of ≤3 seconds; including: J26 (128G, priority 1, 1h), J27 (128G, priority 1, 3h), J28 (128G, priority 1, 2h), J29 (128G, priority 1, 1h), and J30 (128G, priority 1, 3h).

[0087] S207, Cluster computing node status acquisition.

[0088] The built-in load query command of the LSF cluster of the test system is invoked to collect hardware information of each computing node in the cluster in batches, including machine identifier, occupied memory, real-time remaining memory resources, and number of currently resident test tasks; computing nodes with remaining memory below the 100G safety threshold are removed, and a list of available computing nodes is generated in descending order of remaining memory.

[0089] For example, the bhosts and lsload commands are used to query the queue to find a total of 5 computing nodes (M1, M2, M3, M4, M5). The real-time remaining memory of each node is collected and sorted in descending order. After filtering out nodes with less than 100G of remaining memory, an ordered list of available nodes is output.

[0090] S208, Layered bin packing and allocation based on greedy algorithm.

[0091] A greedy bin packing algorithm adapted to power consumption scenarios is adopted, traversing available computing nodes one by one, using the real-time remaining memory resources of the nodes as the upper limit for allocation, and superimposing the following three layers of scheduling constraints for hierarchical allocation: 1. For the first target test task with more than 512GB of memory, a single node is isolated and allocated; 2. For secondary target test tasks with an execution time exceeding 8 hours, distribute them in a round-robin fashion. 3. Prioritize allocating computing nodes to high-priority test tasks.

[0092] For example, the allocation process executes the test tasks in groups A, B, C, and D sequentially. After all test tasks in a single group are allocated, the remaining memory of the corresponding node is deducted in real time, the node's memory status is updated, and then the next group of test tasks is allocated. The four-stage scheduling process is as follows: Execute the A group test task scheduling: The initial remaining memory of each computing node is M1 (1400G), M2 (1700G), M3 (1100G), M4 (1700G), and M5 (1400G). Traverse the remaining memory in descending order: M2→M4→M1→M5→M3. Allocate J01 (530G), J02 (520G), and J03 (512G) in sequence. After allocation, deduct the memory of the corresponding node synchronously and update the remaining memory of the node to M1 (888G), M2 (1170G), M3 (1100G), M4 (1180G), and M5 (1400G). Then proceed to the B group allocation.

[0093] Execute the B group test task scheduling: use M5→M4→M2→M3→M1 to cyclically allocate J04-J10, and deduct the remaining memory of the node immediately after each test task is allocated; after the B group allocation is completed, refresh the node memory to M1 (481G), M2 (740G), M3 (682G), M4 (355G), M5 (544G), and then proceed to the C group allocation.

[0094] Execute C group test task scheduling: allocate J11-J22 in the polling order of M2→M3→M1→M5→M4. If the current node has insufficient memory, it will be postponed to the next node for matching. After each test task is allocated, the remaining memory of the node is updated in real time. After the C group allocation is completed, the remaining memory of the node is updated to M1 (75G), M2 (130G), M3 (98G), M4 (145G), and M5 (7G), and then enters the D group allocation.

[0095] Execute the D group test task scheduling: allocate J23-J30 in a loop according to M1→M2→M3→M4→M5. If a match is successful, the node memory is deducted. After allocation, the remaining memory of the nodes is M2 (2G) and M4 (0G), and the memory of the other nodes remains unchanged. At this point, the four-group tiered allocation process is completed.

[0096] Once all test tasks are boxed, a node mapping table is generated that binds each test task to a specific node, along with a list of batch-submitted test tasks.

[0097] S209. Call the batch submission interface function to submit the node mapping relationship table and the test task list and obtain the running log.

[0098] The node mapping table output by S208 is read, and the batch submission interface function of the LSF cluster is called to distribute each test task to the available compute nodes specified in the mapping table. Furthermore, the `bjobs` command continuously monitors the cluster's test task queuing, running status, and other runtime data, generating standardized submission logs. These logs record information such as test task number, assigned node, memory requirements, and business priority for later operational verification.

[0099] S210, Analyze the runtime logs to extract real metrics.

[0100] After all test tasks in this round are completed, the system automatically parses the runtime logs output by the LSF and power analysis tools, and extracts the actual peak memory (Mem_Peak), actual execution time (Runtime), business priority, and other metrics for each test task.

[0101] S211, Write back and update the task profile database.

[0102] The real metrics obtained from S210 parsing are written back to the task profile database to update the storage records of the corresponding test tasks. The average reference data of the same type of test tasks are updated synchronously to continuously optimize the memory and duration prediction accuracy of subsequent test tasks, forming a self-iterative data closed loop.

[0103] S212, the scheduling loop is waiting for a new round of triggering.

[0104] Once the complete scheduling process for this round is completed, the scheduling script enters a standby state, continuously monitoring three types of trigger signals: manual, timed, and R&D process. Upon receiving a trigger command, it automatically jumps to S201.

[0105] Compared with existing technologies, this application can distinguish between large memory and long-duration test tasks and deploy these test tasks in a distributed manner, avoiding memory overflow, idle computing power and load imbalance caused by similar test tasks being concentrated on the same node. This allows each computing node to bear the load of the entire batch of power consumption test tasks in a balanced manner, thereby shortening the running cycle of the entire batch of power consumption test tasks and effectively improving the speed of project development and delivery.

[0106] Furthermore, as Figures 1 to 2 To provide a specific implementation of the method shown, this embodiment offers a task allocation device, such as... Figure 3 The diagram shows a structural schematic of a task allocation device 300, which includes: an acquisition module 310, a prediction module 320, a selection module 330, and an allocation module 340, wherein: The acquisition module 310 is configured to acquire multiple test tasks to be assigned to the test system.

[0107] The prediction module 320 is configured to predict the memory requirements and execution time data required for multiple test tasks to be executed in the test system.

[0108] Module 330 is configured to, based on memory requirement data and execution time data, determine at least one test task whose memory requirement data is greater than a preset memory threshold from multiple test tasks as at least one first target test task that meets the first target load condition; and, from multiple test tasks, determine at least one test task whose memory requirement data is less than or equal to a preset memory threshold and whose execution time data is greater than a preset time threshold as at least one second target test task that meets the second target load condition.

[0109] The allocation module 340 is configured to allocate at most one first target test task to the available computing nodes in the test system according to the first test task allocation method corresponding to the first target load condition; and to allocate at least one second target test task to each available computing node in the test system according to the second test task allocation method corresponding to the second target load condition.

[0110] Optionally, the allocation module 340 is also configured to update the remaining memory resource information of the available computing nodes allocated to the first target test task based on the memory requirement data of the first target test task.

[0111] Optionally, the allocation module 340 is also configured to traverse the available computing nodes in the test system and allocate at most one first target test task to each available computing node.

[0112] Optionally, the allocation module 340 is also configured to allocate the second target test task to the available computing nodes in a round-robin manner until each second target test task is allocated to the corresponding available computing node.

[0113] Optionally, the prediction module 320 is also configured to obtain historical running data of the test system executing historical test tasks, including historical memory usage data and historical execution time data.

[0114] The system matches the current test task with historical test tasks. If the same historical test task is matched, the system determines the memory requirement and execution time of the current test task based on the historical running data of the same historical test task.

[0115] If no matching historical test task is found for the current test task, the memory requirement and execution time of the current test task are determined based on the preset prediction template.

[0116] Optionally, the allocation module 340 is further configured to generate a node mapping table based on the test task allocation relationship between at least one target test task and at least one different available computing node; the target test task includes a first target test task and / or a second target test task.

[0117] Input the node mapping table into the test system, so that the test system can schedule at least one different available computing node to execute at least one target test task according to the node mapping table.

[0118] Optionally, the allocation module 340 is also configured to sort multiple target test tasks according to business priority from high to low.

[0119] For target test tasks with the same business priority, sort them from largest to smallest memory requirement.

[0120] Target test tasks with the same memory requirements are sorted from longest to shortest execution time to obtain a target test task sequence. This target test task sequence is used to determine the scheduling order of the target test tasks.

[0121] It should be noted that other corresponding descriptions of the functional units involved in the task allocation device provided in this embodiment can be found in [reference needed]. Figures 1 to 2 The corresponding descriptions in [the document] will not be repeated here.

[0122] Based on the above, Figures 1 to 2 Accordingly, this embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. Figures 1 to 2 The method shown.

[0123] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0124] like Figure 4 The diagram shown is a hardware structure schematic of an electronic device according to the present invention, comprising: At least one processor 401; and, A memory 402 is communicatively connected to at least one of the processors 401; wherein, The memory 402 stores instructions that can be executed by at least one of the processors to enable the at least one of the processors to perform the task allocation method as described above.

[0125] Figure 4 Take a processor 401 as an example.

[0126] The electronic device may also include an input device 403 and a display device 404.

[0127] The processor 401, memory 402, input device 403, and display device 404 can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0128] Memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the task allocation method in the embodiments of this application, for example, Figures 1 to 2 The method flow is shown. The processor 401 executes various functional applications and data processing by running non-volatile software programs, instructions, and modules stored in the memory 402, thereby implementing the task allocation method in the above embodiments.

[0129] Memory 402 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created according to the use of the task allocation method, etc. Furthermore, memory 402 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 402 may optionally include memory remotely located relative to processor 401, and these remote memories may be connected via a network to the apparatus performing the task allocation method. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0130] Input device 403 can receive user clicks and generate signal inputs related to user settings for task allocation methods and function control. Display device 404 may include display devices such as a display screen.

[0131] When one or more modules are stored in the memory 402, and are run by one or more processors 401, the task allocation method in any of the above method embodiments is executed.

[0132] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0133] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0134] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. By applying the solution of this embodiment, compared with the prior art, this application can distinguish between large memory and long-duration test tasks and deploy these test tasks in a distributed manner, avoiding memory overflow, idle node computing power, and load imbalance caused by similar test tasks being concentrated on the same node. This allows each computing node to bear the load of the entire batch of power consumption test tasks in a balanced manner, thereby shortening the running cycle of the entire batch of power consumption test tasks and effectively improving the project development and delivery speed.

[0136] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0137] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A task allocation method, characterized in that, include: Retrieve multiple test tasks to be assigned to the test system; Predict the memory requirements and execution time data for the multiple test tasks to be executed in the test system; Based on the memory requirement data and the execution time data, from the plurality of test tasks, at least one test task whose memory requirement data is greater than a preset memory threshold is determined as at least one first target test task that meets the first target load condition; and from the plurality of test tasks, at least one test task whose memory requirement data is less than or equal to the preset memory threshold and whose execution time data is greater than the preset time threshold is determined as at least one second target test task that meets the second target load condition. According to the first test task allocation method corresponding to the first target load condition, at most one first target test task is allocated to the available computing nodes in the test system; According to the second test task allocation method corresponding to the second target load condition, at least one second target test task is allocated to each available computing node in the test system.

2. The method according to claim 1, characterized in that, The available computing nodes are those whose remaining memory resources in the test system meet the requirements for running the test task. After allocating at most one of the first target test tasks to the available computing nodes in the test system according to the first test task allocation method corresponding to the first target load condition, the method further includes: Based on the memory requirement data of the first target test task, update the remaining memory resource information of the available computing nodes allocated to the first target test task.

3. The method according to claim 1, characterized in that, The step of allocating at most one of the first target test tasks to the available computing nodes in the test system according to the first test task allocation method corresponding to the first target load condition includes: Iterate through the available computing nodes in the test system and assign at most one of the first target test tasks to each available computing node.

4. The method according to claim 1, characterized in that, According to the second test task allocation method corresponding to the second target load condition, at least one second target test task is allocated to each available computing node in the test system, including: The second target test task is assigned to the available computing nodes in a round-robin manner until each second target test task is assigned to the corresponding available computing node.

5. The method according to any one of claims 1-4, characterized in that, The predicted memory requirements and execution time data for the multiple test tasks to be executed in the test system include: Obtain historical runtime data of the test system executing historical test tasks, including historical memory usage data and historical execution duration data; The current test task is matched with historical test tasks. If the same historical test task is matched, the memory requirement and execution time of the current test task are determined based on the historical running data corresponding to the same historical test task. If no matching historical test task is found for the current test task, the memory requirement and execution time of the current test task are determined based on a preset prediction template.

6. The method according to any one of claims 1-4, characterized in that, According to the first test task allocation method corresponding to the first target load condition, at most one first target test task is allocated to the available computing nodes in the test system. According to the second test task allocation method corresponding to the second target load condition, at least one second target test task is allocated to each available computing node in the test system, including: A node mapping table is generated based on the test task allocation relationship between at least one target test task and the at least one different available computing node; the target test task includes a first target test task and / or a second target test task. The node mapping table is input into the test system, so that the test system schedules at least one different available computing node to execute at least one of the target test tasks according to the node mapping table.

7. The method according to claim 6, characterized in that, Before generating the node mapping table, the method further includes: The target test tasks are sorted from highest to lowest according to their business priority; For target test tasks with the same business priority, sort them from largest to smallest memory requirement; Target test tasks with the same memory requirements are sorted from longest to shortest execution time to obtain a target test task sequence, which is used to determine the scheduling order of the target test tasks.

8. A task allocation device, characterized in that, include: The acquisition module is configured to acquire multiple test tasks to be assigned to the test system; The prediction module is configured to predict the memory requirements and execution time data required for the execution of the multiple test tasks in the test system; The selection module is configured to, based on the memory requirement data and the execution duration data, determine at least one test task whose memory requirement data is greater than a preset memory threshold from the plurality of test tasks as at least one first target test task that meets the first target load condition; and, from the plurality of test tasks, determine at least one test task whose memory requirement data is less than or equal to the preset memory threshold and whose execution duration data is greater than the preset duration threshold as at least one second target test task that meets the second target load condition. The allocation module is configured to allocate at most one first target test task to the available computing nodes in the test system according to the first test task allocation method corresponding to the first target load condition. According to the second test task allocation method corresponding to the second target load condition, at least one second target test task is allocated to each available computing node in the test system.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. An electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images