Non-blocking automated test scheduling method and system based on heterogeneous file queue

Abstract

This invention provides a non-blocking automated test scheduling method and system based on heterogeneous file queues. The steps include: acquiring upstream test task requests and encapsulating them into task description files; constructing a file directory according to business scenario requirements, and atomically updating task status by moving the task description files between file directories; real-time monitoring of the load operation of execution nodes, generating a scheduling strategy for the task description files based on the load monitoring results and task status, and dynamically optimizing the strategy. Encapsulating test information into independent files and storing them in the directory enables task status recovery; asynchronous communication between the task generation end and the execution end through the file system improves system scalability; dynamically adjusting the scheduling strategy based on the priority and dependency relationships of file content; and optimizing resource utilization by having a daemon process running on the execution node monitor its own load and actively extract and read tasks.

Description

Technical Field

[0001] This invention relates to the field of automated testing technology, and more particularly to a non-blocking automated testing scheduling method and system based on heterogeneous file queues. Background Technology

[0002] In automated testing within a continuous integration / continuous delivery (CI / CD) environment, task scheduling is a critical step. Current mainstream testing frameworks (such as Jenkins) typically employ a centralized scheduling model, where a master node maintains an in-memory task queue. The master node receives task requests, queues them, and distributes tasks to execution nodes (Agents). This scheduling model relies heavily on the master node and has limitations: Firstly, the system has poor reliability (easily lost): once the master node crashes, restarts, or experiences a network partition, the status of all queued tasks in memory will be lost, causing the test to be interrupted and difficult to recover. Secondly, limited scalability (prone to blocking): When the number of test tasks surges, the scheduling thread of the master node may be blocked due to insufficient processing, which will affect the responsiveness of the entire CI / CD system and even prevent other build tasks from being triggered normally. Third, the scheduling strategy is rigid (lacking flexible control): It usually adopts a simple first-in-first-out (FIFO) queue, which makes it difficult to perform dynamic and flexible scheduling based on task priority, business urgency or inter-task dependencies. Fourth, insufficient resource utilization: the execution nodes are in a passive task receiving mode and cannot actively adjust the task retrieval rate according to their own real-time load, which may lead to some nodes being overloaded while others are idle.

[0003] To address the aforementioned shortcomings, there is an urgent need for a non-blocking automated test scheduling method and system based on heterogeneous file queues. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a non-blocking automated test scheduling method and system based on heterogeneous file queues. Test information is encapsulated as independent files and stored in a directory, enabling task state recovery. The task generation and execution ends communicate asynchronously through the file system, improving system scalability. Scheduling strategies are dynamically adjusted based on file content priority and dependency parsing. Resource utilization is optimized by using a daemon process running on the execution node to detect its own load and actively extract and read tasks.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: This invention provides a non-blocking automated test scheduling method based on heterogeneous file queues, comprising the following steps: S1: Obtain the upstream test task request, and serialize and encapsulate all the information required for the test into a task description file according to the test task request; S2: Construct a file directory to store the task description file according to business scenario requirements, and update the task status by atomically moving the task description file between the file directories; S3: Real-time detection of the load operation of the execution node, and based on the load detection results and the task status, generate the scheduling strategy of the task description file in combination with the priority algorithm; S4: Dynamically optimize the scheduling strategy based on real-time status feedback during task execution.

[0006] Preferably, all the information required for the test includes, but is not limited to, the storage path of the firmware under test, the software version number, the target board type, the set of test cases to be executed, the task priority, the retry count, and the parent task ID.

[0007] Preferably, the task description file is in serialized format, and the filename of the task description file contains a timestamp and a hash value, which is globally unique.

[0008] Preferably, the file directory includes a Scan directory for storing all newly generated, pending task description files, which are preferentially stored in the Scan directory after generation; a Processing directory for storing task description files that have been read by the execution node and are being executed; a Completed directory for storing task description files that have been successfully executed; and a Failed directory for storing task description files that failed to execute.

[0009] Preferably, the scheduling strategy for generating the task description file includes: The load status of the node is monitored in real time by a daemon running on the node, and load monitoring results are generated. Based on task priority, task dependency, and number of retries, candidate tasks that meet the execution conditions are filtered from the Scan directory using the critical path method to form a candidate task list; Based on the load detection results and the candidate task list, a task scheduling strategy is generated.

[0010] Preferably, the generation of load detection results includes: Initialize the daemon process, load preset configuration parameters, and define detection rules; The daemon process collects local resource data through the operating system's native interfaces or tool libraries at preset intervals and preprocesses the collected data. The load data is quantified based on the preprocessed data, and the load status of the execution nodes is classified according to preset thresholds to generate structured load constraint information.

[0011] Preferably, the process of forming the candidate task list includes: Read all task description files from the Scan directory, extract the core fields required for filtering, and build a basic task information table; Based on the task basic information table, the dependency relationship between tasks is sorted out based on the parent task field to build a dependency graph, and the completion status of the prerequisite dependencies of each task is verified to generate a dependency satisfaction task pool. Based on the dependency graph and the dependency satisfaction task pool, the total float time is calculated using the critical path method, and critical tasks and critical paths are selected to generate a list of critical tasks and a list of non-critical tasks. Based on the list of critical tasks and the list of non-critical tasks, the filtering order is optimized by combining task priority and retries, a weighted priority sorting table is generated, and the tasks are sorted in descending order to obtain a candidate task list.

[0012] Preferably, the task scheduling strategy includes: A mapping model between task type and resource consumption is established based on the least squares fitting algorithm, and the load judgment criteria are dynamically adjusted. Based on the adjusted load assessment criteria, the future load index change trend is predicted using the ARIMA model. Based on the trend of indicator changes, combined with the load detection results and the candidate task list, a task scheduling strategy is generated using a genetic algorithm.

[0013] Preferably, the dynamic optimization of the scheduling strategy includes: Real-time acquisition of task execution status, execution node load changes, and actual task resource consumption data, and structured processing thereof; The acquired real-time status data is analyzed to identify abnormal states that deviate from expectations and classified according to the degree of impact. For different levels of abnormal states, the original scheduling strategy is optimized by means of local fine-tuning or global recalculation.

[0014] In another aspect, the present invention provides a non-blocking automated test scheduling system based on heterogeneous file queues, including a communication connection: Test task acquisition module: used to acquire upstream test task requests, and serialize and encapsulate all the information required for the test into a task description file according to the test task request; File directory construction module: used to construct a file directory to store the task description file according to business scenario requirements, and to update the task status by atomically moving the task description file between the file directories; Load detection module: Used to detect the load status of execution nodes in real time through a running daemon process and generate load detection results; Scheduling decision module: used to generate a scheduling strategy for the task description file based on the load detection results and the task status, combined with a priority algorithm; Strategy optimization module: Used to dynamically optimize the scheduling strategy based on real-time status feedback during task execution.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: Improve system reliability: Test information is encapsulated as independent files and stored in a directory. The task state changes are realized by moving task files between directories. Even if the entire scheduling system crashes and restarts, all task states can be recovered without loss simply by looking at the directory where the files are located. The task queue is persisted in the file system, avoiding dependence on the master node and completely solving the problem of task loss caused by scheduling node failure, thus achieving lossless recovery of task states.

[0016] Improved scalability: The task generation end and the execution end communicate asynchronously through the file system, without blocking each other, which improves the overall throughput and responsiveness of the system and overcomes the drawback of centralized scheduling being prone to blocking.

[0017] Dynamic scheduling strategy adjustment: Based on the priority and dependency parsing of file content, it can realize complex and dynamic scheduling logic, and can flexibly adjust the scheduling strategy according to evaluation indicators, realizing intelligent dynamic scheduling and breaking the rigid limitations of FIFO queues.

[0018] Improve resource utilization: The daemon running on the execution node can detect the load of its own node in real time, actively and periodically scan the directory based on the detection results, and read task files according to the priority algorithm, thereby optimizing resource utilization and realizing backpressure control based on resource availability. Attached Figure Description

[0019] Figure 1 This is a flowchart of the non-blocking automated test scheduling method based on heterogeneous file queues of the present invention; Figure 2 This is a block diagram of the non-blocking automated test scheduling system based on heterogeneous file queues according to the present invention; Figure 3 This is a flowchart of step S3 of the present invention; Figure 4 This is a flowchart of the execution process of a specific embodiment of the present invention. Detailed Implementation

[0020] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0021] Please see Figure 1 As shown, this invention provides a non-blocking automated test scheduling method based on heterogeneous file queues, comprising the following steps: S1: Obtain the upstream test task request, and serialize and encapsulate all the information required for the test into a task description file according to the test task request; All information required for the test includes, but is not limited to, the storage path of the firmware under test, the software version number, the target board type, the set of test cases to be executed, the task priority, the retry count, and the parent task ID.

[0022] The storage path of the firmware under test points to the specific storage location (local path / network shared path) of the firmware under test (such as software installation package, hardware driver, embedded program). The execution node can accurately obtain the test object through this path, avoid test interruption due to resource shortage, and ensure the location of test resources.

[0023] The software version number is a unique version identifier for the software under test (such as V3.2.1, Build20240608, GitCommitID), used to distinguish test tasks of different versions, support version regression testing and multi-version parallel testing, and avoid result deviations caused by version confusion.

[0024] The target board type refers to the hardware execution environment corresponding to the test task, such as a specific model of server board, embedded development board, or test-specific hardware module. This helps the execution node determine whether its own hardware is compatible, avoids assigning tasks to incompatible nodes, and improves resource matching efficiency.

[0025] The test case set to be executed is the list of test cases to be executed, which includes test case ID, test case name, execution order, and test case script association information. It is used to clarify the test scope and execution logic, and to ensure that the execution nodes complete the test coverage as expected, without missing any key scenarios.

[0026] Task priority is an indicator of the importance / urgency of a task, supporting dynamic scheduling strategies. It prioritizes the execution of critical tasks such as core business operations and emergency return, breaking the rigid limitations of traditional FIFO.

[0027] The retry count is the number of times a task is allowed to retries after it fails. It is used to deal with temporary failures (network fluctuations, temporary hardware occupation), automatically trigger retries, improve the success rate of tasks, and reduce manual intervention.

[0028] The parent task ID (parent_task_id) is a unique ID that identifies the preceding tasks that the current task depends on, such as associated preceding tests and environment preparation tasks. It enables task dependency orchestration, ensuring that the current task is executed only after the preceding tasks are completed, thus avoiding test failures caused by missing dependencies.

[0029] The task description file is in a serialized format, such as JSON, YAML, XML, or other serialized formats. The filename of the task description file contains a timestamp and a hash value, and is globally unique.

[0030] It should be noted that the task description file's filename includes a timestamp, used to distinguish tasks generated at different times and indirectly reflecting the task generation sequence; and a hash code, which is a unique hash value calculated based on the task's core information (such as the firmware path under test, software version number, task priority, etc.), used to completely avoid filename conflicts between different tasks generated at the same time, ensuring a globally unique identifier. In addition to the two core essential pieces of information, the timestamp and hash code, the filename can be supplemented with secondary identifiers based on the actual scenario, such as task type and business line identifier.

[0031] S2: Construct a file directory to store the task description file according to business scenario requirements, and update the task status by atomically moving the task description file between the file directories; The file directory includes a Scan directory, which stores all newly generated, pending task description files. After a task description file is generated, it is preferentially stored in the Scan directory. A Processing directory, which stores task description files that have been read by the execution node and are being executed. A Completed directory, which stores task description files that have been successfully executed. A Failed directory, which stores task description files that failed to execute.

[0032] In this embodiment, four file directories are constructed, corresponding to the four states of a task. After the task description file is generated, it is placed into a preset scan directory (Scan Directory) using atomic operations (such as the `mv` command). The transitions between task states do not depend on the database or memory markers, but are achieved atomically by moving the task description file between the four file directories. For example, when an execution node reads a task from the Scan directory, it moves the read task into the Processing directory by performing an atomic `mv` operation. This mechanism ensures that the task state is clear and persistent at all times. Even if the entire scheduling system crashes and restarts, all task states can be recovered without loss simply by referring to the directories where the files are located, improving system reliability.

[0033] It should be noted that in practical applications, an appropriate number of file directories can be built according to the specific business scenario requirements, and the names of the file directories can be specially adjusted. For example, a Pending directory and a Cancelled directory can be added.

[0034] S3: Real-time detection of the load operation of the execution node, and based on the load detection results and the task status, generate the scheduling strategy of the task description file in combination with the priority algorithm; Please see Figure 3 As shown, the scheduling strategy for generating the task description file includes: The load status of the node is monitored in real time by a daemon running on the node, and load monitoring results are generated. In this embodiment, a daemon process (Watcher) runs on the execution node. The daemon process actively and periodically scans the Scan directory, avoiding passively waiting for the master node to dispatch tasks and reducing the dependence on the master node.

[0035] The generation of load detection results includes: Initialize the daemon process and load the preset configuration parameters; Configure the set of test metrics: Based on the characteristics of the test task (such as CPU-intensive or I / O-intensive tests), select the core metrics to be collected, such as CPU utilization, memory usage, number of available test devices, network I / O throughput, and disk I / O read / write speed.

[0036] Configure sampling parameters: Set the sampling period and data acquisition timeout threshold (e.g., if no data is collected within 300ms, it is judged as a temporary anomaly). Among them, the detection frequency is adaptively adjusted according to load changes based on the PID control algorithm, balancing real-time performance and resource consumption. The degree to which the load deviates from the normal threshold is used as the deviation value (P), and the sampling period is dynamically adjusted by combining the cumulative deviation (I) and the rate of change of deviation (D).

[0037] Configure load thresholds: Set the "normal / overload" thresholds for each indicator. The thresholds can be customized according to business scenarios and dynamically adjusted based on statistical distribution to adapt to dynamic load scenarios.

[0038] The daemon process collects local resource data through the operating system's native interfaces or tool libraries at preset intervals and preprocesses the collected data. In this embodiment, the collected indicators include: CPU utilization: Read the system's / proc / stat file (Linux system), calculate the percentage of CPU idle time per unit time, and deduce the utilization rate in reverse; Memory usage: Read the / proc / meminfo file to get the total physical memory capacity and the used capacity (including cache and process usage), and calculate the ratio of "used capacity / total capacity"; Number of available test devices: Query the number of locally available test hardware (such as target boards and testers) through the device management interface (such as USB device enumeration, board communication protocol) to troubleshoot faulty or occupied devices; Network I / O throughput: Read the / proc / net / dev file to count the number of bytes sent and received by the network card per unit time, and determine whether the network is under high load (if it exceeds 100MB / s, it is determined to be a network bottleneck). Disk I / O read / write rate: Use the iostat tool or the / libaio library to collect the disk read / write counts per second (IOPS) and read / write bandwidth to avoid test task lag caused by disk I / O blocking.

[0039] The collected data undergoes preprocessing. If the raw metrics exhibit instantaneous peaks, the daemon process may misjudge the load as overloaded, missing the opportunity to pull tasks. A moving average algorithm is used to smooth out these instantaneous fluctuations, reducing the probability of misjudgment and ensuring the stability of load assessment. To reflect the reference value of recent data, an exponential smoothing algorithm can be used to assign higher weights to recent data, dynamically tracking load change trends.

[0040] The load data is quantified based on the preprocessed data, and the load status of the execution nodes is classified according to preset thresholds to generate structured load constraint information.

[0041] The node load is divided into three levels based on preset thresholds, and the task pull permissions for each level are clearly defined to avoid node overload and achieve backpressure control. For example: Normal level: all indicators ≤ threshold (CPU ≤ 80%, memory ≤ 75%), available devices ≥ 1, can pull high / medium / low priority tasks, no quantity limit; Light load level, 1-2 non-core indicators exceed the limit (such as CPU=85%, memory is normal), only high / medium priority tasks are pulled, with a maximum of 1 task at a time; If the load level is too high and core metrics exceed limits (e.g., CPU usage ≥ 90% or available devices are 0), pause fetching new tasks and only process already received tasks.

[0042] Based on task priority, task dependencies, and number of retries, candidate tasks that meet the execution conditions are filtered from the Scan directory using the Critical Path Method (CPM) to form a candidate task list; The process of forming a candidate task list includes: Read all task description files from the Scan directory (pool of tasks to be processed), extract the core fields required for filtering, and build a basic task information table; The core fields include task ID, parent task ID, task priority, number of retries, and estimated execution time (duration). Tasks with incorrect file formats or missing core fields are directly marked as invalid and will not participate in subsequent filtering. Priority is quantified into numerical values ​​to facilitate weight calculation. For tasks without historical execution data, a default estimated duration is assigned according to task type (such as interface testing or performance testing), ultimately generating a task basic information table.

[0043] Based on the task basic information table, the dependency relationship between tasks is sorted out based on the parent task field to build a dependency graph, and the completion status of the prerequisite dependencies of each task is verified to generate a dependency satisfaction task pool. Each valid task in the task basic information table is a node. The node attributes include task ID, priority, retry_count, and duration. If the parent_task_id of task B is the task_id of task A, then a directed edge from A to B is added, indicating that "B can only be executed after A is completed". The dependency graph is stored using an adjacency list.

[0044] By querying the Completed directory, it is determined whether the prerequisites for each task have been completed. If there are no dependent tasks, it is directly judged as "dependency satisfied". If there are dependent tasks, it is checked whether the result file corresponding to parent_task_id exists in the Completed directory (if the file exists, the parent task has been completed). If it exists, it is judged as "dependency satisfied"; if it does not exist, it is marked as "dependency not ready" and is not included in the candidate pool. If there is a cycle in the dependency graph (such as T6→T7→T6), all tasks within the cycle are marked as "dependency abnormal" and excluded from the filtering scope. Finally, a dependency satisfied task pool is generated, which only contains valid, dependency satisfied, and non-circular dependency tasks.

[0045] Based on the dependency graph and the dependency satisfaction task pool, the total float time is calculated using the critical path method, and critical tasks and critical paths are selected to generate a list of critical tasks and a list of non-critical tasks. First, based on the dependency graph and the dependency satisfaction task pool definition: Earliest start time (ES) is the earliest time a task can start execution, which requires all preceding dependent tasks to complete; Earliest completion time (EF) = ES + duration (estimated execution time); Latest start time (LS) is the latest time a task can start execution without affecting the overall test cycle; Latest completion time (LF) = LS + duration; Total float time (TF) = LS - ES. ​​TF = 0 indicates that the task has no buffer time and is a critical task; TF > 0 indicates that the task has buffer time and is a non-critical task. The link formed by all critical tasks with TF = 0 constitutes the critical path, and the total duration of this link (the sum of the durations of all tasks) is the longest.

[0046] Forward traversal: Perform topological sorting on the dependency graph, for example, the topological sequence is [T1, T4, T2, T5]. Initialize the earliest start time (ES) of tasks without dependencies to 0 (the task can start immediately; the earliest start time refers to the earliest time a task can begin execution), and the earliest completion time (EF) = ES + duration (estimated execution time). Traverse subsequent tasks and calculate ES and EF. If a task has multiple prerequisites, then ES is the maximum value of the EF of all prerequisite tasks, and this requires all prerequisite tasks to be completed. If a task has only one prerequisite, then ES is the EF of the prerequisite task.

[0047] Reverse traversal: Traverse the topological sequence in reverse, for example, the reverse sequence [T5,T2,T4,T1]. Initialize the LF of a task without subsequent tasks as its own EF (no buffer, latest completion time = earliest completion time), and LS = LF - duration. Traverse the preceding tasks to calculate LF and LS. If a task has multiple subsequent tasks, then LF is the minimum of the ES of all subsequent tasks, and execution time should be reserved for subsequent tasks. If a task has only one subsequent task, then LF is the ES of the subsequent task.

[0048] Calculate the total float time (TF) for each task (LS - ES), select tasks with TF=0 as critical tasks, form the critical path with all critical tasks, and finally generate a list of critical tasks and a list of non-critical tasks, clarifying the core CPM metrics for each task.

[0049] Based on the list of critical tasks and the list of non-critical tasks, the filtering order is optimized by combining task priority and retries, a weighted priority sorting table is generated, and the tasks are sorted in descending order to obtain a candidate task list.

[0050] Weights are assigned to three metrics: CPM score, task priority, and number of retries. A weighted priority score is then calculated using these weighted averages. For CPM score, critical tasks (TF=0) receive 10 points, while non-critical tasks receive a score of (1-TF / maximum TF) × 10 (lower TF results in higher scores). Task priority is directly calculated using the quantified priority value. For retries, retry_count=0 receives 1 point; retry_count=1 receives 5 points; and retry_count≥2 receives 10 points (prioritizing retries). A weighted priority ranking table is generated based on the calculated weighted priority scores, arranging tasks in descending order of weighted priority score to prevent low-priority tasks on the critical path from preempting resources or overlooking high-priority non-critical tasks.

[0051] Based on the load detection results and the candidate task list, a task scheduling strategy is generated.

[0052] The generated task scheduling strategy includes: A mapping model between task type and resource consumption is established based on the least squares fitting algorithm, and the load judgment criteria are dynamically adjusted. From the task execution log database of the execution node, extract historical task types and resource consumption data to form a structured dataset. Task complexity is extracted as an auxiliary feature and normalized. Task types are converted into numerical codes, such as: Interface Test = 1, Performance Test = 2, Regression Test = 3, Smoke Test = 4. A fitting model is selected based on the linear / non-linear relationship between task type and resource metrics, and the fitting parameters are calculated to obtain the mapping model. When the daemon process of the execution node receives a candidate task, it calls the mapping model and, combined with the node's maximum load threshold, dynamically calculates the load capacity threshold for the current task. Specifically, the task type and task complexity are read from the task description file and quantified as X1 and X2, which are then substituted into the mapping model to calculate the estimated resource consumption of the task. The dynamic load threshold is the difference between the node's maximum load threshold (a fixed value, such as a CPU maximum threshold of 80%) and the task's estimated resource consumption.

[0053] Let's take CPU consumption modeling as an example for explanation: If the relationship between "Task Type → CPU Consumption" is linear, select the linear regression model. The formula is: Y = a × X1 + b × X2 + c; In the formula, Y represents the estimated CPU utilization of the task, X1 represents the quantized task type, X2 represents the normalized task complexity, and a / b / c represent the parameters to be fitted.

[0054] The fitting parameters are obtained by minimizing the sum of squared errors between the predicted and actual values, or directly calculated using Python's numpy.linalg.lstsq or scikit-learn's LinearRegression library. The calculated fitting parameters are then substituted into the model, assuming the resulting model is: Estimated CPU utilization = 10 × task type + 20 × task complexity + 5.

[0055] In a specific embodiment, assuming the maximum CPU load threshold of the execution node is 80%, the current candidate task is interface testing (task type = 1), the number of interfaces = 10 (normalized complexity is 0.5), and substituting into the mapping model for calculation: estimated CPU utilization = 10 × 1 + 20 × 0.5 + 5 = 25%; dynamic CPU threshold = 80% - 25% = 55%; if the current CPU utilization of the execution node is ≤ 55%, then the "interface testing" task is allowed; otherwise, it is rejected (to avoid the CPU exceeding 80% after the task is executed).

[0056] Based on the adjusted load assessment criteria, the future load index change trend is predicted using the ARIMA model. It should be noted that the adjusted load assessment metrics are extracted from the previous steps (least squares fitting dynamic adjustment) to obtain the "dynamic load threshold" and related core resource metrics for the current candidate task, such as CPU utilization and memory usage. One or more of these metrics are then predicted separately, prioritizing the metric with the highest task dependency. For example, for CPU-intensive tasks, CPU metrics are predicted first. It is crucial that the predicted metrics and the dynamic threshold have completely consistent definitions. For instance, if the dynamic threshold is average CPU utilization, then the predicted object is also average CPU utilization, not peak utilization.

[0057] An ARIMA model is built and trained based on historical load data to predict the values ​​and trends of load indicators in future periods. The input is the latest load data and the prediction duration; the output is the predicted load indicator values ​​for each future time point (e.g., the predicted CPU utilization every 30 seconds within the next 10 minutes) and the prediction confidence interval. Based on the predicted value sequence, the changing trends of the load indicators (upward trend, downward trend, and stable trend) are analyzed and output in a structured format.

[0058] Based on the trend of indicator changes, combined with the load detection results and the candidate task list, a task scheduling strategy is generated using a genetic algorithm.

[0059] Using genetic algorithms as the optimization tool, three types of constraints—"load trend, current load, and candidate tasks"—are transformed into quantifiable optimization objectives. By simulating the biological evolution process, a globally optimal scheduling strategy is generated, which avoids overloading execution nodes (meeting non-blocking requirements) while maximizing the execution efficiency of test tasks (adapting to high-concurrency scenarios in automated testing). Specifically: The three core scheduling decisions—"task allocation execution node, task execution order, and task start timing"—are encoded as chromosomes, with constraints including task dependency (if task B depends on task A, then A's execution order must be earlier than B's) and resource requirements (task resource requirements ≤ node's current / future remaining load). A fitness function is constructed by integrating load constraints, task priority, and resource utilization as three objectives. Fitness = w1 × Priority Score + w2 × Resource Utilization - w3 × Overload Penalty - w4 × Delay Penalty; Among them, resource utilization rate: the average resource usage of all execution nodes; overload penalty: if the strategy causes the current / future load of the execution node to exceed the threshold, points will be deducted by multiplying the number of overloads by a fixed penalty value; delay penalty: the sum of the difference between the actual completion time and the expected time of the task, the longer the delay, the more points will be deducted; the weights w1~w4 can be adjusted according to the specific business scenario requirements.

[0060] Based on three rules—task allocation (randomly assigning candidate tasks to idle / low-load execution nodes), execution order (randomly sorting tasks while satisfying task dependencies), and start timing (randomly setting based on current load and predicted trends)—the population is initialized, and N initial strategies conforming to the encoding constraints are randomly generated. Using either roulette wheel or elite retention, chromosomes with high fitness are prioritized for retention. The core segments of two high-fitness chromosomes (scheduling strategies) are combined to generate a new strategy. Some genes in the chromosome are randomly adjusted (scheduling decisions). Through repeated selection-crossover-mutation and multiple rounds of evolutionary iteration, the scheduling strategy with the highest fitness is gradually selected, and finally, the chromosome with the highest fitness is output, representing the optimal task scheduling strategy.

[0061] S4: Dynamically optimize the scheduling strategy based on real-time status feedback during task execution.

[0062] The dynamic optimization of the scheduling strategy includes: Real-time acquisition of task execution status, execution node load changes, and actual task resource consumption data, and structured processing thereof; During task execution, the task execution status is fed back in real time, and the scheduling strategy is dynamically adjusted to ensure non-blocking characteristics and resource optimization. If the task is executed successfully, it is moved to the Completed directory, node resources are released, the scheduling decision module updates the remaining number of tasks that can be handled, and the next batch of tasks can be pulled. If the task fails, it is moved to the Failed directory, and whether it is re-added to the Scan directory is determined based on the number of retries (if retry_count > 0, the count is decremented and the task is put back; otherwise, it is marked as a permanent failure).

[0063] The acquired real-time status data is analyzed to identify abnormal states that deviate from expectations and classified according to the degree of impact. Anomalies include task execution anomalies, resource consumption deviation anomalies, execution node load fluctuation anomalies, and task dependency anomalies. Anomalies are classified into three levels, including: Minor anomalies: slight delay in a single task (delay < 10% of expected duration), resource consumption deviation < 30%, slight load fluctuation, low optimization priority, local fine-tuning (no need to recalculate strategy). Moderate anomalies: single task failure / timeout, resource consumption deviation of 30%-50%, brief overload of a single node, slight delay of dependent tasks. Optimization priority is in progress, with local optimization (adjusting individual tasks). Severe anomalies: multiple tasks fail / time out in batches, resource consumption deviation > 50%, multiple nodes continuously overloaded, core dependent tasks fail, optimization priority is high, global recalculation (regeneration of strategy) is performed.

[0064] For different levels of abnormal states, the original scheduling strategy is optimized by means of local fine-tuning or global recalculation.

[0065] For mild and severe anomalies, a local fine-tuning approach is adopted. For example, in a scenario with a large deviation in resource consumption due to moderate anomalies, the optimization operation is as follows: immediately correct the dynamic load threshold corresponding to the task (recalculate according to the actual resource consumption); adjust the allocation of subsequent tasks on the execution node (prioritize the allocation of low resource consumption tasks); and record the deviation data for subsequent least squares model iteration.

[0066] For severe anomalies, a global recalculation approach is adopted, which involves updating the candidate task list, load data, and resource model data, invoking a genetic algorithm, and combining the latest real-time status data to generate a new global scheduling strategy. This strategy is then deployed in batches to nodes (global strategy) or targeted to a single node (local strategy), using a structured format for easy node parsing. After the strategy is deployed, the task execution status is monitored in real time to confirm its effectiveness. The results of each scheduling strategy optimization and real-time status data are stored in a historical database for iterative optimization of the preceding core model, improving the accuracy of subsequent scheduling strategies and forming a closed-loop optimization.

[0067] On the other hand, please see Figure 2 As shown, this invention provides a non-blocking automated test scheduling system based on heterogeneous file queues, including communication connections: Test task acquisition module: used to acquire upstream test task requests, and serialize and encapsulate all the information required for the test into a task description file according to the test task request; File directory construction module: used to construct a file directory to store the task description file according to business scenario requirements, and to update the task status by atomically moving the task description file between the file directories; Load detection module: Used to detect the load status of the node in real time by executing the daemon process running on the node, and obtain the load detection results; Scheduling decision module: used to generate a scheduling strategy for the task description file based on the load detection results and the task status, combined with a priority algorithm; Strategy optimization module: Used to dynamically optimize the scheduling strategy based on real-time status feedback during task execution.

[0068] Please see Figure 4 The following is a flowchart of a specific embodiment of the present invention: Obtain the upstream test task request, and serialize and encapsulate all the information required for the test into a JSON format task description file according to the test task request; after the task description file is generated, store the task description file in the Scan directory by performing an atomic move operation; The execution node periodically scans the Scan directory through a daemon process and simultaneously monitors its own load. This is achieved by using real-time metrics (CPU utilization, memory usage, number of available test devices, network I / O, disk I / O, etc.) to determine the load. If the load is detected to be too high, no tasks are read, and the node waits for the next cycle. If the load is detected to be normal, a priority algorithm is used to read high-priority task description files with retry_count > 0 from the Scan directory, and the read task description files are atomically moved to the Processing directory. The test task is executed based on the task description file in the Processing directory. If the task is executed successfully, the task description file is atomically moved to the Completed directory, and the test task ends. If the task fails, the task description file is atomically moved to the Failed directory, and rescheduling can be determined based on retry_count.

[0069] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A non-blocking automated test scheduling method based on heterogeneous file queues, characterized in that, Includes the following steps: S1: Obtain the upstream test task request, and serialize and encapsulate all the information required for the test into a task description file according to the test task request; S2: Construct a file directory to store the task description file according to business scenario requirements, and update the task status by atomically moving the task description file between the file directories; S3: Real-time detection of the load operation of the execution node, and based on the load detection results and the task status, generate the scheduling strategy of the task description file in combination with the priority algorithm; S4: Dynamically optimize the scheduling strategy based on real-time status feedback during task execution.

2. The non-blocking automated test scheduling method based on heterogeneous file queues according to claim 1, characterized in that, All information required for the test includes, but is not limited to, the storage path of the firmware under test, the software version number, the target board type, the set of test cases to be executed, the task priority, the retry count, and the parent task ID.

3. The non-blocking automated test scheduling method based on heterogeneous file queues according to claim 1, characterized in that, The task description file is in serialized format, and its filename contains a timestamp and a hash value, ensuring global uniqueness.

4. The non-blocking automated test scheduling method based on heterogeneous file queues according to claim 1, characterized in that, The file directory includes a Scan directory, which stores all newly generated and pending task description files. After the task description files are generated, they are preferentially stored in the Scan directory. The Processing directory stores task description files that have been read by the execution node and are being executed. The Completed directory stores description files for tasks that have been successfully completed; the Failed directory stores description files for tasks that failed to complete.

5. The non-blocking automated test scheduling method based on heterogeneous file queues according to claim 1, characterized in that, The scheduling strategy for generating the task description file includes: The load status of the node is monitored in real time by a daemon running on the node, and load monitoring results are generated. Based on task priority, task dependency, and number of retries, candidate tasks that meet the execution conditions are filtered from the Scan directory using the critical path method to form a candidate task list; Based on the load detection results and the candidate task list, a task scheduling strategy is generated.

6. The non-blocking automated test scheduling method based on heterogeneous file queues according to claim 5, characterized in that, The generated load detection results include: Initialize the daemon process, load preset configuration parameters, and define detection rules; The daemon process collects local resource data through the operating system's native interfaces or tool libraries at preset intervals and preprocesses the collected data. The load data is quantified based on the preprocessed data, and the load status of the execution nodes is classified according to preset thresholds to generate structured load constraint information.

7. The non-blocking automated test scheduling method based on heterogeneous file queues according to claim 5, characterized in that, The process of forming the candidate task list includes: Read all task description files from the Scan directory, extract the core fields required for filtering, and build a basic task information table; Based on the task basic information table, the dependency relationship between tasks is sorted out based on the parent task field to build a dependency graph, and the completion status of the prerequisite dependencies of each task is verified to generate a dependency satisfaction task pool. Based on the dependency graph and the dependency satisfaction task pool, the total float time is calculated using the critical path method, and critical tasks and critical paths are selected to generate a list of critical tasks and a list of non-critical tasks. Based on the list of critical tasks and the list of non-critical tasks, the filtering order is optimized by combining task priority and retries, a weighted priority sorting table is generated, and the tasks are sorted in descending order to obtain a candidate task list.

8. The non-blocking automated test scheduling method based on heterogeneous file queues according to claim 5, characterized in that, The generated task scheduling strategy includes: A mapping model between task type and resource consumption is established based on the least squares fitting algorithm, and the load judgment criteria are dynamically adjusted. Based on the adjusted load assessment criteria, the future load index change trend is predicted using the ARIMA model. Based on the trend of indicator changes, combined with the load detection results and the candidate task list, a task scheduling strategy is generated using a genetic algorithm.

9. The non-blocking automated test scheduling method based on heterogeneous file queues according to claim 1, characterized in that, The dynamic optimization of the scheduling strategy includes: Real-time acquisition of task execution status, execution node load changes, and actual task resource consumption data, and structured processing thereof; The acquired real-time status data is analyzed to identify abnormal states that deviate from expectations and classified according to the degree of impact. For different levels of abnormal states, the original scheduling strategy is optimized by means of local fine-tuning or global recalculation.

10. A non-blocking automated test scheduling system based on heterogeneous file queues, applied to the non-blocking automated test scheduling method based on heterogeneous file queues according to any one of claims 1-9, characterized in that, Including communication connections: Test task acquisition module: used to acquire upstream test task requests, and serialize and encapsulate all the information required for the test into a task description file according to the test task request; File directory construction module: used to construct a file directory to store the task description file according to business scenario requirements, and to update the task status by atomically moving the task description file between the file directories; Load detection module: Used to detect the load status of execution nodes in real time through a running daemon process and generate load detection results; Scheduling decision module: used to generate a scheduling strategy for the task description file based on the load detection results and the task status, combined with a priority algorithm; Strategy optimization module: Used to dynamically optimize the scheduling strategy based on real-time status feedback during task execution.

Images