Automatic test method and device based on test sequence generation, equipment and medium

Abstract

The invention provides an automatic testing method and device based on test sequence generation, equipment and a medium, and relates to the technical field of server testing. The method comprises the following steps: acquiring natural language test requirement data, and generating a structured target test step sequence based on a test requirement; then resource state information of the environment to be tested is collected, the resource state information is combined with the target test step sequence, and a test task scheduling instruction used for indicating a step execution strategy is generated; and distributing the test task scheduling instruction to a plurality of distributed execution nodes in a to-be-tested environment, and executing test steps in the scheduling instruction by the distributed execution nodes to obtain a corresponding test execution result. According to the scheme, automatic generation from natural language requirements to executable test steps can be achieved, reasonable scheduling is carried out based on the environment resource condition, and the execution efficiency and accuracy of the test task in the distributed environment are improved.

Description

Technical Field

[0001] This disclosure relates to the field of server testing technology, and more specifically, to an automated testing method, apparatus, device, and medium based on test sequence generation. Background Technology

[0002] In the field of server system testing, verifying the functionality of software and hardware in large-scale cluster environments typically requires executing numerous structured test procedures. A widely adopted approach is for testers to manually write scripts or static configuration files to describe the test tasks, which are then deployed to the target environment for execution. In this model, the design and execution of test content heavily rely on human experience, and the test coverage is limited by the testers' knowledge and scripting abilities, making it difficult to adapt to diverse software version combinations and hardware configuration changes in cluster environments.

[0003] As cluster size increases, testing processes often need to be executed across multiple nodes. Currently, there are solutions based on automation frameworks in related testing technologies, which control the testing process through pre-defined test case combinations and pipeline mechanisms. However, such test cases still rely on pre-defined static definitions, and the test process and scheduling strategies depend on human experience, resulting in low efficiency, high repetition rates, and insufficient test coverage. Furthermore, they cannot exhaustively enumerate or combine complex hardware and software configurations, leading to low accuracy of test results.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide an automated testing method, an automated testing device, an electronic device, and a computer-readable storage medium based on test sequence generation, thereby improving the execution efficiency and accuracy of testing tasks in a distributed environment.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] According to a first aspect of the present disclosure, an automated testing method based on test sequence generation is provided, comprising: Acquire test requirement data in natural language form, and generate a target test step sequence based on the test requirement data. The target test step sequence is an ordered sequence of test actions to be executed, consisting of multiple test steps. Collect resource status information of the environment to be tested, and combine the resource status information with the target test step sequence to generate test task scheduling instructions. The test task scheduling instructions are a set of structured control instructions used to guide the distributed execution nodes in the environment to be tested to execute test tasks. The test task scheduling instruction is assigned to a distributed execution node in the environment to be tested, so that the test steps in the test task scheduling instruction are executed by the distributed execution node to obtain the test execution result.

[0008] In some example embodiments of this disclosure, based on the foregoing scheme, generating the target test step sequence based on the test requirement data includes: The test requirement data in natural language form is preprocessed to obtain test requirement word vectors; The test requirement word vectors are input into the trained test sequence generation model to obtain the target test step sequence.

[0009] In some example embodiments of this disclosure, based on the foregoing scheme, the test sequence generation model includes a semantic encoder and a sequence decoder, and the test sequence generation model is obtained through a first model training process, which includes: Obtain the first training dataset, which includes training and testing requirements data in natural language form and reference test step labels; The training and testing requirements data are input into the semantic encoder to obtain a context-dependent hidden state representation vector; The hidden state representation vector is input into the sequence decoder, and the sequence decoder is forced to learn by inputting the reference test step label, thereby completing the training of the test sequence generation model.

[0010] In some example embodiments of this disclosure, based on the foregoing scheme, the step of generating a test task scheduling instruction by combining the resource status information with the target test step sequence includes: Based on the resource status information and the target test step sequence, a state vector is constructed; The state vector is input into the trained test step optimization model to infer the probability distribution of each current scheduling action. The current scheduling action is used to indicate the execution order of assigning at least two test steps to each distributed execution node. A test task scheduling instruction is generated based on the current scheduling action with the highest probability value in the probability distribution.

[0011] In some example embodiments of this disclosure, based on the foregoing scheme, the test step optimization model during the training phase includes at least a policy network and a value network. The test step optimization model is obtained through a second model training process, which includes: The sampled state vector in the pre-designed state space is input into the policy network, so that the policy network, in combination with the sampled state vector, outputs the sampled scheduling action belonging to the pre-designed action space. The sampled state vector and the sampled scheduling action are input into the value network so that the combination of the sampled state vector and the sampled scheduling action is evaluated by the reward function of the value network to generate an evaluation reward value. The policy network is updated with policy gradients using the evaluation reward value, and the value network is updated with parameter regression using the evaluation reward value. When the loss values ​​of the policy network and the value network are less than or equal to a preset loss threshold, or when the number of iterations in the training process reaches a preset maximum number of iterations, the trained policy network and the value network are obtained. The evaluation reward value is obtained by the reward function based on at least one of the following: test pass rate, resource utilization rate, average task completion time, and improvement in historical coverage.

[0012] In some example embodiments of this disclosure, based on the foregoing scheme, after allocating the test task scheduling instruction to distributed execution nodes in the environment under test, so as to execute the test steps in the test task scheduling instruction through the distributed execution nodes and obtain the test execution result, the method further includes: Based on the test task scheduling instructions and the test execution results, determine the abnormal test path; By combining preset historical failure samples and abnormal feature points in the abnormal test path, the abnormal test path is adjusted to obtain the target test path; The target test path is used as an improvement sample to update the test sequence generation model used to generate the target test step sequence.

[0013] In some example embodiments of this disclosure, based on the foregoing scheme, determining the abnormal test path according to the test task scheduling instruction and the test execution result includes: The test task scheduling instruction and the test execution result are input into the trained anomaly test prediction model, and the probability distribution data of each test path is output. The anomaly test prediction model is obtained by training and modeling based on historical execution logs. The historical test execution logs are log data formed by recording the execution status, execution result and resource usage of the test path or test step corresponding to the test task scheduling instruction during the execution of historical test tasks. Under the constraint of abnormal probability in the probability distribution data, test paths with test coverage or test pass rate less than a preset proportion threshold are selected as abnormal test paths.

[0014] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the step of optimizing and adjusting the abnormal test path by combining preset historical failure samples and abnormal feature points in the abnormal test path to obtain the target test path includes: Extract target failure samples that match the abnormal feature points from the historical failure samples; Based on the target failure samples, test step adjustment suggestions are generated, which include replacing the software source, adding version rollback tests, or adding abnormal simulation scenarios. Based on the proposed test step adjustment, the test steps in the abnormal test path are adjusted to generate the target test path.

[0015] In some exemplary embodiments of this disclosure, based on the foregoing scheme, the step of using the target test path as an improvement sample to update the test sequence generation model used to generate the target test step sequence, so as to achieve self-evolutionary processing of the test step sequence, includes: The target test path is combined with the corresponding natural language test requirement data to form a new training sample pair; The new training sample pair is added to the training dataset of the test sequence generation model; The test sequence generation model is retrained using the updated training dataset to adjust its model parameters.

[0016] In some example embodiments of this disclosure, based on the foregoing scheme, after allocating the test task scheduling instruction to distributed execution nodes in the environment under test, so as to execute the test steps in the test task scheduling instruction through the distributed execution nodes and obtain the test execution result, the method further includes: Based on the test task scheduling instructions and the test execution results, a successful test path is determined. The successful test path is a test path whose test coverage or test pass rate is greater than or equal to a preset ratio threshold. New state-action pair training data are constructed using the test task scheduling instructions and resource status information corresponding to the successful test path. Using the new state-action pair training data, the policy gradient in the policy network of the test step optimization model used to generate the test task scheduling instructions is updated to optimize the model parameters of the policy network.

[0017] In some example embodiments of this disclosure, based on the foregoing scheme, the step of allocating the test task scheduling instruction to the distributed execution nodes in the environment under test includes: Based on the computing resource status information of the distributed execution nodes, the node identification information of each distributed execution node is determined. The node identification information is a node identifier dynamically allocated according to the computing resource status information of each distributed execution node. Based on the node identification information, each of the distributed execution nodes is bound to the test steps in the test task scheduling instruction; The test steps corresponding to the test task scheduling instruction are sent to the bound distributed execution nodes so that each distributed execution node can execute its corresponding test steps to obtain the test execution results.

[0018] According to a second aspect of the present disclosure, an automated testing apparatus based on test sequence generation is provided, comprising: The test step sequence generation module is used to acquire test requirement data in natural language form and generate a target test step sequence based on the test requirement data. The target test step sequence is an ordered sequence of test actions to be executed, consisting of multiple test steps. The test task scheduling module is used to collect resource status information of the environment under test, and combine the resource status information with the target test step sequence to generate test task scheduling instructions. The test task scheduling instructions are a set of structured control instructions used to guide the distributed execution nodes in the environment under test to execute test tasks. The test execution result generation module is used to allocate the test task scheduling instruction to the distributed execution nodes in the environment to be tested, so as to execute the test steps in the test task scheduling instruction through the distributed execution nodes and obtain the test execution result.

[0019] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory storing computer-readable instructions that, when executed by the processor, implement the automated testing method based on test sequence generation in the first aspect.

[0020] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the automated testing method based on test sequence generation in the first aspect.

[0021] The technical solutions provided in this disclosure may have the following beneficial effects: The automated testing method based on test sequence generation in the example embodiments of this disclosure, on the one hand, obtains and transforms test requirement data in natural language form to obtain a target test step sequence, enabling test requirements to be described and input in a way that is closer to human intuition, and automatically transforms them into a structured, executable test process. This reduces the reliance of test design on manual script writing experience, improves the automation level of test case generation, adapts to diverse software version combinations and hardware configuration changes in cluster environments, and overcomes the problem of limited coverage in traditional static script testing. On the other hand, by collecting resource status information of the environment under test and combining the resource status information with the target test step sequence to generate test task scheduling instructions, the test tasks can be scheduled according to the actual situation. The system dynamically optimizes and schedules resources based on real-time conditions, achieving precise matching between test steps and distributed node resources. This improves the parallel efficiency and resource utilization of test execution, reduces duplicate executions or resource conflicts caused by imperfect manual scheduling strategies, and enhances testing efficiency in large-scale cluster environments. Furthermore, by distributing test task scheduling instructions to distributed execution nodes in the test environment, test execution results are obtained. By combining resource status information with the test task scheduling instructions generated from the target test step sequence, the test process can adapt to collaborative operation under a distributed architecture, expanding the scope and concurrency of test execution. This enhances the coverage depth and defect detection probability of the test process for complex hardware and software configurations, thereby improving the accuracy and reliability of test results.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 The illustration shows a flowchart of an automated testing method based on test sequence generation according to some embodiments of the present disclosure.

[0025] Figure 2 The illustration schematically shows a flowchart of a first model training process for training a test sequence generation model according to some embodiments of the present disclosure.

[0026] Figure 3 The illustration shows a flowchart illustrating the self-evolution of a sequence of test steps according to some embodiments of the present disclosure.

[0027] Figure 4 The illustration shows a flowchart of an optimized adjustment to obtain a target test path according to some embodiments of the present disclosure.

[0028] Figure 5 The illustration schematically depicts a process diagram of updating a test sequence generation model to achieve self-evolution of test step sequence generation according to some embodiments of the present disclosure.

[0029] Figure 6 The illustration shows a flowchart of an optimization adjustment strategy network according to some embodiments of the present disclosure.

[0030] Figure 7 The schematic diagram illustrates a flowchart of an automated testing method based on test sequence generation according to some other embodiments of the present disclosure.

[0031] Figure 8 The illustration shows a schematic diagram of an automated testing apparatus based on test sequence generation according to some embodiments of the present disclosure.

[0032] Figure 9 The schematic diagram illustrates the structural schematic of a computer system of an electronic device according to some embodiments of the present disclosure.

[0033] Figure 10 A schematic diagram of a computer-readable storage medium according to some embodiments of the present disclosure is shown.

[0034] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0036] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0037] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0038] Furthermore, the accompanying drawings are for illustrative purposes only and are not necessarily drawn to scale. The block diagrams shown in the drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] According to the automated testing method based on test sequence generation in this example embodiment, on the one hand, by acquiring test requirement data in natural language form and converting it, a target test step sequence is obtained. This allows test requirements to be described and input in a way that is closer to human intuition and is automatically transformed into a structured, executable test process. This reduces the reliance of test design on manual script writing experience, improves the automation level of test case generation, adapts to diverse software version combinations and hardware configuration changes in cluster environments, and overcomes the problem of limited coverage in traditional static script testing. On the other hand, by collecting resource status information of the environment under test and combining the resource status information with the target test step sequence to generate test task scheduling instructions, the test tasks can be scheduled according to real-time... The system dynamically optimizes and schedules resources to achieve precise matching between test steps and distributed node resources, thereby improving the parallel efficiency and resource utilization of test execution, reducing duplicate executions or resource conflicts caused by imperfect manual scheduling strategies, and improving testing efficiency in large-scale cluster environments. On the other hand, by distributing test task scheduling instructions to each distributed execution node in the test environment to obtain test execution results, and by combining resource status information with the test task scheduling instructions generated by the target test step sequence, the test process can adapt to collaborative operation under a distributed architecture, expanding the scope and concurrency of test execution, thereby enhancing the coverage depth of the test process for complex hardware and software configurations and the probability of defect detection, and improving the accuracy and reliability of test results.

[0040] The automated testing method based on test sequence generation in this example embodiment will be further described below. In this example embodiment, an automated testing method based on test sequence generation is first provided. This method can be applied to a control terminal or a server; this embodiment does not impose any special limitations on it. The following description will use the execution of this method by a control terminal as an example. Figure 1 The illustration schematically depicts a flowchart of an automated testing method based on test sequence generation according to some embodiments of the present disclosure. Reference Figure 1 As shown, this automated testing method based on test sequence generation may include the following steps: In step S110, test requirement data in natural language form is obtained, and a target test step sequence is generated based on the test requirement data. The target test step sequence is a sequence of test actions to be executed in an ordered manner, consisting of multiple test steps.

[0041] In one example embodiment of this disclosure, test requirement data refers to natural language text information received to express the test purpose and test scenario description. For example, test requirement data can be test instruction text entered by testers through a command-line interface, test task description entered through a web console, or requirement statements for testing GPU node servers extracted from historical logs. This embodiment does not specifically limit the source of test requirement data in natural language form, as long as it can express the semantics of the test requirements and can be used as model input. In some optional implementations, test requirement data can include hardware model information (e.g., GPU model, server specifications), system platform information (e.g., Ubuntu version number, kernel version), test objectives (e.g., performance testing, stability testing), test scope description (e.g., deployment, inference, temperature control verification), and environmental dependencies (e.g., driver version, software stack version), etc. Of course, test requirement data in natural language form can also include other types of test condition descriptions. This embodiment does not specifically limit the content composition of test requirement data.

[0042] Before transforming the test requirement data, text preprocessing can be performed. Text preprocessing refers to standardizing and structurally decomposing the natural language text input by the user to facilitate semantic modeling in subsequent deep learning. For example, text preprocessing may include character cleaning, word segmentation, special marker recognition (such as GPU model, driver number string), and stop word filtering. Alternatively, subword tokenization can be used to accommodate the numerous technical terms, software package names, and version number formats present in GPU server testing scenarios; this embodiment does not impose specific limitations on this approach.

[0043] After text preprocessing, the preprocessed test requirement data can be converted into test requirement word vectors. Test requirement word vectors are numerical vectors used to represent the semantic information of natural language text. For example, test requirement word vectors can be word embeddings obtained based on pre-trained models (such as Bidirectional Encoder Representations from Transformers, BERT), domain-specific word vectors trained based on the Transformer architecture, or word vectors generated using traditional statistical methods. As long as they can express the semantic features of natural language text, they can be used as valid input data for the test sequence generation model. This embodiment does not impose any special limitations on the generation method of word vectors.

[0044] The target test step sequence refers to an ordered sequence of executable test actions consisting of multiple test steps. For example, the test step sequence may include common test steps in fields such as "hardware configuration check", "system dependency check", "driver loading verification", "model inference test", and "temperature and power consumption monitoring", or it may include test tasks designed for a specific hardware platform or software environment. This embodiment does not make any special limitation on the specific types of steps included in the target test step sequence.

[0045] Optionally, the test requirement data can be input into a pre-trained test sequence generation model for transformation to obtain the target test step sequence. For example, the test sequence generation model can adopt a deep learning model with an encoder-decoder structure, which uses a semantic encoder to perform contextual modeling of the test requirement word vectors and a sequence decoder to generate a structured test step sequence. Of course, the test sequence generation model can also adopt other deep learning architectures that can perform sequence-to-sequence transformation, such as Long Short-Term Memory (LSTM) networks or Gated Recurrent Unit (GRU) networks. This embodiment does not limit the specific structure of the model.

[0046] By transforming test requirement data in natural language form, unstructured test requirement information can be automatically converted into a sequence of executable target test steps. This not only reduces the workload of manually writing scripts, but also ensures that the generation process of test steps maintains consistency, standardization, and traceability, thereby significantly improving the automation level of test tasks and enhancing the system's adaptability in complex test scenarios.

[0047] In step S120, resource status information of the environment to be tested is collected, and test task scheduling instructions are generated by combining the resource status information with the target test step sequence. The test task scheduling instructions are a set of structured control instructions used to guide the distributed execution nodes in the environment to be tested to execute test tasks.

[0048] In one example embodiment of this disclosure, the test environment refers to the hardware and software combination runtime environment used to execute the target test step sequence. For example, the test environment can be a distributed computing environment containing multiple GPU node servers, a single-node server test cluster, a virtualized container environment, or an elastic computing node in a cloud platform. As long as it can support the execution of the test sequence and provide collectable runtime status information, it is acceptable. This embodiment does not impose any special limitations on the deployment method of the test environment.

[0049] Resource status information refers to status data that reflects the current node's available resources, load, and system health indicators. For example, resource status information may include GPU resource status (such as memory usage, temperature, GPU load, power consumption, fan speed, etc.), CPU utilization and available cores, memory usage, disk I / O, network bandwidth usage, the number of running tasks, task queue length, and historical execution time data. Of course, resource status information may also include system version information, software stack version information, driver loading status, etc., related to the execution of test steps. This embodiment does not limit the specific type of resource status information, as long as it reflects the node's execution capabilities and availability. In optional implementations, resource status information can be obtained locally, such as through monitoring interfaces provided by GPU drivers, Linux system-level monitoring tools, container monitoring application programming interfaces (APIs), or customized performance acquisition agents. Alternatively, resource status information can be periodically pushed to the policy module through a centralized monitoring platform (such as the Prometheus monitoring system or a custom monitoring service), where the policy module actively retrieves the resource status information. This embodiment does not specifically limit the resource acquisition method.

[0050] Test task scheduling instructions refer to a set of structured control instructions generated based on resource status information and the sequence of target test steps, used to guide distributed execution nodes in executing test tasks. For example, resource status information can be first associated with the sequence of target test steps, such as constructing a state vector based on the resource status information and the sequence of target test steps. This state vector can be used to describe the current resource status of the test environment and the numerical multidimensional vector data of the test step requirements. Then, this state vector is input into a trained reinforcement learning model to determine the scheduling action of the test task. Finally, test task scheduling instructions can be generated based on this scheduling action. Of course, other optimization strategies can also be used. For example, a multi-constraint solution model can be constructed based on resource status information and the sequence of target test steps to obtain the optimal test path. The optimal test path can then be used to generate test task scheduling instructions. This embodiment does not impose any special limitations on the generation method of test task scheduling instructions. By generating test task scheduling instructions, the test system can dynamically generate the optimal scheduling strategy based on real-time resource conditions, avoid resource conflicts, improve execution efficiency, significantly improve the stability and coverage of the entire automated testing process, and improve the accuracy of test results.

[0051] In step S130, the test task scheduling instruction is assigned to a distributed execution node in the environment to be tested, so that the test steps in the test task scheduling instruction are executed by the distributed execution node to obtain the test execution result.

[0052] In one example embodiment of this disclosure, a distributed execution node refers to a computing node capable of receiving test task scheduling instructions, automatically executing the test steps contained therein, and returning the execution results. For example, a distributed execution node can be a GPU server node, a GPU virtual machine instance, a containerized execution agent, an edge computing node, or any hardware device with a certain computing power and the ability to run test scripts; of course, a distributed execution node can also be a Kubernetes-based computing Pod or an elastic instance based on a virtualization platform, as long as it can execute test tasks and return results. This embodiment does not impose any special limitations on the implementation form of the distributed execution node.

[0053] Test agents can be pre-deployed on distributed execution nodes. A test agent is a software module used to parse test task scheduling instructions, execute corresponding test code, collect runtime information, and return execution results. For example, test agents can be written in Python, Go, or Rust, or they can be constructed by integrating system command execution modules through shell scripts, or they can provide a unified test execution interface through a Remote Procedure Call (RPC) framework (such as gRPC). Of course, as long as they can correctly respond to test task scheduling instructions and execute test steps, this embodiment does not particularly limit the specific implementation method of the test agent.

[0054] Before allocating test task scheduling instructions to distributed execution nodes, each test step can be bound to a target distributed execution node based on the node selection information or current scheduling action contained in the test task scheduling instructions. The test step binding relationship refers to the correspondence between a specific test step and a target execution node. For example, a "system dependency check" step might be bound to node 2, which has a higher resource idle rate, and a "GPU inference test" step might be bound to node 5, which has a specified GPU model. Of course, this binding relationship can also be dynamically generated through a strategy optimization model, and this embodiment does not impose any special restrictions on it.

[0055] After completing the test steps and binding with the distributed execution node, the relevant test steps in the test task scheduling instruction can be sent to the bound distributed execution node. The sending method can be an HTTP request based on the REST API, a binary communication method based on the RPC framework, or a task push method based on message queues (such as Kafka, RabbitMQ). As long as the reliability and consistency of instruction delivery can be guaranteed, it can be applied to this embodiment. This embodiment does not limit the instruction sending method.

[0056] Upon receiving a test task scheduling instruction, the distributed execution node automatically executes the test steps through its internal test agent. During the execution of these test steps, the test agent can load corresponding test scripts, invoke underlying system commands, load test data, deploy dependent services, execute model inference processes, trigger performance monitoring tools, or execute fault simulation scenarios. For example, when executing the GPU inference step, the test agent can invoke the test framework to load model weights, perform inference calculations, and collect inference time; when executing the system configuration check step, it can invoke system commands to verify driver versions and dependency installation status. Of course, different test steps may correspond to different execution methods; this embodiment does not limit the execution process of the test steps.

[0057] After the test steps are completed, the distributed execution nodes will generate test execution results. Test execution results refer to structured data used to represent execution status, output data, performance metrics, or monitoring information. For example, test execution results may include test step identifiers, node identifiers, execution status (success / failure), failure reasons, execution time, maximum temperature, power consumption fluctuations, system log fragments, error codes, peak resource usage, etc. Of course, the specific content of the test execution results can dynamically change according to the test step type, and this embodiment does not impose any special limitations on this. In optional implementations, test execution results can be returned to the control terminal in JSON, YAML, or Protocol Buffer formats, or asynchronously reported through a log collection system (such as Fluentd or Logstash). Of course, as long as the test execution results can be accurately returned and used for subsequent analysis and self-evolution, this embodiment does not limit the return method.

[0058] Through the above execution process, test task scheduling instructions can be executed quickly and automatically on distributed execution nodes, realizing the parallel processing capability of test tasks and thus improving the execution efficiency and response speed of the entire test system. Furthermore, since the test execution results are completely transmitted back to the control terminal, a reliable data foundation can be provided for subsequent anomaly detection, path optimization, and the self-evolution of the generation model.

[0059] The contents of steps S110 to S130 will be described in detail below.

[0060] In one example embodiment of this disclosure, the generation of a target test step sequence based on test requirement data can be achieved through the following steps, specifically including: Test requirement data in natural language form can be preprocessed to obtain test requirement word vectors; these word vectors can then be input into a trained test sequence generation model to obtain the target test step sequence.

[0061] In this embodiment, the test requirement word vector refers to a floating-point vector representation that expresses the semantics of natural language in numerical form. For example, the test requirement word vector can be a context-related embedding generated by a pre-trained bidirectional encoding representation model, a domain-specific embedding obtained by self-training based on a Transformer model, or a simple statistical word vector. Of course, as long as it can express the semantic features of the natural language test requirements, this embodiment does not limit the word vector generation method. The test requirement word vector can be output by a pre-trained word vector model independent of the test sequence generation model and then transmitted to the test sequence generation model for subsequent processing; or it can be connected to a pre-trained input network before the test sequence generation model to preprocess the test requirement data in natural language form to obtain the test requirement word vector. This embodiment does not make any special limitations on this. In optional implementations, the test requirement word vector can be represented with a fixed dimension, such as a 256-dimensional or 512-dimensional embedding; it can also be represented with a variable length and processed by the masking mechanism inside the word vector model; it can also enhance the expressive power of the sequence structure by adding additional position information vectors. This embodiment does not make any special limitations on this.

[0062] A test sequence generation model is a deep learning model used to automatically convert preprocessed test requirement word vectors into a sequence of structured test steps. It can understand natural language test requirements and output a set of directly executable test steps. For example, a test sequence generation model can convert "test the inference stability of a certain model GPU under Ubuntu 22.04" into a series of steps such as "system check, driver detection, dependency installation, model inference, and temperature monitoring." Of course, the test sequence generation model can also generate other forms of test steps according to different scenarios; this embodiment does not impose any special limitations on this.

[0063] The test sequence generation model can be a sequence-to-sequence (Seq2Seq) type deep learning model, which combines deep encoding of the semantics of the input text with the step-by-step generation of the output step sequence to achieve the mapping from natural language to test steps. The test sequence generation model can also be other types of deep learning models that can generate sequences. For example, in some optional implementations, the test sequence generation model can also be a generative adversarial network (GAN) model used to generate test step sequences. This embodiment does not make any special limitation on the type of test sequence generation model.

[0064] By preprocessing test requirement data in natural language form and converting it into test requirement word vectors, and then using a pre-trained test sequence generation model to generate a sequence of target test steps, the system can automatically understand the test objectives described by testers in natural language, transforming unstructured text into a sequence of structured test steps. This reduces the workload of manual script writing, minimizes omissions or errors in the order of steps due to human comprehension bias, and improves the accuracy, consistency, and automation of test step generation.

[0065] In one example embodiment of this disclosure, the test sequence generation model may include a semantic encoder and a sequence decoder. The semantic encoder is a deep learning module used to perform contextual semantic understanding and semantic feature extraction on the test requirement word vectors corresponding to the natural language test requirements. For example, the semantic encoder can be based on a Transformer encoder structure, capturing long-distance dependencies between test requirement word vectors through a multi-head self-attention mechanism, thereby obtaining a hidden state representation with global semantic understanding capabilities. Alternatively, the semantic encoder can be constructed using a Bidirectional Long Short-Term Memory (BiLSTM) network or a Gated Recurrent Unit (GRU). As long as it can effectively encode the semantics of the input natural language, this embodiment does not impose any special limitations on its internal structure.

[0066] For example, a semantic encoder in practical applications may include modules such as an embedding layer, a positional encoding layer, an attention layer, and a feedforward network layer. The embedding layer maps input word vectors to an embedding space of uniform dimension; the positional encoding layer supplements the positional information of the sequence, enabling the encoder to distinguish words at different positions within the sequence; the attention layer models the semantic relationships between words, for example, it can be used to identify the semantic structure within test requirement phrases such as "driver installation" and "power consumption monitoring"; the feedforward network layer further extracts semantic features and forms context-dependent hidden states. Of course, this is merely an exemplary illustration of the semantic encoder structure, and this embodiment does not limit the arrangement or number of these modules within the semantic encoder.

[0067] A sequence decoder is a model component used to progressively generate a sequence of target test steps based on the hidden state representation obtained from a semantic encoder. For example, a sequence decoder can employ a Transformer decoder structure, using self-attention and encoder-decoder attention mechanisms to consider the association between generated steps and the input semantics while generating each test step. This ensures that the generated test steps satisfy both the logical order of the test and reflect the true intent of the input test requirements. Alternatively, a sequence decoder can use an LSTM or GRU architecture to generate the test step sequence in an autoregressive manner; this embodiment does not limit its specific structure. In optional implementations, the sequence decoder can incorporate a masking mechanism to prevent the model from accessing subsequent steps that have not yet been generated while generating the current step, thus maintaining the causality of the output sequence. In another implementation, the sequence decoder can also support diverse generation strategies, such as beam search, temperature sampling, or top-p sampling, to enhance the richness of generated test steps and improve the model's generalization ability under multi-scenario testing requirements. By working together with semantic encoders and sequence decoders, the test sequence generation model can accurately map natural language test requirements into a structured sequence of target test steps, thereby achieving automated test step generation, reducing the workload of manual script maintenance, and improving the system's accuracy and robustness in understanding complex test requirements.

[0068] As will be readily understood by those skilled in the art, the test sequence generation model in this embodiment can be pre-trained through a first model training process. The first model training process refers to the training flow used to optimize the model parameters of the semantic encoder and sequence decoder, enabling them to accurately complete the task of converting natural language testing requirements into target test step sequences. For example, this training process can be based on supervised learning methods, using paired training data for iterative training; of course, this training process can also combine unsupervised or semi-supervised data augmentation methods in some stages. This embodiment does not limit the training mode. The first model training process can use a large number of data samples with labeled relationships to enable the semantic encoder to learn the semantic structure of natural language text, enable the sequence decoder to learn to generate logically reasonable test step sequences, and enable the overall model to have generalization and abstraction capabilities, adapting to different types of testing requirement scenarios. (Reference) Figure 2 As shown, the training process of the first model may include: Step S210: Obtain the first training dataset, which includes training and testing requirement data in natural language form and reference test step labels; Step S220: Input the training and testing requirement data into the semantic encoder to obtain the context-dependent hidden state representation vector; Step S230: Input the hidden state representation vector into the sequence decoder, and force-teach the sequence decoder by inputting the reference test step label to complete the training of the test sequence generation model.

[0069] The first training dataset refers to a set of paired corpora used to train the test sequence generation model. This can include training test requirement data in natural language and corresponding reference test step labels. For example, the training test requirement data in natural language could be test requirement descriptions written by testers in historical systems, such as "check if GPU power exceeds the threshold." Reference test step labels might be test steps manually labeled by experienced test engineers as "hardware detection, power consumption monitoring, result recording." Of course, this data can also be extracted from version control systems, automated pipeline logs, or maintained test case libraries; this embodiment does not impose any particular limitations on this. In an optional implementation, the reference test step labels can also adopt a single-level tiling format, directly listing the test steps executed sequentially; or a hierarchical structure format, dividing the test steps into different levels (such as "initialization steps, functional test steps, monitoring steps"); or an instruction-based representation, encoding each step as a unified operation identifier (such as "CHK_SYS", "DEP_DRIV", "RUN_INF"). This embodiment does not limit the label format, as long as it clearly represents the target test step sequence corresponding to each natural language requirement.

[0070] The hidden state representation vector refers to the internal semantic feature representation generated by the semantic encoder after processing the sequence of word vectors for the test requirements. This feature representation not only contains the semantic information of individual terms, but also the semantic associations, action relationships, and logical structure between the test objectives and dependencies in the entire test requirement statement. For example, when the input statement is "Check whether the GPU driver is correctly loaded and execute the inference stress test", the contextual hidden state representation can reflect the test dependency relationship between "driver check" and "inference stress test". Of course, in other scenarios, this hidden state representation can also capture semantics in areas such as "version number constraints", "hardware environment specific requirements", and "error code features", which are not specifically limited in this embodiment.

[0071] When the hidden state representation vector is input into the sequence decoder, the sequence decoder first obtains the complete semantic representation from the semantic encoder and uses it as the conditional input for generating the test step sequence in the decoding phase. The sequence decoder enables each decoded test step to access all the semantic information of the test requirements through its internal interactive attention mechanism. For example, when generating the "driver check" step, semantic context such as "driver", "version", and "load" can be extracted from the hidden state.

[0072] Teacher forcing refers to a process during model training where the sequence decoder uses not only the hidden state representation vector output by the semantic encoder at the previous time step as input, but also the reference test step label as input when generating the test step sequence. Alternatively, a hybrid teacher forcing strategy with partial probabilities can be used, employing the true label only at certain time steps; this embodiment does not limit this approach. Teacher forcing significantly accelerates the sequence decoder's training process, enabling it to quickly converge to the correct sequence generation pattern and preventing the model from failing to learn the correct step dependency structure due to excessive prediction bias in early training. Furthermore, teacher forcing improves the model's ability to generate long test steps, maintaining system stability when handling complex testing tasks.

[0073] The target loss function is a mathematical function used to quantify the error of the predicted sequence. For example, the target loss function can be cross-entropy loss, used to calculate the difference between the probability distributions of each item between the predicted test step sequence and the true test step sequence. Of course, sequence-level loss functions, such as sequence matching loss, BLEU loss, or edit distance loss, can also be used. This embodiment does not limit this, as long as it can be used to guide model updates. By backpropagating the target loss function, the optimization framework can update the parameters of the semantic encoder and sequence decoder, enabling the test sequence generation model to continuously reduce the error between the predicted test step sequence and the true test step sequence. In optional implementations, Adam, RMSProp, or stochastic gradient descent algorithms with momentum can be used for parameter optimization during training. In another implementation, gradient clipping can be used to prevent the gradient explosion problem. This embodiment does not limit this.

[0074] As the number of training iterations increases, the semantic understanding capability of the test sequence generation model for natural language testing requirements can be continuously enhanced, and the accuracy of the generated target test step sequences gradually improves. Training of the test sequence generation model is considered complete when the target loss function converges to a preset threshold or when the maximum number of iterations is reached. Through model training in the first model training process, the test sequence generation model can ultimately achieve accurate parsing of natural language testing requirements and precise generation of corresponding test steps, ensuring that the generated test step sequences have logical consistency, execution feasibility, and semantic integrity, thus adapting to the actual needs of automated testing processes.

[0075] By setting up a semantic encoder and a sequence decoder, and introducing a first training dataset, contextual hidden state representation, and forced teaching during the training phase, the model can fully learn the correspondence between natural language and test steps during training, accelerate the convergence speed of the loss function, improve the model's understanding and generation capabilities when faced with complex test requirement statements, make the generated test step sequences more in line with the test objectives and more logically rigorous, and further improve the overall automatic generation quality of the system.

[0076] In one example embodiment of this disclosure, the generation of a test task scheduling instruction by combining resource status information with the target test step sequence can be achieved through the following steps, specifically including: A state vector can be constructed based on resource status information and the sequence of target test steps. The state vector is then input into the trained test step optimization model to infer the probability distribution of each current scheduling action. The current scheduling action is used to indicate the execution order of allocating at least two test steps to each distributed execution node. Test task scheduling instructions are generated based on the current scheduling action with the highest probability value in the probability distribution.

[0077] In this context, a state vector is a multi-dimensional feature representation data structure used to characterize the resource usage, load level, node health status, and adaptation relationship between the target test step sequence and the current environment at a given moment in the test environment. State vectors are typically presented as fixed-length or variable-length numerical sequences and can be input into the test step optimization model for scheduling action inference. State vectors can be constructed from multiple resource metrics related to distributed execution nodes through concatenation, stacking, or attention fusion. For example, real-time status information such as GPU temperature, memory usage, computational load, driver loading status, CPU utilization, memory usage, disk I / O, network bandwidth usage, and task queue length can be collected from each distributed execution node and converted into numerical features usable as model input according to a preset normalization rule. Based on this, the features of all nodes can be concatenated in node number order to form a two-dimensional feature matrix, which can then be flattened or encoded into a state vector.

[0078] The test step optimization model refers to an intelligent decision-making model that automatically generates the optimal scheduling action based on the resource status information of the current test environment and the execution requirements of the test steps to be scheduled in the target test step sequence. In this embodiment, the test step optimization model can infer the probability distribution of each scheduling action by inputting the state vector and combining it with the policy network and value network trained internally, so as to support the system in selecting the optimal scheduling action and generating test task scheduling instructions.

[0079] The test step optimization model can include a policy network and a value network. The policy network is a network structure used to map state vectors to a probability distribution of scheduled actions; the value network is a network structure used to evaluate the long-term execution benefits of the current state-action combination. In an optional implementation, the policy network can be constructed using a multilayer perceptron (MLP) structure, a Transformer encoding structure, or a graph neural network (GNN) to better understand resource relationships and scheduling conflicts between different nodes. For example, when multiple nodes have high-concurrency tasks, the policy network can adaptively adjust action probabilities based on the resource competition relationship between nodes, causing scheduling actions to tend to avoid high-load nodes and prioritize idle nodes.

[0080] Value networks can be designed based on long-term return modeling using reward functions, which can be comprehensively evaluated based on test pass rate, resource utilization, average task completion time, and coverage improvement. For example, when a node consistently maintains a high success rate and a predictable resource curve, the value network can assign higher value to the actions of that node, making the model more inclined to use that node to perform similar steps in future inferences.

[0081] During inference, the test step optimization model can generate a set of candidate scheduling actions based on the current input state vector, and generate a corresponding probability value for each scheduling action through the softmax activation mechanism. For example, when a test step is highly resource-sensitive (such as a GPU inference step), and the state vector shows a large difference in the remaining GPU memory between nodes, the policy network will appropriately increase the probability of the action "schedule to a node with higher remaining GPU memory," making the model tend to generate scheduling actions with higher execution efficiency and lower failure risk.

[0082] In another alternative implementation, the test step optimization model can also support an action selection mechanism based on exploration-exploitation balance (such as ε-greedy or probabilistic sampling decoding), so that the model not only relies on historical best nodes in long-term operation, but also maintains a certain exploration capability, thereby further improving coverage and execution stability in complex test scenarios.

[0083] By combining resource status information with the target test step sequence to construct a state vector, and then using this state vector to infer scheduling actions through the test step optimization model, the system can fully consider multi-dimensional factors such as node resource load, step resource requirements, and historical performance during the scheduling process. This results in a more accurate and efficient scheduling strategy, avoiding resource bottlenecks, task congestion, or node mismatch, and improving parallel efficiency and overall test execution success rate in a multi-node execution environment.

[0084] In an exemplary embodiment of this disclosure, the test step optimization model during the training phase includes at least a policy network and a value network. The test step optimization model is obtained through a second model training process, which refers to the training flow used to train the test step optimization model so that it can automatically infer the optimal scheduling action based on the input state vector under different test scenarios. During this training process, by sampling, evaluating, and updating the values ​​and gradients of states and actions in a pre-designed state space, the policy network and value network gradually converge to a scheduling policy with generalization capabilities in real test environments.

[0085] Optionally, the second model training process can be implemented through the following steps, which may specifically include: A sampled state vector from a pre-designed state space can be input into a policy network. The policy network, combined with the sampled state vector, outputs a sampled scheduling action belonging to a pre-designed action space. The sampled state vector and the sampled scheduling action are then input into a value network. The reward function of the value network evaluates the state value of the combination of the sampled state vector and the sampled scheduling action, generating an evaluation reward value. The policy network is then updated with its policy gradient based on the evaluation reward value, and its parameters are updated using regression. The trained policy network and value network are obtained when the loss values ​​of the policy network and value network are less than or equal to a preset loss threshold, or when the number of iterations in the training process reaches a preset maximum number of iterations. The evaluation reward value is obtained by the reward function based on at least one of the following: test pass rate, resource utilization rate, average task completion time, and improvement in historical coverage.

[0086] The sampled state vector is a set of states selected randomly or regularly from a pre-designed state space. This state space is a set of resource state features pre-built during training, covering a large number of possible node resource combinations to simulate various scenarios such as high and low GPU node load, queue congestion, and system resource shortage.

[0087] When inputting the sampled state vector into the policy network, the vector can first be mapped to the input layer of the policy network. The policy network then makes a comprehensive judgment based on features such as changes in node resource strength, task queue distribution, and historical execution performance contained in the state vector. The policy network can include a multilayer perceptron structure or an attention-based network structure. It generates corresponding action output values ​​through layer-by-layer calculation, ultimately outputting sampled scheduling actions belonging to the pre-designed action space. In one optional implementation, the output of the policy network can be normalized using a softmax activation function to ensure a stable and differentiable probability distribution for each sampled scheduling action. Alternatively, an ε-greedy strategy can be used, allowing some training steps to employ exploratory actions to enhance the model's adaptability to unknown states. In another optional approach, sampled scheduling actions can be selected from the action output using a probabilistic sampling method to maintain action diversity during training.

[0088] The pre-designed action space refers to the set of actions defined in advance during the training phase, used to represent all possible test step scheduling strategies. For example, actions such as "assign the test step to node 1", "delay the execution of the test step", and "split the test step into parallel subtasks" can all be elements of the action space. In actual training, the size of the action space can be enriched or simplified according to the type of test scenario, so that the action output conforms to the execution logic of the test steps and does not exceed the final system's executable capabilities.

[0089] After the policy network generates sampled scheduling actions, the sampled state vector and the sampled scheduling action can be input into the value network. The value network then evaluates this state-action combination based on its internally constructed reward function. The value network calculates the long-term benefit of executing the action in the current state, and its output is typically a scalar evaluation reward value. This value reflects the comprehensive performance of the currently selected scheduling action in terms of task completion efficiency, improved test coverage, reduced execution failure rate, and optimized resource utilization. Optionally, during the input process, the sampled state vector and sampled scheduling action can be concatenated or encoded to form an input structure recognizable by the value network. The value network can include a multi-layer neural network structure, performing semantic fusion on the input through layer-by-layer computation to capture the coupling relationship between state features and action effects. For example, when a state vector indicates node resource scarcity, and the sampled scheduling action is "continue scheduling high-load steps to this node," the value network can reduce the evaluation reward value corresponding to this action based on training experience, reflecting that this action may lead to task failure or decreased execution efficiency.

[0090] The reward function is an evaluation criterion within the value network, which can be constructed based on indicators such as test pass rate, average task completion time, resource utilization, and the degree of historical coverage improvement. For example, if a node significantly improves coverage and maintains stability after executing a specified step, the corresponding reward value can be increased; if an action causes resource congestion, the reward value can be decreased accordingly. Of course, this embodiment does not specifically limit the specific formula form of the reward function, as long as it can reflect the impact of the action on the overall system performance.

[0091] In optional implementations, the value network can be updated using Temporal Difference (TD) learning, enabling it to learn the judgment logic that future performance is far greater than current immediate benefits; alternatively, it can be calculated using Monte Carlo methods, allowing the model to reflect long-term effects in test tasks with long sequence structures. This embodiment does not impose any special limitations on the specific value calculation method of the value network.

[0092] After the value network outputs an evaluation reward value, the policy network can be updated with a policy gradient based on this reward value. This allows the policy network to be more inclined to choose scheduling actions that yield higher rewards in subsequent inferences. Specifically, the policy gradient direction can be calculated based on the difference between the evaluation reward value and the probability of the policy network outputting a scheduling action in the current state, thereby updating the policy network parameters. During policy gradient updates, the evaluation reward value can be used as a feedback signal indicating the quality of the action. A higher evaluation reward value indicates that the corresponding scheduling action is closer to the system's desired goal in the current state. For example, in a certain sampled state, scheduling a test step to a node with sufficient resources significantly improves the test pass rate, resulting in a higher reward value; conversely, if an action causes task congestion or increases the failure rate, the reward value is lower. The policy network can automatically adjust its internal parameters based on this feedback signal, increasing the probability of outputting high-reward scheduling actions and causing the model to converge towards a better policy.

[0093] In some alternative implementations, a policy gradient update method based on the Monte Carlo method can be used, which optimizes parameters through multiple rounds of sampling and reward accumulation; alternatively, a policy gradient method based on the advantage function can be used to make the update more stable. For example, the Proximal Policy Optimization (PPO) algorithm can be used to improve the convergence speed while limiting the policy change. This embodiment does not impose any special limitations on this, as long as the policy network parameters can be optimized under the feedback of reward value.

[0094] After obtaining the evaluated reward value, the value network's parameters can be further updated using this reward value to enable the value network to more accurately assess the long-term returns of future actions. In the specific update process, the evaluated reward value can be used as the target output of the value network, allowing it to adjust its internal parameter structure through error calculation, thus making its output more closely resemble the actual returns. In practice, the difference between the current output state value of the value network and the evaluated reward value can be calculated as the value error, and the parameters of each layer of the value network can be updated through backpropagation. For example, the Mean Squared Error (MSE) loss function can be used as the regression target, allowing the value network's predicted values ​​to gradually approach the actual reward values; alternatively, more stable regression methods such as Huber loss can be used to improve the convergence speed in high-noise reward scenarios.

[0095] After completing multiple rounds of policy gradient updates and value regression updates, the convergence of training can be determined based on the loss values ​​of the policy network and value network. When both the policy network loss and the value network loss fall below a preset loss threshold, the model can be considered to have converged to an accuracy level that meets scheduling requirements, and training can be stopped. The policy network loss measures the deviation between the policy network's generated action probability distribution and the ideal action selection; the value network loss measures the gap between its state value prediction and the actual reward. By monitoring these two loss values ​​in real time during training, it can be determined whether they continuously decrease, whether they fluctuate stably, and whether they reach the set threshold requirements, thereby determining whether the training meets the termination conditions.

[0096] In another alternative approach, a maximum number of training iterations can be set as a training cap. When the preset maximum number of iterations is reached, even if the loss has not fully reached the threshold, the model can be considered to have reached a usable stable state, and training can be stopped. This method can prevent the training process from becoming excessively time-consuming or getting stuck in local oscillations. Alternatively, a dual-condition approach can be used, where either the loss meets the threshold or the number of iterations reaches the maximum value, balancing efficiency and stability in the training process.

[0097] By designing a second model training process for the policy network and value network, and using the reward feedback of the sample state-action combination as the basis for updating, the policy network can gradually learn the optimal scheduling action under different resource states, and the value network can more accurately evaluate the long-term benefits of actions. This results in the trained test step optimization model having high scalability, high stability and high generalization ability, enabling it to continuously provide high-quality scheduling strategies in real environments, effectively improving the execution efficiency and robustness of the entire test system.

[0098] In one example embodiment of this disclosure, after obtaining the test execution result by allocating the test task scheduling instruction to the distributed execution nodes in the environment under test, it is also possible to... Figure 3 The steps in this embodiment enable the self-evolution of the automated testing method, refer to... Figure 3 As shown, it can specifically include: Step S310: Determine the abnormal test path based on the test task scheduling instruction and the test execution result; Step S320: Combine the preset historical failure samples and the abnormal feature points in the abnormal test path to adjust the abnormal test path and obtain the target test path. Step S330: Using the target test path as an improved sample, update the test sequence generation model used to generate the target test step sequence.

[0099] An abnormal test path refers to any path segment during the execution of a test task that leads to a decrease in coverage, an increase in execution failure rate, or abnormal fluctuations in resources. For example, if a model's inference step consistently results in memory overflow across multiple nodes, the path containing that inference step can be identified as an abnormal test path. In another scenario, if a system initialization step fails multiple times in different tasks, that initialization path can also be considered an abnormal test path. This embodiment does not impose any special limitations on the occurrence of abnormal test paths.

[0100] For example, the test step sequence, node allocation information, and indicators such as execution status, failure reason, execution time, and resource fluctuation in the test execution results contained in the test task scheduling instructions can be compared item by item to identify path segments in the test process that may have execution anomalies, performance degradation, or insufficient coverage, thereby determining abnormal test paths. Alternatively, test steps with execution statuses of failure, abnormal termination, resource exceedance, or timeout can be marked as candidate abnormal nodes based on the status codes recorded in the test execution results, and test steps with abnormal nodes can be identified as abnormal test paths. In some optional implementations, to facilitate path-level judgment, path aggregation can be performed on step-level anomalies, that is, multiple test steps with consecutive anomalies can be combined into an abnormal test segment. For example, in system deployment testing, if "dependency installation failure, driver verification failure, and model loading failure" occur consecutively, they can be combined into "system initialization path anomaly". This method can more accurately describe the location and scope of the anomaly.

[0101] Anomaly features refer to key indicators that cause failures, anomalies, or abnormal fluctuations during test execution. For example, anomaly features may include "driver version mismatch", "peak memory exceeding limit", "insufficient system privileges", "excessive cold start time", etc. This embodiment does not impose any special restrictions on the types of anomaly features.

[0102] First, based on the category of the abnormal feature points, target failure samples with similar failure patterns can be retrieved from a preset historical failure sample library using feature matching models or rule matching methods. For example, when the abnormal feature point is "driver loading failure," failure cases related to driver version conflicts or missing dependencies can be retrieved from historical samples; when the abnormal feature point is "model inference timeout," failure samples containing reasons such as excessive model size or insufficient hardware bandwidth can be retrieved. After obtaining the target failure samples, test step adjustment suggestions can be generated based on the failure reasons, processing suggestions, or supplementary test steps recorded in the failure samples. For example, adjustment suggestions such as "replacing the software source," "downgrading or rolling back the driver version," "adding abnormal data injection scenarios to the inference step," and "adding a node warm-up step" can be given based on historical experience. It is understood that this is only an illustrative example and should not be construed as a specific limitation on this embodiment. In an optional embodiment of this disclosure, the adjustment suggestions can also be mapped to structural modifications to the abnormal test path, such as inserting an "environment repair test step" before the abnormal step; adding an "abnormal stress test step" after the abnormal step; or replacing the original test steps with a more refined equivalent step structure. For example, the "driver loading check" step can be replaced with the "driver version matching check, dependency library verification, and driver initialization test" steps. This embodiment does not impose any special limitations on the optimization and adjustment methods for abnormal test paths.

[0103] After obtaining the target test path, it can be combined with its corresponding natural language test requirement data to form a new training sample pair. This enables the test sequence generation model to learn from abnormal cases and automatically update its generation strategy. For example, the target test path can be used as an optimized sequence of real test steps, and combined with the original test requirement language text to form a new training pair, supplementing the training dataset of the test sequence generation model. Simultaneously, the pre-stored test case library can be updated using the target test path.

[0104] During model updates, incremental training or full fine-tuning can be performed within the model training framework using new training samples, allowing the test sequence generation model to adjust the original model parameters based on the latest samples. Simultaneously, the weight of the target test path can be determined based on its scarcity in the training data. When the target test path represents a less frequent anomaly repair pattern, the frequency of the new training sample in the training batch can be appropriately increased, enabling the test sequence generation model to adapt to this anomaly type more quickly. Optionally, continuous learning methods can be employed to prevent the test sequence generation model from forgetting existing knowledge during incremental learning; alternatively, knowledge distillation can be used to retain the decision-making capabilities of the old test sequence generation model in the new test sequence generation model to maintain the stability of generation performance. This embodiment does not impose specific limitations on these methods.

[0105] By identifying abnormal test paths after test execution, optimizing and adjusting these abnormal paths based on historical failure samples, and using the optimized target test paths to update the test sequence generation model, the system can acquire self-learning and self-repair capabilities. This allows the test step generation model to continuously absorb failure experience, correct error patterns, and optimize generation logic, thereby achieving self-evolution in test step sequence generation. This makes the system more accurate and stable with use, improving test coverage and anomaly detection capabilities.

[0106] In one example embodiment of this disclosure, the abnormal test path can be determined based on the test task scheduling instruction and the test execution result through the following steps, which may specifically include: Test task scheduling instructions and test execution results can be input into a trained anomaly test prediction model, which outputs probability distribution data for each test path. The anomaly test prediction model is trained and modeled based on historical execution logs. The historical test execution logs are log data that records the execution status, execution results, and resource usage of the test paths or test steps corresponding to the test task scheduling instructions during the execution of historical test tasks. Under the anomaly probability constraint of the probability distribution data, test paths with test coverage or test pass rate less than a preset proportion threshold are selected as anomaly test paths.

[0107] Among them, the anomaly test prediction model is a test path-level prediction model trained on a large number of historical execution logs and failure records. It can use a recurrent neural network (RNN), a long short-term memory network, or a sequence classification model based on the Transformer structure as its architecture. By performing temporal feature extraction and semantic fusion operations on the input features, it outputs the anomaly probability for each test path.

[0108] After generating test execution results, the test task scheduling instructions and the corresponding test execution results can be fed as joint input data into the trained anomaly test prediction model to infer the probability distribution of whether anomalies exist in the test path. First, the test step sequence, dependencies between steps, node allocation information, and execution order recorded in the test task scheduling instructions can be encoded into path structure feature vectors. Simultaneously, the execution status code, execution time, resource fluctuation curves, failure root cause information, and error log fragments from the test execution results can be encoded into execution feature vectors. Concatenating or structurally fusing these two types of vectors forms the input features of the anomaly test prediction model.

[0109] Test coverage can refer to the number of modules, interfaces, or scenarios covered during the execution of a test path; test pass rate can refer to the ratio of the number of successful executions of each step in the test path to the total number of executions. When the test coverage of a test path is lower than a preset coverage threshold, for example, if it only covers part of the logic of the target scenario, it can be considered that the test path has failed to fully verify the target function and is marked as an abnormal path. When the test pass rate of a test path is lower than a preset pass threshold, for example, if it fails frequently in multiple replay executions, it can be determined that the test path has potential defects or execution instability, and it is also regarded as an abnormal test path.

[0110] Historical test execution logs are log data that records the execution status, results, and resource usage of test paths or test steps corresponding to test task scheduling instructions during the execution of historical test tasks. When identifying abnormal test paths, the test task scheduling instructions and test execution results can be input into a trained abnormal test prediction model, which then outputs probability distribution data for each test path. The probability distribution data characterizes the likelihood of an anomaly occurring on each test path under the current test task execution conditions; a higher probability value indicates a greater risk of an anomaly occurring on the corresponding test path.

[0111] In one optional implementation, test paths can be first screened based on probability distribution data. For example, test paths with anomaly probabilities higher than a preset anomaly probability threshold can be identified as a set of candidate abnormal test paths, thereby excluding test paths with low anomaly risk at the probability level and reducing the possibility of misjudgment in subsequent judgment processes. After completing the initial screening based on probability distribution data, a second stage of screening can be performed on the set of candidate abnormal test paths by combining the test coverage or test pass rate recorded in historical test execution logs. For example, for a candidate abnormal test path, it can be determined whether its corresponding test coverage or test pass rate is less than a preset proportion threshold. If a test path is in the high anomaly probability range in the anomaly probability assessment results, and its test coverage or test pass rate is lower than the preset proportion threshold, then the test path can be identified as an abnormal test path.

[0112] By inputting test task scheduling instructions and test execution results into a trained anomaly test prediction model, and filtering anomaly test paths based on the path anomaly probability output by the model combined with coverage and pass rate thresholds, test paths with high failure risk can be proactively identified. This significantly improves anomaly localization efficiency, reduces manual investigation time, and enhances the test system's ability to provide early warnings of latent and chain anomalies, providing accurate input for subsequent repair and path enhancement.

[0113] In one example embodiment of this disclosure, it can be achieved through Figure 4 The steps described in the document combine preset historical failure samples and abnormal feature points in the abnormal test path to optimize and adjust the abnormal test path, thereby obtaining the target test path. (Refer to...) Figure 4 As shown, it can specifically include: Step S410: Extract target failure samples that match the abnormal feature points from the historical failure samples; Step S420: Generate test step adjustment suggestions based on the target failure sample. The test step adjustment suggestions include replacing the software source, adding version rollback tests, or adding abnormal simulation scenarios. Step S430: Based on the test step adjustment suggestions, adjust the test steps in the abnormal test path to generate the target test path.

[0114] The historical failure sample library can consist of a large number of failure paths, failure steps or failure logs that occurred during historical test executions. Each sample can be accompanied by relevant information such as corresponding failure reason annotations, context environment information and verified repair strategies.

[0115] One or more anomalous feature points can be extracted from the abnormal test path, and these can be used as retrieval criteria to extract target failure samples that match their semantic or pattern features from the historical failure sample library. In this process, the anomalous feature points can first be encoded into feature vectors. These feature vectors can be constructed based on multiple dimensions such as anomaly category, failure root cause label, resource fluctuation characteristics, error code sequence, and failure log keywords, so as to map them to the semantic space of historical samples for matching. By calculating the similarity between the anomalous feature point vector and the historical failure sample vector (e.g., cosine similarity, vector distance, or cluster label consistency), several target failure samples most closely matching the current anomaly pattern can be selected.

[0116] After extracting the target failure sample, test step adjustment suggestions related to the current abnormal test path can be generated based on the failure reasons in the sample and historically verified repair methods. The generated adjustment suggestions can include operations such as replacing the software source, adding version rollback tests, or adding abnormal simulation scenarios. The purpose is to make the repaired or enhanced test path more stable and comprehensive in the next round of execution.

[0117] For example, when the target failure sample shows the failure reason as "corrupted software source image" or "incorrect dependency package version", an adjustment suggestion of "replacing the software source" can be generated, replacing the "dependency installation step" in the abnormal test path with installation using an alternative image source, thus obtaining the target test path; when the target failure sample shows the failure reason as "version incompatibility" or "driver upgrade causing function rollback", an adjustment suggestion of "adding version rollback test" can be generated, thus inserting a "version downgrade step" or "driver compatibility rollback verification step" into the current abnormal test path, thus obtaining the target test path; when the failure sample shows that the cause of the anomaly is an unstable operating environment or abnormal data fluctuations, such as GPU temperature peaks causing inference failure, an adjustment suggestion of "adding abnormal simulation scenarios" can be generated, adding "stress test steps", "random noise injection scenarios", "resource fluctuation simulation steps", etc. to the abnormal test path to enhance test coverage, thus obtaining the target test path.

[0118] By matching target failure samples from historical failure samples, adjustment suggestions such as step replacement, rollback verification, or abnormal scenario injection are generated, and the abnormal test path is structurally modified. Targeted compensation can be made at the root cause of the anomaly, making the adjusted test path more comprehensive, with a higher failure reproducibility rate and more thorough scenario simulation. This effectively improves the test path's ability to capture problem scenarios, making anomaly detection more sensitive and path execution more stable.

[0119] In one example embodiment of this disclosure, it can be achieved through Figure 5The steps described in the document involve using the target test path as an improvement sample to update the test sequence generation model used to generate the target test step sequence, thereby achieving self-evolutionary processing of the test step sequence generation. (Refer to...) Figure 5 As shown, it can specifically include: Step S510: Combine the target test path with the corresponding natural language test requirement data to form a new training sample pair; Step S520: Add the new training sample pair to the training dataset of the test sequence generation model; Step S530: Retrain the test sequence generation model using the updated training dataset to adjust the model parameters of the test sequence generation model.

[0120] The target test path refers to the structured test step sequence obtained after anomaly feature point analysis, historical failure sample matching, and test step adjustment. This test step sequence can more effectively characterize the execution process that test requirements should take in complex environments.

[0121] After obtaining the target test path formed based on the repair of the abnormal test path, the target test path can be paired with its corresponding natural language test requirement data to form a new training sample pair. In practice, the test requirement content in natural language form that originally corresponded to the target test path can be found from the test execution record. For example, if the test requirement content in natural language form that originally corresponded to the target test path was "verify the stability of the GPU inference process under the latest driver version", this can be used as the training test requirement data for the training sample pair, while the repaired target test path can be used as the sequence of actual test steps.

[0122] After constructing new training sample pairs, these pairs can be added to the existing training dataset of the test sequence generation model, allowing the training dataset to gradually accumulate experiential knowledge gained from the anomaly repair path. Before adding new training sample pairs to the training dataset, the structure of the training dataset can be checked to avoid adding duplicate or conflicting samples. For example, if samples with the same text as the current requirement but different test step sequences already exist in the training dataset, a version control mechanism can be used to distinguish the samples, or manual annotation can be used to determine which samples to retain. Alternatively, a weighting mechanism can be used to give new training sample pairs a higher probability of occurrence during the training phase, thereby enhancing the model's adaptability to anomaly scenarios.

[0123] After updating the training dataset, the test sequence generation model can be retrained based on the new training dataset, allowing the model parameters to adaptively adjust according to the semantic features and path structure of the latest training sample pairs. During retraining, incremental training can be used, where sub-network parameters are fine-tuned only based on new samples, allowing the model to quickly absorb new knowledge while retaining its original capabilities; alternatively, full training can be used, retraining the model on all samples to ensure the consistency of the overall model's input and output.

[0124] By combining the target test path with the corresponding natural language requirements to form new training samples and adding them to the training dataset, and then retraining the test sequence generation model, a closed-loop optimization of failure, repair, learning, and regeneration is achieved. This enables the model to automatically generate repaired and higher-quality test steps when facing similar requirements in the future, reducing repeated errors, improving model adaptability, and continuously evolving and enhancing the test path generation capability.

[0125] In one example embodiment of this disclosure, it can also be achieved through Figure 6 The steps described in the document optimize the policy network; refer to [reference]. Figure 6 As shown, it can specifically include: Step S610: Based on the test task scheduling instruction and the test execution result, determine the successful test path, wherein the successful test path is a test path with a test coverage rate or test pass rate greater than or equal to a preset ratio threshold. Step S620: Construct new state-action pair training data using the test task scheduling instructions and resource status information corresponding to the successful test path; Step S630: Using the new state-action pair training data, update the policy gradient of the policy network in the test step optimization model used to generate the test task scheduling instruction, so as to optimize the model parameters of the policy network.

[0126] A successful test path refers to a test path in which all or most of the test steps are successfully executed, and the pass rate is greater than or equal to a preset threshold. After the test task scheduling instructions have been assigned to the execution nodes and the execution nodes have completed the step execution, the test paths can be determined as successful test paths based on the test task scheduling instructions and the test execution results. First, the sequence of test steps in the test task scheduling instructions can be aligned sequentially with the return logs of the execution nodes to form a structured path execution record. Then, the path can be analyzed item by item based on the success status code or stability index of the execution results. In an optional implementation, the success or failure of a step can be determined based on the execution status field returned in the test execution results. For example, if the return status of the test execution results is "SUCCESS", "PASS", or "VALIDATED", it can be considered successful; when the return status is "DEGRADED" but resource fluctuations are still within a safe range, it can also be determined as successful based on a threshold strategy.

[0127] After a successful test path is identified, new state-action pair training data can be constructed based on the test task scheduling instructions corresponding to the successful test path and the resource state information collected at that time. This new state-action pair training data is used for policy learning in the reverse reinforcement training phase, enabling the policy network to refer to the decision patterns of the successful path in subsequent inference.

[0128] By extracting state-action pairs from successful test paths and using them for policy gradient updates in the policy network, the policy network's learning of high-quality scheduling policies can be strengthened. This enables the model to prioritize node allocation policies that have been proven effective, thereby improving the scheduling success rate of subsequent test tasks, making test task execution more efficient, node resource utilization more reasonable, and further enhancing the overall system performance and success rate.

[0129] In one example embodiment of this disclosure, the distribution of test task scheduling instructions to distributed execution nodes in the environment under test can be achieved through the following steps, specifically including: Based on the computing resource status information of the distributed execution nodes, the node identification information of each distributed execution node can be determined. The node identification information is dynamically assigned according to the computing resource status information of each distributed execution node. According to the node identification information, each distributed execution node is bound to the test steps in the test task scheduling instruction. The test steps corresponding to the test task scheduling instruction are sent to the bound distributed execution nodes so that each distributed execution node can execute its corresponding test steps to obtain the test execution results.

[0130] Among them, node identification information refers to a flag field used to uniquely identify a certain execution node in the cluster and can be used for binding scheduling instructions. It can be a node number (such as node-01), node IP address, node load level flag or node function label (such as GPU-node, IO-node), etc. This embodiment does not limit the specific encoding method of node identification information, as long as it can maintain uniqueness in the scheduling and execution process.

[0131] Before issuing test task scheduling instructions to distributed execution nodes, their node identification information can be determined based on the computing resource status information of the distributed execution nodes. At this point, the cluster monitoring module can first collect the resource status of each execution node in real time, including CPU utilization, GPU resource usage, remaining memory capacity, disk read / write bandwidth, network latency, current task queue length, and node health status. In the specific implementation, nodes can be identified according to a preset strategy based on the collected node resource status information. For example, when a node has a large amount of remaining GPU memory and is idle, it can be identified as a high-availability node, and this identification information can be recorded in the node status table; when a node's load is close to its limit or it experiences CPU saturation or insufficient memory, it can be identified as a low-availability execution node, so that the scheduling module avoids scheduling resource-sensitive tasks to this execution node in subsequent steps.

[0132] Binding refers to explicitly associating a test step with a specific node, enabling that node to receive the bound test step during subsequent execution. Binding can be based on node identification information and the resource requirements of the test step. For example, when a test step involves GPU inference operations, nodes with high available GPU memory and low GPU temperature can be prioritized; when a test step involves a large number of file read / write operations, nodes with strong I / O performance can be prioritized. By matching resource requirements with node capabilities, the binding results can be more reasonable, improving the test execution success rate.

[0133] For example, the binding process can adopt a sequential binding method, that is, selecting the most suitable node step by step according to the order of the steps; or a multi-step joint binding method, binding multiple steps according to their dependencies, so that steps on the same dependency chain are bound to the same or a group of nodes with good resource consistency. For example, when executing the model inference chain, the three steps of "model loading, model warm-up, and inference testing" can be bound to the same GPU node to avoid the environment switching overhead caused by cross-nodes. In optional implementations, the node binding results can also be cached so that subsequent test steps with similar resource requirements can be quickly bound, improving scheduling efficiency; or a node binding correction mechanism can be used so that when a node encounters a resource anomaly during execution, the unexecuted steps can be rebound to other nodes. This embodiment does not impose any special limitations on this.

[0134] By determining node identification information based on the computing resource status information of distributed execution nodes and binding test steps with node identification, and then sending the corresponding test steps to the bound nodes, it can be ensured that each test step is executed on the most suitable node, making node resource matching more accurate and execution stability higher. At the same time, it avoids resource conflicts, scheduling chaos or incorrect step delivery, thereby improving execution efficiency, task distribution accuracy and overall stability in the distributed testing environment.

[0135] Figure 7 The schematic diagram illustrates a flowchart of an automated testing method based on test sequence generation according to some other embodiments of the present disclosure.

[0136] refer to Figure 7As shown, in step S711, test requirement data in natural language form input by the user is received and used as input information for subsequent conversion processing; in step S712, the test requirement data in natural language form is input to a preset test sequence generation model, wherein the test sequence generation model includes a semantic encoder and a sequence decoder, used for semantic modeling and structured generation of natural language content; in step S713, the target test step sequence corresponding to the natural language input is output by the test sequence generation model, serving as the basis for subsequent scheduling optimization and execution process; in step S714, the resource status information of the current test environment and the target test step sequence construction status are combined. A state vector is generated and input into the test step optimization model. The probability distribution of each current scheduled action is obtained through policy network inference, thereby determining the target scheduled action to allocate the test steps to the distributed execution nodes. In step S715, a test task scheduling instruction is generated based on the inferred scheduling action, ensuring that the instruction accurately describes the selection of execution nodes and the execution order of each test step. In step S716, the generated test task scheduling instruction is allocated to the distributed execution nodes, which then execute their corresponding test steps and return the test execution results. In step S717, the test execution results are evaluated; if the test task is successfully executed, the next step is executed. Step S718, output the test results and end the method flow; if the test task fails, proceed to step S721; in step S718, based on the test task scheduling instructions and test execution results, identify the corresponding abnormal test path to determine the key step combination that caused the test failure; in step S719, the reward feedback of the policy network can be calculated based on the test task scheduling instructions and test execution results, and this reward feedback can be used for policy network updates during the training phase; in step S720, the reward feedback obtained in step S719 can be used to optimize and update the policy network in the test step optimization model. The update method can be based on the policy gradient algorithm, by maximizing the probability of successful actions or The action advantage function value is increased to adjust the model parameters of the policy network towards a better policy. In step S721, failure samples matching abnormal feature points are obtained from a preset historical failure sample library to guide the optimization and repair of abnormal paths. Test step adjustment suggestions are generated based on the target failure samples, including step replacement, version rollback, or abnormal scenario simulation. The abnormal test path is optimized and adjusted using these adjustment suggestions to generate a target test path. In step S722, the target test path can be combined with the corresponding natural language test requirement data to form a new training sample pair. In step S723, the new training sample pair is added to the training dataset of the test sequence generation model.By retraining the test sequence generation model using the updated training dataset, the model parameters can be adjusted and updated, enabling self-evolution of the test step sequence generation.

[0137] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0138] Furthermore, in this example embodiment, an automated testing apparatus based on test sequence generation is also provided. (Refer to...) Figure 8 As shown, the automated testing device 800 based on test sequence generation includes a test step sequence generation module 810, a test task scheduling module 820, and a test execution result generation module 830, wherein: The test step sequence generation module 810 is used to acquire test requirement data in natural language form and generate a target test step sequence based on the test requirement data. The target test step sequence is an ordered sequence of test actions to be executed, consisting of multiple test steps. The test task scheduling module 820 is used to collect resource status information of the environment under test, and combine the resource status information with the target test step sequence to generate test task scheduling instructions. The test task scheduling instructions are a set of structured control instructions used to guide the distributed execution nodes in the environment under test to execute test tasks. The test execution result generation module 830 is used to allocate the test task scheduling instruction to the distributed execution node in the environment to be tested, so as to execute the test steps in the test task scheduling instruction through the distributed execution node and obtain the test execution result.

[0139] In some example embodiments of this disclosure, based on the foregoing scheme, the test step sequence generation module 810 is configured as follows: The test requirement data in natural language form is preprocessed to obtain test requirement word vectors; The test requirement word vectors are input into the trained test sequence generation model to obtain the target test step sequence.

[0140] In some example embodiments of this disclosure, based on the foregoing scheme, the test sequence generation model includes a semantic encoder and a sequence decoder, and the test sequence generation model is obtained through a first model training process, which includes: Obtain the first training dataset, which includes training and testing requirements data in natural language form and reference test step labels; The training and testing requirements data are input into the semantic encoder to obtain a context-dependent hidden state representation vector; The hidden state representation vector is input into the sequence decoder, and the sequence decoder is forced to learn by inputting the reference test step label, so that the target loss function between the output predicted test step sequence and the reference test step label is accelerated, thereby completing the training of the test sequence generation model.

[0141] In some example embodiments of this disclosure, based on the foregoing scheme, the test task scheduling module 820 is configured as follows: Based on the resource status information and the target test step sequence, a state vector is constructed; The state vector is input into the trained test step optimization model to infer the probability distribution of each current scheduling action. The current scheduling action is used to indicate the execution order of assigning at least two test steps to each distributed execution node. A test task scheduling instruction is generated based on the current scheduling action with the highest probability value in the probability distribution.

[0142] In some example embodiments of this disclosure, based on the foregoing scheme, the test step optimization model during the training phase includes at least a policy network and a value network. The test step optimization model is obtained through a second model training process, which includes: The sampled state vector in the pre-designed state space is input into the policy network, so that the policy network, in combination with the sampled state vector, outputs the sampled scheduling action belonging to the pre-designed action space. The sampled state vector and the sampled scheduling action are input into the value network so that the combination of the sampled state vector and the sampled scheduling action is evaluated by the reward function of the value network to generate an evaluation reward value. The policy network is updated with policy gradients using the evaluation reward value, and the value network is updated with parameter regression using the evaluation reward value. When the loss values ​​of the policy network and the value network are less than or equal to a preset loss threshold, or when the number of iterations in the training process reaches a preset maximum number of iterations, the trained policy network and the value network are obtained. The evaluation reward value is obtained by the reward function based on the test pass rate, resource utilization rate, average task completion time, and improvement in historical coverage.

[0143] In some example embodiments of this disclosure, based on the foregoing scheme, the test execution result generation module 830 is further configured as follows: Based on the test task scheduling instructions and the test execution results, determine the abnormal test path; By combining preset historical failure samples and abnormal feature points in the abnormal test path, the abnormal test path is adjusted to obtain the target test path; The target test path is used as an improvement sample to update the test sequence generation model used to generate the target test step sequence.

[0144] In some example embodiments of this disclosure, based on the foregoing scheme, the test execution result generation module 830 is further configured as follows: The test task scheduling instruction and the test execution result are input into the trained anomaly test prediction model, and the probability distribution data of each test path is output. The anomaly test prediction model is obtained by training and modeling based on historical execution logs. The historical test execution logs are log data formed by recording the execution status, execution result and resource usage of the test path or test step corresponding to the test task scheduling instruction during the execution of historical test tasks. Under the constraint of abnormal probability in the probability distribution data, test paths with test coverage or test pass rate less than a preset proportion threshold are selected as abnormal test paths.

[0145] In some example embodiments of this disclosure, based on the foregoing scheme, the test execution result generation module 830 is further configured as follows: Extract target failure samples that match the abnormal feature points from the historical failure samples; Based on the target failure samples, test step adjustment suggestions are generated, which include replacing the software source, adding version rollback tests, or adding abnormal simulation scenarios. Based on the proposed test step adjustment, the test steps in the abnormal test path are adjusted to generate the target test path.

[0146] In some example embodiments of this disclosure, based on the foregoing scheme, the test execution result generation module 830 is further configured as follows: The target test path is combined with the corresponding natural language test requirement data to form a new training sample pair; The new training sample pair is added to the training dataset of the test sequence generation model; The test sequence generation model is retrained using the updated training dataset to adjust its model parameters.

[0147] In some example embodiments of this disclosure, based on the foregoing scheme, the automated testing device 800 further includes a test step optimization model self-evolution module, which is configured as follows: Based on the test task scheduling instructions and the test execution results, a successful test path is determined. The successful test path is a test path whose test coverage or test pass rate is greater than or equal to a preset ratio threshold. New state-action pair training data are constructed using the test task scheduling instructions and resource status information corresponding to the successful test path. Using the new state-action pair training data, the policy gradient in the policy network of the test step optimization model used to generate the test task scheduling instructions is updated to optimize the model parameters of the policy network.

[0148] In some example embodiments of this disclosure, based on the foregoing scheme, the test execution result generation module 830 is configured as follows: Based on the computing resource status information of the distributed execution nodes, the node identification information of each distributed execution node is determined. The node identification information is a node identifier dynamically allocated according to the computing resource status information of each distributed execution node. Based on the node identification information, each of the distributed execution nodes is bound to the test steps in the test task scheduling instruction; The test steps corresponding to the test task scheduling instruction are sent to the bound distributed execution nodes so that each distributed execution node can execute its corresponding test steps to obtain the test execution results.

[0149] The specific details of each module of the automated test device based on test sequence generation mentioned above have been described in detail in the corresponding automated test method based on test sequence generation, so they will not be repeated here.

[0150] It should be noted that although several modules or units of the automated test apparatus based on test sequence generation have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0151] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described automated testing method based on test sequence generation is also provided.

[0152] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be embodied in the following forms: a completely hardware embodiment, a completely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0153] The following reference Figure 9 To describe an electronic device 900 according to such an embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0154] like Figure 9 As shown, the electronic device 900 is presented in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, a bus 930 connecting different system components (including storage unit 920 and processing unit 910), and a display unit 940.

[0155] The storage unit stores program code that can be executed by the processing unit 910, causing the processing unit 910 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 910 can perform actions such as... Figure 1 Step S110 shows the acquisition of test requirement data in natural language form, and the generation of a target test step sequence based on the test requirement data. The target test step sequence is an ordered sequence of test actions to be executed, consisting of multiple test steps. Step S120 shows the acquisition of resource status information of the environment under test, and the generation of test task scheduling instructions by combining the resource status information with the target test step sequence. The test task scheduling instructions are a set of structured control instructions used to guide the distributed execution nodes in the environment under test to execute test tasks. Step S130 shows the allocation of the test task scheduling instructions to the distributed execution nodes in the environment under test, so that the test steps in the test task scheduling instructions can be executed by the distributed execution nodes to obtain test execution results.

[0156] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 921 and / or cache memory (Cache) 922, and may further include read-only memory (ROM) 923.

[0157] Storage unit 920 may also include a program / utility 924 having a set (at least one) of program modules 925, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0158] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0159] Electronic device 900 can also communicate with one or more external devices 970 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0160] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0161] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of this disclosure may also be implemented as a program product including program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0162] refer to Figure 10 As shown, a program product 1000 for implementing the above-described automated testing method based on test sequence generation according to embodiments of the present disclosure is described. This product may be a portable compact disc read-only memory (CD-ROM) and includes program code, and may run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0163] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0164] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0165] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0166] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0167] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0168] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0169] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0170] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An automated testing method based on test sequence generation, characterized in that, include: Acquire test requirement data in natural language form, and generate a target test step sequence based on the test requirement data. The target test step sequence is an ordered sequence of test actions to be executed, consisting of multiple test steps. Collect resource status information of the environment to be tested, and combine the resource status information with the target test step sequence to generate test task scheduling instructions. The test task scheduling instructions are a set of structured control instructions used to guide the distributed execution nodes in the environment to be tested to execute test tasks. The test task scheduling instruction is assigned to a distributed execution node in the environment to be tested, so that the test steps in the test task scheduling instruction are executed by the distributed execution node to obtain the test execution result.

2. The automated testing method according to claim 1, characterized in that, The generation of the target test step sequence based on the test requirement data includes: The test requirement data in natural language form is preprocessed to obtain test requirement word vectors; The test requirement word vectors are input into the trained test sequence generation model to obtain the target test step sequence.

3. The automated testing method according to claim 2, characterized in that, The test sequence generation model includes a semantic encoder and a sequence decoder. The test sequence generation model is obtained through a first model training process, which includes: Obtain the first training dataset, which includes training and testing requirements data in natural language form and reference test step labels; The training and testing requirements data are input into the semantic encoder to obtain a context-dependent hidden state representation vector; The hidden state representation vector is input into the sequence decoder, and the sequence decoder is forced to learn by inputting the reference test step label, thereby completing the training of the test sequence generation model.

4. The automated testing method according to any one of claims 1 to 3, characterized in that, The step of combining the resource status information with the target test step sequence to generate a test task scheduling instruction includes: Based on the resource status information and the target test step sequence, a state vector is constructed; The state vector is input into the trained test step optimization model to infer the probability distribution of each current scheduling action. The current scheduling action is used to indicate the execution order of assigning at least two test steps to each distributed execution node. A test task scheduling instruction is generated based on the current scheduling action with the highest probability value in the probability distribution.

5. The automated testing method according to claim 4, characterized in that, The test step optimization model during the training phase includes at least a policy network and a value network. This test step optimization model is obtained through a second model training process, which includes: The sampled state vector in the pre-designed state space is input into the policy network, so that the policy network, in combination with the sampled state vector, outputs the sampled scheduling action in the pre-designed action space. The sampled state vector and the sampled scheduling action are input into the value network so that the combination of the sampled state vector and the sampled scheduling action is evaluated by the reward function of the value network to generate an evaluation reward value. The policy network is updated with policy gradients using the evaluation reward value, and the value network is updated with parameter regression using the evaluation reward value. When the loss values ​​of the policy network and the value network are less than or equal to a preset loss threshold, or when the number of iterations in the training process reaches a preset maximum number of iterations, the trained policy network and the value network are obtained. The evaluation reward value is obtained by the reward function based on at least one of the following: test pass rate, resource utilization rate, average task completion time, and improvement in historical coverage.

6. The automated testing method according to any one of claims 1 to 3, characterized in that, After allocating the test task scheduling instruction to distributed execution nodes in the environment under test, so as to execute the test steps in the test task scheduling instruction through the distributed execution nodes and obtain the test execution result, the method further includes: Based on the test task scheduling instructions and the test execution results, determine the abnormal test path; By combining preset historical failure samples and abnormal feature points in the abnormal test path, the abnormal test path is adjusted to obtain the target test path; The target test path is used as an improvement sample to update the test sequence generation model used to generate the target test step sequence.

7. The automated testing method according to claim 6, characterized in that, The step of determining the abnormal test path based on the test task scheduling instruction and the test execution result includes: The test task scheduling instructions and the test execution results are input into the trained anomaly test prediction model, and the probability distribution data of each test path is output. The anomaly test prediction model is obtained by training and modeling based on historical execution logs. The historical test execution logs are log data formed by recording the execution status, execution results and resource usage of the test path or test step corresponding to the test task scheduling instructions during the execution of historical test tasks. Under the constraint of abnormal probability in the probability distribution data, test paths with test coverage or test pass rate less than a preset proportion threshold are selected as abnormal test paths.

8. The automated testing method according to claim 6, characterized in that, The step of combining preset historical failure samples and abnormal feature points in the abnormal test path to optimize and adjust the abnormal test path to obtain the target test path includes: Extract target failure samples that match the abnormal feature points from the historical failure samples; Based on the target failure samples, test step adjustment suggestions are generated, which include replacing the software source, adding version rollback tests, or adding abnormal simulation scenarios. Based on the proposed test step adjustment, the test steps in the abnormal test path are adjusted to generate the target test path.

9. The automated testing method according to claim 6, characterized in that, The step of using the target test path as an improvement sample to update the test sequence generation model used to generate the target test step sequence, in order to achieve self-evolution of the test step sequence, includes: The target test path is combined with the corresponding natural language test requirement data to form a new training sample pair; The new training sample pair is added to the training dataset of the test sequence generation model; The test sequence generation model is retrained using the updated training dataset to adjust its model parameters.

10. The automated testing method according to claim 1 or 6, characterized in that, After allocating the test task scheduling instruction to distributed execution nodes in the environment under test, so as to execute the test steps in the test task scheduling instruction through the distributed execution nodes and obtain the test execution result, the method further includes: Based on the test task scheduling instructions and the test execution results, a successful test path is determined. The successful test path is a test path whose test coverage or test pass rate is greater than or equal to a preset ratio threshold. New state-action pair training data are constructed using the test task scheduling instructions and resource status information corresponding to the successful test path. Using the new state-action pair training data, the policy gradient in the policy network of the test step optimization model used to generate the test task scheduling instructions is updated to optimize the model parameters of the policy network.

11. The automated testing method according to any one of claims 1 to 3, characterized in that, The step of allocating the test task scheduling instructions to the distributed execution nodes in the environment under test includes: Based on the computing resource status information of the distributed execution nodes, the node identification information of each distributed execution node is determined. The node identification information is a node identifier dynamically allocated according to the computing resource status information of each distributed execution node. Based on the node identification information, each of the distributed execution nodes is bound to the test steps in the test task scheduling instruction; The test steps corresponding to the test task scheduling instruction are sent to the bound distributed execution nodes so that each distributed execution node can execute its corresponding test steps to obtain the test execution results.

12. An automated testing device based on test sequence generation, characterized in that, include: The test step sequence generation module is used to acquire test requirement data in natural language form and generate a target test step sequence based on the test requirement data. The target test step sequence is an ordered sequence of test actions to be executed, consisting of multiple test steps. The test task scheduling module is used to collect resource status information of the environment under test, and combine the resource status information with the target test step sequence to generate test task scheduling instructions. The test task scheduling instructions are a set of structured control instructions used to guide the distributed execution nodes in the environment under test to execute test tasks. The test execution result generation module is used to allocate the test task scheduling instruction to the distributed execution nodes in the environment to be tested, so as to execute the test steps in the test task scheduling instruction through the distributed execution nodes and obtain the test execution result.

13. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the automated testing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the automated testing method as described in any one of claims 1 to 11.

Images