Method for intelligent arrangement and dynamic execution of automatic test scene of complex system driven by large model

Abstract

The invention relates to the technical field of crossing of computer software testing and artificial intelligence, in particular to a method for intelligent arrangement and dynamic execution of an automatic testing scene of a complex system driven by a large model, which comprises the following steps of: 1, acquiring multi-source data; step 2, data preprocessing; step 3, constructing a system model; 4, training a large model and arranging a scene; and 5, performing test execution and dynamic adjustment. According to the method, a dual-mode data fusion layer is designed to realize unified representation of heterogeneous data, an end-to-end intelligent arrangement framework is constructed to get rid of manual dependence, a reinforcement learning dynamic adjustment mechanism is proposed to shorten abnormal response time, and a multi-target joint optimization strategy is established to balance coverage rate, efficiency and resource consumption. According to the method, the test arrangement efficiency can be improved by 3-5 orders of magnitude, the exception handling response time is shortened to the minute level, the test coverage rate is improved to 90% or above, meanwhile, the resource utilization rate is optimized by 20%-30%, and the test quality and delivery efficiency of a complex system are effectively guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer software testing and artificial intelligence, in particular to a method for intelligent arrangement and dynamic execution of complex system automatic test scenarios driven by large models. BACKGROUND

[0002] The existing automatic test technology mainly contains four categories, each with specific technical implementation, application scenario and obvious defects. The traditional method based on script and test framework is the most basic and widely used, and test engineers write scripts with the help of Selenium, Appium, RestAssured, Pytest and other frameworks to simulate user operations, call interfaces or execute code logic to verify system response. Test cases exist in the form of structured files or code, and are executed by scripts or CI / CD, for example, simulating the shopping process with Selenium in an e-commerce website, testing API functions with RestAssured, and triggering execution by Jenkins, but this method has high cost of script writing and maintenance, UI testing is sensitive to interface changes, it is difficult to handle complex business logic and dependency relationships, and test arrangement lacks intelligence. The rule engine-based method introduces rule definitions for test case generation and execution order, and rules are based on system architecture, interface documentation, business rules, etc., such as ensuring that identity verification tests are executed before transfer tests in a banking system, or allocating more concurrent threads for high-frequency interfaces, but as the complexity of the system increases, the number of rules grows exponentially, the maintenance cost is extremely high, and it is easy to miss or define errors, making it difficult to adapt to system changes. The heuristic algorithm-based method uses genetic algorithms, simulated annealing and other optimization algorithms to optimize test case order or combination, and evaluates the pros and cons through an adaptive function, for example, using genetic algorithms to optimize test sequences in a financial system, and the adaptive function combines coverage and execution time, but the algorithm converges slowly, the adaptive function depends on the experience of engineers, and it cannot learn the deep rules of the system, and it is mostly a local optimization. The method based on simple data statistical analysis collects results and performance data in testing, and uses average, standard deviation and charts for analysis and monitoring, such as monitoring response time changes through trend charts in continuous integration, but it only stays at the basic data level, lacks depth correlation analysis, prediction ability and intelligent decision-making ability.

[0003] In the current complex system testing field, the rapid development of microservice architecture, distributed systems and Internet of Things (IoT) integrated systems is pushing testing practices to unprecedented challenges, focusing on the following three core contradictions, which are difficult to effectively resolve by existing technical means: (1) Exponential growth of system complexity and bottleneck of test efficiency: Modern critical business systems, such as complex industrial control systems carrying Industry 4.0, financial transaction platforms with high concurrency transaction processing, super large e-commerce platforms supporting large promotions such as "Double 11", and intelligent transportation systems with real-time decision making, are experiencing explosive growth in the number of internal components. Specifically, a typical modern industrial control system may integrate more than 5000 real-time I / O nodes, while the backend architecture of a large e-commerce platform may consist of hundreds or even thousands of interdependent microservices. In the face of such a large system size, traditional testing methods rely heavily on engineers manually combing the dependencies between components, writing and arranging test cases. Engineers often need to invest weeks or even months of time just to understand the system's topology, data flow, and interaction protocols, which results in an abnormally long test preparation period, low test efficiency, and is far from matching the current business requirements of rapid iteration and agile delivery. This lag in efficiency directly threatens the timely verification of product quality and the protection of market time.

[0004] (2) Inherent conflict between system dynamic runtime characteristics and static testing strategies: The inherent characteristics of microservice architecture, such as elastic scaling of services (dynamically increasing or decreasing instances based on load), blue-green deployment (running new and old versions in parallel for quick switching), canary release (gradually directing traffic to new versions to control risks), dynamic service discovery and load balancing, all make the system in a state of continuous change. Similarly, the status (online / offline, configuration changes, firmware upgrades) of tens of thousands of devices in an Internet of Things system can also change dynamically at any time. However, most traditional testing strategies are designed based on a static snapshot of the system. They are like "fossilized" scripts, lacking the ability to perceive and adapt to changes in the runtime environment. For example, a well-designed test case may be effective for a specific service instance and environment configuration at the initial deployment stage, but over time, if some service instances are removed due to failure, new instances are dynamically added, the network topology changes due to failure or policy adjustment, or the response characteristics of dependent external services change, the test case may no longer be applicable, even causing false positives or causing test execution to fail, wasting valuable test resources. This disconnection between static strategies and dynamic reality seriously undermines the credibility and effectiveness of test results, making it difficult for testing to accurately reflect the performance of the system in the real running environment.

[0005] (3) Difficulty in reconciling multi-dimensional testing goals with a single optimization method: In actual work, the test team usually needs to pursue multiple and often mutually restrictive optimization goals at the same time, including: striving to cover as many code paths, function points and key business scenarios as possible, for example, setting a target coverage rate of more than 95% to ensure system function integrity and robustness; pursuing the shortest possible test execution period to speed up defect feedback, for example, the goal is to reduce test execution time by more than 30% to support faster development iterations; efficiently executing test tasks in parallel on limited and often costly test infrastructure (such as physical servers, virtual machines, container resources), for example, the goal is to improve resource utilization by more than 20% to reduce test costs; ensuring that the test process itself is stable and reliable, minimizing false positives (reporting normal functions as defects) and false negatives (failing to detect actual defects), while ensuring that the test environment is not damaged or contaminated during the test process, ensuring the accuracy and repeatability of test results. However, relying on individual engineer experience or using a single, isolated traditional optimization method (for example, only pursuing coverage rate to the extreme while ignoring execution speed, or only focusing on speed at the expense of resource utilization), it is difficult to find a balance point between these multi-dimensional and often conflicting goals, known as the "Pareto optimal solution". The limitations of this optimization approach have become a key bottleneck restricting further improvement of the industry's testing capabilities. SUMMARY

[0006] The purpose of the present application is to provide a large model driven complex system automatic test scene intelligent arrangement and dynamic execution method to solve the problems raised in the background art.

[0007] To achieve the above purpose, the present application provides the following technical solutions: A large model driven complex system automatic test scene intelligent arrangement and dynamic execution method, comprising the following steps: Step 1, multi-source data collection: collect static architecture data, dynamic runtime data and business process data, static architecture data includes microservice interface definition, UML component diagram dependency relationship, Docker Compose resource configuration; dynamic runtime data includes distributed call chain information, component resource indicators, Kubernetes event logs; business process data includes user operation logs, transaction transaction logs; Step 2, data preprocessing: clean and fuse the collected heterogeneous raw data, filter abnormal data through the IQR method, fill in missing values with the XGBoost regression model, associate business operations with component calls with transaction ID, and use linear interpolation to achieve multi-frequency data time alignment; Step 3, System Model Construction: Based on the preprocessed data, construct an architecture diagram model and a business process state machine. The architecture diagram model is constructed as a directed graph using NetworkX, clarifying the attributes of component nodes and call edges. The business process state machine extracts process elements according to the BPMN2.0 standard, generating a state transition matrix and flow path. Step 4, Large Model Training and Scene Orchestration: A dual encoder architecture is constructed using graph attention network and Transformer. The model is trained using historical efficient test solutions as positive samples and ineffective solutions as negative samples. An executable test plan is output through dependency resolution, concurrent strategy generation, and test case generation. Step 5, Test Execution and Dynamic Adjustment: The execution environment is built based on Docker+Kubernetes. Distributed task scheduling is implemented through Celery. The reinforcement learning PPO algorithm is adopted. Combined with real-time collected test indicators and system resource data, the number of concurrent threads, test order and resource allocation are dynamically adjusted to handle abnormal scenarios.

[0008] Preferably, the microservice interface definition in step 1 is automatically extracted using the Swagger Parser tool, the UML component diagram dependency relationship is extracted using the EclipseUML2 parser API, and the Docker Compose file is read using the PyYAML library.

[0009] Preferably, in step 1, the distributed call chain information is collected through Jaeger / Zipkin client SDK instrumentation, the component resource metrics are collected through Prometheus Exporter and stored in a time-series database, and the Kubernetes event logs are extracted through client-go API listening.

[0010] Preferably, in step 1, user operation logs are obtained through JavaScript instrumentation on the web frontend and Appium test log parsing; transaction logs include database audit logs and message middleware logs, which are collected and parsed through the Debezium tool and the Kafka Connect client, respectively.

[0011] Preferably, in step 2, data cleaning converts the interface parameters into a unified JSON standard format and serializes the data using Apache Avro; data fusion establishes a multi-dimensional mapping relationship of "user operation - component call - data layer interaction".

[0012] Preferably, the component node attributes of the architecture graph model in step 3 include component_id, technology_stack, and resource_requirements, and the call edge attributes include call_type, data_size, and dependency_level; and the state transition matrix of the business process state machine is based on user operation log statistics to calculate transition probabilities.

[0013] Preferably, the composite loss function of the large model in step 4 is L=Lorder+Lconcurrency+γLcoverage, where Lorder is a sequential generation cross-entropy loss, Lconcurrency is a resource scheduling mean square error loss, and Lcoverage is a coverage penalty loss.

[0014] Preferably, the feature vector collected in real time in step 5 includes scene_id, current_step, response_time_p99, error_rate, cpu_usage, memory_usage, and dependency_status, and a 5-minute sliding window is used to process and extract time sequence rules.

[0015] Preferably, the reward function of the reinforcement learning in step 5 is R=0.6Rcoverage+0.3Refficiency−0.1Rresource, where Rcoverage is a coverage reward, Refficiency is an efficiency reward, and Rresource is a resource penalty; and when an abnormal scene is processed, if three consecutive use cases fail, pretest supplement execution and component health check are triggered.

[0016] Compared with the prior art, the present application has the following beneficial effects: The present application constructs a model based on Transformer to process business process event sequences, capture the evolution rules and long-distance dependency relationships of business logic in the time dimension. In order to break the "semantic gap" between the architecture static graph and the business dynamic flow, the present application designs a dual-mode data fusion layer, which uses transaction ID, user session ID, component call relationship and other key clues to deeply fuse data from different sources and in different formats, and finally forms a unified high-dimensional semantic space. The present application first solves the fusion problem of heterogeneous multi-source data, so that the machine can "understand" a complex system composed of multiple technology stacks like a human expert; it realizes the unified representation of heterogeneous complex systems, and breaks through the limitation of traditional methods which can only process single or simple architectures; and it significantly improves the insight of the model into the deep relationships of the system, laying a solid data foundation for subsequent intelligent arrangement and dynamic adjustment.

[0017] Based on a deep understanding of the system, this invention further develops an intelligent orchestration algorithm, which is the core link in transforming test requirements into efficient execution plans. This invention constructs an adaptive orchestration framework comprising three layers: dependency resolution, concurrent strategy generation, and resource allocation. The most innovative aspect is the design of an end-to-end test sequence generation algorithm leveraging the powerful autoregressive sequence generation capability of the large model. This algorithm can directly process the dynamic dependency graph output from the modeling phase, generating a preliminary test execution order without complex intermediate steps or manual intervention. Furthermore, we propose a multi-objective optimized concurrent strategy generation method, aiming to automatically balance the often conflicting objectives of test coverage, execution efficiency, and resource consumption. Ultimately, the large model can directly output a well-structured, executable test plan that can be used directly. The advantage of this orchestration technology lies in its complete breakthrough from the constraints of traditional methods: it no longer relies on time-consuming and potentially inefficient human experience or slow-converging heuristic algorithms, thus improving orchestration efficiency by several orders of magnitude; the generated strategies are also more optimized and realistic, and can automatically learn the complex mapping relationship between component dependencies, business processes and testing efficiency, thereby getting closer to the Pareto optimal solution while satisfying dependencies; at the same time, it can flexibly handle dynamically changing dependencies, demonstrating adaptability far exceeding that of static rules.

[0018] This invention introduces a powerful dynamic adjustment mechanism, endowing the testing process with self-repair and continuous optimization capabilities. This mechanism first establishes a multi-dimensional data acquisition system encompassing performance metrics, resource usage, and real-time test results, providing a continuous "fuel" for online decision-making. Based on reinforcement learning algorithms, this invention transforms these real-time metrics into the state space of a reinforcement learning model, designing a composite reward function that includes coverage rewards, time penalties, and resource consumption constraints to guide the agent in making optimal decisions. The decision action space is designed to be highly rich, including dynamically adjusting the number of concurrent test instances, reordering scenarios to be executed, and injecting faults at specific nodes to simulate stress. Simultaneously, this invention also constructs an automatic anomaly handling module, capable of detecting anomalies such as dependency conflicts and resource contention, and automatically repairing and rebalancing resources. This dynamic adjustment mechanism achieves closed-loop control of the test execution process, transforming the testing system from a "silent" executor into an intelligent agent with "perception-decision-execution" capabilities. Its most direct benefit is that the response time for exception handling is greatly reduced to the minute level, significantly improving the robustness and adaptability of the testing process. Even when faced with dynamic issues such as resource conflicts, logical conflicts, and component failures, it can ensure the stability and effectiveness of the test, truly realizing the transformation of the testing strategy from static preset to online adaptive.

[0019] To ensure that the entire test process is always efficient around the core goal, the present application specially designs a multi-objective joint optimization strategy, which is the "wisdom brain" connecting intelligent arrangement and dynamic adjustment. The present application proposes a test scene dynamic generation method based on runtime monitoring data, so that the test plan is no longer a rigid script, but a strategy that can evolve according to the real-time state of the system. Through the carefully designed composite reward function, the coverage, execution time, resource consumption and other multi-dimensional targets are included in the unified consideration, and the powerful multi-task learning and sequence generation capability of the large model is used to automatically search for solutions close to the Pareto optimal in the complex decision space. This means that the system can automatically balance multiple often contradictory goals without repeatedly weighing between efficiency, coverage and cost. The beneficial effect of this strategy is also significant: it not only realizes the automatic balance between multiple objectives and avoids the one-sidedness brought by single optimization, but also brings comprehensive improvement in test scheme quality - while ensuring high coverage, significantly shortening execution time and improving resource utilization, forming a win-win situation. More importantly, it makes the test process itself more intelligent and efficient, and the large model can continuously learn from historical data and real-time feedback to optimize the multi-objective balance strategy, thereby better adapting to complex testing needs.

[0020] The present application not only accurately solves the core pain points of the prior art in terms of low efficiency, poor adaptability and insufficient intelligence, but also improves the process of automated testing to a new intelligent level. The scheme can effectively cope with the great challenges brought by modern complex system testing, significantly improve test quality and efficiency, reduce test cost and risk, and provide strong protection for software quality and system reliability. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The system architecture diagram of the present application. DETAILED DESCRIPTION

[0022] As Figure 1 shown, a large model driven complex system automated test scene intelligent arrangement and dynamic execution method includes the following steps: Step 1, multi-source data acquisition: multi-source data acquisition is divided into three modules of static data acquisition, dynamic data acquisition and business process data acquisition, and the collected data and purposes are as follows: a. Static data acquisition: Microservice interface definition is automatically extracted through the analysis tool of Swagger / OpenAPI (such as Swagger Parser), focusing on obtaining interface name, request method, parameter type, response format and other information to provide basis for parameter verification and call format specification of test cases; Dependency extraction of UML component graphs utilizes the API of the Eclipse UML2 parser (such as UMLModelLoader) to load model files, traverse the relationships between components (such as dependencies and generalizations), and convert them into structured data (such as (component_id1, component_id2, dependency_type)) as the basis for calling edge attributes (such as dependency_level) in the architecture graph model; The DockerCompose file is read by a YAML parsing library (such as PyYAML) to extract resource configurations such as container name, image version, CPU / memory limits (such as cpus:0.5, mem_limit:512m), providing parameters for the initial resource scheduling strategy of the test scenario (such as resource quota allocation for concurrent testing).

[0023] b. Dynamic runtime data acquisition: Distributed call chain information is embedded in microservices through the Jaeger / Zipkin client SDK (such as JaegerClient) to collect the service name, call time, time consumption, trace ID, etc. of each call in real time, and build a mapping relationship of "user operation - component call". For example, the user's "order operation" is associated with the call chain of "order service → payment service → inventory service". Component resource metrics are collected through Prometheus Exporters (such as Node Exporter and JMXExporter), including CPU utilization, memory usage, disk I / O, etc., and stored in a time-series database at a frequency of seconds, providing real-time data for resource load assessment during test execution (such as determining whether resource expansion is triggered). Kubernetes event logs are monitored through Kubernetes APIs (such as client-go) to extract scaling events of service instances (such as changes in the replicas field of Deployment), and record scaling time, changes in the number of instances, and triggering reasons (such as HPA rules) to analyze the impact of elastic scaling on the stability of the test environment (such as whether the interface response latency increases during scaling).

[0024] c. Business process data collection: User operation log collection: Web front-ends collect clickstream data through JavaScript event tracking (such as embedding log reporting functions in button click events), including operation time, page URL, element ID, user ID, etc., forming a user interaction path sequence (such as "homepage → product list → details page → place order"), providing data for user operation nodes in the business process state machine; Mobile UI interaction sequences are achieved by analyzing Appium test logs, which contain operation types (such as clicks, inputs), element locators (such as id, xpath), and operation results. Python log parsing tools (such as loguru) are used to extract and standardize the data, supplement the mobile business scenario process data, and ensure the completeness of multi-terminal business process coverage.

[0025] Transaction log collection: Database audit logs (such as MySQL's binlog, PostgreSQL's pgAudit) are parsed by audit tools (such as Debezium) to extract SQL operation types (such as SELECT, UPDATE), involved table names, execution times, and associated user operation IDs. The mapping between "user operation-data layer interaction" is established (such as "order operation→order table INSERT→inventory table UPDATE"). Message middleware logs (such as Kafka's topic logs, RabbitMQ's channel logs) are collected through client tools (such as Kafka Connect) to extract message senders, receivers, topic / queue names, message IDs, and sending times. The dependency relationship of asynchronous business processes is identified (such as "payment completion message→order status update service") to avoid missing asynchronous links during test scenario arrangement.

[0026] Step 2, Data preprocessing: Data preprocessing is directly related to the multi-source data collection in step 1, which is the process of standardizing the collected heterogeneous raw data (system architecture static / dynamic data, business process operation / transaction logs). By solving data anomalies, missing values, inconsistent formats, and time-space misplacement problems, high-quality and high-correlation data are provided for subsequent system model construction. Specifically, it includes: a. Data cleaning: Identify and filter interface response time abnormal data through IQR method (for interface performance indicators in dynamic runtime monitoring data collected in step 1); Use XGBoost regression model to predict and fill missing values of component resource consumption (for Prometheus component resource monitoring data collected in step 1); Convert interface parameters to JSON standard format and use Apache Avro for data serialization (for Swagger interface definitions and unstructured parameters in various logs collected in step 1).

[0027] b. Data fusion: Correlate business operation logs with distributed call chains using transaction IDs as the link, establishing a "user operation-component call" mapping relationship (correlation step 1: collect user operation logs and Jaeger / Zipkin call chain data). For monitoring data with different sampling frequencies (such as Prometheus second-level monitoring and Kafka minute-level logs), use linear interpolation to achieve time alignment, ensuring the spatiotemporal consistency and relevance of the data (processing step 1: collect multi-frequency dynamic data).

[0028] Step 3, System model construction Through the coordination of architecture diagram model and business process state machine, the preprocessed data is converted into a computable model, providing a structured system abstraction for test scenario arrangement.

[0029] a. Architecture diagram model construction Define the attributes of nodes and edges: component node attributes include component_id (unique identifier), technology_stack (technology stack such as Spring Boot, Node.js), and resource_requirements (resource requirements such as minimum CPU core number); call edge attributes include call_type (call type such as synchronous, asynchronous), data_size (data transfer volume), and dependency_level (dependency level such as strong dependency, weak dependency); Build a directed graph based on NetworkX: Through the NetworkX library in Python, first create an empty graph (DiGraph()), then add nodes (add_node(component_id, attributes)) and edges (add_edge(source, target, attributes)) in sequence, for example, add an edge ("User Service", "Authentication Service", call_type="synchronous", dependency_level="strong"), and finally form a visual and computable architecture topology model.

[0030] b. Business process state machine construction According to the BPMN2.0 standard, use parsing tools (such as CamundaModelAPI) to extract activity nodes (such as "Submit Order") and gateways (such as "Payment Success / Failure Judgment") from business process diagrams, as well as sequential flows (such as "From Submit Order to Payment Verification"); Build a state transition matrix based on the extracted elements: the matrix rows represent the current state, the list represents the next state, and the value represents the transition probability (based on the statistical frequency of user operation logs), for example, the transition probability from "Submit Order" to "Payment Verification" is 90%, and to "Order Cancellation" is 10%. State transition sequence generation: According to the state transition matrix, combined with the user operation sequence (such as the click stream collected in step 1), the state transition path conforming to the business logic (such as "initial state → submit order → payment verification → order completion") is generated, which provides the basis for business path coverage of the test scenario.

[0031] Step 4, large model training and scenario arrangement module: First, large model training includes three core steps of architecture design, data construction and training optimization: a. Architecture design: Use graph attention network (GAT) to build architecture graph encoder, set node embedding dimension to 512, configure 8 multi-head attention mechanism, and use Transformer encoder to process business process event sequence, capture time sequence features through Sinusoidal position encoding and 768-dimensional hidden layer. The test sequence generator outputs the test scenario execution order based on the autoregressive model, and uses the mask mechanism to avoid circular dependency; the concurrency strategy generator outputs the resource scheduling vector, including CPU core number, thread number, timeout threshold and other key parameters.

[0032] b. Training data construction: Select efficient test schemes (execution time ≤ expected 80% and coverage rate ≥ 90%) from historical manual arrangement as positive samples, and construct invalid schemes containing circular dependency or resource conflict as negative samples. 15% of event nodes in the business process sequence are randomly masked to simulate incomplete log scenarios; 5% of the dependency edges of the architecture graph are randomly added, deleted and disturbed to improve the robustness of the model.

[0033] Training process optimization: Composite loss function: Use multi-objective weighted loss function L=Lorder+Lconcurrency+γLcoverage, where Lorder is the cross-entropy loss of the sequence generation based on Seq2Seq, Lconcurrency measures the resource scheduling accuracy through mean square error, and Lcoverage imposes a penalty term on the scheme that does not cover the key components.

[0034] The intelligent arrangement execution process covers three steps of dependency analysis, concurrency strategy generation and test case generation: a. Dependency relationship analysis. Generate test dependency graph based on architecture graph, and perform topological sorting through Kahn algorithm to ensure that components with dependency relationship are tested in sequence. For example, when "payment service" depends on "user authentication service", the system automatically executes the user authentication related test scenario in advance to avoid test failure due to unmet dependencies.

[0035] b. Concurrent strategy generation. Ensure that the CPU and memory requirements of each test scenario do not exceed the total resource limits of the environment by the formula ∑i=1n (cpui≤CPUtotal∧memoryi≤MEMtotal).

[0036] Assign the highest concurrent priority to safety-critical scenarios such as financial transaction processes, and use a multi-level feedback queue scheduling algorithm to achieve reasonable resource allocation, ensuring the execution efficiency and stability of high-priority scenarios.

[0037] c. Test case generation: Automatically generate interface test cases based on the OpenAPI specification, and use the Faker library to generate boundary value test data (such as long strings, negative amounts), covering normal and abnormal input scenarios. For each state transition node in the business process state machine, generate abnormal scenario test cases (such as network interruption, insufficient permissions) to comprehensively verify the fault tolerance of the system under fault conditions.

[0038] Step 5, test execution and dynamic adjustment module: First, build an intelligent execution engine. The execution environment and distributed framework building includes two parts: execution environment configuration and distributed execution framework deployment. In the execution environment configuration, deploy test tasks through Docker containerization, and use Kubernetes to realize dynamic scheduling and management of resources; at the same time, configure Prometheus+Grafana monitoring cluster to collect two types of key indicators in real time: one is the test task indicator, covering execution progress, pass rate, average response time, used to evaluate the execution effect of test task; the second is the system resource indicator, including CPU usage rate, memory usage rate, network throughput according to core statistics, to monitor the resource load of test environment. In terms of distributed execution framework, use Celery distributed task queue to support asynchronous distribution and result collection of test tasks, improve the parallelism and efficiency of task processing; at the same time, realize the breakpoint resume function of test cases, automatically record database transaction ID, interface session cookie and other context states when execution is interrupted, ensure that the task can continue from the breakpoint when it is restored, avoid repeated operation and resource waste. Then real-time feedback processing flow. Data collection and dynamic adjustment decision-making includes data processing and model building two major links. In data collection and feature engineering, real-time features such as scene ID, current step, P99 response time, error rate, CPU / memory usage, and dependency state are collected, and real-time data is processed in a 5-minute sliding window to calculate statistical features such as mean and standard deviation to extract time series patterns. The real-time collected feature vector includes: { "scene_id": str, "current_step": int, "response_time_p99": float, "error_rate": float, "cpu_usage": float, "memory_usage": float, "dependency_status": ["ready", "pending", "failed"] } The dynamic adjustment decision model part adopts the reinforcement learning PPO algorithm, takes the real-time feature vector as the state space, and the action space includes adjusting the number of concurrent threads by ± 10% steps, triggering the dependency graph to be reanalyzed to reorder the test scenarios, and simulating the injection of 200 ms network delay and other fault scenarios. The reward function is designed in a multi-objective weighted form, R = 0.6Rcoverage+0.3Refficiency−0.1Rresource, wherein Rcoverage provides a linear reward based on the number of newly added coverage components, Refficiency gives an exponential reward according to the proportion of execution time reduction, and Rresource imposes a penalty term on the CPU overload time. Through continuous learning, the dynamic optimization of the test strategy is realized.

[0039] Finally, the adaptive strategy is combined. The abnormal scene processing and resource dynamic allocation include two mechanisms of abnormal response and resource scheduling. In abnormal scene processing, when "component unavailable" errors occur in 3 consecutive test cases, the system first checks whether the pre-test of the component in the dependency graph is completed, and if not, it automatically triggers the pre-test to be executed. Then, a component health check test scenario is inserted to verify whether the component state has returned to normal. At the same time, the concurrency of the subsequent test scenarios that depend on the component is reduced to avoid more test failures caused by component failure. In terms of resource dynamic allocation, when the CPU utilization exceeds 85% and lasts for 10 minutes, the system starts the resource rebalancing strategy: the CPU-intensive performance stress test and other test scenarios are migrated to idle nodes to balance the cluster load; for test scenarios with high memory occupancy, container resource limits are enabled by setting CPU quota, memory limit and other parameters to prevent excessive resource competition from affecting test stability.

[0040] Although embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent orchestration and dynamic execution of automated test scenarios for complex systems driven by large models, characterized in that, Includes the following steps: Step 1: Multi-source data collection: Collect static architecture data, dynamic runtime data, and business process data. Static architecture data includes microservice interface definitions, UML component diagram dependencies, and Docker Compose resource configurations; dynamic runtime data includes distributed call chain information, component resource metrics, and Kubernetes event logs; business process data includes user operation logs and transaction logs. Step 2, Data Preprocessing: Clean and merge the collected heterogeneous raw data, filter out abnormal data using the IQR method, fill missing values ​​using the XGBoost regression model, associate business operations and component calls with transaction IDs, and use linear interpolation to achieve time alignment of multi-frequency data. Step 3, System Model Construction: Based on the preprocessed data, construct an architecture diagram model and a business process state machine. The architecture diagram model is constructed using NetworkX to create a directed graph, clarifying the attributes of component nodes and call edges. The business process state machine extracts process elements according to the BPMN 2.0 standard and generates a state transition matrix and flow path; Step 4, Large Model Training and Scene Orchestration: A dual encoder architecture is constructed using graph attention network and Transformer. The model is trained using historical efficient test solutions as positive samples and ineffective solutions as negative samples. An executable test plan is output through dependency resolution, concurrent strategy generation, and test case generation. Step 5, Test Execution and Dynamic Adjustment: The execution environment is built based on Docker+Kubernetes. Distributed task scheduling is implemented through Celery. The reinforcement learning PPO algorithm is adopted. Combined with real-time collected test indicators and system resource data, the number of concurrent threads, test order and resource allocation are dynamically adjusted to handle abnormal scenarios.

2. The method for intelligent orchestration and dynamic execution of automated test scenarios for complex systems driven by large models, as described in claim 1, is characterized in that... The microservice interface definitions described in step 1 are automatically extracted using the Swagger Parser tool, the UML component diagram dependencies are extracted using the Eclipse UML2 parser API, and the Docker Compose files are read using the PyYAML library.

3. The method for intelligent orchestration and dynamic execution of automated test scenarios for complex systems driven by large models, as described in claim 1, is characterized in that... In step 1, the distributed call chain information is collected through the Jaeger / Zipkin client SDK, the component resource metrics are collected through the Prometheus Exporter and stored in the time series database, and the Kubernetes event logs are extracted through the client-go API.

4. The method for intelligent orchestration and dynamic execution of automated test scenarios for complex systems driven by large models, as described in claim 1, is characterized in that... In step 1, user operation logs are obtained through JavaScript instrumentation on the web frontend and Appium test log parsing; transaction logs include database audit logs and message middleware logs, which are collected and parsed through the Debezium tool and the Kafka Connect client, respectively.

5. The method for intelligent orchestration and dynamic execution of automated test scenarios for complex systems driven by large models, as described in claim 1, is characterized in that... In step 2, data cleaning converts interface parameters into a unified JSON standard format and serializes the data using Apache Avro; data fusion establishes a multi-dimensional mapping relationship of "user operation - component call - data layer interaction".

6. The method for intelligent orchestration and dynamic execution of automated test scenarios for complex systems driven by large models, as described in claim 1, is characterized in that... The component node attributes of the architecture graph model described in step 3 include component_id, technology_stack, and resource_requirements, and the call edge attributes include call_type, data_size, and dependency_level; the state transition matrix of the business process state machine is based on the transition probability statistics of user operation logs.

7. The method for intelligent orchestration and dynamic execution of automated test scenarios for complex systems driven by large models, as described in claim 1, is characterized in that... In step 4, the composite loss function of the large model is L = Lorder + Lconcurrency + γLcoverage, where Lorder is the sequential generation cross-entropy loss, Lconcurrency is the resource scheduling mean square error loss, and Lcoverage is the coverage penalty loss.

8. The method for intelligent orchestration and dynamic execution of automated test scenarios for complex systems driven by large models, as described in claim 1, is characterized in that... The feature vectors collected in real time in step 5 include scene_id, current_step, response_time_p99, error_rate, cpu_usage, memory_usage, and dependency_status. A 5-minute sliding window is used to extract the time sequence pattern.

9. The method for intelligent orchestration and dynamic execution of automated test scenarios for complex systems driven by large models, as described in claim 1, is characterized in that... In step 5, the reward function for reinforcement learning is R = 0.6Rcoverage + 0.3Refficiency − 0.1Rresource, where Rcoverage is the coverage reward, Refficiency is the efficiency reward, and Rresource is the resource penalty. In handling abnormal scenarios, if three consecutive test cases fail, the pre-tests are re-executed and the component health check is triggered.

Images