Reproduction method, system and related equipment for production environment incidental bug

Abstract

This invention provides a method, system, and related equipment for reproducing intermittent bugs in a production environment. The method first captures fault context data related to the intermittent bug in the production environment. Then, it uses a multimodal spatiotemporal graph neural model to analyze the fault context data and infer the root cause of the fault. Next, in an isolated test environment, based on the fault context and root cause, a generative artificial intelligence model automatically aligns the environment configuration and data states to create a reproduction environment. Then, a reinforcement learning agent actively intervenes in the non-deterministic behavior of the system to explore the deterministic execution path that triggers the intermittent bug. Finally, based on the deterministic execution path and the reproduction environment, an executable reproduction test script is automatically generated. This invention, by integrating artificial intelligence models and reinforcement learning agents, transforms the elusive intermittent faults in the production environment into deterministic and reproducible problems, shortening fault resolution time and improving the reliability of the software system.

Description

Technical Field

[0001] This invention relates to the field of software engineering, and in particular to a method, system, and related equipment for reproducing occasional bugs in a production environment. Background Technology

[0002] With the rapid evolution of modern software engineering and cloud computing technologies, software system architecture has shifted from traditional monolithic applications to microservices, serverless computing, and highly distributed cloud-native topologies. While this architectural upgrade has significantly improved system scalability and development agility, it has also led to an exponential increase in the complexity of system state space and component interactions, fundamentally changing the manifestation of software defects (bugs) and posing serious challenges to software quality assurance (QA) and debugging.

[0003] In traditional deterministic software systems, most defects are "Bohr bugs," which are highly deterministic and stable, reproducible under the same initial state and input parameters, and exhibit predictable behavior. However, in today's distributed, high-concurrency production environments, most critical failures are caused by "Heisenbugs" or "Mandelbugs." The core characteristic of Heisenbugs is nondeterminism; their triggering depends on extremely stringent and difficult-to-detect hidden conditions, such as specific multi-threaded scheduling interleaving, microsecond-level network latency jitter, garbage collection (GC) pauses, and unexpected states of underlying hardware.

[0004] The most challenging issue in debugging Heisenberg defects is the "observer effect" or "probe effect": when developers use conventional methods such as inserting logs, attaching debuggers, and reducing compiler optimization levels, they inevitably alter the program's execution timing, memory address layout, or register allocation strategy. For example, floating-point numbers stored in high-precision registers may be flushed to main memory after a debugger is connected, thus changing subsequent floating-point comparison results. This timing and state deviation caused by troubleshooting methods can significantly reduce or even eliminate the probability of triggering defects such as race conditions and deadlocks, misleading developers into believing that the defects have been fixed and severely hindering root cause analysis. Studies show that in modern production environments, 65% of critical software system crashes can be traced back to such sporadic defects that are difficult to reproduce reliably in test environments.

[0005] In current industry practice, if defects cannot be accurately reproduced in the development or testing environment, developers can only rely on discrete logs and incomplete stack traces to guess the system state. This blind repair is not only time-consuming but also prone to introducing new regression errors. In the IT operations core metric Mean Time To Resolve (MTTR), more than 70% of the cycle is consumed in the "diagnosis and attempt to reproduce" phase.

[0006] To address the aforementioned challenges, existing technical solutions still suffer from the following insurmountable bottlenecks and shortcomings: First, the "evenness or oddness" of production and testing environments is difficult to align precisely. Traditional fault reproduction relies on manually building test sandboxes, but the production environment is a complex, dynamically evolving ecosystem containing implicit environment variables, specific database index states, undocumented differences in operating system kernel patches, and third-party API rate limiting policies. When the test environment configuration or data distribution deviates from the production baseline, the triggering chain of sporadic defects will be completely broken, creating an ineffective troubleshooting loop of "it runs fine on my machine." Furthermore, due to data privacy and compliance requirements such as GDPR and HIPAA, real production data cannot be directly imported into the test environment, and traditional data anonymization or fake data generation methods cannot preserve the entity referential integrity and extreme long-tail distribution of business data, further reducing the success rate of reproduction.

[0007] Second, existing deterministic replay technologies face performance and applicability barriers in distributed production environments. Although tools such as the rr debugger and the Pernosco platform can achieve nanosecond-level time travel debugging by intercepting system calls, signals, and context switches, this incurs a performance overhead of 1.2 to 5 times or even higher on the host machine, making it unsuitable for global deployment in high-throughput, low-latency-sensitive microservice production environments. Furthermore, the lack of application-layer semantic context in underlying system call tracing renders single-process replay technology completely ineffective when facing cross-microservice failures.

[0008] Third, traditional monitoring and rule-based root cause analysis (RCA) lack deep causal reasoning capabilities. With the development of observability technologies, systems generate massive amounts of logs, metrics, and distributed tracing data. However, existing systems rely on static threshold alerts or shallow machine learning anomaly detection, which easily triggers "alert storms" and exhibits poor stability when fusing cross-modal data. These tools only operate at the level of statistical correlation and cannot construct causal dependency graphs for microservice calls, resulting in root cause localization often focusing on downstream symptoms of the fault propagation chain rather than the true triggering source.

[0009] Fourth, the quality of automated defect reproduction test scripts (BRTs) is low, and dynamic interactive feedback is lacking. After a defect report is generated, the testing team needs to write regression test cases. Existing static generation methods based on Large Language Models (LLMs) have extremely low usability. For example, the industry-leading LIBRO method has a reasonable BRT generation rate of only about 10% in real industrial-grade codebases. The core reason is that LLMs are limited by context windows when dealing with extremely large codebases, making it impossible to accurately capture the global program dependency graph (PDG). Furthermore, the code generated in a single instance is prone to syntax illusions or missing dependencies, lacks a closed-loop feedback loop of "generation-verification-correction," and cannot be run directly.

[0010] In summary, existing technologies face a series of interconnected challenges when dealing with occasional bugs in production environments, such as incomplete state capture, inaccurate root cause localization, inconsistent environment reproduction, reliance on luck for defect triggering, and low quality of test script generation. These challenges result in persistently high average fault resolution times. Therefore, there is an urgent need in this field for an innovative method that can systematically and automatically solve the above problems. Summary of the Invention

[0011] This invention provides a method, system, and related equipment for reproducing occasional bugs in a production environment. The main purpose of this invention is to solve the technical problem that occasional bugs in a production environment are difficult to reproduce in the prior art.

[0012] The first aspect of this invention provides a method for reproducing intermittent bugs in a production environment, comprising: In a production environment, when a pre-defined abnormal event is detected, fault context data related to intermittent bugs is captured. Based on a multimodal spatiotemporal neural network model, the fault context data is analyzed to infer the root cause of the occasional bug. In an isolated test environment, based on the fault context data and the fault root cause, a generative artificial intelligence model is used to automatically align the environment configuration and data status, creating a reproduction environment consistent with the production environment baseline. In the reproducible environment, by actively intervening in the nondeterministic behavior of the system through reinforcement learning agents, a deterministic execution path that stably triggers the sporadic bug is explored and determined. Based on the deterministic execution path and the reproduction environment, a reproduction test script is automatically generated to stably reproduce the occasional bug.

[0013] In an optional embodiment of the first aspect of the present invention, the step of capturing fault context data related to intermittent bugs when a preset abnormal event is detected in a production environment includes: By using probes deployed in the operating system kernel and application-layer proxies deployed in the microservice mesh, the system state within the fault time window is intercepted in a bypass manner, and a structured, portable replay data packet is generated. The portable replay data package includes the following logical layers: inventory and tracing layer, construction and sealing context, input and file system layer, environment and configuration snapshot, network transaction transcription, time and random number control, concurrency and scheduling metadata, end-to-end telemetry data layer, and isolated replay execution engine.

[0014] In an optional embodiment of the first aspect of the present invention, the step of analyzing the fault context data based on a multimodal spatiotemporal neural network model to infer the root cause of the intermittent bug includes: The fault context data is preprocessed using a multimodal spatiotemporal neural network model through feature engineering. Based on the feature vector data containing logs and time-series performance metrics obtained from the preprocessing, a dynamic service dependency graph is constructed with microservice instances as nodes and inter-service calls as edges. By fusing diffusion graph convolution and causal inference algorithms on the dynamic service dependency graph, a targeted fault subgraph is output to locate the source of the fault.

[0015] In an optional embodiment of the first aspect of the invention, the automatic alignment of environment configuration and data state through a generative artificial intelligence model includes: Using a first generative artificial intelligence model, the environmental snapshot in the fault context data is parsed to automatically generate or adjust the infrastructure code manifest to align with the underlying infrastructure configuration; By utilizing a second generative artificial intelligence model based on generative adversarial networks or variational autoencoders, the joint distribution characteristics of production data are learned to synthesize test data in the test environment that maintains entity referential integrity and business logic constraints and does not contain personally identifiable information.

[0016] In an optional embodiment of the first aspect of the present invention, the step of exploring and determining a deterministic execution path that stably triggers the sporadic bug by actively intervening in the nondeterministic behavior of the system through a reinforcement learning agent in the reproduction environment includes: The system's thread scheduling and network event timing control problems are modeled as Markov decision processes; Define the state space, action space, and reward function of the reinforcement learning agent. The state space includes thread instruction pointers, lock states, and distances to the root cause of the fault. The action space includes blocking or waking up specific threads, injecting delays between network requests, and changing the context switching strategy. The reward function is set based on whether the program execution path is close to the target exception stack. Based on the definition, the reinforcement learning agent is driven to explore and obtain a sequence of scheduled actions that can reliably trigger the occasional bug as the deterministic execution path.

[0017] In an optional embodiment of the first aspect of the present invention, the automatic generation of the reproduction test script for stably reproducing the intermittent bug is achieved through a closed-loop multi-functional intelligent agent collaborative process including generation, verification, and correction, wherein the closed-loop multi-functional intelligent agent collaborative process includes: The agent is retrieved by context slicing, and the minimal relevant code context is extracted from the source code library based on the program dependency graph; An intelligent agent is generated from the test script, and candidate reproducible test scripts are generated based on the relevant code context and the deterministic execution path; The closed-loop verification agent executes the candidate reproducibility test script in the sandbox and detects whether the target bug can be triggered. The reflection and correction agent iteratively corrects the candidate reproducibility test script based on the feedback information of verification failure until the verification is successful.

[0018] In an optional embodiment of the first aspect of the present invention, when the reinforcement learning agent is exploring, a dynamic partial order reduction mechanism is introduced. By identifying and removing equivalent scheduling sequences whose execution order does not affect the final state during the scheduling exploration process, the exploration efficiency of the reinforcement learning agent in discovering the target BUG trigger path is improved.

[0019] A second aspect of the present invention provides a system for reproducing occasional bugs in a production environment, the system comprising: The fault context acquisition module is used to capture fault context data related to intermittent bugs in the production environment when a preset abnormal event is detected. The fault root cause analysis module is used to analyze the fault context data based on a multimodal spatiotemporal neural network model and infer the fault root cause of the occasional bug. The reproduction environment creation module is used to automatically align the environment configuration and data status with the production environment baseline in an isolated test environment based on the fault context data and the fault root cause through a generative artificial intelligence model. The BUG path exploration module is used to explore and determine the deterministic execution path that stably triggers the occasional BUG in the reproduced environment by actively intervening in the nondeterministic behavior of the system through reinforcement learning agents. The test script generation module is used to automatically generate a test script for stably reproducing the intermittent bug based on the deterministic execution path and the reproduction environment.

[0020] A third aspect of the present invention provides a device for reproducing occasional bugs in a production environment, the device comprising: a memory and at least one processor, wherein the memory stores instructions and the memory and the at least one processor are interconnected via a circuit; The at least one processor invokes the instructions in the memory to cause the production environment intermittent bug reproduction device to execute the production environment intermittent bug reproduction method as described in any one of the first aspects of the present invention.

[0021] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a method for reproducing occasional bugs in a production environment as described in any one of the first aspects of the present invention.

[0022] Beneficial Effects: This invention provides a method, system, and related equipment for reproducing intermittent bugs in a production environment. The method first captures fault context data related to the intermittent bug in the production environment. Then, it uses a multimodal spatiotemporal graph neural model to analyze the fault context data and infer the root cause of the fault. Next, in an isolated test environment, based on the fault context and root cause, a generative artificial intelligence model automatically aligns the environment configuration and data states to create a reproduction environment. Then, a reinforcement learning agent actively intervenes in the non-deterministic behavior of the system to explore the deterministic execution path that triggers the intermittent bug. Finally, based on the deterministic execution path and the reproduction environment, an executable reproduction test script is automatically generated. This invention, by integrating artificial intelligence models and reinforcement learning agents, transforms the elusive intermittent faults in the production environment into deterministic and reproducible problems, shortening fault resolution time and improving the reliability of the software system. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of an embodiment of a method for reproducing occasional bugs in a production environment according to the present invention; Figure 2 This is a schematic diagram of an embodiment of a system for reproducing occasional bugs in a production environment according to the present invention; Figure 3 This is a schematic diagram of an embodiment of a device for reproducing occasional bugs in a production environment according to the present invention. Detailed Implementation

[0024] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] See Figure 1 The first aspect of this invention provides a method for reproducing occasional bugs in a production environment, comprising: S100. In a production environment, when a preset abnormal event is detected, fault context data related to the intermittent bug is captured. In this invention, this step may include: bypassing the system state within the fault time window through a probe deployed in the operating system kernel and an application layer agent deployed in the microservice mesh, and generating a structured portable replay data packet; wherein, the portable replay data packet includes the following logical layers: inventory and tracing layer, construction and sealing context, input and file system layer, environment and configuration snapshot, network transaction transcription, time and random number control, concurrency and scheduling metadata, end-to-end telemetry data layer, and isolated replay execution engine.

[0026] Specifically, in order to accurately reproduce the fault in the test environment, it is necessary to fully obtain the boundary conditions and non-deterministic factors at the time of the fault occurrence, while controlling the performance overhead of the production environment. This invention abandons the traditional global memory dump (Core Dump) or full instruction tracing method and constructs a structured "Portable Replay Bundle" to build fault context data.

[0027] This invention achieves non-intrusive bypass interception through operating system kernel eBPF (Extended Berkeley Packet Filter) probes and sidecar proxies mounted on microservice meshes (such as Istio / Envoy). When the upstream observability platform detects abnormal indicators (such as a surge in HTTP 5xx errors, uncaptured Panic anomalies, or long-tail latency mutations), the capture module is triggered, persisting the pre-configured ring buffer data and constructing a high-fidelity replay packet containing nine logical layers, as shown in Table 1 below: Table 1. Logical layering table of the portable playback data packets of this invention

[0028] The layered capture mechanism described above avoids irrelevant memory dumps, minimizing storage and transmission overhead, and constructs physically isolated "information bubbles" to ensure the pure deterministic nature of the playback process and prevent external nondeterministic entropy sources from leaking into the debugging process.

[0029] S200. Based on a multimodal spatiotemporal neural network model, analyze the fault context data to infer the root cause of the intermittent bug. In this invention, this step may include: performing feature engineering preprocessing on the fault context data using a multimodal spatiotemporal neural network model; constructing a dynamic service dependency graph with microservice instances as nodes and inter-service calls as edges based on the feature vector data containing logs and time-series performance indicators obtained from the preprocessing; and fusing diffusion graph convolution and causal inference algorithms on the dynamic service dependency graph to output a targeted fault subgraph for locating the fault source.

[0030] Specifically, in complex microservice architectures, edge node failures can trigger cascading crashes, resulting in hundreds of cross-service alerts. To narrow down the scope of system reproduction, this system integrates a deep learning root cause analysis engine, which merges captured heterogeneous multimodal data to infer the true source of the failure.

[0031] In this invention, firstly, step S200 performs feature engineering preprocessing on the data: considering that production logs are mostly unstructured text, a fine-tuned pre-trained language model log parser such as Mistral or Qwen is used to automatically cluster the massive logs and extract them into standardized log templates and parameter sequences; then, the log templates and events are transformed into high-dimensional dense semantic embeddings through a Transformer architecture (such as BERT); at the same time, outlier detection and normalization are performed on the collected time series performance indicators (such as CPU utilization and network throughput).

[0032] After feature extraction is completed, a dynamic service dependency graph is constructed, with nodes corresponding to microservice instances and directed edges corresponding to inter-service network RPC or API calls. Considering that fault states both spread in the topology space and accumulate in the time axis, this invention adopts a spatiotemporal graph model that integrates diffusion convolution operations and recurrent neural network architecture (such as diffusion convolution recurrent neural network DCRNN).

[0033] In the DCRNN network, the diffusion graph convolutional layer is responsible for capturing the spatial topological dependency propagation of abnormal indicators and error logs between different microservice nodes along the directed call edges of services; the gated recurrent unit (GRU) is responsible for feature aggregation of the historical time steps of each node to capture the time dependency of fault incubation. In order to improve the accuracy of root cause diagnosis and eliminate false positives caused by statistical correlation, the system superimposes the Peter-Clark causal inference algorithm on the GNN layer. By performing strict conditional independence tests on the latent variables of nodes, pruning the pseudo-causal edges caused by common pre-driving, and retaining the core directional fault causal links.

[0034] Ultimately, the root cause inference module outputs a "targeted fault subgraph," which accurately locates the microservice that caused the anomaly, the specific method-level stack, and the key time window that triggered the anomaly. This not only significantly filters out irrelevant noise alarms but also defines the minimum code slice range for subsequent reproduction script generation.

[0035] S300. In an isolated test environment, based on the fault context data and the fault root cause, a generative artificial intelligence model is used to automatically align the environment configuration and data state, creating a reproduction environment consistent with the production environment baseline. In this invention, this step may include: using a first generative artificial intelligence model to parse the environment snapshot in the fault context data, automatically generating or adjusting the infrastructure i.e., code manifest to align the underlying infrastructure configuration; and using a second generative artificial intelligence model based on a generative adversarial network or variational autoencoder to learn the joint distribution characteristics of the production data, in order to synthesize test data in the test environment that maintains entity referential integrity and business logic constraints and does not contain personally identifiable information.

[0036] Specifically, after obtaining the replay data packet and root cause range, a reproduction environment needs to be prepared in the test cluster. If the underlying configuration and data distribution of the test environment cannot be strictly consistent with the production environment (environment parity alignment), occasional bugs are difficult to trigger again. This invention completely revolutionizes the environment preparation process through generative AI.

[0037] Intelligent Infrastructure as Code (IaC) Alignment: The environment alignment agent parses the build context and environment snapshot in the replay data packet and autonomously compares it with the current baseline of the test cluster. If environmental differences are found, LLM automatically writes or adjusts the Terraform manifest file and Kubernetes deployment specifications. For example, it can precisely specify specific versions of middleware (such as Redis and PostgreSQL images), set consistent memory quota limits, and even generate equivalent virtual network topology configurations to automatically clone miniature production system slices in the test isolation area.

[0038] Production-grade multimodal synthetic data generation: In some business scenarios, defects are only exposed when there are specific historical dirty data or extreme long-tail data combinations in the database (such as complex transaction order management modules). Directly copying production data not only results in a large amount of data but also violates privacy regulations. The system of this invention also integrates a test data synthesis engine based on GAN and variational autoencoder (VAE). Within the security boundary, it performs joint distributed learning on the production database, not only learning the data format of individual fields but also gaining a deeper understanding of referential integrity and business logic constraints across tables and entities. The generator can quickly generate tens of thousands of synthetic records in the test database. Their statistical attributes, correlations, and boundary conditions are highly consistent with real production snapshots and do not contain any traceable personally identifiable information, providing an ideal sandbox base for exploring the edge paths of business logic.

[0039] S400. In the reproduced environment, by actively intervening in the nondeterministic behavior of the system through a reinforcement learning agent, a deterministic execution path that stably triggers the occasional bug is explored and determined. In this invention, this step may include: modeling the system's thread scheduling and network event timing control problem as a Markov decision process; defining the state space, action space, and reward function of the reinforcement learning agent, wherein the state space includes thread instruction pointers, lock states, and distances to the root cause of the fault; the action space includes blocking or waking up specific threads, injecting delays between network requests, and changing the context switching strategy; and the reward function is set based on whether the program execution path is close to the target exception stack. Through the definition, the reinforcement learning agent is driven to explore and obtain a scheduling timing action sequence that can stably trigger the occasional bug as the deterministic execution path.

[0040] Specifically, traditional methods for reproducing the Heisenberg defect rely on repeated execution and random delayed injection, which are extremely inefficient. This invention entrusts the operating system thread scheduling and the timing of network events between microservices to the reinforcement learning (RL) agent to actively seek optimization control, transforming the search for competitive conditions into a systematic mathematical optimization problem.

[0041] This invention models nondeterministic state scheduling as a Markov decision process (MDP), and the relevant definitions in reinforcement learning agents are as follows: State Space: The system state vector is composed of multi-dimensional features of the test environment, including the current instruction pointer (PC) of each active thread, the distribution of nodes covered by the abstract syntax tree (AST), the request and holding matrix of mutexes and semaphores, the depth of the network packet sending and receiving buffers, and the distance heuristic metric for the distance to the anomaly stack provided by the root cause inference module.

[0042] Action Space: The actions that an agent can take at each discrete time step are strong interventions in the system scheduling, including blocking or waking up specific threads, injecting microsecond-level artificial delays between microservice RPC requests, dynamically changing the context switching time slice length, and reversing the execution order of asynchronous event callback functions.

[0043] Reward function: A hierarchical design is adopted to guide the agent to find vulnerabilities. If the scheduling makes the program execution path close to the characteristics of the exception stack, or detects an illegal variable state consistent with the production environment (such as the precursor to null pointer dereferencing), a positive dense reward is given; if the program causes the expected target exception (such as accurately reproducing deadlock or segmentation fault), an extremely high final reward is given; if the program terminates normally or gets stuck in an irrelevant infinite loop, a negative penalty is imposed.

[0044] Furthermore, regarding agent model training and inference, considering the continuity and high-dimensionality of the action space, the Deep Deterministic Policy Gradient (DDPG) algorithm can be adopted. To address the exponential explosion problem of the concurrent program state space, a Dynamic Partial Order Reduction (DPOR) mechanism is introduced. Through static and dynamic code analysis, "pure operations" with no side effects on the global memory state and independent thread events are identified. Pruned equivalent scheduling sequences are merged, allowing the reinforcement learning (PL) agent to skip redundant and ineffective explorations. An "eager race state reversal" strategy is employed to directly target shared memory access conflict points. After multiple iterations, when the RL agent discovers a scheduling trajectory that can trigger a target anomaly, the system persists this scheduling action sequence as a "deterministic script," transforming low-probability, sporadic Heisenberg defects into reproducible defects that can be 100% stably triggered by replaying the script in the test environment. (That is, during the reinforcement learning agent's exploration, a dynamic partial order reduction mechanism is introduced to identify and prune equivalent scheduling sequences whose execution order does not affect the final state during the scheduling exploration process, thereby improving the exploration efficiency of the reinforcement learning agent in discovering the target bug trigger path.)

[0045] S500. Based on the deterministic execution path and the reproduction environment, a reproduction test script for stably reproducing the intermittent bug is automatically generated. In this invention, this step can be implemented through a closed-loop multi-functional intelligent agent collaborative process that includes generation, verification, and correction. The closed-loop multi-functional intelligent agent collaborative process includes: retrieving an intelligent agent from the context slice and extracting the minimal relevant code context from the source code library based on the program dependency graph; generating a test script generation intelligent agent and generating candidate reproduction test scripts based on the relevant code context and the deterministic execution path; executing the candidate reproduction test scripts in a sandbox and detecting whether the target bug can be triggered by a closed-loop verification intelligent agent; and iteratively correcting the candidate reproduction test scripts based on the feedback information of verification failures until verification is successful by a reflection and correction intelligent agent.

[0046] Specifically, after the underlying environment data and trigger timing are determined, the complex underlying state needs to be converted into an automated reproducible test script (BRT) that is readable by developers and can be seamlessly integrated into existing regression test suites. When a single LLM generates such long-context, highly rigorous business logic code, it is prone to "illusions". To address this, this invention designs a multi-functional LLM agent collaborative pipeline, adopting a "generate-verify-correct" closed-loop mechanism. The collaborative process of the multi-functional agent in this invention is shown in Table 2 below: Table 2. Collaboration Mechanism of the Multifunctional Intelligent Agent in this Invention

[0047] This invention, through continuous collaboration among multiple agents, ultimately outputs a highly reliable BRT automated test script, along with a detailed system state transition report and vulnerability timing analysis diagram. This script can be automatically submitted (Pull Request) to the code repository for developer review or integrated with an Automated Program Remediation (APR) system for vulnerability patching. The innovative solution for reproducing occasional bugs in production environments proposed in this invention deeply integrates multimodal large language models (LLMs), spatiotemporal graph neural networks (STGNN), generative adversarial networks (GAN), and reinforcement learning (RL). This achieves an end-to-end closed loop from production fault capture, causal root cause diagnosis, environmental data alignment, concurrent scheduling and control to automated reproduction script generation, significantly shortening fault resolution time and improving software system robustness.

[0048] More specifically, in comparing this invention with existing technologies, it comprehensively examines and transcends the traditional technological paradigms in the fields of software debugging, automated testing, and AI observability. The specific comparison is as follows: Firstly, in the field of fault state capture and system replay: although traditional low-level tools (such as rr) can achieve process-level time travel debugging by recording system calls, they have high performance overhead and cannot adapt to large-scale distributed microservice topologies, which limits their application in production environments. The portable replay data packet design of this invention, through business boundary proxy and eBPF bypass observation, breaks through the single-process limitation while maintaining "bit-by-bit consistency" and achieves cross-service state encapsulation.

[0049] Secondly, in the direction of root cause analysis (RCA): traditional AIOps tools rely on log regular expression matching and simple time series regression, which have problems such as multimodal data fragmentation and easy causal inversion. This invention combines DCRNN and the Peter-Clark causal inference framework, and for the first time integrates microservice spatiotemporal evolution and topological dependence to eliminate false correlations in alarm storms from the root.

[0050] Finally, regarding the automatic generation of test scripts: early academic research relied solely on LLM zero-shot / few-shot inference to generate test code, lacking real-world semantic verification. The solution architecture of this invention integrates PDG dynamic backward slicing and multi-agent closed-loop pipeline, eliminating the logical illusion of single-model from a mechanism perspective, and enabling the generated reproducible test cases to meet enterprise-level CI / CD integration standards.

[0051] This invention proposes an AI-based bug testing environment reproduction solution for production environments, which overturns the traditional troubleshooting paradigm dominated by subjective human experience. Through automated holographic state capture and proactive intelligent state deduction, it significantly improves software quality assurance efficiency. It demonstrates a significant generational advantage in the verification of industrial-grade microservices, cloud computing infrastructure, and high-frequency trading systems. The core effects are as follows: 1. Breaking through the bottleneck of Heisenberg defect reproduction and achieving a reduction in the scope of nondeterminism to determinism: In traditional concurrent system defect detection, even with high-frequency random fuzz testing, the reproduction rate of extreme race condition vulnerabilities is only in the single digits. The RL scheduling exploration module of this invention, combined with DPOR redundant timing pruning, can directly target the intersection of abnormal threads. Comparative experiments show that in the verification set of distributed systems such as Hadoop and OpenStack, the overall trigger recall rate of this system in locating and reproducing extreme race condition vulnerabilities reaches more than 68.8%, and the accuracy of deterministic reproduction testing reaches 92%, which is an order of magnitude higher than the performance of traditional tools.

[0052] 2. Significantly reduced MTTR, optimizing core IT operations and maintenance efficiency: This invention eliminates tedious work such as manually collecting logs, guessing dependencies, and setting up test environments, significantly compressing MTTR. Through multimodal data and STGNN causal correlation, it can filter 50%-70% of noise and false alarms, focusing on source services. Industrial deployment cases show that core troubleshooting indicators are significantly improved, as detailed in Table 3 below: Table 3. Comparison of the effectiveness of the present invention and traditional solutions in core obstacle removal indicators.

[0053] 3. Generate high-quality automated test scripts to prevent quadratic regression errors: Early single LLM code generation solutions (such as LIBRO) had a reasonable BRT generation rate of only about 10%. This system effectively eliminates the illusion of LLM logic and syntactic defects through PDG context slicing retrieval and a multi-agent "generate-verify-correct" pipeline. In authoritative datasets such as SWE-bench and real enterprise defect ticket tests, the high-quality executable BRT generation rate reaches 28%~74% (depending on the depth of defect logic), which is more than 19.0% higher than the original solution. The generated scripts can be directly integrated into the CI / CD process to prevent regression errors.

[0054] 4. Low-overhead production environment tracking and absolute privacy compliance: The capture module based on eBPF and sidecar proxy has excellent lightweight characteristics. On high-concurrency servers or computing clusters such as Apache, the additional performance overhead is as low as less than 0.17%, which can run normally in critical production environments. At the same time, the synthetic data engine automatically generates structured test data, preventing PII from leaving the country from the source, and perfectly meeting the stringent privacy regulations such as GDPR.

[0055] In summary, the solution of this invention theoretically overcomes the challenges of system state observation and reconfiguration, and has broad application prospects in technology-intensive fields with extremely high requirements for software stability, business continuity, and troubleshooting timeliness. Large-scale cloud-native SaaS and e-commerce platforms: In highly dynamic distributed cloud platforms composed of thousands of containers and microservices, network latency jitter and service degradation are frequent sources of intermittent failures. This system can automatically launch an equivalent environment through IaC, accurately locate performance bottlenecks and abnormal configurations in multi-hop call chains, and ensure the smooth operation of high-concurrency services.

[0056] Intelligent connected vehicles and ADAS systems: The autonomous driving system adopts an edge-cloud collaborative architecture, involving the fusion of massive multimodal sensor data and low-latency nondeterministic algorithm control. This system can automatically identify concurrent scheduling blind spots on a simulation cloud platform through timing manipulation and RL, generating defensive regression test cases for long-tail edge scenarios.

[0057] Fintech and High-Frequency Quantitative Trading Systems: Financial trading infrastructure demands extremely high levels of transaction consistency and response speed; even minor concurrency issues can lead to significant financial losses. Furthermore, financial data is sensitive, and developers are prohibited from accessing real transaction records. This invention provides a highly realistic synthetic data module and deterministic replay mechanism, offering a compliant and high-fidelity troubleshooting and reconstruction simulation platform.

[0058] Healthcare platforms and public health information networks: System updates involving Electronic Health Records (EHRs) and Protected Health Information (PHIs) require extreme caution. This technology, while absolutely protecting patient privacy, utilizes AI to synthesize equally distributed medical test data, efficiently tracking and reproducing deadlocks and crashes caused by asynchronous workflows or high-concurrency queries, ensuring 24 / 7 high availability of the system.

[0059] See Figure 2 The second aspect of the present invention provides a system for reproducing occasional bugs in a production environment, the system comprising: The fault context acquisition module 10 is used to capture fault context data related to intermittent bugs in the production environment when a preset abnormal event is detected. The fault root cause analysis module 20 is used to analyze the fault context data based on a multimodal spatiotemporal neural network model and infer the fault root cause of the occasional bug. The reproduction environment creation module 30 is used to automatically align the environment configuration and data status with the production environment baseline in an isolated test environment based on the fault context data and the fault root cause through a generative artificial intelligence model. The BUG path exploration module 40 is used to explore and determine the deterministic execution path that stably triggers the occasional BUG in the reproduced environment by actively intervening in the nondeterministic behavior of the system through a reinforcement learning agent. The test script generation module 50 is used to automatically generate a test script for stably reproducing the occasional bug based on the deterministic execution path and the reproduction environment.

[0060] In an optional embodiment of the second aspect of the present invention, the fault context acquisition module includes: The portable replay data packet generation unit is used to bypass and intercept the system status within the fault time window through probes deployed in the operating system kernel and application layer proxies deployed in the microservice mesh, and generate structured portable replay data packets. The portable replay data package includes the following logical layers: inventory and tracing layer, construction and sealing context, input and file system layer, environment and configuration snapshot, network transaction transcription, time and random number control, concurrency and scheduling metadata, end-to-end telemetry data layer, and isolated replay execution engine.

[0061] In an optional embodiment of the second aspect of the present invention, the root cause analysis module includes: The data preprocessing unit is used to perform feature engineering preprocessing on the fault context data using a multimodal spatiotemporal neural network model; The spatiotemporal graph construction unit is used to construct a dynamic service dependency graph with microservice instances as nodes and inter-service calls as edges based on the feature vector data containing logs and time-series performance indicators obtained by the preprocessing. The causal inference unit is used to fuse diffusion graph convolution and causal inference algorithm on the dynamic service dependency graph to output a targeted fault subgraph for locating the source of the fault.

[0062] In an optional embodiment of the second aspect of the present invention, the reproduction environment creation module includes: An environment configuration unit is used to use a first generative artificial intelligence model to parse the environment snapshot in the fault context data and automatically generate or adjust the infrastructure i.e. code manifest to align with the underlying infrastructure configuration. The data state configuration unit is used to learn the joint distribution characteristics of production data using a second generative artificial intelligence model based on generative adversarial networks or variational autoencoders, in order to synthesize test data in the test environment that maintains entity referential integrity and business logic constraints and does not contain personally identifiable information.

[0063] In an optional embodiment of the second aspect of the present invention, the BUG path exploration module includes: The problem modeling unit is used to model the system's thread scheduling and network event timing control problems as Markov decision processes. The agent setting unit is used to define the state space, action space, and reward function of the reinforcement learning agent. The state space includes thread instruction pointers, lock states, and distances to the root cause of the fault. The action space includes blocking or waking up specific threads, injecting delays between network requests, and changing the context switching strategy. The reward function is set based on whether the program execution path is close to the target exception stack. An exploratory learning unit is used to drive the reinforcement learning agent to explore and obtain a sequence of scheduled actions that can reliably trigger the sporadic bug as the deterministic execution path, based on the definition.

[0064] In an optional embodiment of the second aspect of the present invention, the automatic generation of the reproduction test script for stably reproducing the intermittent bug is achieved through a closed-loop multi-functional intelligent agent collaborative process including generation, verification, and correction, wherein the closed-loop multi-functional intelligent agent collaborative process includes: The agent is retrieved by context slicing, and the minimal relevant code context is extracted from the source code library based on the program dependency graph; An intelligent agent is generated from the test script, and candidate reproducible test scripts are generated based on the relevant code context and the deterministic execution path; The closed-loop verification agent executes the candidate reproducibility test script in the sandbox and detects whether the target bug can be triggered. The reflection and correction agent iteratively corrects the candidate reproducibility test script based on the feedback information of verification failure until the verification is successful.

[0065] In an optional embodiment of the second aspect of the present invention, when the reinforcement learning agent is exploring, a dynamic partial order reduction mechanism is introduced. By identifying and removing equivalent scheduling sequences whose execution order does not affect the final state during the scheduling exploration process, the exploration efficiency of the reinforcement learning agent in discovering the target BUG trigger path is improved.

[0066] In summary, the overall architecture of the system portion of this invention can be divided into a data plane and a control plane, spanning both production and testing environments. It consists of five highly collaborative core functional modules, which communicate through standardized APIs and an event flow bus to ensure efficient data flow and strict privacy isolation.

[0067] The core functions are as follows: First, "lightweight and portable replay data packet capture" deployed at the edge of the production environment, which intercepts the instantaneous context of faults, external inputs, and non-deterministic entropy sources with low performance overhead through kernel-level probes and application-layer proxies, and generates high-fidelity encrypted data packets; Second, "multimodal spatiotemporal graph neural network root cause inference" that processes massive amounts of heterogeneous telemetry data, constructs a microservice evolution causal graph, and accurately locates the initial source of faults; Third, "test environment alignment and data synthesis based on generative AI" that uses Infrastructure as Code (IaC) intelligent generation and synthesis data technology to restore the production baseline at the pixel level on isolated test nodes and eliminate environmental parity differences; Fourth, the core "reinforcement learning-driven concurrent scheduling exploration and occasional bug triggering" that takes over the operating system threads and network scheduling rights, simulates extreme interleaved timing sequences through Markov decision processes, and forcibly induces hidden race conditions; Fifth, "multi-agent collaborative reproduction script generation and closed-loop verification" that outputs stable and executable automated test code through program dependency graph slicing and multi-role LLM continuous adversarial correction.

[0068] The production environment bug testing environment reproduction system proposed in this invention aims to build a highly automated and intelligent closed-loop troubleshooting pipeline. This system eliminates the need for manual discrete troubleshooting and blind guessing. Through system-level holographic telemetry data fusion, it actively reconstructs and intervenes in the system state in the test sandbox, transforming intermittent faults (Heisenberg defects) that are difficult to capture in the production environment into deterministic defects that are 100% reproducible in the test environment, and automatically outputs high-quality defect reproduction test scripts (BRT).

[0069] Figure 3 This is a schematic diagram of a device for reproducing intermittent bugs in a production environment, provided by an embodiment of the present invention. This device can vary significantly due to differences in configuration or performance, and may include one or more processors 60 (central processing units, CPUs) (e.g., one or more processors) and memory 70, and one or more storage media 80 (e.g., one or more mass storage devices) for storing applications or data. The memory and storage media can be temporary or persistent storage. The program stored in the storage media may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the device for reproducing intermittent bugs in the production environment. Furthermore, the processor may be configured to communicate with the storage media to execute the series of instruction operations in the storage media on the device for reproducing intermittent bugs in the production environment.

[0070] The device for reproducing intermittent bugs in the production environment according to the present invention may further include one or more power supplies 90, one or more wired or wireless network interfaces 100, one or more input / output interfaces 110, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The illustrated device structure for reproducing occasional bugs in the production environment does not constitute a limitation on the device for reproducing occasional bugs in the production environment. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0071] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the method for reproducing occasional bugs in the production environment.

[0072] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system or system / unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0073] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reproducing occasional bugs in a production environment, characterized in that, include: In a production environment, when a pre-defined abnormal event is detected, fault context data related to intermittent bugs is captured. Based on a multimodal spatiotemporal graph neural network model, the fault context data is analyzed to infer the root cause of the occasional bug. In an isolated test environment, based on the fault context data and the fault root cause, a generative artificial intelligence model is used to automatically align the environment configuration and data status, creating a reproduction environment consistent with the production environment baseline. In the reproducible environment, by actively intervening in the nondeterministic behavior of the system through reinforcement learning agents, a deterministic execution path that stably triggers the sporadic bug is explored and determined. Based on the deterministic execution path and the reproduction environment, a reproduction test script is automatically generated to stably reproduce the occasional bug.

2. The method for reproducing occasional bugs in the production environment according to claim 1, characterized in that, In the production environment, when a preset abnormal event is detected, capturing fault context data related to intermittent bugs includes: By using probes deployed in the operating system kernel and application-layer proxies deployed in the microservice mesh, the system state within the fault time window is intercepted in a bypass manner, and a structured, portable replay data packet is generated. The portable replay data package includes the following logical layers: inventory and tracing layer, construction and sealing context, input and file system layer, environment and configuration snapshot, network transaction transcription, time and random number control, concurrency and scheduling metadata, end-to-end telemetry data layer, and isolated replay execution engine.

3. The method for reproducing occasional bugs in the production environment according to claim 1, characterized in that, The analysis of the fault context data based on the multimodal spatiotemporal neural network model to infer the root cause of the intermittent bug includes: The fault context data is preprocessed using a multimodal spatiotemporal neural network model through feature engineering. Based on the feature vector data containing logs and time-series performance metrics obtained from the preprocessing, a dynamic service dependency graph is constructed with microservice instances as nodes and inter-service calls as edges. By fusing diffusion graph convolution and causal inference algorithms on the dynamic service dependency graph, a targeted fault subgraph is output to locate the source of the fault.

4. The method for reproducing occasional bugs in the production environment according to claim 1, characterized in that, The automatic alignment of environment configuration and data state through generative artificial intelligence models includes: Using a first generative artificial intelligence model, the environmental snapshot in the fault context data is parsed to automatically generate or adjust the infrastructure code manifest to align with the underlying infrastructure configuration; By utilizing a second generative artificial intelligence model based on generative adversarial networks or variational autoencoders, the joint distribution characteristics of production data are learned to synthesize test data in the test environment that maintains entity referential integrity and business logic constraints and does not contain personally identifiable information.

5. The method for reproducing occasional bugs in the production environment according to claim 1, characterized in that, In the reproducible environment, the process of exploring and determining a deterministic execution path that stably triggers the occasional bug by actively intervening in the nondeterministic behavior of the system through a reinforcement learning agent includes: The system's thread scheduling and network event timing control problems are modeled as Markov decision processes; Define the state space, action space, and reward function of the reinforcement learning agent. The state space includes thread instruction pointers, lock states, and distances to the root cause of the fault. The action space includes blocking or waking up specific threads, injecting delays between network requests, and changing the context switching strategy. The reward function is set based on whether the program execution path is close to the target exception stack. Based on the definition, the reinforcement learning agent is driven to explore and obtain a sequence of scheduled actions that can reliably trigger the occasional bug as the deterministic execution path.

6. The method for reproducing occasional bugs in the production environment according to claim 1, characterized in that, The automatic generation of reproduction test scripts for reliably reproducing the intermittent bugs is achieved through a closed-loop multi-functional intelligent agent collaborative process that includes generation, verification, and correction. This closed-loop multi-functional intelligent agent collaborative process includes: The agent is retrieved by context slicing, and the minimal relevant code context is extracted from the source code library based on the program dependency graph; An intelligent agent is generated from the test script, and candidate reproducible test scripts are generated based on the relevant code context and the deterministic execution path; The closed-loop verification agent executes the candidate reproducibility test script in the sandbox and detects whether the target bug can be triggered. The reflection and correction agent iteratively corrects the candidate reproducibility test script based on the feedback information of verification failure until the verification is successful.

7. The method for reproducing occasional bugs in the production environment according to claim 5, characterized in that, When the reinforcement learning agent explores, a dynamic partial order reduction mechanism is introduced. By identifying and removing equivalent scheduling sequences whose execution order does not affect the final state during the scheduling exploration process, the exploration efficiency of the reinforcement learning agent in discovering the target BUG trigger path is improved.

8. A system for reproducing occasional bugs in a production environment, characterized in that, The system for reproducing occasional bugs in the production environment includes: The fault context acquisition module is used to capture fault context data related to intermittent bugs in the production environment when a preset abnormal event is detected. The fault root cause analysis module is used to analyze the fault context data based on a multimodal spatiotemporal neural network model and infer the fault root cause of the occasional bug. The reproduction environment creation module is used to automatically align the environment configuration and data status with the production environment baseline in an isolated test environment based on the fault context data and the fault root cause through a generative artificial intelligence model. The BUG path exploration module is used to explore and determine the deterministic execution path that stably triggers the occasional BUG in the reproduced environment by actively intervening in the nondeterministic behavior of the system through reinforcement learning agents. The test script generation module is used to automatically generate a test script for stably reproducing the intermittent bug based on the deterministic execution path and the reproduction environment.

9. A device for reproducing occasional bugs in a production environment, characterized in that, The device for reproducing the occasional bugs in the production environment includes: a memory and at least one processor, wherein the memory stores instructions and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the instructions in the memory to cause the production environment intermittent bug reproduction device to execute the production environment intermittent bug reproduction method as described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements a method for reproducing occasional bugs in the production environment as described in any one of claims 1-7.

Images