A service process level playback verification method based on full-link tracking data

Abstract

The application discloses a kind of business process level playback verification methods based on full-link tracking data, it is related to software testing technical field.For the internal state leakage detection of regression verification in code refactoring and technology stack migration, state system distortion, it is difficult to consider performance and precision, inefficient problem of difference audit, collect full-link data with TraceID, intercepting agent is directed to collect SQL, after desensitization cleaning, request is aggregated as process level case set according to business dimension ID;Generate benchmark program self-checking, new version fixed TraceID is simulated according to historical time sequence Dependent call replay;Difference decision module outputs and internal state such as database, cache, message queue, distributed lock across request comparison, automatically filter irrelevant field, expected difference is matched script first, and new script is generated by large language model for failure.The application realizes high coverage, high accuracy, high efficiency, high security and automated playback verification, improves refactoring migration test quality and efficiency.

Description

Technical Field

[0001] This invention relates to the field of software testing technology, and specifically to a business process-level replay verification method based on end-to-end tracing data. Background Technology

[0002] With the popularization of microservice architecture and continuous iteration of software systems, code refactoring has become a routine operation in the software development process. Refactoring requires optimizing the internal code structure, upgrading the technology stack, or improving system performance while keeping the external business behavior of the software unchanged. Regression verification after refactoring is a core part of controlling deployment risks. Current mainstream verification solutions have many inherent defects. Manually writing regression test cases is highly dependent on the testers' business analysis ability and experience, has limited scenario coverage, and is difficult to reproduce edge, anomaly, and concurrent timing scenarios in the production environment. At the same time, manually writing test cases requires a large manpower investment and a long cycle, which cannot support the rapid refactoring and iteration of large systems. Traditional traffic replay tools can only complete the black-box input and output comparison of interfaces, and cannot identify hidden defects such as database, cache, message queue, etc., which have abnormal internal states but return normal interfaces. They only support the independent replay of single requests, and cannot reproduce the timing and concurrency relationship of production requests. They are difficult to verify cross-request state-dependent business processes such as creation before querying and transaction before refund, and are not suitable for stateful systems that rely on databases, caches, and distributed locks.

[0003] Existing publicly available traffic replay solutions, such as the invention patent "A Novel Automated Testing Method and System for Software Quality Inspection Using Traffic" (patent application number CN202511143295.2), record production environment traffic using probes and replay it to the test environment. This supports field-level comparison of interface return values ​​and database changes, improving test coverage depth to some extent. However, this type of solution still has several technical limitations: First, it still treats each request as an independent test unit, failing to aggregate related requests based on business dimensions to form complete business process test cases. This makes it impossible to guarantee the execution order and state transmission across requests, resulting in insufficient accuracy for long-process business verification in stateful systems. Second, this type of solution uses a full-volume collection strategy for database operation data, recording all SQL execution parameters. This incurs significant performance overhead in high-concurrency production environments, making large-scale, routine deployment difficult. Third, the request link identifiers in this type of solution are randomly generated, making it impossible to reuse link identifiers consistent with historical data during replay testing. This makes data alignment and problem tracing difficult between multiple tests. Fourth, the difference filtering rules of this type of solution rely on manual configuration and maintenance. When faced with massive amounts of repetitive differences, testers need to review and confirm each one individually, resulting in low efficiency in error attribution. It also lacks the self-learning ability to automatically generate matching rules based on error characteristics and accumulate analytical experience. Fifth, this type of solution does not differentiate between time-sensitive requests. Long-cycle scenarios such as scheduled tasks and batch processing need to be executed at the original time intervals, leading to long testing times and low efficiency. Sixth, the test data source for this type of solution is singular, using only production environment traffic. This fails to cover low-frequency edge anomaly scenarios in production, and the desensitization mechanism for sensitive production data is inadequate, posing data compliance risks.

[0004] In existing technologies, mainstream end-to-end monitoring APM tools such as SkyWalking and Pinpoint are designed only for performance monitoring and troubleshooting, lacking the ability to replay traffic and automatically compare differences. Their native JDBC collection plugins use full SQL parameter collection logic, which incurs significant performance overhead in high-concurrency production environments, making large-scale data collection impossible. Solutions that rely on large language models to directly generate test cases suffer from content illusion, resulting in deviations between generated test cases and real production business scenarios, leading to insufficient reliability of test results. Considering these various technical shortcomings, existing regression testing methods cannot meet the testing requirements of high coverage, high accuracy, high efficiency, and compliant security in code refactoring scenarios. Summary of the Invention

[0005] The purpose of this invention is to address the technical problems of regression verification techniques in code refactoring and technology stack migration scenarios, including missed detection of internal state defects, test distortion of stateful systems, difficulty in balancing data acquisition performance and accuracy, low efficiency in reviewing massive differences, and an inability to balance test authenticity and data compliance. To address these issues, a business process-level replay verification method based on end-to-end tracing data is proposed. This method includes: Collect a full-link dataset with TraceID tags generated by the business program in the historical environment. The dataset contains input and output parameters, internal subroutine call order, timestamps and internal state transition data within the complete lifecycle of a single request. SQL operation data is collected by interceptors to collect complete parameterized logs of data operation statements within a specified range. Sensitive field desensitization processing is performed on the full-link dataset. Data cleaning is performed based on TraceID and business dimension ID to filter closed-loop test cases that can be generated by the business data within the process. Based on the business dimension ID, multiple requests with business associations are aggregated and sorted according to their actual execution order to form a business process-level test case set. A benchmark test program is generated based on the historical version of the business program and the full-link dataset. After running the program in the historical version environment to verify that the benchmark output is consistent with the output of the historical version program, it is used as the comparison benchmark. In the new version environment to be tested, the benchmark program initiates a replay test, assigning a TraceID consistent with the historical data to each test data. The simulation replay module simulates the dependent call response based on the TraceID according to the historical call order and time interval. The difference decision module compares the output and internal state change data of the new version program with the baseline, automatically filters differences in business-irrelevant fields, performs cause matching based on error matching scripts for unexpected differences, and calls a large language model to analyze the cause of the difference and generate a corresponding matching script when the matching fails, outputting a structured difference report.

[0006] Preferably, the data cleaning and filtering closed-loop test cases specifically include: associating all TraceID requests under the same business process based on the business dimension ID, identifying the characteristics of the business process's initial request, filtering out incomplete lifecycle data that only contains subsequent operations such as queries, refunds, and order closures but lacks the initial transaction creation request, and retaining only the complete process that starts from the business initial request and whose business data can be generated within the process as valid test cases, so that the test case set is independent of the existing transaction data at the time of collection; after the benchmark test program is generated, a benchmark self-verification process is executed: the data-cleaned test case set is run completely once in a historical version of the program that is consistent with the version of the collection environment, using the input parameters collected in the production environment as test input, comparing the output of the test environment with the collected output of the production environment and whether the internal state is completely consistent, verifying the completeness of the test dataset and the correctness of the initial error matching script, and only after all self-verifications pass can it be used for comparative testing of the new version of the program. By associating all TraceID requests within the same process using business dimension IDs, the system identifies the characteristics of initial business requests and filters out incomplete lifecycle data containing only subsequent operations such as queries, refunds, and order closures. Only closed-loop test cases that can be generated automatically within the process are retained. Simultaneously, a benchmark self-verification is performed before formal testing, running the cleaned dataset in the same version of the program to verify data integrity and the correctness of the initial attribution script. This ensures the self-containment of test cases from the data source, making the test set independent of existing data at the time of collection. Furthermore, the pre-verification mechanism avoids test result distortion caused by missing data or rule deviations, significantly improving the reliability and credibility of replay test results.

[0007] Preferably, during the historical data collection phase, a globally unique TraceID is assigned to each request and persists throughout the entire call chain. Each replay test mandates the use of a TraceID that is completely identical to the historical data, ensuring that the link identifier remains consistent across multiple replays of the same test data. This supports data comparison and problem localization across test cycles. This solves the problem of misalignment and comparison of identical test data in multiple replay tests, providing a unified link identifier foundation for repeated testing, problem reproduction, cross-test data comparison, and difference localization, making it easier for testers to track and locate specific defects.

[0008] Preferably, a targeted interception mechanism is implemented using MyBatis Interceptor as the interceptor. The interception scope is limited to pre-configured data tables, Mapper interfaces, and data operation types. Complete SQL execution logs with parameters are recorded for operations within the interception scope. For query operations and data operations outside the interception scope, only the call behavior is recorded, achieving a balance between data comparison completeness and collection performance. The logging logic of all request-response entry points is rewritten using the SPI mechanism, covering all middleware call data such as HTTP, Dubbo, RabbitMQ, databases, caches, and distributed locks. Structured extraction and storage of logs are achieved through ES Pipeline. During the data collection phase, the time period of environment configuration parameter changes is recorded synchronously. Requests within the configuration change time period are skipped during the replay phase to avoid false positives caused by configuration differences. MyBatis Interceptor is used to implement application-layer targeted interception of SQL operations, recording complete parameterized logs only for data tables and write operations within the configuration scope. Simultaneously, the logging logic of various middleware such as HTTP, Dubbo, MQ, databases, caches, and distributed locks is rewritten non-intrusively through the SPI mechanism to achieve end-to-end collection, and related requests are automatically skipped upon detecting configuration change time periods. This approach allows for targeted interception to replace the full JDBC data collection of the APM framework, while ensuring the accuracy of comparisons at the critical field level. This keeps the performance overhead of the production environment within an acceptable range, while also enabling non-intrusive data collection of the entire middleware without affecting business code and avoiding false alarms caused by differences in environment configuration.

[0009] Preferably, the test dataset sources include real business data successfully executed in historical production environments, and test data from testers constructing coverage edge scenarios and low-frequency anomaly scenarios in historical test environments. Sensitive data masking is performed by automatically identifying sensitive fields based on regular expressions and field semantics, using an irreversible hash algorithm. Interface logic containing sensitive information that cannot be masked is excluded from the test dataset. The masked data maintains the integrity of business logic, supporting the program's normal processing and judgment of business processes. Multi-source data fusion compensates for the insufficient coverage of low-frequency anomaly scenarios by pure production traffic, improving the comprehensiveness of test scenarios. Irreversible masking ensures the security and compliance of production data, and the masked data retains the integrity of business logic, without affecting the program's normal processing and judgment of business processes.

[0010] Preferably, the business dimension ID includes identifiers such as order number, payment order number, user ID, and transaction serial number, which can uniquely identify the complete business process. Multiple independent requests under the same business dimension ID are sorted according to the actual execution timestamps of the historical environment, forming complete business process test cases across requests and services. This is suitable for verifying long processes and state dependencies in microservice systems. During the replay phase, the execution order of each request within the business process is strictly guaranteed, ensuring cross-request state transmission and business logic consistency. This overcomes the limitations of traditional solutions that use a single entry request or a single online transaction as the replay unit, making it suitable for verifying long processes and state dependencies in microservice systems. It can accurately verify the correctness of order-sensitive business logic such as creation before querying or transaction before refunding, solving the problem of untestable cross-request state transmission in stateful systems.

[0011] Preferably, during the replay process, when the simulation module encounters an external call request that is not covered by historical data, the current test process is immediately terminated, and an alarm message containing the complete call context is generated to notify the testers. For non-real-time requests with low time sensitivity, traffic compression technology is used to compress the request interval while ensuring that the request execution order remains unchanged, thereby shortening the replay time of long-cycle business processes and improving test execution efficiency. The rigorous mechanism of terminating if not covered avoids test result distortion caused by unknown responses, ensuring the rigor of the test. At the same time, time compression reduces the test time of long-cycle business processes such as scheduled tasks and batch processing from hours to minutes, significantly improving test execution efficiency.

[0012] Preferably, the difference decision module incorporates a dynamic difference filtering mechanism based on pre-configured field filtering rules. This mechanism automatically identifies and ignores differences in fields such as timestamps, random numbers, server IDs, log IDs, and request IDs that affect the correctness of business logic. The comparison scope of internal state transition data covers state changes of request-related fields in the database, changes in relevant key-value pairs in the cache, the production and consumption of relevant messages in the message queue, and the acquisition and release of distributed locks. The comparison performs correlation verification across the request business process dimension, verifying the state transmission and data consistency between multiple requests under the same business dimension ID, rather than comparing independently within a single request. This expands the comparison scope from the black-box interface return value to all middleware state changes, enabling the discovery of hidden defects where internal state errors occur but external returns are normal. Simultaneously, cross-request correlation verification verifies the state transmission and data consistency between multiple requests under the same business, resulting in a verification depth and scope significantly greater than traditional single-request comparison schemes. Furthermore, field semantic filtering significantly reduces the false positive rate of difference comparison.

[0013] Preferably, the specific process of error analysis assisted by the large language model is as follows: When a new version is detected to be inconsistent with the baseline output and the difference field is not within the expected whitelist, the error analysis process is triggered; error matching scripts are traversed and executed, prioritizing grouping by the requested business method address; if a script executes successfully and matches the current difference feature, the error reason corresponding to the script is returned directly; if all existing scripts fail to match, the request input parameters, output difference, business method address, and call chain context are input into the large language model, which analyzes the difference reason and generates a new error matching script; the newly generated script is associated with the error reason and stored and maintained independently by grouping by business method address; the system records the error reason for each difference, and during the manual review stage, only the logical correctness of the corresponding matching script needs to be verified to confirm the reasons for similar differences in batches. Positioning the large language model as an auxiliary role rather than a direct decision-maker ensures the rigor of the test results. Simultaneously, the automatic matching and batch confirmation of error reasons are achieved through the accumulation and reuse of matching scripts, reducing the workload of manual review from confirming differences one by one to only verifying the logic of new scripts, forming a self-learning capability to continuously improve the efficiency of difference handling.

[0014] Preferably, during the replay process, each sub-service uses the collected production environment input parameters as the test environment input, and compares the consistency between the test environment output and the production environment output hop-by-hop. The test environment only needs to be configured with system parameters that do not change frequently, and the test case set is independent of the existing business data and the complete production database snapshot at the time of collection, without the need to restore the full production data state. This achieves weak environment dependency for replay testing, completely eliminating the strong dependency of traditional traffic replay solutions on the complete database snapshot and the full existing data. The test environment setup is simplified from needing to synchronize the full production data to only configuring basic parameters, reducing the environment preparation time from weeks to hours. At the same time, the hop-by-hop comparison mechanism ensures the verifiability of the output of each service node under the microservice architecture.

[0015] In summary, by adopting the aforementioned technical solution, this invention addresses the regression verification needs in code refactoring and technology stack migration scenarios. Compared to existing technologies, it aggregates and associates requests using business dimension IDs to form complete business process-level test cases. Combined with a fixed TraceID time-series replay mechanism, it accurately reproduces the request order and state transmission in the production environment, effectively solving the problem that traditional single-request replay cannot verify the consistency of cross-request business logic in stateful systems. Through full-link internal state white-box comparison, the verification scope is expanded from interface output to all-dimensional state changes such as databases, caches, message queues, and distributed locks, breaking through the limitations of black-box testing and significantly reducing the false negative rate of internal state anomalies. By using MyBatis to intercept and collect SQL parameters within a specified range, it ensures the integrity of the SQL statements. By controlling the performance overhead of data acquisition while comparing segment-level accuracy, the inherent contradiction between the performance loss of full-volume data acquisition and the requirement for comparison accuracy is resolved. Through a script-first AI-assisted error analysis mechanism, the cause of errors is automatically matched and batch confirmed, significantly reducing the repetitive workload of manual review. At the same time, the independent accumulation of matching scripts forms a self-learning capability, continuously improving the efficiency of difference handling. By integrating real production data and test edge scenario data and performing irreversible sensitive field desensitization, the comprehensive coverage of test scenarios is ensured while meeting data security and compliance requirements. With the help of traffic compression technology, the test time of long-cycle business is shortened. Overall, it achieves high coverage, high accuracy, high efficiency, and high security of automated playback verification, greatly improving the test quality and delivery efficiency of large-scale system reconstruction and migration.

[0016] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below can be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other.

[0017] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description

[0018] The accompanying drawings are not drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures may be denoted by the same reference numeral. For clarity, not every component is labeled in each figure. Embodiments of various aspects of the invention will now be described by way of example and with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall process of the business process-level replay verification method based on end-to-end tracing of the present invention; Figure 2This is a schematic diagram of the data acquisition module for the business process-level replay verification method based on end-to-end tracing of the present invention; Figure 3 This is a schematic diagram of the simulation replay module of the business process-level replay verification method based on end-to-end tracing of the present invention; Figure 4 This is a schematic diagram of the overall data model structure of the business process-level replay verification method based on end-to-end tracing of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art.

[0020] The terms "first," "second," and similar words used in the specification and claims of this patent application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, unless the context clearly indicates otherwise, the singular forms of "an," "a," or "the," etc., do not indicate a quantity limitation, but rather indicate the presence of at least one. Terms such as "comprising" or "including" mean that the element or object preceding "comprising" encompasses the features, integrals, steps, operations, elements, and / or components listed following "comprising" or "including," and do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0021] To better understand the technical solution of the present invention, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0022] The main solution of this invention is as follows: Addressing the regression verification needs in code refactoring and technology stack migration scenarios, it utilizes end-to-end TraceID technology to collect complete runtime data of business programs in historical environments; it uses interceptors to collect SQL operation parameters in a targeted manner; and it collects end-to-end data from each middleware through the SPI mechanism. Based on TraceID and business dimension ID, it performs data cleaning, filters closed-loop test cases that can be generated automatically within the process, and aggregates multiple requests under the same business dimension into a business process-level test case set according to their execution sequence. It generates a benchmark test program and performs benchmark self-verification in the same version environment to verify data integrity and regression. The script's correctness is used as the comparison benchmark. When replaying in the new version environment, a TraceID consistent with historical data is forcibly specified. Dependency calls are simulated according to the historical call order and time interval. Each sub-service compares the output hop by hop with the production input as the test input. The test environment only needs to configure basic system parameters and does not need to restore the production database snapshot. The difference decision module correlates and verifies the output and the internal states of the database, cache, message queue, distributed lock, etc. across the cross-request business process dimension. It automatically filters business-irrelevant fields, prioritizes the execution of error matching scripts for unexpected differences, and calls the large language model to analyze and generate a new script when the match fails, outputting a structured difference report.

[0023] The executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a server or personal computer, or an electronic device capable of performing the above functions. The following uses a code refactoring migration verification system as an example to illustrate this embodiment and the subsequent embodiments.

[0024] Since this invention involves a large number of business terms, to avoid ambiguity, a unified explanation of the terms is provided here: TraceID: A globally unique identifier in distributed end-to-end tracing, generated at the request entry point and passed through the call chain to all downstream services, used to connect all services and middleware calls traversed by a single request. This invention modifies the standard TraceID mechanism: during replay testing, it forces the specification of the same TraceID as historical data, achieving data alignment across multiple tests.

[0025] End-to-end tracing: A distributed system monitoring technology that uses bytecode enhancement or code embedding to collect information such as all service nodes traversed by a single request, call order, time consumption, and parameters, for performance monitoring and troubleshooting. Mainstream implementations include SkyWalking and Pinpoint. This invention extends this technology from the monitoring field to the software testing field.

[0026] Traffic replay: An automated testing technique that records real request traffic from a production or historical environment, retransmits it in the test environment, and compares the results to achieve automated testing based on real business scenarios. This invention implements business process-level replay, which differs from traditional single-request replay.

[0027] Code refactoring: The process of adjusting and optimizing the internal code structure, architecture, and technology stack without changing the external business behavior of the software is the core application scenario of this invention.

[0028] MyBatis Interceptor: A plugin interception mechanism provided by the MyBatis persistence layer framework, which can intercept and process SQL before and after execution. This invention uses this mechanism to selectively collect parameterized SQL statements for specified tables and operations, replacing the full collection by the APM framework and solving performance issues.

[0029] SPI (Service Provider Interface): A service discovery extension mechanism provided by Java, allowing third parties to provide implementations for interfaces without modifying existing code. This invention uses the SPI mechanism to non-intrusively rewrite logging logic, achieving end-to-end data collection.

[0030] Closed-loop test cases: In this invention, a closed-loop test case refers to a complete business process that starts from the initial business request and whose business data can be generated automatically within the process, and whose execution does not depend on existing data or database snapshots outside the process.

[0031] Benchmark self-verification: In this invention, it refers to the process of running the cleaned test case set completely once in an older version of the program that is consistent with the version of the data collection environment to verify the integrity of the test data and the correctness of the initial error matching script.

[0032] ES Pipeline: Elasticsearch's data processing pipeline.

[0033] MDC: Mapped Diagnostic Context, a diagnostic context mapping mechanism in SLF4J / Logback.

[0034] LLM: an abbreviation for Large Language Model.

[0035] Reference Figure 1 The first embodiment of the present invention provides a business process-level replay verification method based on end-to-end tracing data, comprising the following steps: Step 1: End-to-end data collection: like Figure 2As shown, in the old version of the business application before the refactoring, the logging logic of all request and response entry points was rewritten using the Java SPI mechanism. The collection scope covered all middleware call data, including HTTP entry points, Dubbo internal service calls, RabbitMQ message consumption and production, MySQL database operations, Redis cache read and write, and distributed lock acquisition and release. TraceIDs were generated at the request entry point and passed through the entire call chain via the MDC mechanism, ensuring that each request's TraceID was globally unique.

[0036] SQL operations are intercepted via MyBatis Interceptor. The interception scope is limited by configuration to pre-specified data tables, Mapper interfaces, and data operation types (insert, update, delete). For write operations within the interception scope, a complete SQL execution log with parameters is recorded, including the SQL statement itself, bound parameter values, execution time, and the number of rows affected. For query operations and data operations outside the interception scope, only the call behavior is recorded, not the specific parameters, ensuring the accuracy of key field comparisons while controlling performance overhead in the production environment.

[0037] The collected logs are extracted and stored in a structured manner through ES Pipeline, encapsulating all data associated with the same TraceID into a complete single request lifecycle dataset. During the data collection phase, the time periods of changes in environment configuration parameters are recorded synchronously, including parameter modifications in the configuration center, switch switching, interest rate adjustments, etc., so that requests within this time period can be skipped in the subsequent replay phase, avoiding false positives in tests caused by configuration differences.

[0038] Data anonymization is performed synchronously during the data collection process. The anonymization module automatically identifies sensitive fields, including user nicknames, shipping addresses, and email addresses, based on regular expressions and field semantics, and performs anonymization using an irreversible hash algorithm. Interface logic that explicitly contains sensitive information that cannot be anonymized, such as payment password verification and biometric identification, is excluded from the test dataset. The anonymized data maintains the integrity of business logic and does not affect the program's normal processing and judgment of business processes. Business dimension ID fields, such as order numbers, payment order numbers, and transaction serial numbers, are not included in the anonymization process to ensure that requests can be correctly aggregated and associated based on business dimension IDs during the data cleaning phase.

[0039] Step 2: Data cleaning and test case set construction.

[0040] The test dataset consists of two parts. One part is real business data from historical production environments that has been successfully executed, covering normal business processes. The other part is test data constructed by testers in the test environment, specifically covering edge scenarios and low-frequency anomaly scenarios such as exceptions, refund failures, payment timeouts, insufficient inventory, network retries, and distributed transaction rollbacks, to compensate for the insufficient coverage of abnormal paths in production data.

[0041] After data anonymization, data cleaning is performed. Using business-dimensional IDs, such as order numbers, payment order numbers, user IDs, and transaction serial numbers—identifiers that uniquely identify a complete business process—all TraceID requests under the same business process are associated. The characteristics of the initial request for the business process are identified, such as the "create order" request in e-commerce scenarios and the "initiate transaction" request in banking scenarios. Incomplete lifecycle data that only includes subsequent operations such as queries, refunds, order closures, and reversals but lacks the initial transaction creation request is filtered out. The characteristics of the initial request for the business process are specified through configuration, such as pre-configuring specific interface names, URI patterns, or request method names as business start markers. During data cleaning, incomplete processes are identified and filtered based on the configuration, retaining only complete processes that start from the initial business request and whose business data can be generated automatically within the process as valid test cases. This ensures that the test case set is independent of the existing transaction data at the time of collection and the complete production database snapshot.

[0042] After data cleaning, multiple independent requests under the same business dimension ID are sorted according to their actual execution timestamps in the historical environment, forming complete business process test cases across requests and services. For example, in an e-commerce scenario, requests such as order creation, inventory deduction, payment initiation, payment callback, and logistics notification under the same order number are sorted by time to form a full lifecycle test case for an order from creation to completion. Each test case corresponds to a complete business, including the input parameters, expected output, internal call order, time intervals, and internal state transition data of all requests under that business.

[0043] Step 3: Benchmark program generation and benchmark self-verification.

[0044] Based on the old version of the business application code and the cleaned end-to-end dataset, a benchmark test program is automatically generated. The benchmark test program is a standalone, runnable test program containing three core components: a dataset processing module, a simulation replay module, and a difference decision module. The benchmark test program is a pre-developed, general-purpose testing framework that can be run simply by configuring and loading the connection information of the system under test, the test case set, and the matching rule base; no separate test code needs to be written for each system under test.

[0045] After the benchmark test program is generated, a benchmark self-verification process is executed first. The cleaned test case set is deployed and run completely in an older version of the program environment that is identical to the data acquisition environment. Using the input parameters collected from the production environment as test input, the output of the test environment is compared item by item with the collected production environment output, as well as the internal states of the entire chain, including the database, cache, and message queue, to ensure complete consistency. The purpose of benchmark self-verification is twofold: first, to verify the integrity of the test dataset, ensuring that the cleaned data can drive the older version of the program to fully execute the business process; and second, to verify the correctness of the initial error matching script, ensuring that the predefined field filtering rules and known error matching rules function correctly in the benchmark environment. After all self-verifications pass, the verified output results and internal states are used as the comparison benchmark for comparative testing of the new version of the program. If inconsistencies are found during self-verification, the problems in the data acquisition or cleaning process are traced back, corrected, and the self-verification is re-executed.

[0046] Step 4: Replay the new version.

[0047] like Figure 3 As shown, the benchmark program is directed to the new version of the business application environment to initiate a replay test. The test environment only needs to be configured with basic system parameters that do not change frequently, such as service address, connection timeout, and middleware connection string. There is no need to restore the full production database snapshot and existing business data.

[0048] During replay, each test data point is forcibly assigned a TraceID that is completely identical to the historical data, ensuring that the link identifier remains consistent across multiple replays of the same test data. The simulated replay module returns simulated responses based on the TraceID, strictly following the call order and time intervals of the historical environment. Simulated objects include external third-party interfaces such as payment gateways, SMS platforms, and logistics queries, as well as internal service calls and middleware responses. During replay, each sub-service uses the collected production environment input parameters as the test environment input, comparing the output hop-by-hop at each service node. This means that the actual output of each service node is compared with the historical records, rather than only performing a black-box comparison at the final entry point.

[0049] If the simulation module encounters an external call request that is not covered by historical data during the replay process, it will immediately terminate the execution of the current test case and generate an alarm message containing the complete call context (TraceID, call stack, request parameters) to notify the tester, so as to avoid the test results being distorted due to unknown responses.

[0050] For non-real-time requests with low time sensitivity, such as scheduled status queries, batch notifications, and end-of-day reconciliation, traffic compression technology is used to compress the request interval while ensuring that the request execution order remains unchanged. For example, logistics status polling, which takes 10 minutes in a production environment, can be compressed to once every 10 seconds; end-of-day batch processing, which originally took 8 hours, can be compressed to complete in 15 minutes, significantly shortening the replay time of long-cycle business processes.

[0051] The replay scheduler shortens the waiting time between adjacent requests according to the configured compression ratio, and the business timestamp in the request message remains unchanged from the historical collection value; for business logic that depends on the current system time, the compressed virtual time is returned through the Mock time source to avoid logical anomalies such as timeout judgment.

[0052] Step 5: Intelligent difference comparison and attribution.

[0053] The difference decision module compares the output results and internal state change data of the new version of the program with the baseline field by field. The module has a built-in dynamic difference filtering mechanism based on pre-configured field filtering rules, which automatically identifies and ignores field differences such as timestamps, random numbers, server IDs, log IDs, request IDs, and traceIds that do not affect the judgment of the correctness of business logic.

[0054] The comparison scope of internal state transition data covers: state changes of request-related fields in the database, changes of relevant key-value pairs in the cache, the production and consumption of relevant messages in the message queue, and the acquisition and release of distributed locks. This comparison performs correlation verification across the request business process dimension, that is, verifying the state transmission and data consistency between multiple requests under the same business dimension ID, such as whether inventory deduction is correctly triggered after order creation, and whether the order status correctly flows after successful payment, rather than comparing independently within a single request.

[0055] When an unexpected discrepancy is detected, an error analysis process is triggered. First, the system groups the discrepancies by the requested business method address, iterates through the maintained error matching scripts, and executes them. If a script executes successfully and matches the current discrepancy characteristic, it directly returns the corresponding error reason and marks the discrepancy. If all existing scripts fail to match, the request input parameters, output discrepancies, business method address, and call chain context are input into a large language model. The large language model analyzes the discrepancy reason and generates a new error matching script. The newly generated script, associated with the error reason, is stored and maintained independently, grouped by business method address, without modifying existing scripts. The system records the error reason for each discrepancy. During the manual review phase, only the logical correctness of the newly generated matching script needs to be verified to confirm the reasons for all similar discrepancies in batches, eliminating the need to check for recurring errors one by one. The newly generated script is initially in a pending review state. After manual review confirms its logical correctness, it is set to an effective state. Only scripts that have become effective can be used for subsequent automatic discrepancy matching.

[0056] After the test is completed, a structured difference report is output, which includes the TraceID, business dimension ID, difference field, baseline value and new value, call chain position where the difference occurred, internal state transition comparison, error cause classification and corresponding matching script ID for each difference.

[0057] Specific Implementation Example 1: Scenario of Reconstructing a Monolithic E-commerce Transaction System into Microservices This embodiment is applied to a project to refactor an e-commerce platform's order transaction system from a monolithic architecture to a microservice architecture. The goal of the refactoring is to split the original monolithic application into three independent microservices: order service, payment service, and inventory service. The refactoring process requires that no business logic be changed and that the user be unaware of the changes.

[0058] During the data acquisition phase, the request and response entry logs in the old monolithic application were rewritten using the SPI mechanism to collect end-to-end data from HTTP, Dubbo, RabbitMQ, MySQL, Redis, and distributed locks. SQL collection used MyBatisInterceptor to intercept insert, update, and delete operations in four core tables: order, payment, inventory, and order log. Complete SQL statements with parameters were recorded; for query operations, only the call behavior was recorded. The collection period covered a full 7-day business cycle, including peak traffic during promotional periods and off-peak traffic. Configuration change periods were also recorded synchronously.

[0059] During the dataset construction phase, the test set includes 7 days of successful production transaction data and abnormal scenario data constructed by testers, such as exceptions, refund failures, payment timeouts, and insufficient inventory. The data masking module identifies fields such as shipping address and user nickname and masks them using SHA-256 hashes. The payment password interface is excluded from the testing scope. All TraceID requests are associated with the order number as the business dimension ID, and "create order" is identified as the starting request feature. Incomplete data containing only queries, refunds, and order closures is filtered out, retaining only the complete order process starting from order creation. Order creation, inventory deduction, payment initiation, payment callback, and logistics notification under the same order number are sorted by timestamp to form full lifecycle use cases for the order.

[0060] In the benchmark self-verification phase, a benchmark test program is generated based on the old version code and the cleaned dataset. It is then run completely once in the old version monolithic environment. The benchmark is confirmed after comparing the output item by item and ensuring that it is completely consistent with the internal state.

[0061] During the replay phase, the benchmark program points to the new version of the microservice cluster. The test environment is configured only with service addresses and timeout parameters; the production database is not restored. Each test specifies a TraceID consistent with the historical data. The simulation module returns simulated responses from the payment gateway, logistics query, and SMS platform in production chronological order. Each sub-service compares its output hop-by-hop with the production input as the test input. Any external calls not covered are immediately terminated and an alarm is triggered. For requests like logistics polling, traffic compression is used, reducing the test duration for a single order from 30 minutes to 2 minutes. Test cases with configuration change time periods are skipped.

[0062] During the discrepancy comparison phase, fields such as timestamps, log IDs, and server IPs are automatically filtered. The comparison scope covers interface return values, order table status, inventory deduction quantities, payment transaction status, Redis order cache, MQ order messages, and distributed lock status. It also correlates and verifies the status transmission between creation, payment, and refund requests across the entire order process. For known errors such as incorrect inventory deduction order and delayed payment status updates, the script's return reason is directly matched. For new errors such as discrepancies in order amount calculation precision, LLM analyzes and generates new scripts; manual verification of the new script logic is sufficient to batch confirm all precision errors.

[0063] This embodiment achieved automatic coverage of 98% of business scenarios, discovered 17 internal state inconsistencies that were missed by manual testing, and shortened the testing cycle from 2 months to 2 weeks.

[0064] Specific Implementation Example 2: Migration Scenario of Domestic Database in Bank Core Accounting System This example is applied to a city commercial bank's core accounting system migration project from an Oracle database to a domestic distributed database. The goal of the migration is to complete the database replacement without changing any business logic, ensuring zero errors in accounting data.

[0065] During the data collection phase, the transaction entry point, service calls, and database access layer logs in the old core system were rewritten using the SPI mechanism. This collected end-to-end data from the counter HTTP interface, Dubbo service calls, MQ batch notifications, Oracle operations, Redis hot account caching, and distributed transaction locks. SQL collection used MyBatis Interceptor to intercept write operations such as balance updates, transaction insertions, and status changes in the account balance table, transaction log table, customer information table, and batch task table. For query operations, only the transaction code was recorded, not individual parameters. The collection period covered one month, including special time points such as monthly settlement, interest settlement, and batch payouts.

[0066] During the dataset construction phase, the test set includes one month of successful production transaction data and edge scenario data constructed by testers, such as reversals, refunds, account freezes / unfreezes, inter-institutional transfers, and large-scale anomalies. The anonymization module identifies customer names and uses the national cryptographic SM3 algorithm for anonymization, excluding passwords and biometric interfaces. All TraceID requests are associated with the transaction serial number as the business dimension ID, identifying "initiated transaction" as the starting feature. Incomplete data containing only queries, reversals, and refunds is filtered out, retaining only complete transaction flows from transaction initiation where the accounting data can be automatically generated within the process.

[0067] During the benchmark self-verification phase, the benchmark test program runs completely in the old version of Oracle environment, comparing the output with the internal status one by one, verifying the balance calculation accuracy to four decimal places, and confirming the data integrity and the correctness of the initial script.

[0068] During the replay phase, the benchmark program was directed to a new domestic database environment. The test environment was configured only with database connection and transaction timeout parameters, without migrating all historical data. Each transaction was assigned a TraceID consistent with the historical data, and the simulation module returned simulated responses from the central bank payment system, UnionPay clearing, and SMS platform in production chronological order. Each sub-service compared output consistency hop-by-hop. Any calls not covered were immediately terminated with an alarm. Hourly tasks such as end-of-day batch processing and interest calculations were compressed to the second level, and 8-hour batch processing was shortened to 15 minutes. Use cases during interest rate adjustments and system upgrade periods were skipped.

[0069] During the discrepancy comparison phase, fields such as transaction timestamps, log serial numbers, node numbers, and random checksums are automatically filtered. The comparison scope covers interface return codes, four decimal places of account balances, transaction status, customer information updates, cached hot account balances, MQ clearing messages, and distributed transaction lock status. It also correlates and verifies the status transmission between requests for initiation, accounting, notification, and reconciliation across the entire transaction process. Known errors such as distributed transaction timing issues and differences in character set sorting rules are directly matched to the script. For new errors such as tail-counting errors in batch interest calculations, LLM analysis generates new scripts, which are then manually verified to confirm the tail-counting errors in tens of thousands of interest calculation transactions.

[0070] This embodiment achieves 100% coverage of core transaction scenarios, identifies 23 accounting discrepancies caused by database incompatibility, and fixes all discrepancies before going live, achieving zero accounting errors during database migration.

[0071] like Figure 4As shown, the database table hierarchy of the rule engine in the differential decision module is as follows: the rule set version snapshot table rule_set is the top level, which is associated with the rule table rule. rule is then associated with the applicable scope table rule_applicable_scope, the matching condition table rule_match_condition, the rule-level normalization strategy table rule_normalization_strategy, and the hit action configuration table rule_action_config. The matching condition table is further extended to the numerical comparison rule table rule_numeric_value_rule and the conditional normalization verification table rule_normalization_strategy. Each sub-table is labeled with N, 0 / 1, and 1 to indicate the one-to-many, zero or one, and one-to-one association with the parent table.

[0072] Example of data structure for error matching script: The following is a partial SQL database table structure of the error matching script, used to illustrate the implementation of the script-priority difference attribution mechanism: CREATE TABLE `rule_set` ( `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Primary key ID', `rule_version` VARCHAR(20) NOT NULL COMMENT 'rule version number', `rule_generation_time` DATETIME NOT NULL COMMENT 'rule generation time', `llm_model_name` VARCHAR(100) NOT NULL COMMENT 'Model name of the generated rule', `llm_model_version` VARCHAR(50) NOT NULL COMMENT 'Model version number', `llm_input_context` JSON NOT NULL COMMENT 'List of input documents for generating rules', `created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Creation Time', `updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATECURRENT_TIMESTAMP COMMENT 'updated time', PRIMARY KEY (`id`), UNIQUE KEY `uk_rule_version` (`rule_version`), KEY `idx_generation_time` (`rule_generation_time`) )ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Error Matching Rule Set Main Table'; CREATE TABLE `rule` ( `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Primary key ID', `rule_set_id` BIGINT UNSIGNED NOT NULL COMMENT 'The ID of the rule set to which it belongs', `rule_id` VARCHAR(30) NOT NULL COMMENT 'Unique identifier for the rule', `rule_name` VARCHAR(200) NOT NULL COMMENT 'rule name', `rule_type` ENUM('FIELD_IGNORE','VALUE_NORMALIZE','ERROR_MATCH','TIME_COMPRESSION','CONFIG_SKIP') NOT NULL COMMENT 'Rule type', `business_method` VARCHAR(500) NOT NULL COMMENT 'Business method address, used for group matching', `priority` TINYINT UNSIGNED NOT NULL COMMENT 'rule priority', `status` ENUM('ACTIVE','INACTIVE','PENDING_REVIEW') NOT NULL DEFAULT'PENDING_REVIEW' COMMENT 'Rule status', `error_category` VARCHAR(100) DEFAULT NULL COMMENT 'Error reason category', `error_description` TEXT DEFAULT NULL COMMENT 'Detailed description of the error reason', `created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Creation Time', `updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATECURRENT_TIMESTAMP COMMENT 'updated time', PRIMARY KEY (`id`), UNIQUE KEY `uk_rule_id` (`rule_id`), KEY `idx_rule_set_id` (`rule_set_id`), KEY `idx_business_method` (`business_method`), KEY `idx_status` (`status`) )ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Error Matching Rules Table'; CREATE TABLE `rule_applicable_scope` ( `id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT COMMENT 'Primary key ID', `rule_id` BIGINT UNSIGNED NOT NULL COMMENT 'Related Rule ID', `call_chain_position` ENUM('DUBBO','HTTP','MYSQL','REDIS','RABBITMQ') NOT NULL COMMENT 'call chain position', `interface_name` VARCHAR(500) DEFAULT NULL COMMENT 'interface / method / table name', `direction` ENUM('REQUEST','RESPONSE','BOTH','INTERNAL') NOT NULL DEFAULT 'BOTH' COMMENT 'Message Direction', `field_path` VARCHAR(1000) DEFAULT NULL COMMENT 'Field path', `business_dimension_id` VARCHAR(200) DEFAULT NULL COMMENT 'Business Dimension ID', PRIMARY KEY (`id`), KEY `idx_rule_id` (`rule_id`) )ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='Rule Scope Table'; Developers can quickly locate defects introduced during refactoring or migration based on the variance report, fix them, and rerun the tests until all unexpected variances are identified or eliminated.

[0073] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A business process-level replay verification method based on end-to-end tracing data, characterized in that, The methods include: Collect a full-link dataset with TraceID tags generated by the business program in the historical environment. The full-link dataset includes input and output parameters, internal subroutine call order, timestamps and internal state transition data within the complete lifecycle of a single request. SQL operation data is collected by interceptors to collect complete parameterized logs of data operation statements within a specified range. Sensitive field desensitization processing is performed on the full-link dataset. Data cleaning is performed based on TraceID and business dimension ID to filter closed-loop test cases that can be generated by the business data within the process. Based on the business dimension ID, multiple requests with business associations are aggregated and sorted according to their actual execution order to form a business process-level test case set. A benchmark test program is generated based on the historical version of the business program and the full-link dataset. After running the program in the historical version environment to verify that the benchmark output is consistent with the output of the historical version program, it is used as the comparison benchmark. In the new version environment to be tested, the benchmark program initiates a replay test, assigning a TraceID consistent with the historical data to each test data. The simulation replay module simulates the dependent call response based on the TraceID according to the historical call order and time interval. The difference decision module compares the output and internal state change data of the new version program with the baseline, automatically filters differences in business-irrelevant fields, performs cause matching based on error matching scripts for unexpected differences, and calls a large language model to analyze the cause of the difference and generate a corresponding matching script when the matching fails, outputting a structured difference report.

2. The business process-level replay verification method based on end-to-end tracing data according to claim 1, characterized in that, The data cleaning and filtering closed-loop test cases specifically include: associating all TraceID requests under the same business process based on the business dimension ID, identifying the characteristics of the business process's initial request, filtering out incomplete lifecycle data that only contains subsequent operations such as queries, refunds, and order closures but lacks the initial transaction creation request, and retaining only the complete process that starts from the business initial request and whose business data can be generated automatically within the process as valid test cases, making the test case set independent of the existing transaction data at the time of collection; after the benchmark test program is generated, a benchmark self-verification process is executed: the data-cleaned test case set is run completely once in a historical version of the program that is consistent with the collection environment version, using the input parameters collected in the production environment as test input, comparing the output of the test environment with the collected output of the production environment and whether the internal state is completely consistent, verifying the completeness of the test dataset and the correctness of the initial error matching script, and only after all self-verifications pass can it be used for comparative testing of the new version of the program.

3. The business process-level replay verification method based on end-to-end tracing data according to claim 1, characterized in that, During the historical data collection phase, a globally unique TraceID is assigned to each request and is used throughout the entire call chain. Each time a test is replayed, a TraceID that is completely consistent with the historical data is forcibly specified to ensure that the link identifier remains consistent in multiple replays of the same test data, supporting data comparison and problem localization across tests.

4. The business process-level replay verification method based on end-to-end tracing data according to claim 1, characterized in that, A targeted interception mechanism is implemented using MyBatis Interceptor, limiting the interception scope to pre-configured data tables, Mapper interfaces, and data operation types. Complete SQL execution logs with parameters are recorded for operations within the interception scope. For query operations and other data operations outside the interception scope, only the call behavior is recorded without recording specific parameters, achieving a balance between data comparison completeness and collection performance. The logging logic of all request-response entry points is rewritten using the SPI mechanism, covering all middleware call data such as HTTP, Dubbo, RabbitMQ, databases, caches, and distributed locks. Structured log extraction and storage are achieved through ESPipeline. During the data collection phase, the time period of environment configuration parameter changes is recorded synchronously. Requests within the configuration change time period are skipped during the replay phase to avoid false positives caused by configuration differences.

5. The business process-level replay verification method based on end-to-end tracing data according to claim 1, characterized in that, The test dataset sources include real business data that has been successfully executed in historical production environments, and test data from testers constructing coverage edge scenarios and low-frequency abnormal scenarios in historical test environments. The de-identification process automatically identifies sensitive fields based on regular expressions and field semantics, and performs de-identification using an irreversible hash algorithm. Interface logic containing sensitive information that cannot be de-identified is excluded from the scope of the test dataset. The de-identified data maintains the integrity of business logic and supports the program's normal processing and judgment of business processes.

6. The business process-level replay verification method based on end-to-end tracing data according to claim 1, characterized in that, Business dimension IDs include identifiers such as order number, payment order number, user ID, and transaction serial number that can uniquely identify a complete business process. Multiple independent requests under the same business dimension ID are sorted according to the actual execution timestamps of the historical environment to form complete business process test cases across requests and services, which are suitable for verifying long processes and state dependencies in microservice systems. The replay phase strictly ensures the execution order of each request within the business process, ensuring the consistency of state transmission and business logic across requests.

7. The business process-level replay verification method based on end-to-end tracing data according to claim 1, characterized in that, During the replay process, when the simulation module encounters an external call request that is not covered by historical data, it immediately terminates the current test process and generates an alarm message containing the complete call context to notify the testers. For non-real-time requests with low time sensitivity, traffic compression technology is used to compress the request interval while ensuring that the request execution order remains unchanged, thereby shortening the replay time of long-cycle business processes and improving test execution efficiency.

8. The business process-level replay verification method based on end-to-end tracing data according to claim 1, characterized in that, The difference decision module has a built-in dynamic difference filtering mechanism based on pre-configured field filtering rules, which automatically identifies and ignores differences in fields such as timestamps, random numbers, server IDs, log IDs, and request IDs that affect the judgment of the correctness of business logic. The comparison scope of internal state transition data covers state changes of fields related to requests in the database, changes of key-value pairs in the cache, production and consumption of related messages in the message queue, and acquisition and release of distributed locks. The comparison performs correlation verification across the request business process dimension, verifying the state transmission and data consistency between multiple requests under the same business dimension ID, rather than comparing independently within a single request.

9. The business process-level replay verification method based on end-to-end tracing data according to claim 1, characterized in that, The specific process of large language model-assisted error analysis is as follows: when a new version is detected to be inconsistent with the baseline output and the difference field is not in the expected whitelist, the error analysis process is triggered. Prioritize grouping by the requested business method address, traverse and execute the maintained error matching scripts. If a script executes successfully and matches the current difference feature, return the corresponding error reason directly. If all existing scripts fail to match, input the request input parameters, output difference, business method address, and call chain context into the large language model. The large language model analyzes the difference reason and generates a new error matching script. The newly generated script is associated with the error reason and stored and maintained independently by grouping by business method address. The system records the error reason for each difference. During the manual review stage, only the logical correctness of the corresponding matching script needs to be verified to confirm the reasons for similar differences in batches.

10. The business process-level replay verification method based on end-to-end tracing data according to claim 1, characterized in that, During the replay process, each sub-service uses the collected production environment input parameters as the test environment input, and compares the consistency between the test environment output and the production environment output hop by hop. The test environment only needs to be configured with system parameters that do not change frequently. The test case set is independent of the existing business data and the complete production database snapshot at the time of collection, and there is no need to restore the full production data state.

Images