Database request playback method and equipment based on time axis

Abstract

The invention relates to the technical field of database performance testing and problem diagnosis, and provides a database request playback method and device based on a time axis, and the method comprises the steps: loading playback configuration, and building a time axis mapping relation between recording time and playback time; analyzing the historical request data, performing request screening and adaptation according to a mapping relation between playback configuration and a time axis, and calculating a target triggering moment of each request; scheduling control is carried out on the request, and the request is submitted and executed in a serialized mode according to a time sequence relation and a target triggering moment; and acquiring a request execution result and time consumption information, performing slow SQL judgment and selective result comparison, and outputting a statistical report when a termination condition is met. According to the invention, high-fidelity, controllable, stable and easily-verified playback of the database request behavior is realized.

Description

Technical Field

[0001] This invention relates to the field of database performance testing and problem diagnosis technology, and in particular to a time-axis-based database request replay method and device. Background Technology

[0002] In modern database operations, system migration, version upgrades, and performance optimization, the ability to accurately reproduce the execution process of historical database requests is of great significance. By replaying real business loads, database performance can be effectively evaluated, compatibility with new versions or environments can be verified, and online problems can be accurately reproduced, providing a reliable basis for problem localization and resolution. Therefore, database request replay technology has become an important tool in database operations, testing, and development.

[0003] Currently, most common database request replay methods are based on a "queue consumption + fixed rate" implementation model. This method typically loads historical request records into a memory queue, retrieves requests from the queue at fixed time intervals or a fixed rate, and executes them. While this method is simple to implement and can simulate a certain load pressure, it has significant shortcomings in realistically reproducing the time characteristics and concurrency structure of the source requests, specifically in the following aspects:

[0004] First, existing methods struggle to maintain the true temporal distribution and natural concurrency structure of source requests. In real-world production environments, database requests arrive randomly, burstily, and with temporal correlations. Fixed-rate replay cannot recreate the true time intervals between requests, nor can it accurately simulate the temporal relationships of concurrent requests from the source. Especially in business scenarios with alternating peak and off-peak hours, fixed-rate replay often leads to timing distortion and fails to accurately reflect the actual load characteristics of the system.

[0005] Second, in parallel playback scenarios, existing methods lack effective order preservation and backpressure control mechanisms. Without control, parallel playback can easily lead to disordered request execution order, especially for requests within the same session or transaction. Out-of-order execution may cause data consistency issues or abnormal states. In addition, due to the lack of backpressure control, when a large number of requests arrive in a short period of time, the executor can be overwhelmed by instantaneous high concurrency, leading to system resource exhaustion, a surge in response latency, or even service crashes, resulting in a significant deviation from the actual behavior of the source system.

[0006] Third, existing solutions have limited capabilities in the collaborative processing of recording, parsing, and playback, especially lacking unified management of the global timeline. In scenarios requiring simultaneous recording and playback or reproduction at source time intervals, existing methods often need to wait for recording to complete before playback, failing to achieve true streaming processing and resulting in significant end-to-end latency, thus failing to meet real-time or near-real-time verification requirements.

[0007] Fourth, existing technologies have significant shortcomings in terms of functional completeness and configurability. Most playback tools lack standardized processing for time units (such as nanoseconds, microseconds, milliseconds, and seconds), making it difficult to adapt to time recording sources with different precisions. Furthermore, they often lack systematic configuration and scheduling support in areas such as speed scaling, preloading window control, task backlog management, whitelist / blacklist filtering, user / pattern filtering, pattern mapping, slow SQL detection, and query result comparison. This results in poor controllability, weak verification capabilities, and insufficient adaptability during the playback process.

[0008] Chinese patent publication number CN121070977A discloses a method, medium, product and device for replaying database load files. This solution solves the problem of local compensation for execution time drift, but has significant shortcomings in global timeline management, concurrency control, functional integrity, streaming processing and systematic design.

[0009] Therefore, how to provide a database request replay method that can be based on a unified timeline and support multiple functions such as time sequence preservation, concurrency control, back pressure management, dynamic filtering, and result verification, so as to achieve high-fidelity, controllable, and stable reproduction of source database request behavior and meet the diverse needs of modern database operation and maintenance and testing, has become an urgent technical problem to be solved. Summary of the Invention

[0010] In view of this, in order to overcome the shortcomings of the prior art, the present invention aims to provide a time-axis-based database request playback method and device.

[0011] According to a first aspect of the present invention, a time-axis-based database request playback method is provided, the method comprising:

[0012] Step S1: Load the playback configuration and establish a timeline mapping relationship between the recording time and the playback time;

[0013] Step S2: Parse historical request data, filter and adapt requests according to replay configuration and timeline mapping, and calculate the target trigger time for each request;

[0014] Step S3: Schedule and control the requests, and submit them for execution in a sequential manner according to the target trigger time;

[0015] Step S4: Collect the request execution results and time information, determine slow SQL and compare selective results, and output a statistical report when the termination conditions are met.

[0016] Optionally, the time-axis-based database request playback method of the present invention includes step S1: loading a playback configuration containing time units, speed ratios, preloading windows, backlog limits and filtering rules; normalizing the timestamps and durations in the recording stream according to the playback configuration; and establishing a time-axis mapping relationship between the recording start time and the playback start time.

[0017] Optionally, in the time-axis-based database request playback method of the present invention, step S1, establishing a time-axis mapping relationship between recording time and playback time, includes:

[0018] Read the time unit parameters of the normalized baseline used to specify the timestamp and duration from the playback configuration;

[0019] The original timestamp and original duration of each historical request in the record stream are converted into a unified numerical representation according to the time unit to form standardized time data.

[0020] Determine the start time of the record, which is the normalized timestamp of the first request processed in the record stream;

[0021] Set the current system time or a fixed time specified in the configuration as the playback start time, and establish a linear mapping relationship between the recording start time and the playback start time.

[0022] Optionally, in the time-axis-based database request playback method of the present invention, step S2 includes:

[0023] Based on the configured whitelist and blacklist rules, the request model is matched and judged against the session identifier, user identifier, or database pattern identifier, and requests that meet the whitelist conditions and are not excluded by the blacklist are filtered.

[0024] Based on the configured pattern mapping table, replace the source database pattern name of the filtered requests with the corresponding pattern name of the target database.

[0025] For each filtered request, calculate the time offset between its normalized timestamp and the record start time. Divide the time offset by the configured speed ratio parameter for scaling. Add the scaled offset to the playback start time to obtain the target trigger time of the request on the playback timeline.

[0026] Optionally, in the time-axis-based database request playback method of the present invention, step S3 includes:

[0027] The scheduler prefetches and schedules requests that fall within the preloading window at the time of target triggering.

[0028] The scheduler continuously monitors the number of tasks in the queue of tasks to be executed. When the number exceeds the configured backlog upper limit threshold, it suspends the retrieval of new requests from upstream until the number of tasks in the queue falls back below the threshold before resuming prefetching.

[0029] The scheduler groups requests based on session or transaction identifiers in the request model, and submits all requests within the same session or transaction group to the executor in the order of their target trigger times.

[0030] Optionally, in the time-axis-based database request replay method of the present invention, in step S3, the request is executed by the executor thread pool in the following manner:

[0031] The executor thread pool creates a fixed number of worker threads during initialization based on the configured parallel pool size parameter;

[0032] When the target trigger time of the request arrives, the scheduler encapsulates the request as a task object and submits it to the task queue of the executor thread pool.

[0033] The worker threads in the executor thread pool retrieve tasks from the task queue, establish a connection with the target database, execute the SQL statements or database operation instructions corresponding to the request, and obtain the execution results.

[0034] Optionally, in the time-axis-based database request replay method of the present invention, slow SQL is determined in step S4 as follows:

[0035] Record the actual execution time of each request from submission to the executor to the return of the result;

[0036] The actual execution time is compared with the configured slow SQL judgment threshold. If the actual execution time exceeds the threshold, the request is marked as slow SQL.

[0037] Collect information on all marked slow SQL requests, including request text, execution time, actual time elapsed, and session identifier, and store it in a slow SQL record list;

[0038] After the playback ends, a slow SQL analysis report is generated based on the list. The slow SQL analysis report includes the number of slow SQL queries, details of the top N slowest requests, and statistics on the time consumption distribution.

[0039] Optionally, in the time-axis-based database request replay method of the present invention, the selective result comparison in step S4 is performed in the following manner:

[0040] After the executor thread pool obtains the execution result of the request, it extracts the expected result or baseline result corresponding to the request from the record stream;

[0041] The current execution result is compared with the expected result field by field or as a whole. The comparison includes data content, number of rows, column structure and execution status.

[0042] If a difference is detected, record the difference type, difference details, request identifier, and occurrence time, and save it to the difference record collection;

[0043] After playback, a difference report is generated based on the set of difference records. The difference report includes all requests with differences and their specific difference information.

[0044] Optionally, in the time-axis-based database request playback method of the present invention, the termination condition in step S4 includes one or a combination of the following:

[0045] The continuous running time has reached the configured maximum working duration limit;

[0046] All requests in the record stream have been processed, and there are no tasks waiting to be executed or currently being executed in the executor thread pool;

[0047] It receives a user-sent active stop command or an interrupt signal sent by an external system.

[0048] According to a second aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect of the present invention.

[0049] The time-axis-based database request replay method of this invention has the following beneficial technical effects:

[0050] 1. Achieve high-fidelity time-series reproduction: By introducing a unified timeline management mechanism, the timestamps and durations of source requests are standardized, and time offsets are precisely mapped based on a configurable speed ratio, enabling the accurate reproduction of the time distribution and interval characteristics of source requests. This overcomes the time-series distortion problem caused by traditional fixed-rate playback modes, fully preserving the natural arrival patterns and concurrency structure of source requests during playback, thus providing a highly approximate real load environment for performance testing, capacity assessment, and problem reproduction.

[0051] 2. Achieve flexible and controllable concurrent execution: Supports both serial and parallel replay strategies, and can enable a sequence-preserving mechanism at the session or transaction level as needed. When sequence preservation is enabled, requests within the same session or transaction will be executed serially, avoiding state disorder caused by concurrent overlap; when sequence preservation is disabled, they will be executed concurrently naturally according to the timeline, fully leveraging the multi-threaded execution capabilities. This configurable concurrency control mechanism makes the method of this invention suitable for business scenarios requiring strict order, and can also effectively support the stress testing needs of high concurrency loads.

[0052] 3. Achieving stable system throughput and resource protection: A backpressure control mechanism combining a preloading window and a task backlog limit effectively smooths request traffic, preventing instantaneous peaks from overwhelming the executor. The preloading window limits the visibility of future tasks, while task backlog control pauses task prefetching when the queue length exceeds a threshold, resuming once the execution pressure eases. This dynamic adjustment mechanism ensures the stability and controllability of the replay process under various load scenarios, preventing system crashes or performance degradation due to resource overload.

[0053] 4. Comprehensive Result Verification and Diagnostic Support: Built-in slow SQL detection and query result comparison functions can automatically identify requests with poor execution performance based on preset thresholds, and support difference detection between replay results and results recorded on the source end. This not only helps in locating and analyzing performance bottlenecks, but also effectively verifies data consistency and business correctness in scenarios such as migrating from old to new systems and version upgrades, significantly improving the verifiable value of the replay process and the efficiency of problem diagnosis.

[0054] 5. Enables end-to-end streaming collaborative processing: Supports pipelined collaborative work across recording, parsing, and playback stages, enabling low-latency processing with simultaneous recording and playback. Through dynamic maintenance of a unified timeline and real-time response from the scheduler, it can reproduce ongoing database requests in near real-time, providing effective technical support for continuous monitoring, real-time load testing, and rapid problem localization.

[0055] 6. Highly configurable filtering and adaptation capabilities: Provides multi-level, customizable filtering rules, including whitelist and blacklist filtering based on session, user, and database schema, and supports schema mapping between the source and target ends. Users can flexibly select the playback range, adapt to different database environments, and effectively improve the applicability and scenario coverage of the method. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart illustrating a time-axis-based database request playback method according to an exemplary embodiment 1 of the present invention.

[0058] Figure 2 A schematic diagram of the system architecture for implementing the time-axis-based database request playback method according to Exemplary Example 2 of the present invention;

[0059] Figure 3 This is a flowchart illustrating a time-axis-based database request playback method according to Exemplary Embodiment 3 of the present invention.

[0060] Figure 4 This is a schematic diagram of the structure of the device provided by the present invention. Detailed Implementation

[0061] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0062] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0063] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0064] Example 1

[0065] Exemplary embodiment 1 of the present invention provides a database request replay method based on a timeline. Figure 1 This is a flowchart illustrating a time-axis-based database request playback method according to Exemplary Embodiment 1 of the present invention, as shown below. Figure 1 As shown, in this embodiment, the time-axis-based database request replay method is implemented according to the following steps:

[0066] Step S1: Load the playback configuration and establish a timeline mapping relationship between the recording time and the playback time.

[0067] In this embodiment, the playback configuration includes time unit, speed ratio, preload window, backlog limit and filtering rules. After loading the playback configuration, the timestamps and durations in the recording stream are normalized according to the playback configuration, and a time axis mapping relationship between the recording start time and the playback start time is established.

[0068] In this embodiment, the time unit is used to standardize the recording of timestamps and durations into a unified metric (ns / us / ms / s). The speed ratio is used to control the scaling of the time axis, supporting acceleration / deceleration reproduction. The preloading window limits the range of future tasks visible to the scheduler. The backlog limit is used to prevent excessive task accumulation from causing executor blocking.

[0069] As an optional example, in this embodiment, the s-parameter used to specify the normalized baseline for the timestamp and duration is read from the playback configuration; the original timestamp and original duration of each historical request in the recording stream are converted into a unified numerical representation according to the time unit to form normalized time data; the recording start time is determined, which is the normalized timestamp of the first processed request in the recording stream; the current system time or a fixed time specified by the configuration is set as the playback start time, and a linear mapping relationship is established between the recording start time and the playback start time.

[0070] Step S2: Parse historical request data, filter and adapt requests according to replay configuration and timeline mapping, and calculate the target trigger time for each request.

[0071] In this embodiment, based on the configured whitelist and blacklist rules, the session identifier, user identifier, or database pattern identifier contained in the request model is matched and judged to filter requests that meet the whitelist conditions and are not excluded by the blacklist; according to the configured pattern mapping table, the source database pattern name of the filtered requests is replaced with the corresponding pattern name of the target; for each filtered request, the time offset between its normalized timestamp and the recording start time is calculated, the time offset is divided by the configured speed ratio parameter for scaling, and the scaled offset is added to the playback start time to obtain the target trigger time of the request on the playback timeline.

[0072] Step S3: Schedule and control the requests, and submit them for execution in a sequential manner according to the target trigger time.

[0073] In this embodiment, step S3 includes preloading window management, backpressure control, and order maintenance mechanisms. Specifically, it is implemented as follows: The scheduler prefetches and schedules requests whose target trigger times fall within the preloading window range; the scheduler continuously monitors the number of tasks in the queue to be executed, and when this number exceeds the configured backlog upper limit threshold, pausing the retrieval of new requests from upstream is suspended until the number of tasks in the queue falls below the threshold before resuming prefetching; the scheduler groups requests according to the session identifier or transaction identifier in the request model, and submits all requests within the same session or transaction group serially to the executor in the order of their target trigger times. This embodiment uses serialization at the session or transaction level to avoid concurrent crossover.

[0074] It should be noted that in this embodiment, the executor thread pool executes the request in the following manner: during initialization, the executor thread pool creates a fixed number of worker threads according to the configured parallel pool size parameter; when the target trigger time of the request arrives, the scheduler encapsulates the request as a task object and submits it to the task queue of the executor thread pool; the worker threads in the executor thread pool retrieve the task from the task queue, establish a connection with the target database, execute the SQL statement or database operation instruction corresponding to the request, and obtain the execution result.

[0075] Step S4: Collect the request execution results and time information, determine slow SQL and compare selective results, and output a statistical report when the termination conditions are met.

[0076] In this embodiment, when determining slow SQL queries, the actual execution time of each request from submission to the executor to return a result is recorded. This actual execution time is compared with a configured slow SQL query threshold. If the actual execution time exceeds the threshold, the request is marked as a slow SQL query. Information is collected from all marked slow SQL queries, including request text, execution time, actual execution time, and session identifier, and stored in a slow SQL query record list. After playback, a slow SQL query analysis report is generated based on this list. The slow SQL query analysis report includes the number of slow SQL queries, details of the top N slowest requests, and statistics on the distribution of execution time.

[0077] In this embodiment, when performing selective result comparison, after the executor thread pool obtains the execution result of the request, the expected result or benchmark result corresponding to the request is extracted from the record stream; the current execution result is compared with the expected result field by field or as a whole, and the comparison content includes data content, number of rows, column structure and execution status; if a difference is detected, the difference type, difference details, request identifier and occurrence time are recorded and saved to the difference record set; after the playback is completed, a difference report is generated based on the difference record set, and the difference report includes all requests with differences and their specific difference information.

[0078] It should be noted that, in this embodiment, the termination condition includes one or a combination of the following:

[0079] The continuous running time has reached the configured maximum working duration limit;

[0080] All requests in the record stream have been processed, and there are no tasks waiting to be executed or currently being executed in the executor thread pool;

[0081] It receives a user-sent active stop command or an interrupt signal sent by an external system.

[0082] When any termination condition is met, the system first stops the scheduler from prefetching new requests, then waits for all existing tasks in the executor thread pool to complete, and finally safely shuts down the thread pool and releases all database connection resources. In this embodiment, the maximum working time setting is used to limit process remnants and ensure convergence in streaming scenarios.

[0083] Example 2

[0084] Exemplary embodiment 2 of the present invention provides a database request replay method based on a timeline. Figure 2 This is a schematic diagram of the system architecture for implementing the time-axis-based database request playback method according to Exemplary Embodiment 2 of the present invention. Figure 2 As shown, the recycling system in this embodiment includes the following modules:

[0085] Configuration module: Used to configure parameters such as data source mode (JSON / database), replay strategy (serial / parallel), order hold switch, parallel pool size, speed ratio, time unit, preload window, task backlog limit, whitelist / blacklist, user / pattern filtering, pattern mapping, slow SQL threshold, working time limit, reproduction by source time interval, and result comparison.

[0086] Record storage module: Used to store historical request data, provided in the form of files or databases.

[0087] The parsing module is used to convert records into request models (session identifier, request text, record timestamp, duration, category tags, etc.) and normalize them according to time units.

[0088] Filtering module: Used for filtering based on whitelists / blacklists or user / pattern filtering, as well as pattern mapping.

[0089] Timeline Management Module: Used to maintain the mapping relationship between "record start time" and "playback start time"; calculate the "target trigger time" for each request according to the speed ratio; maintain the processed and skipped counts.

[0090] Timeline scheduler: Used to maintain thread pools and task queues; prefetch tasks based on preload windows; trigger backpressure to pause prefetching when task backlog exceeds the limit; submit tasks to the executor according to the timeline trigger time; limit concurrent crossover at the session or transaction level when sequential maintenance is enabled.

[0091] Executor thread pool: Used to execute requests and return results.

[0092] Monitoring and statistics module: used for slow SQL judgment, sampling, selective result comparison and process statistics.

[0093] Results output module: Used to output statistics, slow SQL reports, and difference reports.

[0094] In practical applications, the method of this embodiment also supports a streaming processing mode of recording and playback simultaneously. Specifically, a streaming data channel is established between the parsing module and the recording module. The recording module captures source database requests in real time and writes them into the recording stream, while the parsing module continuously reads newly arriving request data from the recording stream. The timeline management module dynamically adjusts the timeline mapping based on the continuously updated recording stream and calculates the target trigger time for newly arriving requests in real time. The scheduler and executor schedule and execute based on the real-time calculated trigger time, enabling the playback process to closely follow the recording progress and achieve near real-time or near real-time request reproduction and verification.

[0095] In practical applications, the method of this embodiment also supports dynamic configuration updates during playback. Specifically, it provides a configuration management interface that allows dynamic modification of some configuration parameters during playback, including speed ratio, slow SQL threshold, and filtering rules. For dynamic adjustment of the speed ratio, the system will recalculate the target trigger time for all requests that have not yet been triggered based on the new ratio parameter. For dynamic updates of filtering rules, the system applies the new rules to subsequent request flows, while requests that have already entered the scheduling process are still processed according to the original rules. Dynamic configuration updates ensure that the playback process can flexibly adjust behavior strategies according to real-time needs without restarting.

[0096] Example 3

[0097] Exemplary embodiment 3 of the present invention provides a database request replay method based on a timeline. Figure 3 This is a flowchart illustrating a time-axis-based database request playback method according to Exemplary Embodiment 3 of the present invention, as shown below. Figure 3 As shown, the method of this embodiment is implemented according to the following steps:

[0098] 1. Load the configuration, standardize the time unit and speed parameters, and initialize the thread pool size and threshold;

[0099] 2. Initialize the timeline and determine the mapping between the recording start point and the playback start point;

[0100] 3. Parse the records and perform filtering and pattern mapping;

[0101] 4. Map the recording time offset to the playback trigger moment according to the speed ratio;

[0102] 5. Maintain the preload window and backlog limit, and dispatch tasks in sequence; serialize sessions / transactions while maintaining order;

[0103] 6. Execute requests and collect data; determine slow SQL queries and compare results; output statistics and reports;

[0104] 7. Terminate when the working time is reached or the input stream ends and the backlog is cleared.

[0105] The following is a pseudocode example of an exemplary implementation of method 2 according to the present invention:

[0106] Input: Configuration C, Recording Stream R

[0107] initialization:

[0108] Set time unit:

[0109] tsUnit=C.sqlTimestampUnit,durUnit=C.sqlDurationUnit

[0110] Set the speed ratio: speed=C.timeSpeedRatio

[0111] Set the preload window: preload=C.preloadWindowMs

[0112] Set the backlog cap: backlogMax=C.schedulerMaxBacklogTasks

[0113] Create a thread pool: size = C.maxPoolSize

[0114] Record start point: recordStart = timestamp of the first record (normalized)

[0115] Replay Start: replayStart = current time

[0116] deal with:

[0117] For each record in R, req:

[0118] If the session / user / pattern blacklist is matched or the whitelist is not matched => Skip

[0119] Application pattern mapping and target library configuration

[0120] ts = normalized(req.timestamp, tsUnit)

[0121] dur = standardization(req.duration, durUnit)

[0122] offset=ts-recordStart

[0123] fireTime=replayStart + offset / speed

[0124] If fireTime is not in [current time, current time + preload] => wait or poll

[0125] If the number of tasks to be executed exceeds backlogMax, then prefetching (backpressure) should be paused.

[0126] If sequential preservation is enabled => commit by session / transaction serialization

[0127] Otherwise => Submit concurrently in chronological order.

[0128] The actuator executes and collects the time and results.

[0129] If you enable results comparison => perform difference detection

[0130] Slow SQL Detection and TOPN Sampling

[0131] termination:

[0132] Reaching C.replayMaxTime or the end of the input stream and clearing the backlog => disabling scheduling and thread pool.

[0133] Output statistics and slow SQL reports

[0134] Example 4

[0135] Exemplary Example 4 of the present invention provides a database request replay method based on a timeline. The method of this embodiment is implemented according to the following steps:

[0136] 1. Load configuration: Read parameters such as replay strategy, speed ratio, time unit, parallel pool size, preload window, backlog limit, filtering rules, and working duration;

[0137] 2. Time parameter standardization: unify the recorded timestamps and durations to a specified unit, and determine the time axis scaling according to the speed ratio;

[0138] 3. Initialize the timeline: Determine the mapping relationship between the recording start point and the playback start point, and establish the trigger time calculation benchmark;

[0139] 4. Parse records: Read historical requests from the data source and construct a request model that includes session identifier, request text, timestamp, and duration;

[0140] 5. Filtering and Pattern Mapping: Perform whitelist / blacklist and user / pattern filtering to map source patterns to target patterns;

[0141] 6. Calculate the trigger time: Calculate the target execution time for each request using the formula: fireTime = replayStart + (ts - recordStart) / speed;

[0142] 7. Preload window determination: Only requests within the range of "current time to the upper limit of the preload window" are prefetched and scheduled; requests outside the window wait.

[0143] 8. Backlog and back pressure control: When the number of tasks to be executed exceeds the backlog limit, prefetching is paused and the executor is allowed to fall back to prevent throughput abnormalities and memory bloat;

[0144] 9. Sequence Preservation Strategy: When enabled, commits are serialized at the session or transaction level to avoid concurrent overlap; when disabled, concurrency is naturally resumed according to time sequence.

[0145] 10. Parallel dispatch and execution: Submit requests to the thread pool for parallel execution, collect execution results and time consumption information, and support maximum latency threshold tolerance to reduce frequent polling;

[0146] 11. Monitoring, statistics, and result comparison: Perform slow SQL judgment (threshold or time difference rule), TOPN sampling; when result comparison is enabled, detect and record the differences in query results;

[0147] 12. Termination Conditions and Closure: Terminate when the working time is reached or the input stream ends and the backlog is cleared, shut down the scheduler and thread pool, and output statistics and reports.

[0148] The time-axis-based database request replay method of this invention has the following beneficial technical effects:

[0149] 1. Achieve high-fidelity time-series reproduction: By introducing a unified timeline management mechanism, the timestamps and durations of source requests are standardized, and time offsets are precisely mapped based on a configurable speed ratio, enabling the accurate reproduction of the time distribution and interval characteristics of source requests. This overcomes the time-series distortion problem caused by traditional fixed-rate playback modes, fully preserving the natural arrival patterns and concurrency structure of source requests during playback, thus providing a highly approximate real load environment for performance testing, capacity assessment, and problem reproduction.

[0150] 2. Achieve flexible and controllable concurrent execution: Supports both serial and parallel replay strategies, and can enable a sequence-preserving mechanism at the session or transaction level as needed. When sequence preservation is enabled, requests within the same session or transaction will be executed serially, avoiding state disorder caused by concurrent overlap; when sequence preservation is disabled, they will be executed concurrently naturally according to the timeline, fully leveraging the multi-threaded execution capabilities. This configurable concurrency control mechanism makes the method of this invention suitable for business scenarios requiring strict order, and can also effectively support the stress testing needs of high concurrency loads.

[0151] 3. Achieving stable system throughput and resource protection: A backpressure control mechanism combining a preloading window and a task backlog limit effectively smooths request traffic, preventing instantaneous peaks from overwhelming the executor. The preloading window limits the visibility of future tasks, while task backlog control pauses task prefetching when the queue length exceeds a threshold, resuming once the execution pressure eases. This dynamic adjustment mechanism ensures the stability and controllability of the replay process under various load scenarios, preventing system crashes or performance degradation due to resource overload.

[0152] 4. Comprehensive Result Verification and Diagnostic Support: Built-in slow SQL detection and query result comparison functions can automatically identify requests with poor execution performance based on preset thresholds, and support difference detection between replay results and results recorded on the source end. This not only helps in locating and analyzing performance bottlenecks, but also effectively verifies data consistency and business correctness in scenarios such as migrating from old to new systems and version upgrades, significantly improving the verifiable value of the replay process and the efficiency of problem diagnosis.

[0153] 5. Enables end-to-end streaming collaborative processing: Supports pipelined collaborative work across recording, parsing, and playback stages, enabling low-latency processing with simultaneous recording and playback. Through dynamic maintenance of a unified timeline and real-time response from the scheduler, it can reproduce ongoing database requests in near real-time, providing effective technical support for continuous monitoring, real-time load testing, and rapid problem localization.

[0154] 6. Highly configurable filtering and adaptation capabilities: Provides multi-level, customizable filtering rules, including whitelist and blacklist filtering based on session, user, and database schema, and supports schema mapping between the source and target ends. Users can flexibly select the playback range, adapt to different database environments, and effectively improve the applicability and scenario coverage of the method.

[0155] like Figure 4 As shown, the present invention also provides a device including a processor 310, a communication interface 320, a memory 330 for storing processor-executable computer programs, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 implements the aforementioned time-axis-based database request playback method by running the executable computer program.

[0156] The computer program in memory 330, when implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0157] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected based on actual needs to achieve the purpose of this embodiment. Those skilled in the art can understand and implement this without any creative effort.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0159] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A time-axis-based database request replay method, characterized in that, The method includes: Step S1: Load the playback configuration and establish a timeline mapping relationship between the recording time and the playback time; Step S2: Parse historical request data, filter and adapt requests according to replay configuration and timeline mapping, and calculate the target trigger time for each request; Step S3: Schedule and control the requests, and submit them for execution in a sequential manner according to the target trigger time; Step S4: Collect the request execution results and time information, determine slow SQL and compare selective results, and output a statistical report when the termination conditions are met.

2. The time-axis-based database request playback method according to claim 1, characterized in that, Step S1 includes: loading the playback configuration containing time unit, speed ratio, preload window, backlog limit and filtering rules; normalizing the timestamps and duration in the recording stream according to the playback configuration; and establishing a time axis mapping relationship between the recording start time and the playback start time.

3. The time-axis-based database request playback method according to claim 2, characterized in that, In step S1, the timeline mapping relationship between recording time and playback time is established, including: Read the time unit parameters of the normalized baseline used to specify the timestamp and duration from the playback configuration; The original timestamp and original duration of each historical request in the record stream are converted into a unified numerical representation according to the time unit to form standardized time data. Determine the start time of the record, which is the normalized timestamp of the first request processed in the record stream; Set the current system time or a fixed time specified in the configuration as the playback start time, and establish a linear mapping relationship between the recording start time and the playback start time.

4. The time-axis-based database request playback method according to claim 1, characterized in that, Step S2 includes: Based on the configured whitelist and blacklist rules, the request model is matched and judged against the session identifier, user identifier, or database pattern identifier, and requests that meet the whitelist conditions and are not excluded by the blacklist are filtered. Based on the configured pattern mapping table, replace the source database pattern name of the filtered requests with the corresponding pattern name of the target database. For each filtered request, calculate the time offset between its normalized timestamp and the record start time. Divide the time offset by the configured speed ratio parameter for scaling. Add the scaled offset to the playback start time to obtain the target trigger time of the request on the playback timeline.

5. The time-axis-based database request playback method according to claim 1, characterized in that, Step S3 includes: The scheduler prefetches and schedules requests that fall within the preloading window at the time of target triggering. The scheduler continuously monitors the number of tasks in the queue of tasks to be executed. When the number exceeds the configured backlog upper limit threshold, it suspends the retrieval of new requests from upstream until the number of tasks in the queue falls back below the threshold before resuming prefetching. The scheduler groups requests based on session or transaction identifiers in the request model, and submits all requests within the same session or transaction group to the executor in the order of their target trigger times.

6. The time-axis-based database request playback method according to claim 1, characterized in that, In step S3, the request is executed through the executor thread pool in the following manner: The executor thread pool creates a fixed number of worker threads during initialization based on the configured parallel pool size parameter; When the target trigger time of the request arrives, the scheduler encapsulates the request as a task object and submits it to the task queue of the executor thread pool. The worker threads in the executor thread pool retrieve tasks from the task queue, establish a connection with the target database, execute the SQL statements or database operation instructions corresponding to the request, and obtain the execution results.

7. The time-axis-based database request playback method according to claim 1, characterized in that, In step S4, slow SQL queries are determined as follows: Record the actual execution time of each request from submission to the executor to the return of the result; The actual execution time is compared with the configured slow SQL judgment threshold. If the actual execution time exceeds the threshold, the request is marked as slow SQL. Collect information on all marked slow SQL requests, including request text, execution time, actual time elapsed, and session identifier, and store it in a slow SQL record list; After the playback ends, a slow SQL analysis report is generated based on the list. The slow SQL analysis report includes the number of slow SQL queries, details of the top N slowest requests, and statistics on the time consumption distribution.

8. The time-axis-based database request playback method according to claim 1, characterized in that, In step S4, the selective result comparison is performed as follows: After the executor thread pool obtains the execution result of the request, it extracts the expected result or baseline result corresponding to the request from the record stream; The current execution result is compared with the expected result field by field or as a whole. The comparison includes data content, number of rows, column structure and execution status. If a difference is detected, record the difference type, difference details, request identifier, and occurrence time, and save it to the difference record collection; After playback, a difference report is generated based on the set of difference records. The difference report includes all requests with differences and their specific difference information.

9. The time-axis-based database request playback method according to claim 1, characterized in that, In step S4, the termination conditions include one or a combination of the following: The continuous running time has reached the configured maximum working duration limit; All requests in the record stream have been processed, and there are no tasks waiting to be executed or currently being executed in the executor thread pool; It receives a user-sent active stop command or an interrupt signal sent by an external system.

10. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method according to any one of claims 1-9.

Images