Log playback method and device, computer equipment, storage medium and program product

Abstract

The invention discloses a log playback method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: analyzing a to-be-processed log file, and extracting current target transaction features in real time; according to the features of the current target transaction, adopting a trained lock prediction model to predict lock information in the current target transaction and a conflict probability between the current active transaction and the lock information; constructing a lock thermodynamic diagram according to the lock information and the conflict probability, and dividing a non-conflict area and a conflict area; merging the transactions in the non-conflict area, and allocating the transactions to a plurality of space-time fragments to concurrently execute playback tasks; the transactions in the conflict area are divided into a plurality of time window intervals, and if it is judged that no conflict exists in the target time window interval, the transactions in the target time window interval are allocated to a plurality of space-time fragments so as to concurrently execute the playback task. According to the technical scheme provided by the invention, the log playback speed can be increased, the resource consumption can be reduced, and the problem of concurrency conflicts can be effectively solved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of database technology, and in particular to a log playback method, apparatus, computer equipment, storage medium and program product. Background Technology

[0002] In database systems, the binary log (Binlog) is used to record database changes, such as data insertion, updates, and deletions. The Binlog plays a crucial role in data recovery, data replication, and data synchronization. However, with the continuous growth of data volume and the increasing complexity of business operations, traditional Binlog replay methods face numerous performance challenges. Firstly, when processing large amounts of Binlog data, the traditional single-threaded replay method is inefficient, leading to excessively long data recovery or synchronization times, severely impacting business continuity and availability. For example, in some large-scale e-commerce databases, a complete Binlog replay may take hours or even days, during which time the business may be unavailable or the data inconsistent. Secondly, long Binlog replay processes consume significant system resources, including CPU, memory, and disk I / O. This not only affects the performance of the database system itself but may also negatively impact other applications running on the same server. For instance, in resource-constrained cloud server environments, the Binlog replay process may cause excessive system load, resulting in slow response times or even malfunctions for other business applications. Furthermore, when attempting to use multi-threaded parallel replay of Binlog events, concurrency conflicts easily arise due to the dependencies between Binlog events, such as multiple operations within a transaction needing to be executed sequentially. Existing concurrency control mechanisms often require the introduction of numerous locks to handle these complex dependencies, further reducing replay concurrency performance. For instance, in some high-concurrency write database scenarios, frequent lock contention occurs during multi-threaded Binlog replay, resulting in replay speeds that are slower than single-threaded replays. In summary, existing Binlog replay technologies are slow, resource-intensive, and difficult to handle concurrency conflicts, failing to meet the ever-increasing demands of data processing. Summary of the Invention

[0003] This invention provides a log replay method, apparatus, computer device, storage medium, and program product to improve Binlog replay speed, reduce resource consumption, and effectively resolve concurrency conflict issues.

[0004] In a first aspect, embodiments of the present invention provide a log playback method, the method comprising: The log file to be processed is parsed, and the characteristics of the current target transaction are extracted in real time; Based on the characteristics of the current target transaction, the trained lock prediction model is used to predict the lock information in the current target transaction and the probability of conflict with the current active transaction. A lock heatmap is constructed based on the lock information and the conflict probability, and non-conflict zones and conflict zones are divided. Transactions in the non-conflict zone are merged and distributed to multiple spatiotemporal slices for concurrent execution of playback tasks; The transactions in the conflict zone are divided into multiple time window intervals. If it is determined that there is no conflict within the target time window interval, the transactions within the target time window interval are allocated to multiple spatiotemporal slices to execute the replay task concurrently.

[0005] Optionally, before predicting the lock information in the current target transaction and the probability of conflict with the currently active transaction using a trained lock prediction model based on the characteristics of the current target transaction, the method further includes: In production or load testing environments, enable full SQL audit logs and lock wait logs. Associate all SQL statements within the same transaction with their pre-compiled parameters, execution timestamps, and commit / rollback results; From the start of a transaction to its commit, extract the lock characteristics actually held and waited for by the transaction to form a transaction-lock relationship pair; The lock prediction model is trained using training sample data generated based on the transaction-lock relationship.

[0006] Optionally, the current target transaction characteristics include static characteristics, dynamic context characteristics, and historical behavior characteristics; the static characteristics include SQL statement fingerprints, display lock hints, and operation types and objects; the dynamic context characteristics include input parameter values / ranges and the current database status, which includes data distribution, index status, and system load; the historical behavior characteristics include historical lock matrices and historical conflict records.

[0007] Optionally, the method further includes: Real-time monitoring of actual lock conflicts; If a conflict is detected between two transactions, one of the transactions will be rolled back and replayed. If the number of conflicts in a target spatiotemporal segment exceeds a preset threshold, the target spatiotemporal segment is split based on the spatiotemporal distribution of the conflicts, and / or the time window interval is adjusted.

[0008] Optionally, the method further includes: If a conflict is detected, new training sample data is generated based on the conflicting transaction, and the lock prediction model is incrementally trained.

[0009] Optionally, parsing the log file to be processed includes: The log format of the log file to be processed is identified based on predefined syntax rules and semantic templates; Based on the identification results, the log file to be processed is split into multiple sub-files; The multiple sub-files are parsed in parallel.

[0010] Secondly, embodiments of the present invention also provide a log playback device, the device comprising: The feature extraction module is used to parse the log file to be processed and extract the features of the current target transaction in real time; The lock information and conflict prediction module is used to predict the lock information in the current target transaction and the probability of conflict with the current active transaction based on the characteristics of the current target transaction using a trained lock prediction model. A lock heatmap construction module is used to construct a lock heatmap based on the lock information and the conflict probability, and to divide the area into non-conflict zones and conflict zones; The non-conflict zone transaction replay module is used to merge transactions in the non-conflict zone and distribute them to multiple spatiotemporal slices for concurrent execution of replay tasks. The conflict zone transaction replay module is used to divide the transactions in the conflict zone into multiple time window intervals. If it is determined that there is no conflict within the target time window interval, the transactions within the target time window interval are allocated to multiple spatiotemporal slices to execute the replay task concurrently.

[0011] Thirdly, embodiments of the present invention also provide a computer device, the computer device comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the log playback method provided in any embodiment of the present invention.

[0012] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the log playback method provided in any embodiment of the present invention.

[0013] Fifthly, embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the log playback method provided in any embodiment of the present invention.

[0014] This invention provides a log replay method. First, the log file to be processed is parsed, and the characteristics of the current target transaction are extracted in real time. Then, based on the extracted characteristics, a trained lock prediction model is used to predict the lock information in the current target transaction and the probability of conflict with currently active transactions. A lock heatmap is then constructed based on the obtained lock information and conflict probabilities, dividing the heatmap into non-conflict and conflict zones. Transactions in the non-conflict zones are merged for direct concurrent execution of the replay task. Transactions in the conflict zones are further divided into multiple time windows, and when no conflict is determined within a target time window, the replay task is executed concurrently for those transactions. The log replay method provided by this invention, through lock and conflict prediction, can identify concurrently executable transactions in the log, thereby improving Binlog replay speed, reducing resource consumption, and effectively resolving concurrency conflict issues to meet the ever-increasing data processing demands. Attached Figure Description

[0015] Figure 1 This is a flowchart of the log playback method provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the log playback device provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0016] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0017] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0018] Example 1 Figure 1This is a flowchart of the log replay method provided in Embodiment 1 of the present invention. This embodiment is applicable to scenarios requiring log replay, such as data recovery, data replication, and data synchronization, and is particularly suitable for scenarios involving master-slave synchronization and significant synchronization latency, thus reducing latency to some extent. This method can be executed by the log replay device provided in this embodiment of the invention. This device can be implemented in hardware and / or software, and is generally integrated into a computer device. Figure 1 As shown, the specific steps include the following: S11. Parse the log file to be processed and extract the characteristics of the current target transaction in real time.

[0019] S12. Based on the characteristics of the current target transaction, a trained lock prediction model is used to predict the lock information in the current target transaction and the probability of conflict with the current active transaction.

[0020] S13. Construct a lock heatmap based on the lock information and the conflict probability, and divide the area into non-conflict zones and conflict zones.

[0021] S14. Merge the transactions in the non-conflict zone and distribute them to multiple spatiotemporal slices to execute the playback task concurrently.

[0022] S15. Divide the transactions in the conflict zone into multiple time window intervals. If it is determined that there is no conflict within the target time window interval, allocate the transactions within the target time window interval to multiple spatiotemporal slices to execute the playback task concurrently.

[0023] Specifically, the Binlog stream can be parsed in real time. When a new transaction is found during parsing, it is treated as the current target transaction. Optionally, parsing the log file to be processed includes: identifying the log format of the log file to be processed according to predefined syntax rules and semantic templates; splitting the log file to be processed into multiple sub-files based on the identification results; and parsing the multiple sub-files in parallel. Because Binlog log formats are diverse and complex, and the Binlog formats generated by different versions of databases may differ, this poses a challenge to log parsing. By using a log format recognition algorithm based on rule matching and semantic analysis, different Binlog log formats can be quickly and accurately identified. Specifically, by using predefined syntax rules and semantic templates, the type (such as transaction start, data change, transaction end, etc.) and structure of log records can be quickly determined during log reading, avoiding multiple scans of log content and complex format conversion operations in traditional parsing methods, greatly improving parsing efficiency. For example, for ROW, STATEMENT, and MIXED, matching can be performed based on the characteristics of each log, such as STATEMENT format using SQL records, ROW using row records, etc. When reading events, you can jump to the corresponding processing module based on the event type, without needing to fully parse all fields of all events. Additionally, the fixed length of the Binlog event header can be used to skip unnecessary events. For example, when only row events are relevant, you can read the event header and skip other events based on its length. Furthermore, based on the format recognition results, the log file to be processed can be split into multiple sub-files according to preset rules, such as transaction blocks, time segments, etc., and then distributed to multiple parallel parsing threads for processing. Each parsing thread independently parses its assigned log sub-file to generate corresponding parsing results, such as parsed transaction objects, data change operation lists, etc. By adopting a parallel processing approach, the computing resources of multi-core CPUs can be fully utilized, significantly reducing the overall parsing time of Binlog logs.

[0024] Furthermore, the current target transaction features can be extracted in real time based on the parsing results. Optionally, the current target transaction features include static features, dynamic context features, and historical behavior features; the static features include SQL statement fingerprints, display lock hints, and operation types and objects; the dynamic context features include input parameter values / ranges and the current database state, which includes data distribution, index status, and system load; the historical behavior features include historical lock matrices and historical conflict records. The SQL statement fingerprint is a unique identifier generated by hashing the parameterized SQL text (e.g., SELECT * FROM orders WHERE id = ?) using a hash algorithm (e.g., MD5, SHA-256), and can be used to associate historical execution patterns. The lock hints are lock pattern intentions extracted directly from the SQL syntax tree, such as row-level exclusive lock intentions for `FOR UPDATE`, row-level shared lock intentions for `FOR SHARE / LOCK IN SHARE MODE`, table-level metadata exclusive lock intentions for `ALTER TABLE`, and possible snapshot read / shared lock intentions (which need to be combined with the isolation level) for regular `SELECT` statements. The lock patterns can be extracted during the SQL pre-compilation stage so that the lock range can be predicted before the transaction is actually committed. Operation type and object can include the parsed operations (such as `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `DDL`) and the target table, partition name, etc. Input parameter values / ranges are the bind variable values ​​in the pre-compiled SQL. For range queries (such as `id BETWEEN ? AND ?`), this is crucial for predicting the lock range. Data distribution can include statistics of the target table (such as the number of rows, key value distribution histogram), which can be used to estimate the number of data pages or rows that the `WHERE` condition may involve. Index information can include whether the SQL uses an index and which index is used. Locks are usually applied to indexes, and the index information can determine the physical range of the lock. System load can include the number of currently active transactions, lock wait queue length, etc., and can be used to assess the overall system conflict pressure. The historical lock matrix can be extracted from the historical log, specifically the set of locks actually acquired during past executions of the corresponding SQL fingerprint + parameter pattern (e.g., locking pages P1 and P2 of table A, and rows R1-R100). Historical conflict records can include which other transaction patterns the corresponding transaction pattern has conflicted with (identified by SQL fingerprint) and the conflict frequency.

[0025] After extracting the features of the current target transaction, a multi-dimensional feature vector can be constructed based on these features as input to a lock prediction model (such as LightGBM). The lock prediction model can dynamically predict the lock information in the current target transaction and the probability of conflict with the currently active transaction. Specifically, the prediction can be completed before the database transaction officially enters the commit phase and actually acquires locks, providing a basis for decision-making regarding transaction scheduling, pre-rollback, or resource reservation. Specifically, the lock prediction model can output a structured prediction result, where the lock information can include the lock mode and lock range. The predicted lock mode is {lock type: probability}, such as {row-level exclusive lock (X): 0.95, intention exclusive lock (IX): 0.95, table-level exclusive lock: 0.02}. The lock range can be predicted according to a specified granularity. At a coarse granularity, the list of data objects involved can be predicted, such as [table: Orders, index: idx_customer]. At a fine granularity, the range of key values ​​or the list of page IDs that may be locked can be predicted, such as [primary key value: 1000-1500]. A scalar value with a conflict probability of 0-1 can represent the overall probability that the current target transaction will have a lock conflict with the set of active transactions in the current system. In addition, it can also predict the main conflict opponents, such as the IDs or characteristics of the 1-3 currently active transactions that are most likely to conflict with the current target transaction.

[0026] This allows for the continuous construction of a lock heatmap in real-time based on newly predicted lock information and conflict probabilities. The horizontal axis represents time windows, and the vertical axis represents data sharding. Specifically, hotspots with high conflict probabilities and / or frequently occurring locks are designated as conflict zones, while other colder areas are designated as non-conflict zones. Transactions within the non-conflict zones can then be merged in real-time and concurrently replayed. Transactions within conflict zones can be further divided along a time dimension into multiple overlapping time window intervals. The presence of conflicts within each time window interval is then determined based on the predicted conflict probabilities. If no conflicts exist, transactions within the corresponding target time window interval can be concurrently replayed, enabling cross-time-period transaction pipeline execution. If conflicts occur, the conflicting transactions can be replayed serially. For example, transaction T1's original operation is [R1,R2,C1,W1,W2,C2,R3], divided into three time window intervals: window A includes [R1,R2,C1], window B includes [C1,W1,W2], and window C includes [W2,C2,R3]. The subsequent transaction T2's original operation is [R3,R4,C3,W3,W4,C4,R5], divided into three time window intervals: [R3,R4,C3], [C3,W3,W4], and [W4,C4,R5]. The time window for [R3,R4,C3] spans both windows A and B, and the time window for [C3,W3,W4]... The time window spans windows B and C. [C1] of T1 and [R3,R4,C3] of T2 can run in parallel (no data dependency), while [W2] of T1 and [W4] of T2 must run serially (due to write conflicts to the same table). An exemplary time window interval has a window size of 50 operation steps, an overlap ratio of 50% (25 overlapping operations), a minimum safe interval of 5 operations (to ensure dependencies), and a maximum advance of 100 operations (to prevent starvation). For concurrently executable replay tasks, transactions can be distributed to various spatiotemporal slices for execution by corresponding threads. Each slice can contain 50-200 conflict-free predictive transactions and can enable look-ahead execution, allowing out-of-order execution of transactions within a slice.

[0027] Based on the above technical solution, optionally, before predicting the lock information in the current target transaction and the probability of conflict with the currently active transaction using the trained lock prediction model according to the current target transaction characteristics, the method further includes: enabling full SQL audit logs and lock wait logs (such as MySQL's performance_schema related tables) in a production or stress testing environment; associating all SQL statements within the same transaction with their pre-compiled parameters, execution timestamps, and commit / rollback results; extracting the lock characteristics actually held and waited by the transaction from the start of the transaction to the commit period to form transaction-lock relationship pairs; and training the lock prediction model using training sample data generated based on the transaction-lock relationship pairs. Each transaction execution instance can be used as a training sample, and its transaction characteristics can refer to the content of the current target transaction characteristics mentioned above, i.e., the corresponding time snapshot. Its labels can be the actual lock mode, lock range, and conflict results generated by the corresponding transaction, etc. Furthermore, feature engineering can be used to transform training sample data into feature vectors suitable for model input. This includes serialization, range quantization, and historical feature aggregation. Serialization converts non-numerical features (such as table names and SQL fingerprints) into numerical IDs or embedding vectors. Range quantization can bin parameter values, transforming continuous ranges into discrete categories, such as converting customer_id BETWEEN 1000 AND 2000 into range_bucket_10. Historical feature aggregation can bin the data according to SQL fingerprints and parameters, aggregating historical lock ranges (such as finding the union or calculating frequency) and conflict counts to generate statistical features. Training a lock prediction model using historical execution patterns allows dependency analysis to be completed before the transaction enters the commit phase.

[0028] Based on the above technical solution, the method may optionally further include: real-time monitoring of actual lock conflicts; if two transactions are detected to be in conflict, one of the transactions is selected for rollback and replay; if the number of conflicts in a target spatiotemporal slice exceeds a preset threshold, the target spatiotemporal slice is split based on the spatiotemporal distribution of the conflicts, and / or the time window interval is adjusted.

[0029] Specifically, lock prediction models may contain errors, meaning the model incorrectly assumes two transactions will not conflict when a conflict actually occurs. This issue can be addressed by adding a smart replay compensation mechanism. Specifically, this can be monitored in real-time during the replay process by querying lock status tables such as INNODB_TRX, INNODB_LOCKS, and INNODB_LOCK_WAITS to detect actual lock contention. If a lock conflict is detected, it indicates an error in the lock prediction model, and different compensation mechanisms can be triggered based on the conflict type.

[0030] If a conflict is detected between two transactions, a local transaction rollback and replay mechanism can be used to select one of the transactions for rollback and replay. The replay process will then be executed according to the above procedure again. It may be divided into another time-space slice for execution. Preferably, the transaction that started later is selected, or it can be selected according to business priority. For example, suppose two transactions T1 and T2 are allocated to the same spatiotemporal shard (S1). The model predicts no conflict, but during actual execution, T2 requests a lock while T1 holds the lock (or vice versa), causing T2 to wait. The monitoring system detects this lock wait (by querying the lock status table, it finds that T2's status is LOCK WAIT and the waiting time exceeds the threshold). At this point, it can be decided to roll back T2 (starting later, rollback has a lower cost). After the rollback, a re-evaluation is performed, that is, T2's features are re-extracted (some features may change, such as data distribution, system load, etc.), and the model (specifically, the updated model described below) is used to re-predict the probability of conflict between T2 and the currently active transaction (which may include T1). Then, based on the new prediction results, T2 is allocated to a new spatiotemporal shard (such as S2) and scheduled for execution.

[0031] If the number of conflicts in a target spatiotemporal partition exceeds a preset threshold, indicating frequent actual conflicts, it suggests that the partition is not properly divided. In such cases, a dynamic spatiotemporal partition splitting mechanism can be used to further split the target spatiotemporal partition or adjust the corresponding time window interval. For example, if actual conflicts occur frequently in a partition (such as S1), the system can record these conflicts and analyze the spatial and temporal distribution of the conflicting transactions. If it is found that conflicts always occur between transactions within a certain data range (such as customer_id between 1000 and 2000), this data range can be split from the original partition to form a new partition, while adjusting the range of the original partition accordingly. Alternatively, from a temporal perspective, if conflicts always occur within a certain time window, the size or overlap ratio of that time window can be adjusted.

[0032] Based on the above technical solution, optionally, the method further includes: if a conflict is detected, generating new training sample data based on the conflicting transaction, and incrementally training the lock prediction model. Specifically, based on the intelligent replay compensation mechanism, subsequent prediction accuracy can be improved by updating the model parameters. Specifically, conflict samples can be collected to form a new training dataset, and these new data can be used for incremental training of the model at regular intervals (e.g., hourly). Alternatively, the model can be retrained periodically (e.g., daily) using the full dataset (both new and old data). For each actual conflict, the corresponding transaction features (as input) and the conflicting object (as a label, i.e., the conflict the model should predict) can be recorded. Then, data cleaning and feature engineering can be performed, similar to the initial model training. Incremental updates can then be performed using online learning and other methods, and a validation set can be used to evaluate the performance of the updated model. Finally, the updated model is deployed to the production environment. Through continuous monitoring, compensation, and updates, the model can gradually learn previously unconsidered conflict patterns, thereby gradually improving prediction accuracy and reducing the occurrence of actual conflicts. In addition, in practical applications, since model updates themselves may introduce instability, processes such as A / B testing or canary releases can be added.

[0033] The technical solution provided in this invention first parses the log file to be processed and extracts the characteristics of the current target transaction in real time. Then, based on the extracted characteristics, a trained lock prediction model is used to predict the lock information in the current target transaction and the probability of conflict with currently active transactions. A lock heatmap is then constructed based on the obtained lock information and conflict probabilities, dividing the heatmap into non-conflict and conflict zones. Transactions in the non-conflict zones are merged for direct concurrent execution of replay tasks. Transactions in the conflict zones are further divided into multiple time window intervals, and replay tasks are concurrently executed for transactions within a target time window interval when no conflict is detected. Through lock and conflict prediction, concurrently executable transactions in the log can be identified, thereby improving Binlog replay speed, reducing resource consumption, and effectively resolving concurrency conflict issues to meet the ever-increasing data processing demands.

[0034] Example 2 Figure 2 This is a schematic diagram of the log playback device provided in Embodiment 2 of the present invention. This device can be implemented in hardware and / or software, and is generally integrated into a computer device to execute the log playback method provided in any embodiment of the present invention. Figure 2 As shown, the device includes: The feature extraction module 21 is used to parse the log file to be processed and extract the current target transaction features in real time; The lock information and conflict prediction module 22 is used to predict the lock information in the current target transaction and the probability of conflict with the current active transaction based on the characteristics of the current target transaction using a trained lock prediction model. The lock heatmap construction module 23 is used to construct a lock heatmap based on the lock information and the conflict probability, and to divide the non-conflict area and the conflict area. The non-conflict zone transaction replay module 24 is used to merge transactions in the non-conflict zone and distribute them to multiple spatiotemporal slices for concurrent execution of replay tasks. The conflict zone transaction replay module 25 is used to divide the transactions in the conflict zone into multiple time window intervals. If it is determined that there is no conflict within the target time window interval, the transactions within the target time window interval are allocated to multiple spatiotemporal slices to execute the replay task concurrently.

[0035] The technical solution provided in this invention first parses the log file to be processed and extracts the characteristics of the current target transaction in real time. Then, based on the extracted characteristics, a trained lock prediction model is used to predict the lock information in the current target transaction and the probability of conflict with currently active transactions. A lock heatmap is then constructed based on the obtained lock information and conflict probabilities, dividing the heatmap into non-conflict and conflict zones. Transactions in the non-conflict zones are merged for direct concurrent execution of replay tasks. Transactions in the conflict zones are further divided into multiple time window intervals, and replay tasks are concurrently executed for transactions within a target time window interval when no conflict is detected. Through lock and conflict prediction, concurrently executable transactions in the log can be identified, thereby improving Binlog replay speed, reducing resource consumption, and effectively resolving concurrency conflict issues to meet the ever-increasing data processing demands.

[0036] Based on the above technical solution, optionally, the device further includes: The log collection module is used to enable full SQL audit logs and lock wait logs in the production or stress testing environment before predicting the lock information in the current target transaction and the probability of conflict with the current active transaction using a trained lock prediction model based on the characteristics of the current target transaction. The session association module is used to associate all SQL statements within the same transaction with their pre-compiled parameters, execution timestamps, and commit / rollback results; The relationship pair generation module is used to extract the lock characteristics actually held and waited for by a transaction from the start of the transaction to the commit, and form transaction-lock relationship pairs. The model training module is used to train the lock prediction model based on the generated training sample data according to the transaction-lock relationship.

[0037] Based on the above technical solution, optionally, the current target transaction characteristics include static characteristics, dynamic context characteristics, and historical behavior characteristics; the static characteristics include SQL statement fingerprints, display lock hints, and operation types and objects; the dynamic context characteristics include input parameter values / ranges and the current database status, the current database status including data distribution, index status, and system load; the historical behavior characteristics include historical lock matrices and historical conflict records.

[0038] Based on the above technical solution, optionally, the device further includes: The lock conflict monitoring module is used to monitor actual lock conflicts in real time. The rollback and replay module is used to select one of the transactions to roll back and replay if a conflict between two transactions is detected. The spatiotemporal splitting module is used to split the target spatiotemporal segment based on the spatiotemporal distribution of the conflict if the number of conflicts in the target spatiotemporal segment exceeds a preset threshold, and / or adjust the time window interval.

[0039] Based on the above technical solution, optionally, the device further includes: An incremental training module is used to generate new training sample data based on the conflicting transactions if a conflict is detected, and to incrementally train the lock prediction model.

[0040] Based on the above technical solution, optionally, the feature extraction module 21 is specifically used for: The log format of the log file to be processed is identified based on predefined syntax rules and semantic templates; Based on the identification results, the log file to be processed is split into multiple sub-files; The multiple sub-files are parsed in parallel.

[0041] The log playback device provided in the embodiments of the present invention can execute the log playback method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0042] It is worth noting that in the above-described embodiments of the log playback device, the various units and modules are divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0043] Example 3 Figure 3 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention, showing a block diagram of an exemplary computer device suitable for implementing the embodiments of the present invention. Figure 3 The computer device shown is merely an example and should not be construed as limiting the functionality or scope of the embodiments of the present invention. Figure 3 As shown, the computer device includes a processor 31, a memory 32, an input device 33, and an output device 34; the number of processors 31 in the computer device can be one or more. Figure 3 Taking a processor 31 as an example, the processor 31, memory 32, input device 33, and output device 34 in a computer device can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.

[0044] The memory 32, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the log playback method in this embodiment of the invention (e.g., the feature extraction module 21, lock information and conflict prediction module 22, lock heatmap construction module 23, non-conflict zone transaction playback module 24, and conflict zone transaction playback module 25 in the log playback device). The processor 31 executes various functional applications and data processing of the computer device by running the software programs, instructions, and modules stored in the memory 32, thereby implementing the above-described log playback method.

[0045] The memory 32 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 32 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 32 may further include memory remotely located relative to the processor 31, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0046] Input device 33 can be used to acquire log files to be processed, and to generate key signal inputs related to user settings and function control of the computer device. Output device 34 can be used to distribute playback tasks, etc.

[0047] Example 4 Embodiment 4 of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a log replay method, the method comprising: The log file to be processed is parsed, and the characteristics of the current target transaction are extracted in real time; Based on the characteristics of the current target transaction, the trained lock prediction model is used to predict the lock information in the current target transaction and the probability of conflict with the current active transaction. A lock heatmap is constructed based on the lock information and the conflict probability, and non-conflict zones and conflict zones are divided. Transactions in the non-conflict zone are merged and distributed to multiple spatiotemporal slices for concurrent execution of playback tasks; The transactions in the conflict zone are divided into multiple time window intervals. If it is determined that there is no conflict within the target time window interval, the transactions within the target time window interval are allocated to multiple spatiotemporal slices to execute the replay task concurrently.

[0048] Storage media can be any type of memory device or storage device. The term "storage media" is intended to include: mounting media, such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disks or optical storage); registers or other similar types of memory elements. Storage media may also include other types of memory or combinations thereof. Furthermore, storage media may reside in a computer system in which the program is executed, or may reside in a different second computer system connected to the computer system via a network (such as the Internet). The second computer system can provide program instructions to the computer for execution. The term "storage media" can include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). Storage media may store program instructions (e.g., specifically implemented as a computer program) that can be executed by one or more processors.

[0049] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the method operations described above, but can also perform related operations in the log playback method provided in any embodiment of the present invention.

[0050] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0051] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0052] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0053] Example 5 Embodiment 5 of the present invention also provides a computer program product, which includes a computer program (also referred to as code or instructions). The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it is used to execute the log playback method provided in any of the above embodiments, and has the corresponding beneficial effects of the execution method.

[0054] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. A log playback method, characterized in that, include: The log file to be processed is parsed, and the characteristics of the current target transaction are extracted in real time; Based on the characteristics of the current target transaction, the trained lock prediction model is used to predict the lock information in the current target transaction and the probability of conflict with the current active transaction. A lock heatmap is constructed based on the lock information and the conflict probability, and non-conflict zones and conflict zones are divided. Transactions in the non-conflict zone are merged and distributed to multiple spatiotemporal slices for concurrent execution of playback tasks; The transactions in the conflict zone are divided into multiple time window intervals. If it is determined that there is no conflict within the target time window interval, the transactions within the target time window interval are allocated to multiple spatiotemporal slices to execute the replay task concurrently.

2. The log playback method according to claim 1, characterized in that, Before predicting the lock information in the current target transaction and the probability of conflict with the currently active transaction using a trained lock prediction model based on the characteristics of the current target transaction, the method further includes: In production or load testing environments, enable full SQL audit logs and lock wait logs. Associate all SQL statements within the same transaction with their pre-compiled parameters, execution timestamps, and commit / rollback results; From the start of a transaction to its commit, extract the lock characteristics actually held and waited for by the transaction to form a transaction-lock relationship pair; The lock prediction model is trained using training sample data generated based on the transaction-lock relationship.

3. The log playback method according to claim 1, characterized in that, The characteristics of the current target transaction include static characteristics, dynamic context characteristics, and historical behavior characteristics; The static features include SQL statement fingerprints, display lock hints, and operation type and object; the dynamic context features include input parameter values / ranges and the current database state, which includes data distribution, index status, and system load; the historical behavior features include historical lock matrices and historical conflict records.

4. The log playback method according to claim 1, characterized in that, The method further includes: Real-time monitoring of actual lock conflicts; If a conflict is detected between two transactions, one of the transactions will be rolled back and replayed. If the number of conflicts in a target spatiotemporal segment exceeds a preset threshold, the target spatiotemporal segment is split based on the spatiotemporal distribution of the conflicts, and / or the time window interval is adjusted.

5. The log playback method according to claim 4, characterized in that, The method further includes: If a conflict is detected, new training sample data is generated based on the conflicting transaction, and the lock prediction model is incrementally trained.

6. The log playback method according to claim 1, characterized in that, The parsing of the log file to be processed includes: The log format of the log file to be processed is identified based on predefined syntax rules and semantic templates; Based on the identification results, the log file to be processed is split into multiple sub-files; The multiple sub-files are parsed in parallel.

7. A log playback device, characterized in that, include: The feature extraction module is used to parse the log file to be processed and extract the features of the current target transaction in real time; The lock information and conflict prediction module is used to predict the lock information in the current target transaction and the probability of conflict with the current active transaction based on the characteristics of the current target transaction using a trained lock prediction model. A lock heatmap construction module is used to construct a lock heatmap based on the lock information and the conflict probability, and to divide the area into non-conflict zones and conflict zones; The non-conflict zone transaction replay module is used to merge transactions in the non-conflict zone and distribute them to multiple spatiotemporal slices for concurrent execution of replay tasks. The conflict zone transaction replay module is used to divide the transactions in the conflict zone into multiple time window intervals. If it is determined that there is no conflict within the target time window interval, the transactions within the target time window interval are allocated to multiple spatiotemporal slices to execute the replay task concurrently.

8. A computer device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the log playback method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the log playback method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the log playback method as described in any one of claims 1-6.

Images