A database exception diagnosis method and device based on database comparison

Abstract

This specification discloses a database anomaly diagnosis method and device based on database comparison, relating to the field of database comparison technology, and is used to solve the decision-making risks that existing methods easily lead to in database migration acceptance. The method includes: acquiring a comparison event stream based on an introspection engine, generating a comparison state snapshot of the comparison event stream, and persistently storing the comparison state snapshot; synchronously collecting real-time business indicators of the source database and the target database, and comparing them with the corresponding rhythmic baseline to obtain bilateral real-time deviation signals; acquiring the bilateral real-time deviation signals based on a probe engine, and comparing them with the persistently stored comparison state snapshot to execute a preset three-level anomaly judgment logic to obtain anomaly judgment results; if a comparison anomaly exists in the anomaly judgment results, querying the business activity surge event corresponding to the bilateral real-time deviation signal within a preset time window, and attributing it to the time period of the comparison anomaly to determine the source of the external disturbance.

Description

Technical Field

[0001] This specification relates to the field of database comparison technology, and in particular to a database anomaly diagnosis method and device based on database comparison. Background Technology

[0002] In database migration, data governance, and system replacement, database comparison is a common consistency verification method. It involves scanning data in the source and target databases row by row or block by block within a specified range, and determining whether the data at both ends is consistent by comparing data values ​​or checksums. For example, during database migration, database comparison can determine whether the data migration has been completed completely and accurately based on the compared data values ​​or checksums.

[0003] Existing database comparison tools typically employ a method where the comparison process internally maintains a progress counter, periodically outputting the comparison percentage or the amount of data processed, and logging errors in case of anomalies. The operating system or task scheduler then determines the process's viability. However, this approach relies on the comparison process itself for status reporting. When the process encounters anomalies due to memory overflow, connection loss, or other reasons, the reporting mechanism fails synchronously, and users receive no notification. Furthermore, it completely lacks insight into changes in business activity on both the source and target sides during the comparison period, making it impossible to assess the impact of business writes on the reliability of the comparison conclusions. This can easily lead to decision-making risks during database migration acceptance testing. Summary of the Invention

[0004] To address the aforementioned technical problems, this specification provides one or more embodiments of a database anomaly diagnosis method and device based on database comparison.

[0005] One or more embodiments of this specification employ the following technical solutions: This specification provides one or more embodiments of a database anomaly diagnosis method based on database comparison, the method comprising: During the execution of the comparison task, the comparison event stream is obtained based on the introspection engine, a comparison state snapshot of the comparison event stream is generated, and the comparison state snapshot is persistently stored; wherein, the persistent storage is independent of the comparison process memory space; Synchronously collect real-time business metrics from both the source and target databases; The real-time business metrics of the source database and the target database are compared with the corresponding rhythm baseline to obtain the two-sided real-time deviation signal; wherein, the corresponding rhythm baseline is constructed based on the historical business metrics of the corresponding database. The probe engine acquires the real-time deviation signals from both sides and compares them with the state snapshots in the persistent storage to execute a preset three-level anomaly judgment logic and obtain anomaly judgment results. If a comparative anomaly is found in the anomaly judgment result, the probe engine queries the business activity surge event corresponding to the real-time deviation signal on both sides within a preset time window, and associates the business activity surge event with the time period of the comparative anomaly to determine the source of the external disturbance.

[0006] Optionally, in one or more embodiments of this specification, obtaining a comparison event stream based on an introspection engine, generating a comparison state snapshot of the comparison event stream, and persistently storing the comparison state snapshot specifically includes: The comparison event stream is output by the comparison adapter layer of the comparison task obtained based on the introspection engine. A snapshot of the key fields of the comparison event stream is generated by taking snapshots of the key fields of the comparison event stream at a fixed frequency; wherein the key fields include: comparison identification information, progress information, health information, inconsistency information, location information, and snapshot metadata. The comparison state snapshot is continuously written to persistent storage that is independent of the comparison process's memory space, based on the file format; wherein, the persistent storage is an embedded persistent storage based on the local file system.

[0007] Optionally, in one or more embodiments of this specification, real-time business metrics of the source database and the target database are collected simultaneously, specifically including: Real-time business metrics of the source database and the target database are collected synchronously through the read-only, non-intrusive interface between the source database and the target database; wherein, the real-time business metrics are real-time business activity metrics.

[0008] Optionally, in one or more embodiments of this specification, before performing a rolling comparison of the real-time business indicators of the source database and the target database with the corresponding rhythm baseline to obtain the two-sided real-time deviation signal, the method further includes: Before the comparison task is executed, the estimated execution time of the comparison task is obtained; Centered on the time period of the expected execution time, historical business metrics of corresponding databases for multiple similar periods adjacent to the time period are obtained; wherein, the similar period is a historical time period that is similar to the expected execution time. The historical business metrics are segmented based on a fixed time granularity to obtain multiple time granularity units; wherein, the historical business metrics are historical business activity metrics. Statistical characteristics of historical business activity indicators within each time granularity unit are obtained, and corresponding rhythmic baselines are constructed based on these statistical characteristics.

[0009] Optionally, in one or more embodiments of this specification, statistical characteristics of historical business activity indicators within each time granularity unit are obtained to construct a corresponding rhythmic baseline based on the statistical characteristics, specifically including: Obtain the statistical characteristics of historical business activity indicators within each time granularity unit; wherein, the statistical characteristics include: the mean activity and the standard deviation of activity; Based on the mean activity level and the standard deviation of activity level, a statistical distribution model of normal activity level within the corresponding time granularity unit of the database is constructed. If there are time points in the historical business activity index within the time granularity unit that are greater than the activity range, then the time point corresponding to the time granularity unit is marked as a known high activity time point; wherein, the activity range corresponds to the sum of the activity mean and twice the activity standard deviation; The statistical distribution model of normal activity is integrated with the known high activity time points to obtain the corresponding rhythm baseline.

[0010] Optionally, in one or more embodiments of this specification, the real-time business indicators of the source database and the target database are compared with the corresponding rhythm baselines in a rolling manner to obtain a two-sided real-time deviation signal, specifically including: Based on the statistical characteristics of the corresponding time granularity unit in the corresponding rhythm wiring, the threshold for sudden increase in activity outside the baseline is determined; If the real-time business metric exceeds the baseline out-of-baseline activity surge threshold, and the current time point corresponding to the real-time business metric does not belong to the known high activity time point, then it is determined that there is a business activity surge event. When a sudden increase in business activity occurs, a real-time deviation signal for the corresponding database is output; wherein, the real-time deviation signal includes: event occurrence timestamp, duration, deviation from baseline, and source type; The real-time deviation signals output by the source database and the target database are used as the two-sided real-time deviation signals. When outputting the two-sided real-time deviation signal, the method further includes: calculating the evaluation value of the write activity of the source database and the target database based on multi-evaluation dimension indicators, so as to use the evaluation value as a reference indicator for the credibility of the comparison conclusion.

[0011] Optionally, in one or more embodiments of this specification, the real-time deviation signals obtained from both sides are compared with the persistent state snapshots based on the probe engine to execute a preset three-level anomaly judgment logic and obtain anomaly judgment results, specifically including: The probe engine reads-only access to the persistent storage to obtain new comparison state snapshots; Based on the snapshot sequence number and snapshot generation timestamp of the newly added comparison state snapshot, determine whether the snapshot sequence number continuously increases within the maximum tolerance interval to obtain the first anomaly judgment result; If the data stream heartbeat is determined to be normal based on the first anomaly judgment result, then the most recent comparison timestamp in the comparison status snapshot is obtained based on the probe engine, and it is determined whether the comparison timestamp is continuously updated within the preset maximum stagnation tolerance time to obtain the second anomaly judgment result. Based on the probe engine, the real-time comparison rate is calculated according to the actual comparison increment of multiple adjacent cycles of the same type in the comparison status snapshot, and the expected completion time is estimated by combining the remaining comparison data volume; it is then determined whether the expected completion time exceeds the preset maximum allowable comparison time window to obtain the third anomaly judgment result.

[0012] Optionally, in one or more embodiments of this specification, based on the probe engine querying the business activity surge events corresponding to the bilateral real-time deviation signals within a preset time window, and associating the business activity surge events with the time periods of the contrast anomalies to determine the source of the external disturbance, specifically including: If any anomaly exists, the execution time of the comparison task from its start to the time the anomaly is triggered, as well as the fixed backtracking limit, are obtained. The execution time is compared with the fixed backtracking limit to obtain a query time window. Within the query time window, based on the probe engine, detect each business activity surge event corresponding to the two-sided real-time deviation signal, so as to determine whether the start and end time periods of each business activity surge event overlap with the comparison rate decrease period. If there is a time overlap, the source of the external disturbance is determined based on the overlapping time period.

[0013] Optionally, in one or more embodiments of this specification, after the probe engine queries the business activity surge event corresponding to the bilateral real-time deviation signal within a preset time window, and associates the business activity surge event with the time period of the contrast anomaly to determine the source of the external disturbance, the method further includes: The probe engine is used to obtain report composition data corresponding to the comparison anomaly; wherein, the report composition data includes: event description, context corresponding to the external disturbance source side, credibility of comparison conclusion, decision window description, and the completed comparison range and inconsistency distribution; The report components are encapsulated into a structured data format to obtain a diagnostic report, which is then delivered to the user via a notification output layer based on the probe engine.

[0014] This specification provides one or more embodiments of a database anomaly diagnosis device based on database comparison. The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described above.

[0015] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: By generating and persistently storing comparison state snapshots in a memory space independent of the comparison process through an introspection engine, the reliability of monitoring is improved. This avoids the problem that current status reporting depends on the survival of the comparison process itself, which can easily lead to the synchronous failure of the reporting mechanism and users receiving no notifications. By synchronously collecting real-time business metrics from both the source and target ends and performing rolling comparisons with historical rhythm baselines, changes in business activity on both sides can be continuously sensed, outputting real-time deviation signals. Based on these real-time deviation signals and comparison state snapshots, anomaly detection is performed. This avoids false positives or false negatives caused by single-dimensional judgments, significantly improving the accuracy and comprehensiveness of anomaly detection. After determining the existence of a comparison anomaly, the probe engine queries and matches business activity surge events corresponding to the anomaly period within a preset time window. By temporally associating the anomaly with external business events, intelligent attribution of the anomaly source helps to quickly locate the source of external business disturbances, improving problem localization efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart illustrating a database anomaly diagnosis method based on database comparison, provided as an embodiment of this specification; Figure 2 A schematic diagram of a database anomaly diagnosis logic based on database comparison provided in the embodiments of this specification; Figure 3 This is a schematic diagram of the structure of a database anomaly diagnosis device based on database comparison, provided as an embodiment of this specification. Detailed Implementation

[0017] This specification provides a database anomaly diagnosis method and device based on database comparison.

[0018] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0019] like Figure 1 As shown in the figure, this specification provides a flowchart of a database anomaly diagnosis method based on database comparison. Based on... Figure 1 As can be seen, in one or more embodiments of this specification, a database anomaly diagnosis method based on database comparison includes: S101: When the comparison task is executed, the comparison event stream is obtained based on the introspection engine, a comparison state snapshot of the comparison event stream is generated, and the comparison state snapshot is persistently stored; wherein, the persistent storage is independent of the comparison process memory space.

[0020] The introspection engine consumes the comparison event stream produced by the adapter layer and builds comparison state snapshots at a fixed snapshot generation period, continuously writing them to persistent storage. Persistent storage is a lightweight persistent storage such as SQLite or LMDB that ensures data remains accessible to the probe engine even after the comparison process crashes. Data is written to disk as files, independent of the comparison process's memory space, and uses WAL mode to ensure that completed records are not corrupted in case of a crash during the write process.

[0021] In the embodiments described in this specification, after the comparison task is started, the comparison process begins to read and compare data from the source and target ends. All comparison events generated during the process are pushed to the introspection engine in real time through the adapter layer. The introspection engine extracts key information of each comparison event from the collected comparison event stream according to preset parsing rules, integrates the parsed structured comparison event stream data according to a fixed snapshot generation cycle, and generates a comparison status snapshot. The comparison status snapshot is then persistently stored independently of the comparison process's memory space. Because the persistent storage is independent of the comparison process's memory space, when the comparison process experiences a "frozen" state due to lock waiting, IO saturation, or other reasons (i.e., the process is alive but unable to continue performing comparison operations), or when it crashes, the introspection engine stops generating new snapshots. However, the comparison snapshot data already written to persistent storage can still be accessed by the independently deployed probe engine. The probe engine directly mounts the same storage file in read-only mode, obtains read permissions through a file lock, and reads the last complete comparison status snapshot as the basic data source for subsequent anomaly judgment and on-site diagnosis.

[0022] Specifically, in one or more embodiments of this specification, the comparison event stream is obtained based on the introspection engine, a comparison state snapshot of the comparison event stream is generated, and the comparison state snapshot is persistently stored, specifically including: The comparison adapter layer, based on the built-in engine, outputs a comparison event stream for comparison tasks. It takes snapshots of key fields in the comparison event stream at a fixed frequency to generate a comparison status snapshot. These key fields include: comparison identification information, progress information, health information, inconsistency information, location information, and snapshot metadata. The comparison identification information includes the unique ID of this comparison task, the comparison tool type, the source and target database identifiers, the comparison task start timestamp, and the comparison range. Progress information includes the amount of data compared, the estimated total amount of data compared, the current comparison rate, and the actual comparison increment in the most recent sampling period. Health information includes the error event count, the retry event count, and the timestamp of the last successful comparison operation. The difference between the last successful comparison timestamp and the snapshot generation timestamp is a key indicator for determining a "dead" state, independent of the progress percentage reported by the comparison tool. Inconsistency information includes the location of the most recent inconsistency. This field is continuously updated during the comparison process and is used to describe the inconsistency distribution in the on-site diagnostic report. Location information is the location of the last checkpoint after the comparison is completed, recorded in the native format of the comparison tool, serving as data for the user's decision to continue the comparison. The snapshot metadata includes the snapshot generation timestamp and the snapshot sequence number. The sequence number is used by the probe engine to determine whether the data stream heartbeat is continuously advancing.

[0023] The comparison state snapshots are continuously written to persistent storage that is independent of the comparison process's memory space, based on the file format; the persistent storage is an embedded persistent storage based on the local file system.

[0024] Because the generation and storage of the comparison state snapshot are independent of the comparison process's memory space, a crash in the comparison process will not result in the loss of monitoring data. Furthermore, the snapshot in persistent storage records the last checkpoint position, allowing users to resume the process from its breakpoint after a restart. In addition, the monotonically increasing sequence number provides the probe engine with a basis for judging the heartbeat of the data stream, effectively identifying situations where the comparison process is alive but the data stream is frozen.

[0025] S102: Synchronously collect real-time business metrics from the source database and the target database.

[0026] like Figure 2 The database comparison process requires simultaneous reading from both the source and target databases. A sudden increase in business activity on either side will affect the comparison rate and the reliability of the comparison conclusions. This dictates that the business rhythm perception during the comparison process must be bidirectional. Therefore, while obtaining a snapshot of the comparison status based on step S101, real-time business metrics of the source and target databases are collected only through existing data interfaces such as read-only views, slow query logs, and audit logs, based on the principle of read-only and non-intrusive operation. No data is written to either the source or target database during this collection process.

[0027] Specifically, in one or more embodiments of this specification, the synchronous collection of real-time business metrics from the source database and the target database includes: Real-time business metrics of the source and target databases are collected synchronously through read-only, non-intrusive interfaces. Since a sudden increase in business activity on either side of the database can affect the comparison rate and the reliability of the comparison conclusions, the real-time business metrics are real-time business activity metrics to achieve business rhythm awareness during the comparison process.

[0028] S103: The real-time business indicators of the source database and the target database are compared with the corresponding rhythm baseline to obtain the two-sided real-time deviation signal; wherein the corresponding rhythm baseline is constructed based on the historical business indicators of the corresponding database.

[0029] To address the issue that existing tools are completely unaware of changes in business activity on both the source and target sides during the comparison period, unable to establish a correlation between external business disturbances and a decrease in the comparison rate, and unable to assess the impact of business writes on the reliability of comparison conclusions—especially during dual-track operation or incremental data verification phases where the source side continuously writes data and the target side may also have synchronous tasks—the lack of assessment of the reliability of the comparison conclusions directly leads to decision-making risks during migration acceptance. This specification's embodiments perform a rolling comparison of the real-time business metrics of the source and target databases with the corresponding rhythmic baselines to obtain two-sided real-time deviation signals. It should be noted that the corresponding rhythmic baselines are constructed based on the historical business metrics of the corresponding databases before the comparison task is executed.

[0030] Specifically, in one or more embodiments of this specification, before performing a rolling comparison of the real-time business indicators of the source database and the target database with the corresponding rhythm baseline to obtain the two-sided real-time deviation signal, the method further includes: Before executing the comparison task, obtain the estimated execution time of the comparison task. The estimated execution time can be obtained manually through the comparison task configuration interface, or automatically generated based on preset task plans such as scheduled execution strategies or batch execution strategies.

[0031] Centered on the time period of the expected execution time, obtain historical business metrics from the corresponding databases of multiple similar periods adjacent to the time period; among them, historical business metrics are historical business activity metrics, and similar periods are historical time periods that are similar to the expected execution time. For example, if the task is scheduled to be executed on Monday morning, then extract the activity data of the same time period of multiple past Monday mornings, such as extracting the activity data of the same time period of 4 to 8 past Monday mornings.

[0032] Then, historical business metrics are segmented based on a fixed time granularity to obtain multiple time granularity units. The fixed time granularity can be customized by operators based on the current business scenario through configuration files or API parameters, or it can be set based on the database monitoring sampling period. In some scenarios, the time granularity is set to 5 minutes.

[0033] Statistical characteristics of historical business activity indicators within each time granularity unit are obtained to construct a corresponding rhythmic baseline based on these statistical characteristics. Further, in one or more embodiments of this specification, obtaining statistical characteristics of historical business activity indicators within each time granularity unit to construct a corresponding rhythmic baseline specifically includes: Obtain the statistical characteristics of historical business activity indicators within each time granularity unit. These statistical characteristics include: the mean activity value μ and the standard deviation of activity σ. Based on the mean and standard deviation of activity levels, a statistical distribution model of normal activity within the corresponding time granularity unit of the database is constructed. Then, periodic peak time points in historical data are identified. During the identification process, if there are time points within a time granularity unit where the historical business activity index exceeds the activity range, the time point corresponding to that time granularity unit is marked as a known high-activity time point. The activity range corresponds to the sum of the mean activity level and twice the standard deviation of activity level, i.e., μ+2σ. Marking known high-activity time points helps to exempt such time points from subsequent real-time comparisons, preventing sudden increase alarms. Finally, the statistical distribution model of normal activity is integrated with the known high-activity time points to obtain the corresponding rhythmic baseline.

[0034] Furthermore, such as Figure 2 The business rhythm awareness module shown is used to evaluate the credibility of the comparison conclusion. When there is continuous business write activity on one side during the comparison execution, the consistency state reflected by the comparison conclusion may deviate from the actual state. This module provides users with a reference for the credibility of the comparison conclusion by continuously monitoring the business activity on both sides. During the comparison execution, the business rhythm awareness module simultaneously collects real-time business activity indicators from the source and target ends, performs rolling comparisons with their respective rhythm baselines, and independently outputs two real-time deviation signals. That is, the probe engine and the business rhythm awareness module exchange bilateral deviation signals through a local IPC channel. The business rhythm awareness module continuously writes the signals to the shared memory area, and the probe engine reads them on demand, using read-write locks to protect concurrent access security. In one or more embodiments of this specification, the real-time business indicators of the source database and the target database are compared with the corresponding rhythm baselines to obtain bilateral real-time deviation signals, specifically including the following process: Based on the statistical characteristics of the corresponding time granularity unit in the corresponding rhythm wiring, the threshold for sudden increase in activity outside the baseline is determined. This threshold is the baseline mean plus N times the standard deviation, where N is a configurable parameter. The statistical meaning of N is: under the assumption of a normal distribution, N=2 corresponds to approximately 95% of the historical activity distribution range, and N=3 corresponds to approximately 99.7% of the range. The default value for N is recommended to be 2. Its configuration is based on two dimensions: first, business tolerance. Batch processing or non-core business systems have larger activity fluctuations, so it is recommended that N be set to 2.5 to 3 to reduce false alarms; core systems such as financial transactions have stable activity, so it is recommended that N be set to 1.5 to 2 to improve sensing sensitivity. Second, historical data quality. If the historical collection period is less than 4 periods of the same type, N should be appropriately increased or the collection period extended before starting the comparison task. N can be configured independently for the source and target ends to adapt to the differences in business characteristics between the two sides.

[0035] If a real-time business metric on a certain database side exceeds the activity surge threshold outside the baseline for that side, and the current time point corresponding to the real-time business metric does not belong to a known high-activity time point, then a business activity surge event is determined to exist. When a business activity surge event exists, the real-time deviation signal for the corresponding database is output; the real-time deviation signal includes the event occurrence timestamp, duration, extent exceeding the baseline, and source type of the business activity surge event. The real-time deviation signals output by the source database and the target database are used as the bilateral real-time deviation signal.

[0036] It should also be noted that when outputting the real-time deviation signal between the two sides, a comprehensive evaluation value of the write activity of both sides is continuously calculated as a reference indicator for the credibility of the current comparison conclusion. Specifically, the method further includes: calculating the evaluation value of the write activity of the source database and the target database based on multi-evaluation dimension indicators, and using this evaluation value as a reference indicator for the credibility of the comparison conclusion. It can be understood that when both sides are in a low-activity state during the comparison execution, the credibility evaluation is high; while when there is continuous write activity on either side, the credibility evaluation decreases accordingly.

[0037] Specifically, the evaluation value is obtained using a hierarchical weighted calculation method. In the single-sided evaluation phase, transaction volume deviation rate D_tx (weight 0.4), active connection number deviation rate D_conn (weight 0.2), log generation rate deviation rate D_log (weight 0.2), and lock wait count deviation rate D_lock (weight 0.2) are collected for each side. The deviation rate of each indicator is defined as (real-time value - baseline mean) / baseline mean. The single-sided comprehensive evaluation value A_side = 0.4 × D_tx + 0.2 × D_conn + 0.2 × D_log + 0.2 × D_lock. In the double-sided fusion phase, C_decay = max(A_src, A_dst), and the final credibility output C = max(0, 1 - C_decay), normalized to the [0,1] interval, ensuring that any business interference on either side will affect the credibility of the comparison conclusion.

[0038] S104: Based on the probe engine, the real-time deviation signal of both sides is obtained and compared with the state snapshot in the persistent storage to execute the preset three-level anomaly judgment logic and obtain the anomaly judgment result.

[0039] like Figure 2 After acquiring the bilateral real-time deviation signals, the business rhythm perception module does not trigger a notification independently. Instead, it appends the bilateral deviation signals as an interpretation layer to the probe judgment conclusion. Therefore, based on the bilateral real-time deviation signals acquired by the probe engine, a comparison snapshot of the state in persistent storage is performed to execute the pre-set three-level anomaly judgment logic and obtain the anomaly judgment result.

[0040] Specifically, in one or more embodiments of this specification, a probe engine is used to acquire real-time deviation signals from both sides and compare them with a snapshot of the state in persistent storage to execute a preset three-level anomaly judgment logic and obtain anomaly judgment results, specifically including: The probe engine uses read-only access to persistent storage to obtain the sequence number and generation timestamp of the latest snapshot, which is read at fixed intervals. Based on the snapshot sequence number S_curr and the snapshot generation timestamp T_snap, it determines whether the snapshot sequence number continuously increases within the maximum tolerance interval, obtaining the first anomaly judgment result. The first-level anomaly judgment result includes: a data flow heartbeat interruption anomaly corresponding to a non-continuous increase, and a normal data flow heartbeat corresponding to a continuous increase. The maximum tolerance interval T_heartbeat is a configurable parameter that supports dynamic adjustment based on business rhythm. The system maintains a baseline heartbeat threshold T_base, with a recommended default value of 3 times the introspection engine snapshot generation interval to tolerate normal network jitter and write latency. During comparison execution, when the business rhythm awareness module detects a sudden increase in activity on either side outside the baseline, the probe engine dynamically widens the current effective T_heartbeat to T_base multiplied by a configurable expansion coefficient K (default K=2) to avoid false alarms due to slower comparison process response during peak business periods. Once bilateral activity levels return to baseline, T_heartbeat is restored to T_base. This dynamic adjustment logic only applies to the first-level survival assessment and does not affect the thresholds for progress validity assessment and completion probability assessment.

[0041] In other words, if the snapshot sequence number S_curr of the newly added comparison snapshot is greater than the local variable S_last maintained by the probe engine during the first-level logical judgment process, it indicates that the heartbeat is normal. At this time, S_last is updated to S_curr, and this round passes. However, if the snapshot sequence number S_curr of the newly added comparison snapshot is equal to the local variable S_last maintained by the probe engine, it further judges whether the difference between the current time and T_snap exceeds the maximum tolerance interval T_heartbeat. If it does not exceed the limit, no alarm is triggered; otherwise, the heartbeat is judged to have been interrupted. It should also be noted that after completing the first-level judgment logic, if the heartbeat is judged to be abnormally interrupted, there is no need to wait for subsequent judgments; the notification process is directly initiated, and the activity status of both sides before the heartbeat interruption is attached to the report.

[0042] If the data flow heartbeat is determined to be normal based on the first-level judgment logic described above, the probe engine then obtains the most recent comparison timestamp from the comparison status snapshot and determines whether this comparison timestamp continues to update within the preset maximum stagnation tolerance time to obtain the second anomaly judgment result. The second anomaly judgment result includes: stagnant comparison progress without continuous updates and normal comparison progress with continuous updates. The preset maximum stagnation tolerance time is a configurable item, pre-configured by operators based on historical experience. This second-level judgment logic verifies whether the last successful comparison timestamp in the snapshot continues to advance. If the timestamp does not change within the configured maximum stagnation tolerance time T_stall, a progress stagnation anomaly is determined, triggering a notification. Simultaneously, the third-level completion probability assessment conclusion and the business rhythm context from both sides are attached to the report.

[0043] The third level involves using a probe engine to calculate the real-time comparison rate based on the actual comparison increments of multiple adjacent cycles of the same type in the comparison status snapshot, and then combining this with the remaining comparison data to estimate the expected completion time. It then determines whether the expected completion time exceeds a preset maximum allowable comparison time window, obtaining a third anomaly judgment result. If the expected completion time exceeds the pre-configured maximum allowable comparison time window, the completion probability is deemed insufficient, triggering a notification and attaching a source label for the two-sided deviation signal. The triggering priority of the above three levels of judgment is: survival judgment is higher than progress validity judgment, and progress validity judgment is higher than completion probability judgment. When a high-priority judgment is triggered, the conclusion of the low-priority judgment is attached to the report as supplementary information and does not trigger a separate notification.

[0044] Specifically, in a certain scenario, the probe engine executes each detection cycle in the following order: First, it reads the current snapshot sequence number S_curr and the generation timestamp T_snap. If the difference between the current time and T_snap exceeds T_heartbeat, the heartbeat is considered interrupted, and the process jumps to report generation without executing subsequent steps. Second, it reads the last successful comparison timestamp T_last_success from the snapshot. If this timestamp has not changed within T_stall, the process is considered stalled, and the process jumps to report generation. Third, it takes the actual comparison increment of the last N sampling cycles and calculates the current actual rate V_current = total increment / N cycles. Combined with the estimated remaining data volume R, it calculates ETA = R / V_current. If ETA exceeds T_max, the probability of completion is deemed insufficient, and the process enters report generation. If none of the three steps are triggered, the current detection cycle ends normally.

[0045] S105: If there is a comparative anomaly in the anomaly judgment result, the probe engine queries the business activity surge event corresponding to the bilateral real-time deviation signal within a preset time window, and associates the business activity surge event with the time period of the comparative anomaly to determine the source side of the external disturbance.

[0046] Specifically, in one or more embodiments of this specification, based on the probe engine querying the business activity surge event corresponding to the bilateral real-time deviation signal within a preset time window, and associating the business activity surge event with the time period of the contrast anomaly to determine the source of the external disturbance, specifically including: It should also be noted that after any level of the three-level judgment triggers an anomaly but before the report generation process starts, the probe engine immediately initiates an active query to the business rhythm perception module. During the normal detection cycle, it does not initiate an active query to avoid the performance overhead caused by frequent IPC. Specifically, if any anomaly exists, the execution time of the comparison task from its inception to the anomaly trigger time, along with a fixed backtracking upper limit, is obtained. The smaller of the two values ​​is used as the query time window. Within the query time window, based on the probe engine, each business activity surge event corresponding to the real-time deviation signal on both sides is detected to determine whether the start and end times of each business activity surge event overlap with the period of the decrease in the comparison rate. If there is a time overlap, the source of the external disturbance is determined based on the overlapping period, and thus marked as a possible external disturbance cause and its source in the diagnostic report.

[0047] Furthermore, specifically, when one of the probe engine's three-level judgments triggers an anomaly, the probe engine actively queries the business rhythm perception module's output on both sides during the relevant time period, appending the surge event record to the external environment context field of the report, clearly indicating whether the change in activity occurred at the source, target, or both sides simultaneously. If the rate decay period overlaps with the surge event period on one side, the report clearly indicates the possible external causes and source side of the rate decay, allowing users to determine whether the rate can recover on its own after the external disturbance ends. The reliability of the comparison conclusion is output as an independent field as a regular output. Even if no anomalies are triggered in any of the three-level judgments and the comparison is completed normally, the reliability assessment result of the comparison conclusion is still output as an additional field in the comparison completion report, allowing users to evaluate the reference value of this comparison conclusion.

[0048] When an exception is triggered at any level of the pre-defined three-level exception judgment logic, the probe engine actively queries the output of the business rhythm perception module on both sides during the relevant time period, and attaches the sudden event record to the external environment context field of the report, clearly indicating whether the change in activity occurred at the source end, the target end, or both sides simultaneously.

[0049] Specifically, in one or more embodiments of this specification, after querying the business activity surge event corresponding to the bilateral real-time deviation signal within a preset time window based on the probe engine, and associating the business activity surge event with the time period of the contrast anomaly to determine the source of the external disturbance, the method further includes: The report components are obtained based on the probe engine and compared with the anomalies. The report components include: event description, context corresponding to the source of external disturbance, credibility of comparison conclusions, decision window description, and the range of completed comparisons and inconsistency distribution. The event description records the time point when the comparison stopped or anomaly occurred, the specific reason for triggering the judgment, and the last known state of the comparison tool; the context corresponding to the source of the external disturbance records the output conclusions of both sides of the business rhythm perception module during the period of the anomaly, clearly indicating which side of the environment, the source or the target, has a time overlap relationship with the rate anomaly; the credibility of the comparison conclusion is based on the activity data of both sides recorded by the business rhythm perception module throughout the comparison execution, outputting the credibility reference of the conclusions of the part that has been completed by the time the comparison stopped, and indicating whether there were any two-sided business write activities that affected the credibility during the comparison execution; the decision window description informs the user whether both sides are still in an active state, whether the external disturbance has ended or when it is expected to end, helping the user to judge whether it is necessary to wait for the environment to stabilize before continuing or re-executing the comparison; the completed comparison range and inconsistency distribution are based on the last checkpoint position recorded by the introspection engine, describing the comparison range that has been completed when the comparison task stopped; outputting the number of inconsistent records found by the time of stop and their distribution by table or field type, providing the user with preliminary clues for problem localization. The data from the above reports is encapsulated into a structured data format to obtain a diagnostic report, which is then delivered to the user via the notification output layer based on the probe engine.

[0050] like Figure 3 As shown in the diagram, this specification provides a schematic diagram of the structure of a database anomaly diagnosis device based on database comparison. Based on... Figure 3 As can be seen, in one or more embodiments of this specification, a database anomaly diagnosis device based on database comparison includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described above.

[0051] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, and therefore described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0052] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0053] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A database anomaly diagnosis method based on database comparison, characterized in that, The method includes: During the execution of the comparison task, the comparison event stream is obtained based on the introspection engine, a comparison state snapshot of the comparison event stream is generated, and the comparison state snapshot is persistently stored; wherein, the persistent storage is independent of the comparison process memory space; Synchronously collect real-time business metrics from both the source and target databases; The real-time business metrics of the source database and the target database are compared with the corresponding rhythm baseline to obtain the two-sided real-time deviation signal; wherein, the corresponding rhythm baseline is constructed based on the historical business metrics of the corresponding database. The probe engine acquires the real-time deviation signals from both sides and compares them with the state snapshots in the persistent storage to execute a preset three-level anomaly judgment logic and obtain anomaly judgment results. If there is a comparative anomaly in the anomaly judgment result, the probe engine queries the business activity surge event corresponding to the real-time deviation signal on both sides within a preset time window, and associates the business activity surge event with the time period of the comparative anomaly to determine the source side of the external disturbance. The probe engine acquires the real-time deviation signals from both sides and compares them with the persistent state snapshot to execute a pre-set three-level anomaly judgment logic to obtain anomaly judgment results, specifically including: The probe engine reads-only access to the persistent storage to obtain new comparison state snapshots; Based on the snapshot sequence number and snapshot generation timestamp of the newly added comparison state snapshot, determine whether the snapshot sequence number continuously increases within the maximum tolerance interval to obtain the first anomaly judgment result; If the data stream heartbeat is determined to be normal based on the first anomaly judgment result, then the most recent comparison timestamp in the comparison status snapshot is obtained based on the probe engine, and it is determined whether the comparison timestamp is continuously updated within the preset maximum stagnation tolerance time to obtain the second anomaly judgment result. Based on the probe engine, the real-time comparison rate is calculated according to the actual comparison increment of multiple adjacent cycles of the same type in the comparison status snapshot, and the expected completion time is estimated by combining the remaining comparison data volume; it is then determined whether the expected completion time exceeds the preset maximum allowable comparison time window to obtain the third anomaly judgment result.

2. The database anomaly diagnosis method based on database comparison according to claim 1, characterized in that, The process involves acquiring a comparison event stream based on an introspection engine, generating a comparison state snapshot of the comparison event stream, and persistently storing the comparison state snapshot. Specifically, this includes: The comparison event stream is output by the comparison adapter layer of the comparison task obtained based on the introspection engine. A snapshot of the key fields of the comparison event stream is generated by taking snapshots of the key fields of the comparison event stream at a fixed frequency; wherein the key fields include: comparison identification information, progress information, health information, inconsistency information, location information, and snapshot metadata. The comparison state snapshot is continuously written to persistent storage that is independent of the comparison process's memory space, based on the file format; wherein, the persistent storage is an embedded persistent storage based on the local file system.

3. The database anomaly diagnosis method based on database comparison according to claim 1, characterized in that, Synchronously collect real-time business metrics from both the source and target databases, specifically including: Real-time business metrics of the source database and the target database are collected synchronously through the read-only, non-intrusive interface between the source database and the target database; wherein, the real-time business metrics are real-time business activity metrics.

4. The database anomaly diagnosis method based on database comparison according to claim 1, characterized in that, Before performing a rolling comparison of the real-time business metrics of the source database and the target database with the corresponding rhythm baseline to obtain the two-sided real-time deviation signal, the method further includes: Before the comparison task is executed, the estimated execution time of the comparison task is obtained; Centered on the time period of the expected execution time, historical business metrics of corresponding databases for multiple similar periods adjacent to the time period are obtained; wherein, the similar period is a historical time period that is similar to the expected execution time. The historical business metrics are segmented based on a fixed time granularity to obtain multiple time granularity units; wherein, the historical business metrics are historical business activity metrics. Statistical characteristics of historical business activity indicators within each time granularity unit are obtained, and corresponding rhythmic baselines are constructed based on these statistical characteristics.

5. The database anomaly diagnosis method based on database comparison according to claim 4, characterized in that, Obtain the statistical characteristics of historical business activity indicators within each time granularity unit, and construct the corresponding rhythmic baseline based on the statistical characteristics, specifically including: Obtain the statistical characteristics of historical business activity indicators within each time granularity unit; wherein, the statistical characteristics include: the mean activity and the standard deviation of activity; Based on the mean activity level and the standard deviation of activity level, a statistical distribution model of normal activity level within the corresponding time granularity unit of the database is constructed. If there are time points in the historical business activity index within the time granularity unit that are greater than the activity range, then the time point corresponding to the time granularity unit is marked as a known high activity time point; wherein, the activity range corresponds to the sum of the activity mean and twice the activity standard deviation; The statistical distribution model of normal activity is integrated with the known high activity time points to obtain the corresponding rhythm baseline.

6. The database anomaly diagnosis method based on database comparison according to claim 5, characterized in that, The real-time business metrics of the source database and the target database are compared with the corresponding rhythm baselines in a rolling manner to obtain the two-sided real-time deviation signal, specifically including: Based on the statistical characteristics of the corresponding time granularity unit in the corresponding rhythm wiring, the threshold for sudden increase in activity outside the baseline is determined; If the real-time business metric exceeds the baseline out-of-baseline activity surge threshold, and the current time point corresponding to the real-time business metric does not belong to the known high activity time point, then it is determined that there is a business activity surge event. When a sudden increase in business activity occurs, a real-time deviation signal for the corresponding database is output; wherein, the real-time deviation signal includes: event occurrence timestamp, duration, deviation from baseline, and source type; The real-time deviation signals output by the source database and the target database are used as the two-sided real-time deviation signals. When outputting the two-sided real-time deviation signal, the method further includes: calculating the evaluation value of the write activity of the source database and the target database based on multi-evaluation dimension indicators, so as to use the evaluation value as a reference indicator for the credibility of the comparison conclusion.

7. The database anomaly diagnosis method based on database comparison according to claim 1, characterized in that, Based on the probe engine, the system queries the business activity surge events corresponding to the bilateral real-time deviation signals within a preset time window, and associates these business activity surge events with the time periods of the comparative anomalies to determine the source of external disturbances. Specifically, this includes: If any anomaly exists, the execution time of the comparison task from its start to the time the anomaly is triggered, as well as the fixed backtracking limit, are obtained. The execution time is compared with the fixed backtracking limit to obtain a query time window. Within the query time window, based on the probe engine, detect each business activity surge event corresponding to the two-sided real-time deviation signal, so as to determine whether the start and end time periods of each business activity surge event overlap with the comparison rate decrease period. If there is a time overlap, the source of the external disturbance is determined based on the overlapping time period.

8. The database anomaly diagnosis method based on database comparison according to claim 1, characterized in that, Based on the probe engine querying the business activity surge events corresponding to the bilateral real-time deviation signals within a preset time window, and after associating the business activity surge events with the time periods of the comparative anomalies to determine the source of the external disturbance, the method further includes: The probe engine is used to obtain report composition data corresponding to the comparison anomaly; wherein, the report composition data includes: event description, context corresponding to the external disturbance source side, credibility of comparison conclusion, decision window description, and the completed comparison range and inconsistency distribution; The report components are encapsulated into a structured data format to obtain a diagnostic report, which is then delivered to the user via a notification output layer based on the probe engine.

9. A database anomaly diagnosis device based on database comparison, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in any one of claims 1-8.

Images