Bidirectional replication consistency checking and repairing method based on record source identifier

Abstract

The invention discloses a bidirectional replication consistency checking and repairing method based on a record source identifier, particularly relates to the technical field of data consistency of a distributed memory database, and is used for solving the problem that a bidirectional replication cluster is difficult to accurately identify a record source, missing and conflicts in a non-stop scene, so that the reliability of the bidirectional replication cluster is improved. Therefore, the problems of difficulty in consistency verification, high possibility of false difference and repeated modification are solved. A deletion mark, a lock mark and a source mark are coded in a record control area in a centralized manner, a primary key field and a source pseudo column are used as anchor points, and an idempotent identifier constructed based on a primary key and a version field, a fragment comparison process divided according to a verification window and a primary key range and an automatic repair decision oriented to missing records and records with inconsistent fields are combined; therefore, the real difference can still be accurately recognized under the condition of bidirectional replication continuous operation, replication loopback and false inconsistency are avoided, and the consistency of the business data of the memory database clusters on the two sides is kept on the premise that business writing is not stopped.

Description

Technical Field

[0001] This invention relates to the field of distributed memory database data consistency technology, specifically a bidirectional replication consistency verification and repair method based on record source identifier. Background Technology

[0002] In existing technologies, distributed in-memory database clusters are widely used in latency-sensitive business scenarios such as financial transactions, e-commerce orders, and real-time risk control. To improve availability and disaster recovery capabilities, bidirectional replication is often used to synchronize insert, modify, and delete operations between two clusters. In existing solutions, consistency assurance mainly relies on the confirmation mechanism of the replication link itself, transaction commit logs, and post-event sampling verification. When the data volume is small, methods such as full table comparison, primary key comparison plus field comparison, and checksum comparison are often used to check whether the records on both sides are consistent. However, these checks usually exist in the form of offline tasks or low-frequency scripts, lacking verification window management that matches the actual business operation status. The characterization of various write sources, version evolution, and resource constraints during bidirectional replication is also relatively crude, making it difficult to meet the data consistency management requirements of large-scale clusters running online for a long time.

[0003] In bidirectional replicated distributed in-memory database clusters, business writes and replication writes coexist. The replication chain often adopts an asynchronous mode, inevitably leading to replication delays, network fluctuations, temporary failures of some nodes, and duplicate writes caused by manual replay. Existing consistency verification methods mostly rely on simple comparisons of primary keys and business fields, lacking a unified write source identifier and version anchor to assist decision-making. This makes it difficult to distinguish between situations such as "local business has been updated but the other side has not caught up" and "replication loop or accidental write." At the same time, under the premise of continuous business writes, the lack of configurable verification windows and sharding strategies easily leads to a large number of false differences caused by changes occurring while comparisons are being made. After differences are discovered, common repair methods mainly rely on manual derivation and manual scripts, lacking an automatic repair loop that is coordinated with idempotent control and sequential control. There is also a lack of structured difference records and audit evidence chains, making the repair process prone to duplicate modifications, conflict amplification, or difficulty in tracing responsibility.

[0004] Therefore, under the above application context, the main technical problems of the current technology can be summarized as follows: In a bidirectional replicated distributed in-memory database cluster, there is a lack of a consistency verification mechanism that can, without interrupting business operations, combine the write source attributes and version evolution, and within a controlled verification window, finely identify, idempotently repair, and form an auditable chain of evidence for the differences between the two sides of the data, resulting in the difficulty in timely, accurate, and traceable detection and handling of data inconsistencies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a bidirectional replication consistency verification and repair method based on record source identifiers, thereby resolving the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for bidirectional replication consistency verification and repair based on record source identifier, comprising: S1. In the in-memory database cluster, reserve a control area for records, compress the delete flag, lock flag, and source flag into the first two bytes, and specify the primary key field and the source pseudo-column as anchor points; S2. Configure source identification rules for the replication client and business client, adjust the source flag value according to the identification result before the write operation, and generate an idempotent flag based on the primary key and version to be stored with the record; S3. During consistency verification, set a verification window for both clusters, divide the table into shards according to the primary key range, create a row location array for each shard in the compared cluster, and write a miss status for each row. S4. Scan records in the first cluster by shard, retrieve records in the second cluster based on the primary key, compare the source pseudo-column with the business field and remove non-business fields, and update the row positioning array status according to the comparison results. S5. For missing records and records with inconsistent fields, generate repair decisions according to the source mark order and version, construct insert, update, and delete instructions with idempotency flags for the records, and execute them in the target cluster. S6. After completing the first cluster scan, traverse the row location array to find records that are not found, perform repair, and generate a detailed file of differences in primary key values, field differences, source markers, repair instructions, and execution results as a consistency verification evidence chain.

[0007] Furthermore, S1 includes: In the database kernel, two bytes of control space are reserved for the header of each record; The first byte uses all bits as a deletion marker, the second byte uses the zeroth bit as a lock marker, the first bit as a source marker, and the remaining bits are fixed to zero. During logical deletion, the deletion flag is changed from the in-use state to the idle state, while the physical space of the corresponding record is temporarily not reclaimed, and the source flag derives the source pseudo-column; The source pseudo-column and the primary key field together serve as anchors for consistency checks and conflict arbitration; The layout of the control area and the meaning of the tag values ​​are written into the metadata table in the form of configuration records and locked by the cluster configuration center through the version number.

[0008] Furthermore, S2 includes: In the in-memory database cluster, a connection management module is set up. Based on the source identification rules, the connection source is identified when the connection is established and written into the session context. The rule number and the current version field value are saved. Before business client connection and replication client connection initiate writing, business records and replication records are distinguished in the source flag bit of the record control area according to the session context rule number; After the source tag is set, an idempotent identifier is generated by combining the primary key field and the version field and then written to the memory page along with the record; Register the source identification rule version number and the idempotency identifier generation rule version number in the metadata table so that consistency verification and repair can restore the source partitioning method and idempotency identifier construction method according to the corresponding rule version.

[0009] Furthermore, S3 includes: The consistency verification module receives a verification task from the scheduling platform, which includes the source cluster identifier, the target cluster identifier, the target table name set, and the start and end times of the verification window. Verify the primary key field and record control area format of the target table in the metadata table. If the check fails, return an error code and record the failure information. When the check passes, lock the control area format version number, source identification rule version number, and idempotency identifier generation rule version number; In the comparison cluster, the target table is divided into table shards according to the primary key range, and the primary key field and local row location marker are scanned sequentially in each table shard. Write the local row location markers into the row location array in primary key order and initialize the array items to a miss state to mark the comparison coverage status of records in the verification window.

[0010] Furthermore, S4 includes: After the verification window and table shards are determined, the consistency verification module scans records on the first cluster in the order of table shards, constructs access conditions carrying the start and end times of the verification window based on the primary key field value, and retrieves primary key matching records in the second cluster. For primary key matching records, compare the source pseudocolumn and business field of the records in the two clusters. The business field excludes update timestamp, auto-increment version number and audit field based on the table structure meta information. When the source pseudo-column and all business fields are consistent, the status of the array to be compared in the cluster row is updated from missing to missing; When the primary key matches but the values ​​of the business fields are inconsistent, write a difference flag in the row location array, and keep the record in a miss state when no primary key matching record is found.

[0011] Furthermore, S5 includes: After completing the comparison of records within the table shards, the consistency verification module classifies the records to be repaired into missing record categories and field inconsistency categories based on the source marker and version field in the verification window. While maintaining consistency between the primary key fields of the first and second clusters, an insertion-type repair instruction is generated for missing records, and an update-type or deletion-type repair instruction is used for records with inconsistent fields based on the source tag and version field. Each repair instruction carries an idempotent identifier formed by the combination of the primary key field value and the version field value, as well as a rule version number.

[0012] Furthermore, after receiving a repair instruction carrying an idempotent flag, the repair execution module searches the repair log table for repair records with the same idempotent flag and a successful status, based on the idempotent flag and the primary key field. If the repair record exists, the business table will not be modified and the repair instruction will be marked as completed. If the repair record does not exist, the primary key record in the target table will be located according to the repair action type, and the corresponding type of repair instruction will be executed. Before execution, a site snapshot is generated and the original field values ​​are recorded in the audit log table. After execution, a repair log record containing an idempotency flag and a rule version number is registered in the repair log table, and a status code indicating successful completion and indicating that an idempotency flag conflict has been discarded is written.

[0013] Furthermore, S6 includes: After the first cluster completes the shard scan, the consistency verification module traverses the row location array of the compared clusters and generates the corresponding instruction type among the insert, delete and update instructions for the missing records based on the source flag and version field. In the repair execution module, the repair log table is checked based on the idempotency identifier and rule version number, and the repair action is performed only when there is no record with the same idempotency identifier and the status is successful. After the repair is completed, a detailed record of differences is constructed, including the primary key field value, business field difference, source mark value, version field value, repair instruction type, execution status, and failure reason. This record is then written into the centralized audit database as a consistency verification evidence chain record.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. By centrally encoding deletion markers, lock markers, and source markers in the record control area, using primary key fields and source pseudo-columns as anchors, and combining idempotent identifiers constructed based on primary key and version fields, a sharding comparison process divided by verification window and primary key range, and automatic repair decisions for missing records and records with inconsistent fields, the system can accurately identify real differences, avoid replication loops and false inconsistencies under continuous bidirectional replication operation, and maintain the consistency of business data in the two-sided in-memory database clusters without stopping business writes.

[0015] 2. By implementing version locking for source identification rules, control area format, and idempotent identifier generation rules, the row positioning array, repair log table, and difference detail file are used to fully record the comparison status, repair actions, and rule version information of each record during consistency verification and repair. This facilitates fault location, risk assessment, rollback decision-making, and compliance auditing on the operations and maintenance side in complex production and maintenance scenarios, and improves the long-term operational reliability of large-scale distributed in-memory database clusters. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a bidirectional replication consistency verification and repair method based on record source identifier according to the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example: A method for bidirectional replication consistency verification and repair based on record source identifier, comprising: S1. In the in-memory database cluster, reserve a control area for records, compress the delete flag, lock flag, and source flag into the first two bytes, and specify the primary key field and the source pseudo-column as anchors. The specific implementation is as follows: In a real-world production environment, the in-memory database cluster employs a self-developed distributed in-memory database. Each record is stored in a fixed-length format within a memory page. Two bytes are reserved at the beginning of each record as a control area, used to centrally store deletion flags, lock flags, and source flags. The in-memory database cluster is a database system composed of multiple server nodes interconnected via a high-speed network, using memory as the primary storage medium. A memory page is the smallest allocation unit used to organize record storage in this database. The record header is a fixed area located at the very beginning of each record, used to store management information. The control area is a contiguous byte space within the record header used to centrally store status bit information.

[0019] The delete flag indicates whether a record is valid in business logic; the lock flag indicates whether a record is held by a transaction (a transaction is a set of operations committed atomically in the database); and the source flag indicates whether the record originates from bidirectional replication traffic. Bidirectional replication refers to the process of two in-memory database clusters sending insert, update, and delete operations to each other at the record level and writing them to the other side. Replication traffic refers to the sequence of these operations transmitted through the replication channel.

[0020] The primary key field is defined as a combination of one or more fields that can uniquely identify a record within a single table. Its physical meaning is to locate the record within any cluster and serve as the primary anchor point for bidirectional replication comparison. The source pseudo-column is defined as a logical field derived from the source marker in the second byte of the control area. It does not occupy additional storage space, and its value reflects the write source of the record, serving as an auxiliary anchor point in conflict arbitration and repair decisions. This implementation is suitable for in-memory database clusters that support primary key constraints, support extending control bits in the record header, and employ a bidirectional replication mechanism. During the deployment phase, the cluster configuration center uniformly distributes the control area format and related thresholds.

[0021] The cluster configuration center refers to the management module responsible for centrally maintaining configuration information and distributing it to each node in the management network. The metadata table refers to the system management table in the database used to store table structure, control area layout, and rule configuration.

[0022] During project implementation, two bytes are reserved in the header of each record in the record format definition of the database kernel. All bits of the first byte are used to represent the deletion flag, where a zero value indicates an idle state and a one value indicates an in-use state. Preferably, it can be set so that logical deletion of the same record only changes the deletion flag without immediately reclaiming physical space. Space reclamation is completed through background batch processing to reduce the accumulation of fragmentation caused by frequent allocation and release of memory pages.

[0023] The zeroth bit of the second byte is defined as the lock flag, the first bit is defined as the source flag, and the remaining six high-order bits are uniformly filled with zeros. Filling the high-order bits with zeros means that they will not be allocated to other uses in the current version, thereby ensuring the consistency of the control area format in all business tables and reserving space for future expansion.

[0024] To ensure consistency across different business databases and tables, the cluster configuration center writes the layout of the two bytes of the control area, the meaning of each flag value, and the corresponding version number into the metadata table as configuration records when creating business databases and business tables. The version number is used to identify the effective version of the current control area format. When it is necessary to expand the meaning of the control bits in the future, the evolution is carried out by adding a new version number and migration strategy to avoid ambiguity in the parsing of historical records.

[0025] During the record writing process, the business side generates primary key values ​​based on the pre-configured primary key field set. When constructing a record, the database kernel determines the initial values ​​of the deletion flag and the lock flag according to the current transaction state and the operation type passed in by the business interface. For example, the deletion flag of a newly inserted record can be set to "in use," and the lock flag is in an occupied state during the transaction holding period and is released uniformly when the transaction is committed. The value of the source flag is updated by the subsequent source identification rules. In this step, it is only guaranteed that the three flag bits in the control area are physically reserved and can be modified safely by subsequent logic.

[0026] To improve access efficiency in high-concurrency write scenarios, it is preferable to arrange the record header at the alignment boundary of the memory page so that the control area falls in a position that is convenient for single read and write access. At the same time, the relative offset of the control area is recorded in the page-level metadata so that the byte where the control area is located can be quickly located by pointer during the consistency check and conflict arbitration phases.

[0027] To address the issue of differing primary key field definitions across different business lines, a unified requirement is in place during the table creation phase for business tables participating in bidirectional replication to configure a combination of primary key fields. When a business has historical tables without a primary key, it can be configured to use a combination of unique constraint fields or a row identifier field automatically assigned by the system to replace the primary key field for anchoring, ensuring that each record has a unique anchor point within the cluster.

[0028] To ensure the applicability of this step to both full and incremental data, preferably, when performing historical data migration or reloading records from persistent files, the record headers are also padded or converted according to the above control area format, so that the records already in memory have a unified control area layout, and do not need to be processed separately according to the record source during subsequent bidirectional replication and consistency verification.

[0029] By employing the above method, without significantly increasing the memory overhead of a single record, a stable physical anchor and state carrier are provided for subsequent source identification, consistency verification, conflict arbitration, and repair decisions. Preferably, the two-byte size of the control area remains a fixed value, the length and type of the primary key field are determined according to the business table field definitions, and the offset relationship between the record header and the control area is version-locked using constants in the database kernel. For example, when performing performance evaluation on a business table on a server node, the memory page size can be set to several kilobytes, the record length can be set to tens of bytes, and the control area is always located at the beginning of the record. By observing the changes in time overhead and memory usage caused by control area access under different concurrent connection numbers and different write rhythms, it can be verified that the latency limit, throughput, and resource consumption of this structure in online business scenarios are all within acceptable ranges, thus proving the feasibility and scope of application of this step in actual production environments.

[0030] S2. Configure source identification rules for the replication client and the business client. Before the write operation, adjust the source flag value according to the identification result, and generate an idempotent identifier based on the primary key and version to be stored with the record. The specific implementation is as follows: In the business environment, business client connections and replication client connections typically coexist on the same in-memory database cluster. Business client connections refer to sessions that carry write requests submitted by the front-end business system, while replication client connections refer to sessions that carry replication traffic from the peer cluster. To distinguish write behaviors from different sources in a bidirectional replication scenario, the system sets up a connection management module at the in-memory database layer. The capability of the connection management module is to identify the source attributes of each connection and maintain a stable source identifier during the connection establishment and connection persistence phases.

[0031] The connection management module configures source identification rules. These rules are a set of rules that classify connections as either business sources or replication sources based on at least one of the following characteristics: connection account name, the role of the connection account, the client address range, the port number, and the certificate fingerprint. Each source identification rule is assigned a fixed rule number, which is bound to the connection's session context after the connection is established and remains unchanged until the session is released. The session context refers to a set of session state information maintained for each connection, used to store key state variables such as the rule number, the current transaction state, and the current version field value. This allows subsequent write operations to records to obtain stable source information without additional table lookups.

[0032] Before a business client or replication client initiates an insert, modify, or delete operation, the in-memory database first reads the rule number stored in the current session context. Based on the rule number, it determines whether the current session belongs to a business source or a replication source, and accordingly sets or clears the source flag in the second byte of the record control area. The record control area consists of two bytes written in the record header. The second byte is used to carry the lock flag and the source flag. The source flag occupies a pre-specified bit in the second byte, and its physical meaning is a source status flag that indicates whether the record originates from replication traffic.

[0033] The source flag value is updated throughout the record's lifecycle as the write source changes, guiding the conflict arbitration order during subsequent consistency checks. After setting the source flag, the system specifies a set of primary key fields on the record and generates an idempotent flag for that record. The primary key field is a combination of fields that can uniquely identify a record within a single table; its physical meaning is a unique anchor point used for quickly locating records in an in-memory database cluster, for bidirectional replication comparison anchoring, and for deduplication in repair actions. The idempotent flag is an identifier value formed by combining the primary key field value and the version field, used to identify the same record version and avoid duplicate activation when repeatedly performing repair actions or receiving replication traffic.

[0034] The version field is an integer field that monotonically increments with the business modification rhythm, with its unit being the distance distance. Each time a business modification is performed, this field increments by one, used to distinguish different historical versions under the same primary key. When generating the idempotent identifier, the system combines the current value of the primary key field and the current value of the version field in a fixed order according to pre-registered rules to form a fixed-length code, ensuring the uniqueness and decodability of the idempotent identifier within the cluster. After generation, the idempotent identifier is written to the corresponding memory page along with the record. The memory page is a storage unit of fixed size in the in-memory database. Records are stored sequentially in a fixed-length format within the memory page. The control area and business fields maintain a consistent layout to facilitate subsequent scanning and comparison according to page alignment.

[0035] To ensure the stability and traceability of the aforementioned source identification rules and idempotency identifier generation rules during cluster evolution, the system registers the currently effective source identification rule version number and idempotency identifier generation rule version number in the metadata table. The metadata table is a dedicated system table that records cluster configuration and rule definitions, used to lock different versions of rules and their applicable time ranges. This allows for the restoration of the source partitioning method and idempotency identifier construction method used at the time when performing consistency checks, troubleshooting, and historical audits.

[0036] Preferably, the source flag of the business client can be set to zero, and the source flag of the replication client can be set to one. Sufficient encoding space is reserved in the rule number to support the expansion of multiple replication channels. The version field can be set to be initialized to one when the record is first written, and incremented by one step after each business modification action is completed. In high-concurrency scenarios, the monotonicity of the version field under the same primary key can be guaranteed by the increment control logic inside the cluster.

[0037] Preferably, the source identification rules can be configured by combining account prefix division and address range division. For example, connections with account names containing a specific prefix and source addresses falling within a preset network segment range are classified as replication sources, while other connections are classified as business sources. By recording the correspondence between rule numbers and rule content in the metadata table, a complete rule version chain is formed when rules are upgraded by adding new versions and retaining old versions.

[0038] Preferably, in complex scenarios where a record undergoes multiple cross-cluster replications and local modifications, the combination of the source tag and the version field can be used to identify whether the current record is the result of the most recent business modification. When generating subsequent repair actions, the idempotency flag is brought into the repair instruction, so that repeatedly arriving repair instructions are identified as executed records if the idempotency flag is consistent, thereby avoiding data offset caused by repeated modifications. Furthermore, by writing the idempotency flag and the corresponding rule version number into the difference details file and audit log, a traceable chain of evidence is formed, providing a basis for data consistency verification and compliance auditing during long-term cluster operation.

[0039] S3. During consistency verification, a verification window is set for both clusters, the table is sharded according to the primary key range, and a row location array is created for each shard in the compared cluster, and a miss status is written for each row. The specific implementation is as follows: When organizing a consistency check for a bidirectional replication scenario, the operations and maintenance side registers the check task with the consistency check module through the scheduling platform. The consistency check module is a logical unit on the in-memory database cluster side that is specifically responsible for organizing the comparison process. The check task includes at least the source cluster identifier, the target cluster identifier, the target table name set, and the start and end times of the check window. The source cluster identifier and the target cluster identifier are used to uniquely identify the two in-memory database clusters participating in this round of comparison. The target table name set is used to limit the range of business tables that need to be compared. The physical meaning of the check window is the time range of records allowed to participate in this round of comparison, which is used to lock the comparison process within a stable time interval and reduce the risk of false differences caused by continuous business writes.

[0040] For tables with time field constraints, you can set up filtering based on the creation time or last modification time of the record and the start and end times of the verification window. Only records that fall within the verification window will participate in this round of comparison. For tables without time field constraints, you can set up splitting of records by logical version field or by business day, and include records with version number or business date falling within the specified range in this round of comparison.

[0041] When registering a verification task, the consistency verification module first checks the metadata table to see if the target table has a configured primary key field and whether the record control area format is consistent with the aforementioned control area format description record. If it is found that the target table is missing a primary key field, the control area format is not registered, or the target table does not exist, it returns at least one error code through the verification task registration interface: missing parameters, table not found, or missing primary key. The time, initiator, and reason for the failure of this registration are recorded in the task list to form a traceable chain of evidence. Only when all the preconditions are met is a verification task number assigned to the task, and the control area format version number, source identification rule version number, and idempotent identifier generation rule version number used in this round of verification are locked in the metadata table to ensure that the subsequent comparison process is executed under a fixed rule version.

[0042] Based on the registered set of table names, the consistency verification module divides each target table into several table shards in the compared cluster according to the primary key range. A table shard refers to a record range within the current table, anchored by the primary key field. Each table shard corresponds to a non-overlapping primary key range. The upper and lower boundaries of the primary key range are based on the actual distribution of the primary key in the field business and are adjusted in combination with the number of records in a single shard and the allocable memory capacity to ensure that the memory usage of a single scan does not exceed the preset resource limit. The resource limit can be locked in the scheduling platform through configuration parameters, such as limiting the maximum number of records in a single shard, the maximum number of concurrent shards in a single task, and the maximum memory capacity that a single task can occupy.

[0043] For each partitioned table shard, the consistency verification module performs a sequential scan on the compared cluster, containing only the primary key field and local row location markers. The local row location marker refers to the location information that uniquely points to the physical location of the record in the memory page within the current cluster. This location information can be formed by combining the page number and the offset within the page, and is used to quickly access the corresponding record in the subsequent repair phase. During the sequential scan, the local row location markers of each scanned record are filled into the row location array in memory in primary key order. The row location array is defined as a set of markers within the current verification window and the current table shard, indicating whether each row in the compared cluster has been compared in the other cluster. Each item in the array corresponds to a row record, and the status of all array items is uniformly set to a miss state during array initialization.

[0044] As the shard scan progresses on the source cluster side, when the source cluster scans a record within the specified verification window and finds a record with the same primary key in the target cluster and completes the field comparison, the consistency verification module will update the status of the array item in the corresponding row location array of the target cluster from the missed state to the hit state, in order to mark that the record has been covered in this round of comparison; when this round of comparison is completed, the array item that is still in the missed state corresponds to the record in the target cluster that has not been covered by the source cluster record in the current verification window and the current table shard, which is used to identify missing records and redundant records in the future.

[0045] To adapt to different business scenarios, the consistency verification module supports specifying the maximum number of records per shard and the upper limit of concurrent shards when registering tasks. Preferably, the number of records in a single table shard can be set to 10,000 to 100,000, and the length of a single verification window can be set to half an hour to one hour, which can be adjusted according to the memory capacity and processing power of the cluster nodes. The memory occupied by a single item in the row positioning array can be set to several bytes. At the same time, the sharding parameters and row positioning array layout parameters corresponding to this round of verification tasks are registered in the metadata table as the basis for subsequent verification result analysis and performance evaluation.

[0046] In a production system for a type of financial transaction settlement business, preferably, the primary key of each transaction master table can be set as the transaction number, with a daily average record volume of several million records. The nighttime verification task is triggered by the scheduling platform during the business off-peak period. The scheduling platform specifies a verification window length of one hour, divides each table into multiple table shards with approximately 50,000 records each, and sets the upper limit of the number of concurrent shards to a certain number to ensure that the memory usage of a single node is lower than the estimated upper limit. Under this condition, the row positioning array size corresponding to each table shard is approximately several megabytes. A single round of consistency verification is completed within the limited window. The configuration information, partitioning rules, row positioning array status statistics, and returned error codes during the execution of the verification task are all persisted in the system audit database with the task number as the index, providing a complete chain of evidence for the subsequent generation of consistency verification reports and fault tracing reports.

[0047] S4. Scan records in the first cluster by shard, retrieve data in the second cluster based on the primary key, compare the source pseudo-column with the business field and remove non-business fields, and update the row positioning array status based on the comparison results. The specific implementation is as follows: When organizing a consistency verification task, the consistency verification module, within the task constraints issued by the scheduling platform, first selects one in-memory database cluster as the first cluster and the other in-memory database cluster as the second cluster. The first cluster is used to sequentially scan records, and the second cluster is used to cooperate in completing the comparison operation.

[0048] The consistency verification module pre-maintains the definitions of verification windows, table shards, and row location arrays. The verification window refers to the time range of records allowed to participate in the comparison in this round of consistency verification, used to limit the comparison parties to being in the same business period. Table shards refer to several primary key intervals divided according to the value of the primary key field, and each primary key interval corresponds to an independently shard unit. The row location array refers to the set of status flags assigned to each record in each shard on the comparison cluster side, used to record whether the record has established a one-to-one correspondence with the record in the first cluster.

[0049] With the verification window parameters and table shard information ready, the consistency verification module selects table shards on the first cluster one by one and initiates scanning actions within the parallelism limit set by the scheduling platform. For each table shard, records are read in the order of local storage. For each scanned record, access conditions are constructed based on the value of the primary key field in the record. These access conditions use the name of the primary key field and its value as the unique location basis, and carry the start and end times of the current verification window as filtering constraints when accessing the second cluster.

[0050] When accessing the second cluster, the system locates the corresponding target table and primary key range in the second cluster according to the table sharding definition in the first cluster. It only searches for records that match the primary key value within the time limit of the verification window, ensuring that the records of both sides are matched under the same business cycle and the same primary key range.

[0051] Once a record matching the primary key is found in the second cluster, the system compares the predefined source pseudo-column and business field values ​​in the records of the first and second clusters in a one-to-one correspondence manner. The source pseudo-column is a logical field derived from the second byte of the record control area, used to characterize whether the record originates from replication traffic. The business fields are a set of fields that are directly related to the actual business meaning and need to be consistent across clusters. In the initial processing stage, the system classifies each field according to the application table structure metadata, classifying the update timestamp field, auto-increment version number field, and audit field as non-business fields and removing non-business fields from the comparison range, thereby avoiding false inconsistencies that cannot be eliminated in the long term due to the natural progression of time, auto-incrementing versions, or differences in audit records.

[0052] During the comparison process, if the primary key matches and the values ​​of the source pseudo-column and all business fields meet the consistency judgment rules, the consistency verification module will change the status of the record in the row location array on the compared cluster side from "missed" to "hit," indicating that the record has been found within the current verification window and the business content is consistent. If no record matching the primary key is found in the second cluster, the status of the corresponding record in the row location array will remain "missed," and the record will be classified as a missing record in subsequent steps. If the primary key matches but the business fields differ, the record will be recorded using a difference marker in the row location array, enabling subsequent repair steps to generate corresponding repair decisions based on the difference type.

[0053] During the process of proceeding in the order of records, the consistency verification module also needs to handle access failure scenarios. For network access failures, response timeouts, or lock wait times exceeding preset thresholds that occur during access to the second cluster, the system uses a pre-generated idempotent identifier in the record to register the current comparison attempt, adds the current primary key and corresponding shard to the retry queue, and performs cross-cluster retrieval again without exceeding the concurrency and retry limit set by the scheduling platform. When the number of retryes corresponding to the same idempotent identifier exceeds the preset limit, the system will no longer continue automatic comparison, register the primary key and its corresponding shard as objects of manual concern, and write the corresponding error code in the status record of this verification task.

[0054] In this embodiment, error codes are used to characterize the types of anomalies during the automatic comparison process. Preferably, they can be set to three categories: peer access unreachable, lock wait time exceeded, and structure mismatch. Peer access unreachable is used to characterize the situation where the network link is interrupted or the target node is inaccessible. Lock wait time exceeded is used to characterize the situation where the record in the second cluster is occupied for a long time and cannot be read. Structure mismatch is used to characterize the situation where the table structure or business field set definition of the two parties is inconsistent.

[0055] The above comparison process proceeds sequentially within each table shard until all records in that shard have completed the comparison attempt or reached the retry limit. This enables the consistency of the business fields of both clusters to be checked one by one under the constraint of the verification window, using the primary key field and the source pseudo-column as anchors. By changing the state of the row positioning array, a complete on-site evidence chain is provided for subsequent identification of missing records, decision-making on repairing discrepancies, and summarization of manually monitored objects.

[0056] In a preferred embodiment, the number of records in a single table shard can be set to several thousand to tens of thousands, the maximum number of retries can be set to two to three, the lock wait time threshold can be set to several seconds, and exceeding the above threshold is registered as a lock wait over-limit error code. The status field of each record in the row positioning array can be set to occupy several bits to mark three states: miss, hit, and difference. The verification window length can be set to half an hour to one hour to take into account the online business write frequency and the acceptable time overhead of the verification task. With the above parameter combination, on-site operation and maintenance personnel can complete the shard comparison of critical business tables with controlled resource consumption under the condition of continuous bidirectional replication. When network fluctuations, lock contention intensifies, or table structure is adjusted, the problem concentration area can be quickly identified through the error code and status distribution in the row positioning array, and subsequent repair and manual processing can be arranged.

[0057] S5. For missing records and records with inconsistent fields, generate repair decisions according to the source tag order and version, construct insert, update, and delete instructions with idempotency flags for the records, and execute them on the target cluster. The specific implementation is as follows: After completing the record comparison within the table shards, the consistency verification module enters the repair phase. Within the same verification window, it iterates through each missing record and record with inconsistent fields obtained from the summarized comparison results. While maintaining consistency in the primary key caliber between the first and second clusters, it generates a repair decision based on the source marker and version field. The source marker, already fixed in the second byte of the record control area in the previous steps as either a business source or a replication source, physically identifies whether the record was generated by a business client on this cluster or written by the peer cluster via bidirectional replication, distinguishing between business modifications and replication traffic in bidirectional replication scenarios. The version field, defined in the previous steps as a version number that increments at a uniform pace under the same primary key, physically distinguishes the evolution state of the same business object at different times.

[0058] In this step, the consistency verification module first classifies missing records: For records that exist in the first cluster but are missing in the second cluster, if the source marker of the record indicates that it originates from a local business source, then within the current verification window, it is preferentially identified that there is a missing record under the primary key in the second cluster. The consistency verification module generates a repair decision for the second cluster based on the primary key field value, version field value, and source marker of the record, marks the decision type as new record repair, and constructs an insert-type repair instruction carrying an idempotent flag; If the source marker of the record in the first cluster indicates that it originates from a replication source, but there is no corresponding record in the second cluster, then it is identified that there is a redundant record in the first cluster. The consistency verification module generates a deletion-type repair decision on this cluster, constructs a deletion-type repair instruction carrying an idempotent flag, and retains the corresponding source marker and rule version number in the repair decision for subsequent auditing and backtracking.

[0059] For records with inconsistent fields, the consistency verification module already provides the differences in the primary key fields, source pseudocolumns, and business fields of the records on both sides in the comparison results. At this time, priority is first determined based on the source marker: when the source marker of one record is business source and the source marker of the other record is replication source, the business source side is the priority side by default. The consistency verification module generates an update-type repair decision on the peer cluster based on the current business field value of the record on the business source side. When the source markers of both records are business source, the newness of the version fields on both sides is compared. The record with the newer version field is regarded as the side with business advantage. An update-type repair decision for the other cluster is generated based on the value of the business field on that side, and the repair decision records whether the version field difference falls within the pre-set acceptable tolerance range. When the source markers of both records are replication source and the version field difference is not within the acceptable tolerance range, the consistency verification module no longer performs automatic repair. Instead, it marks the record corresponding to the primary key as a manual arbitration category, writes the manual arbitration marker and related comparison information into the subsequent difference details file, and hands it over to the operation and maintenance personnel to make a final judgment based on the business background.

[0060] The idempotency identifier has already been generated in the previous steps by combining the primary key field value and the version field. It is used to uniquely identify the same repair action. All repair instructions generated by the repair decisions in this step carry this idempotency identifier to ensure that the same repair action will not be executed repeatedly in scenarios of network fluctuations, scheduling retries, and cross-cluster calls.

[0061] Repair instructions are uniformly received and executed by the repair execution module on the target cluster. Upon receiving a repair instruction, the repair execution module first searches the local repair log table for a repair record with the same idempotency identifier and a successful status, based on the idempotency identifier and primary key field. If a record already exists, the repair instruction is marked as completed and backfilled to the consistency verification module without further modification to the business tables. If no successful record is found, the module locates the corresponding primary key record in the target table based on the repair action type in the repair instruction. Before executing insert, modify, or delete actions, the module generates a snapshot of the relevant record or records the original field values ​​in the local audit log table according to the cluster-level security policy. This allows the system to restore the system to its pre-repair state in case of repair policy misconfiguration or cluster rollback.

[0062] After the repair is completed, the repair execution module registers a repair log record in the repair log table. This record includes at least an idempotency flag, a primary key field value, a target cluster identifier, a repair action type, a rule version number, an execution time range, and an execution result flag. An execution status code is assigned to each repair to indicate whether the repair action was successfully completed, rejected due to permission issues, aborted due to resource limitations, discarded due to idempotency flag conflicts, or intercepted due to rule version inconsistency. Rule version inconsistency refers to a discrepancy between the rule version number carried in the repair command and the version number registered in the current cluster configuration center. In such cases, the repair execution module does not modify the business tables; it only records the rule version inconsistency status in the repair log table and feeds it back to the consistency verification module, which then provides a rule version inconsistency warning in the difference details.

[0063] In the main embodiment, the consistency verification module and the repair execution module collaborate through the service call channel within the cluster. The submission frequency of the repair command is subject to the concurrency and resource quota allocated by the scheduling platform for the verification task. The repair execution module executes the repair actions on the same primary key in idempotent order to avoid cross-cluster cyclic rewriting and disordered order of the same record.

[0064] Preferably, an idempotency identifier can be set to correspond one-to-one with the version field under the same primary key. The acceptable tolerance of the version field difference can be set to no more than a certain number of version steps. The number of retries for repair execution can be set to no more than three. The allowable proportion of manual arbitration records in a single repair task can be set to no more than 10% of the total number of difference records. When the proportion of manual arbitration records exceeds this threshold, the consistency verification module marks this task as a task that needs to be checked in the verification task status record and adds a prompt message to the end of the difference details file.

[0065] Through the above arrangements, this step can form a repeatable automatic repair loop for missing records and inconsistent fields in large-scale distributed in-memory database clusters in actual production sites. At the same time, it leaves clear security and compliance space at the boundaries of rule version locking, idempotent control and manual arbitration. Based on this, engineering technicians can configure source tags, version fields, tolerance ranges, retry counts and status code sets in accordance with the conventional database management methods in this field. Without relying on the commercial implementation of a specific vendor, the repair mechanism can be deployed to the self-developed distributed in-memory database cluster.

[0066] S6. After completing the first cluster scan, iterate through the row location array to find records that were not found, perform repair, and generate a detailed file of differences in primary key values, field differences, source markers, repair instructions, and execution results as a consistency verification evidence chain. The specific implementation is as follows: After the first cluster completes a full shard scan, the consistency verification module locks the current verification window and table shard set according to the aforementioned verification task parameters. It then iterates through the row location arrays constructed for the target table in the comparison cluster one by one. The array items that are still in a miss state are identified as records that have not been covered by the records of the first cluster in this round of comparison. Combining the source marker and version field in the record, the module determines the role of the record in the bidirectional replication relationship, whether it is a redundant record in the comparison cluster or a missing record in the source cluster.

[0067] For records marked as missing in the row location array, the consistency verification module generates a repair decision according to the aforementioned repair rules. Specifically, when the record source marker indicates that it is from a normal business source and there is no corresponding primary key in the first cluster, an insert instruction is generated to complete the record in the first cluster; when the record source marker indicates that it is from a replication source and there is no corresponding primary key in the first cluster, a delete instruction is generated to delete the record in the comparison cluster; when a record that needs to be aligned with the comparison cluster is found in the row location array, an update instruction is generated based on the newer version of the version field.

[0068] All insert, update, and delete commands carry the idempotency flag constructed in the preceding steps. Upon receiving the command, the repair execution module first checks the local repair log table according to the idempotency flag and rule version number. The business action is only executed if there is no historical record with the same idempotency flag and the execution status is successful. After the execution is completed, the execution result and timestamp in the repair log table are written back to ensure that there will be no duplicate modifications or disordered order under the conditions of scheduling retries, network jitter, or node switching.

[0069] After the repair action is completed, the consistency verification module constructs a difference detail record based on the primary key field value, the difference between the business fields on both sides, the source flag value, the version field value, the repair instruction type, the execution status and the reason for failure. The physical meaning of the difference detail record is to establish an audit entry for each record that has a difference and triggers a repair decision within a single verification task, which is used to completely restore the state evolution process of the record in this round of verification and repair.

[0070] The difference details are appended to the difference details file according to a preset format. The difference details file preferably adopts a text format or a columnar storage format. The fields include the verification task number, table name, primary key field name and value, source pseudo-column value, version field value, summary of field values ​​before repair, summary of field values ​​after repair, repair rule version number and timestamp. The file is indexed in the centralized audit database according to the verification task number and table name. In this method, the centralized audit database is defined as a set of dedicated database tables used to centrally store the difference details files of each verification task. The database is linked with the upstream compliance audit system through pull and query interfaces.

[0071] After receiving the discrepancy details file, the upstream compliance audit system can determine the scope of the verification and the impact of the repair action based on the number of discrepancy entries, the range of primary keys involved, and the proportion of failed repair records. It then decides whether to trigger a business rollback strategy or add manual review based on preset security boundaries. To facilitate implementation of this method by those skilled in the art under different scales and load conditions, this implementation supports configuring the retention period of the discrepancy details file, the maximum number of discrepancy entries per task, and the capacity threshold of the centralized audit database in the scheduling platform. When the capacity threshold is exceeded, an archiving strategy is triggered using the rule version number and timestamp, migrating historical discrepancy details files to low-cost storage media.

[0072] In one preferred embodiment, the target of a single verification task can be set to one million records in a core business table, the verification window length is one hour, the proportion of the number of difference detail records to the total number of records is controlled at the level of one ten-thousandth, the difference detail files of the most recent three months are retained in the centralized audit database, and the remaining files are archived to offline storage. Field tests show that, under the above parameter configuration, this method can complete a full table comparison and repair within a few minutes without interrupting business writes, and form a complete consistency verification evidence chain, providing a stable data foundation for subsequent compliance audits and fault tracing.

[0073] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for bidirectional replication consistency verification and repair based on record source identifier, characterized in that, include: S1. In the in-memory database cluster, reserve a control area for records, compress the delete flag, lock flag, and source flag into the first two bytes, and specify the primary key field and the source pseudo-column as anchor points; S2. Configure source identification rules for the replication client and business client, adjust the source flag value according to the identification result before the write operation, and generate an idempotent flag based on the primary key and version to be stored with the record; S3. During consistency verification, set a verification window for both clusters, divide the table into shards according to the primary key range, create a row location array for each shard in the compared cluster, and write a miss status for each row. S4. Scan records in the first cluster by shard, retrieve records in the second cluster based on the primary key, compare the source pseudo-column with the business field and remove non-business fields, and update the row positioning array status according to the comparison results. S5. For missing records and records with inconsistent fields, generate repair decisions according to the source mark order and version, construct insert, update, and delete instructions with idempotency flags for the records, and execute them in the target cluster. S6. After completing the first cluster scan, traverse the row location array to find records that are not found, perform repair, and generate a detailed file of differences in primary key values, field differences, source markers, repair instructions, and execution results as a consistency verification evidence chain.

2. The bidirectional replication consistency verification and repair method based on record source identifier according to claim 1, characterized in that, S1 includes: In the database kernel, two bytes of control space are reserved for the header of each record; The first byte uses all bits as a deletion marker, the second byte uses the zeroth bit as a lock marker, the first bit as a source marker, and the remaining bits are fixed to zero. During logical deletion, the deletion flag is changed from the in-use state to the idle state, while the physical space of the corresponding record is temporarily not reclaimed, and the source flag derives the source pseudo-column; The source pseudo-column and the primary key field together serve as anchors for consistency checks and conflict arbitration; The layout of the control area and the meaning of the tag values ​​are written into the metadata table in the form of configuration records and locked by the cluster configuration center through the version number.

3. The bidirectional replication consistency verification and repair method based on record source identifier according to claim 1, characterized in that, S2 include: In the in-memory database cluster, a connection management module is set up. Based on the source identification rules, the connection source is identified when the connection is established and written into the session context. The rule number and the current version field value are saved. Before business client connection and replication client connection initiate writing, business records and replication records are distinguished in the source flag bit of the record control area according to the session context rule number; After the source tag is set, an idempotent identifier is generated by combining the primary key field and the version field and then written to the memory page along with the record; Register the source identification rule version number and the idempotency identifier generation rule version number in the metadata table so that consistency verification and repair can restore the source partitioning method and idempotency identifier construction method according to the corresponding rule version.

4. The bidirectional replication consistency verification and repair method based on record source identifier according to claim 1, characterized in that, S3 includes: The consistency verification module receives a verification task from the scheduling platform, which includes the source cluster identifier, the target cluster identifier, the target table name set, and the start and end times of the verification window. Verify the primary key field and record control area format of the target table in the metadata table. If the check fails, return an error code and record the failure information. When the check passes, lock the control area format version number, source identification rule version number, and idempotency identifier generation rule version number; In the comparison cluster, the target table is divided into table shards according to the primary key range, and the primary key field and local row location marker are scanned sequentially in each table shard. Write the local row location markers into the row location array in primary key order and initialize the array items to a miss state to mark the comparison coverage status of records in the verification window.

5. The bidirectional replication consistency verification and repair method based on record source identifier according to claim 1, characterized in that, S4 includes: After the verification window and table shards are determined, the consistency verification module scans records on the first cluster in the order of table shards, constructs access conditions carrying the start and end times of the verification window based on the primary key field value, and retrieves primary key matching records in the second cluster. For primary key matching records, compare the source pseudocolumn and business field of the records in the two clusters. The business field excludes update timestamp, auto-increment version number and audit field based on the table structure meta information. When the source pseudo-column and all business fields are consistent, the status of the array to be compared in the cluster row is updated from missing to missing; When the primary key matches but the values ​​of the business fields are inconsistent, write a difference flag in the row location array, and keep the record in a miss state when no primary key matching record is found.

6. The bidirectional replication consistency verification and repair method based on record source identifier according to claim 1, characterized in that, S5 include: After completing the comparison of records within the table shards, the consistency verification module classifies the records to be repaired into missing record categories and field inconsistency categories based on the source marker and version field in the verification window. While maintaining consistency between the primary key fields of the first and second clusters, an insertion-type repair instruction is generated for missing records, and an update-type or deletion-type repair instruction is used for records with inconsistent fields based on the source tag and version field. Each repair instruction carries an idempotent identifier formed by the combination of the primary key field value and the version field value, as well as a rule version number.

7. The bidirectional replication consistency verification and repair method based on record source identifier according to claim 6, characterized in that: After receiving a repair instruction carrying an idempotency flag, the repair execution module searches the repair log table for repair records with the same idempotency flag and a successful status, based on the idempotency flag and the primary key field. If the repair record exists, the business table will not be modified and the repair instruction will be marked as completed. If the repair record does not exist, the primary key record in the target table will be located according to the repair action type, and the corresponding type of repair instruction will be executed. Before execution, a site snapshot is generated and the original field values ​​are recorded in the audit log table. After execution, a repair log record containing an idempotency flag and a rule version number is registered in the repair log table, and a status code indicating successful completion and indicating that an idempotency flag conflict has been discarded is written.

8. The bidirectional replication consistency verification and repair method based on record source identifier according to claim 1, characterized in that, S6 include: After the first cluster completes the shard scan, the consistency verification module traverses the row location array of the compared clusters and generates the corresponding instruction type among the insert, delete and update instructions for the missing records based on the source flag and version field. In the repair execution module, the repair log table is checked based on the idempotency identifier and rule version number, and the repair action is performed only when there is no record with the same idempotency identifier and the status is successful. After the repair is completed, a detailed record of differences is constructed, including the primary key field value, business field difference, source mark value, version field value, repair instruction type, execution status, and failure reason. This record is then written into the centralized audit database as a consistency verification evidence chain record.

Images