Micro-service database gray scale double-writing switching method based on flow marking

Abstract

The invention discloses a micro-service database gray scale double-writing switching method based on a flow mark, particularly relates to the technical field of micro-service database switching and migration, and aims to solve the problem that when a service database is migrated from a source database to a target database under a micro-service architecture, the service database cannot be migrated from the source database to the target database. And flow switching and data synchronization are difficult to uniformly control under a non-stop condition, so that continuous consistency of data of a source library and a target library is ensured, and a rapid rollback capability is achieved. A stage state machine is established in a configuration center, and closed-loop control is formed by combining a flow mark, a stage routing matrix, a local message table and a data verification and repair unit, so that flow switching and data synchronization are controlled at the same time by a unified mechanism in the process of migrating a micro-service database from a source library to a target library; the effect that non-stop progressive switching is continuously consistent with data of a source library and a target library is achieved, and when indexes are abnormal, the safety stage is quickly recovered by relying on strategy rollback is achieved.

Description

Technical Field

[0001] This invention relates to the field of microservice database switching and migration technology, specifically a microservice database grayscale dual-write switching method based on traffic marking. Background Technology

[0002] In existing technologies, microservice architecture has been widely adopted. Business systems are typically divided into multiple independent databases according to services, and each microservice completes online transaction processing such as order, account, and electricity customer management through its own business database. With the growth of business volume, database version upgrades, and cloud transformation, operators need to migrate the business database of a microservice from the existing source database to a new target database. The common practice is to rely on database synchronization tools to perform full and incremental synchronization in the background, and during business downtime, operations personnel switch the database connection once using scripts or operations platforms. If necessary, manual rollback or emergency scripts are used to restore it to the source database. Although this method can complete the migration in some scenarios, the switching action, business traffic control, and consistency verification between the source and target databases are often scattered across different systems, relying on manual experience for coordination, resulting in poor overall observability and controllability.

[0003] Existing solutions also attempt to introduce dual-write mechanisms or canary release mechanisms on the application side. For example, writing to two databases simultaneously in the application code, or redirecting a portion of requests to the new database proportionally through a gateway, to verify the read and write capabilities of the target database before switching. However, these solutions typically only focus on one aspect and lack a unified constraint relationship with the configuration center, monitoring system, and data verification module. On the one hand, the granularity of experimental traffic selection is too coarse, making it difficult to stably control the traffic ratio at the user or session level. On the other hand, the results comparison, difference repair, and phase advancement conditions between the source and target databases often lack fixed standards and unified strategies, making it impossible to form a closed-loop control of the series of actions from "read-only verification, read-write dual-write, and gradual traffic switching to the new database" without downtime. Furthermore, they lack the ability to quickly roll back to a safe state based on a unified strategy when indicators deteriorate.

[0004] Therefore, when migrating databases under a microservice architecture, the core problem that existing technologies generally have is the lack of a unified collaborative mechanism driven by a configuration center that connects experimental traffic selection, request routing control, source and target database result verification, difference repair, and phased advancement and rollback decisions. It is difficult to complete the gradual switch from the source database to the target database safely and controllably without interrupting business operations, while taking into account data consistency and rapid rollback capabilities. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a microservice database canary dual-write switching method based on traffic marking, in order to solve the problems mentioned in the background technology.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a microservice database canary dual-write switching method based on traffic marking, comprising: S1. The configuration center registers the microservice database switching task, establishes a phase state machine, generates and distributes a switching strategy carrying source database and target database identifiers and a phase routing matrix. S2. The microservice entry point calculates a hash value based on the request identifier to select experimental traffic and adds a traffic tag containing the traffic type, experimental flag, and stage identifier to the request. S3. The database routing component parses the stage identifier in the traffic tag, determines the access mode according to the stage routing matrix, executes the selected access mode between the source database access mode and the source database-target database dual-path access mode on the experimental traffic, and generates a verification task. S4. During the read-write dual-write phase, the microservice writes to the business table and the local message table simultaneously within the source database transaction. The background task asynchronously writes from the local message table to the target database according to the phase status and updates the message status according to the retry strategy. S5. The data verification and repair unit compares the source database results and the target database results in the verification task, generates repair instructions for the difference records, and performs insert, update and delete operations in the target database. At the same time, it calculates the difference rate and the number of repair failures. S6. The configuration center aggregates difference rate, error rate and latency indicators according to the time window to drive the stage state machine migration. When the advanced conditions are met, the next stage strategy is issued. When the indicators exceed the alarm threshold, the migration back to the safe stage is completed and the rollback strategy is issued.

[0007] Furthermore, S1 includes: The configuration center registers microservice database switching tasks based on the existing network configuration repository and monitoring system, and generates a switching strategy that includes source database identifier, target database identifier, stage identifier and stage routing matrix; The stage routing matrix is ​​arranged by stage and by normal traffic and experimental traffic (read-only traffic and read-write traffic) as columns. Each unit is used to select between source database access mode and source database / target database dual access mode. The switching strategy, along with the difference rate threshold, error rate threshold, latency threshold, and dwell time threshold, is stored in the configuration center task library and distributed to the corresponding microservices through a unified interface; Before each deployment, the configuration center performs a consistency check based on the task identifier, stage status, and policy version number, and records the serial number, operator identifier, and policy content summary.

[0008] Furthermore, S2 includes: The microservice entry point obtains the phase identifier and experimental traffic ratio from the switching strategy; Extract a request identifier for each service request, consisting of the user ID, tenant ID, and session ID; The hash value is calculated according to the pre-agreed hash rules, and the requests are divided into two categories, experimental traffic and normal traffic, according to the hash interval corresponding to the proportion of experimental traffic. The microservice entry point adds traffic tags to the requests, which include traffic type, experiment flag, and phase identifier; Write the request header fields and keep them consistent in the thread context. The database routing component selects the access mode in the stage routing matrix based on the stage identifier in the traffic tag.

[0009] Furthermore, S3 includes: The database routing component obtains the phase routing matrix from the switching policy and parses the traffic type, experiment flag, and phase identifier from the traffic tags; Based on the stage routing matrix, the read-only traffic and read-write traffic of normal traffic and experimental traffic are respectively determined to be one of the source database access mode and the source database and target database dual-path access mode. When performing dual-path access mode for source and target databases on experimental traffic, the source and target database results, along with the request identifier, task identifier, and stage identifier, are used to generate a verification task and write it into the verification queue.

[0010] Furthermore, S4 includes: During the read-write dual-write phase, for requests marked as read-write traffic, the microservice updates the business table and inserts message records into the local message table within the same source database transaction; The message record includes task identifier, stage identifier, business primary key, target database table name, operation type, field value snapshot, message status, retry count, and deduplication flag; The deduplication tag consists of a task identifier, a business primary key, and an operation type, and a unique constraint is set on the local message table to achieve idempotent control over writing the same business primary key to the target database within the same switching task.

[0011] Furthermore, after the source database transaction is committed, the background task scans the message records in the local message table that are in the pending state and corresponding to the read-write dual-write stage at fixed intervals. Execute insert, update, and delete operations in the target database according to the insertion time sequence, corresponding to the operation type. When the target database returns a successful response, the message status is updated to "completed". When the target database returns a failed response, the number of retries is increased according to the preset retry strategy and a delayed sending is set. When the number of retries exceeds the limit, mark the message status as permanently failed and write the task identifier, business primary key, operation type and failure reason to the monitoring log. The background task limits the number of messages read from the local message table and the number of write batches executed in the target database in each cycle by using fixed concurrency and throttling parameters.

[0012] Furthermore, S5 includes: Set up a data verification and repair unit to sequentially read verification tasks from the verification queue, which include task identifier, stage identifier, primary key set, source database result field value set, and target database result field value set. Based on the primary key, determine the difference record and generate a repair instruction carrying the task identifier, stage identifier, target database table name, primary key value and field value set, and write it to the repair queue with unique constraints on the task identifier, stage identifier and primary key value; Within a fixed time window, the number of records participating in verification and the number of difference records are counted to obtain the difference rate. The task identifier, stage identifier, start and end time of the time window, difference rate, and number of repair failures are written into the monitoring database.

[0013] Furthermore, S6 includes: The configuration center aggregates difference rate, error rate, and latency indicators from the monitoring database based on task and stage identifiers according to the time window to form a stage operation snapshot for the observation window; When a phase snapshot meets the preset advanced conditions, the phase state machine is migrated from the current phase to the next phase and a new policy version is generated. When any indicator exceeds the alarm threshold, the stage state machine is transitioned to a safe stage and a rollback strategy is issued.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. By establishing a phased state machine in the configuration center and combining it with traffic marking, phased routing matrix, local message table, and data verification and repair unit to form a closed-loop control, a unified mechanism can be used to simultaneously control traffic switching and data synchronization during the migration of the microservice database from the source database to the target database. This achieves a gradual switch without downtime and continuous consistency of data between the source and target databases. Furthermore, it enables rapid recovery to a safe phase by relying on policy rollback when indicators are abnormal.

[0015] 2. Experimental traffic is selected by calculating hash values ​​based on request identifiers at the microservice entry point. In the database routing component, dual-path access and asynchronous writing are performed only on experimental traffic according to the stage routing matrix. At the same time, the background task performs rate limiting to scan the local message table and cooperates with data verification and repair to statistical difference rate, error rate and latency indicators. This achieves the effect of gradually verifying the target database read and write capabilities, continuously collecting high-quality verification samples and providing quantitative basis for stage advancement, while ensuring the main business experience and controllable resource usage. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a microservice database canary dual-write switching method based on traffic marking according to the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example: A method for switching between canary dual-write operations in a microservice database based on traffic marking, comprising: S1. The configuration center registers the microservice database switching task, establishes a phase state machine, generates and distributes a switching strategy carrying source and target database identifiers and a phase routing matrix. The specific implementation is as follows: In a real production environment, the configuration center, as the unified scheduling unit of the microservice cluster, is responsible for registering, generating, and distributing microservice database switching tasks. A microservice database switching task refers to a complete migration activity for a single microservice from a source database to a target database. The source database is the database instance currently carrying the read and write operations of the microservice, and the target database is another database instance that is planned to take over the business after the migration is completed. The scope of application includes the splitting of monolithic databases, migration from local data centers to cloud databases, and cross-cluster migration within the same cloud platform.

[0019] The configuration center first collects basic information related to the microservice from the live configuration repository and monitoring system. The basic information includes the microservice name, source database connection information, target database connection information, the names of the difference rate indicators, error rate indicators, and latency indicators used in the monitoring system to characterize the consistency and stability between the source and target databases, as well as the sampling rhythm and historical value range of the above indicators in the monitoring system. The difference rate can be understood as the proportion of the number of difference records to the number of records participating in the comparison within a fixed observation window. The error rate can be understood as the proportion of the number of failed accesses to the target database to the number of access requests. The latency indicator can be understood as the statistical value of the latency of accessing the target database within the observation window.

[0020] The configuration center organizes the names, definitions, and units of measurement in the above information, unifies the sampling period of each indicator to a fixed duration, fills in the upper and lower limits of the range of difference rate, error rate, and latency, and stores the organized results in the configuration center task library with the task identifier as the primary key. The task identifier is a number that can uniquely identify the microservice database switch task.

[0021] Based on the organized basic information, the configuration center establishes a phase state machine for each task. The phase state machine is a directed graph that describes the transferable relationships between the initialization phase, read-only dual-write phase, read-write dual-write phase, read-to-target-database phase, full-to-target-database phase, and security phase. The initialization phase is used to complete basic configuration and connectivity checks. The read-only dual-write phase is used to compare the query results of the source database and the target database without changing the write path. The read-write dual-write phase is used to gradually introduce writes to the target database while keeping the source database transactions as the standard. The read-to-target-database phase is used to migrate read operations to the target database. The full-to-target-database phase is used to migrate all read and write operations to the target database. The security phase is used to restore access to the source database-centric mode in case of unexpected situations.

[0022] The configuration center configures a difference rate threshold, an error rate threshold, a delay threshold, and a dwell time threshold for each of the above stages. The difference rate threshold, error rate threshold, and delay threshold are used to constrain the operational quality of the stage, while the dwell time threshold is used to constrain the shortest operating time of the stage, thereby enabling the stage state machine to advance or regress within controllable boundaries.

[0023] Subsequently, the configuration center generates a switching strategy carrying source database identifiers, target database identifiers, and a stage routing matrix. The source database identifier and target database identifier are unique identifiers of the corresponding database instances within the configuration center. The stage routing matrix is ​​an access mode mapping table with stages as rows and read-only traffic and read-write traffic of normal traffic and experimental traffic as columns. Normal traffic consists of business requests not extracted for experimental purposes, experimental traffic consists of business requests extracted according to rules to carry out the trial operation of the new strategy, read-only traffic consists of requests that only trigger query operations, and read-write traffic consists of requests that may trigger both query and update operations. Each cell of the stage routing matrix records whether the request uses the source database access mode or the source-target database dual-path access mode under the corresponding stage. The source database access mode initiates only one access to the source database, while the source-target database dual-path access mode initiates one access to each of the source and target databases simultaneously.

[0024] The switching strategy is stored in the configuration center task library in the form of key-value pairs containing task identifier, strategy version number, stage identifier, stage routing matrix, monitoring threshold set and security stage identifier. The strategy version number is a monotonically increasing version number. The stage identifier refers to the specific stage in the current stage state machine. The monitoring threshold set includes difference rate threshold, error rate threshold, latency threshold and dwell time threshold. The security stage identifier refers to the stage that the stage state machine should migrate to when rollback is required.

[0025] The configuration center distributes the switching policy to the corresponding microservice through a unified interface for microservices. The interface has fixed fields such as task identifier, policy version number, stage identifier, stage routing matrix, monitoring threshold set, and update time. The update time is the policy generation time. After receiving the switching policy with the newest policy version number, the microservice updates the locally cached policy content.

[0026] The configuration center assigns a serial number to each policy switching action and records the serial number, task identifier, policy version number, stage identifier, operator identifier, and policy content summary in the operation and maintenance audit database to form an evidence chain for subsequent traceability. Before the policy takes effect, the configuration center performs a consistency check on the task identifier, stage status, and policy version number. If the task identifier is missing, the stage status is inconsistent with the record in the task database, or the policy version number is lower than the existing version in the task database, the configuration center returns a pre-agreed error identifier through the interface and rejects the policy change, thereby restricting cross-task operations, unauthorized switching, and contention for the same task by multiple control terminals at the boundary.

[0027] In a preferred embodiment, for the scenario of migrating order-type microservices from a local relational source database to a cloud target database, the difference rate threshold can be set to 0.1%, the error rate threshold can be set to 0.01%, the latency threshold can be set to a certain multiple of the high quantile value not exceeding the normal value, and the dwell time threshold can be set to multiple observation windows in hours. Initially, the stage routing matrix only allows experimental traffic to use the source database and target database dual-path access mode in the read-only dual-write stage, while ordinary traffic maintains the source database access mode. As the stage state machine progresses, it is gradually relaxed to enable dual-path access and target database access for all traffic, thereby smoothly completing the database switch without changing the microservice business interface form.

[0028] S2. The microservice entry point selects experimental traffic based on the hash value calculated from the request identifier, and appends a traffic tag containing the traffic type, experimental flag, and stage identifier to the request. Specifically, the implementation is as follows: In a real production environment, the microservice entry point serves as the unified entry point for handling business requests. It can take the form of the interface layer exposed by the microservice itself, or it can be a gateway node deployed at the front end. This allows for the selection and labeling of experimental traffic without altering the business interfaces. After receiving the switching policy from the configuration center, the microservice entry point reads the stage identifier and experimental traffic ratio corresponding to the current microservice database switching task. The stage identifier indicates the current stage of the stage state machine, and the experimental traffic ratio represents the proportion of requests used to carry the trial operation strategy among all business requests.

[0029] To ensure a stable experience for different users during the switching process, the microservice entry point first determines the request identifier for each business request when processing it. The request identifier is a marker that can stably identify the same user session in the business scenario. It is preferably obtained by concatenating the user number, tenant number and session number in an agreed order, so that the same user or the same session has a consistent request identifier in multiple requests.

[0030] The microservice entry point maps request identifiers to integers within a fixed range, based on pre-agreed hashing rules. These integers serve as the hash value. The hashing rules remain consistent across all microservices and are version-locked through a configuration center, ensuring the same request identifier yields the same hash value across different instances. The microservice entry point calculates an experimental interval based on the proportion of experimental traffic and compares the hash value with this interval. If the hash value falls within the experimental interval, the request is classified as experimental traffic, and in subsequent phases, it can preferentially handle new strategies such as read-only dual-write, read-write dual-write, or read-switching to the target database. If the hash value is outside the experimental interval, the request is classified as normal traffic, and in the corresponding phase, it continues to use the source database-centric access method.

[0031] To enable downstream database routing components to select access modes without being aware of upstream implementation details, the microservice entry point constructs a traffic tag for each request after separating experimental traffic from normal traffic. The traffic tag is a set of structured information attached to the request, which includes at least the traffic type, experimental flag, and stage identifier. The traffic type is used to distinguish between read-only traffic and read-write traffic, and is preferably obtained by parsing the business method or operation type of the request. The experimental flag is used to indicate whether the request belongs to experimental traffic or normal traffic. The stage identifier is consistent with the stage status in the switching strategy, so that the database routing component can apply the corresponding stage routing matrix in different stages.

[0032] Traffic tags are preferably written to request header fields or thread context variables. The field name, value encoding method, and allowed value range are agreed upon between microservices through interface specifications, and the version number is registered in the configuration center. Subsequent changes are controlled by the version number for rolling upgrades to avoid access mode decision deviations due to inconsistent field meanings.

[0033] For situations where stable user IDs cannot be obtained in certain business scenarios, the microservice entry point can construct a request identifier by combining the source address and time slice. The source address can be the network address of the access terminal, and the time slice can be a time period identifier divided into fixed durations. By limiting the maximum number or maximum proportion of experimental traffic from the same source address within a time slice through rules, the concentration of experimental traffic on specific tenants or specific network exits can be reduced, thereby mitigating the impact of experimental strategies on local user experience.

[0034] During operation, the microservice entry point periodically pulls the latest switching strategy from the configuration center. When a strategy version number update is detected, the phase identifier and experimental traffic ratio in the local cache are replaced. Subsequent new requests are automatically divided into experimental traffic and normal traffic according to the updated ratio. For requests that are already being processed, the traffic marker bound when they enter the process is used and is not changed midway to ensure that the behavior of a business call is consistent within the same phase.

[0035] In a preferred embodiment, for order-type microservices, the experimental traffic ratio can be set to one percent of the total number of requests during the read-only dual-write phase. The microservice entry point calculates the experimental range according to this ratio. After running for several observation windows, when the configuration center determines that the experimental range can be expanded based on the difference rate and error rate, the experimental traffic ratio is adjusted to ten percent. The microservice entry point only needs to update the experimental range according to the new ratio without modifying the business code logic. This allows the experimental traffic coverage to be automatically expanded in subsequent requests, achieving smooth adjustment and controllable expansion of experimental traffic during database switching.

[0036] S3. The database routing component parses the stage identifier in the traffic marker, determines the access mode based on the stage routing matrix, executes the selected access mode between the source database access mode and the source-target database dual-path access mode on the experimental traffic, and generates a verification task. The specific implementation is as follows: In a real-world production environment, the database routing component is deployed within the access layer of a microservice. It controls the flow of each database access based on a switching strategy, without altering the upper-layer business interface. The database routing component is a logical module that maintains a routing matrix locally and selects between a source database access mode and a dual-path access mode (source and target database). The source database access mode involves initiating a single database access to the source database, while the dual-path access mode involves simultaneously initiating one database access to the source database and one to the target database within the same business call.

[0037] The database routing component pulls and caches the switching strategy corresponding to the current microservice database switching task from the configuration center. The switching strategy includes the task identifier, strategy version number, stage identifier, and stage routing matrix with stages as rows and normal traffic and experimental read-only traffic and read-write traffic as columns.

[0038] Normal traffic consists of business requests not selected for trial operation; experimental traffic consists of business requests selected according to rules to carry the new strategy; read-only traffic consists of requests that only trigger query operations; and read-write traffic consists of requests that may trigger both query and update operations. The microservice entry point has already attached a traffic tag to each request in the previous stage. The traffic tag includes at least the traffic type, experimental flag, and stage identifier. When the database routing component receives a call carrying a traffic tag, it first parses the traffic type, experimental flag, and stage identifier in the traffic tag. It then compares the stage identifier with the stage status in the local cache switching strategy. If they match, it locates the corresponding stage row in the stage routing matrix and then, based on whether it's normal traffic or experimental traffic, and read-only traffic or read-write traffic, locks the specific access mode entry under that row.

[0039] For normal traffic, the database routing component executes either source database access mode or source-target database dual-path access mode according to the corresponding entry in the stage routing matrix. For experimental traffic, the database routing component executes either source database access mode or source-target database dual-path access mode according to the corresponding entry in the stage routing matrix. When using the source-target database dual-path access mode, the query results of the source database and the query results of the target database, along with the request identifier, task identifier, and stage identifier, are assembled into a verification task.

[0040] A verification task is a record unit used for subsequent comparison of the source database results and the target database results. Preferably, it includes the query conditions for this access, the primary key set and corresponding field value set, the generation time, and a deduplication flag. The deduplication flag can be a combination of a task identifier, a request identifier, and a stage identifier. The database routing component appends verification tasks to the verification queue. The verification queue can be implemented as a message middleware topic or a database table, ensuring that verification tasks are stored in order of generation time and that each verification task is assigned a unique identifier so that downstream data verification and repair units can read them sequentially and prevent duplicate processing.

[0041] To mitigate the impact on online access latency, the database routing component prioritizes returning the source database query results to the upper-layer business logic after the source database returns its results. Then, a verification task is constructed in a separate execution unit and pushed to the verification queue. This ensures that the additional overhead from dual-path access is primarily concentrated in the backend channel. For read / write traffic, during the dual-write phase, the database routing component still uses a phased routing matrix to determine whether a read access to the target database is needed to generate a verification task. The specific actions of writing to the target database are handled by a subsequent local message table mechanism.

[0042] To ensure consistent behavior, when a new policy version is detected from the configuration center, the database routing component only uses the new stage routing matrix for newly arriving requests, and continues to complete the call for requests that have already entered the execution stage according to the stage identifier and access mode bound at the time of entry.

[0043] In a preferred embodiment, for order-type microservices in the read-only dual-write phase, the phase routing matrix can be set to use source-target database dual-path access mode only for experimental read-only traffic, and source database access mode for normal traffic and all read-write traffic. Under the premise that the peak request volume per second is kept at the existing level, the database routing component controls the additional resource consumption generated by dual-path access within a limited range, and at the same time continuously provides verification tasks containing request identifiers, task identifiers and phase identifiers for subsequent data verification and repair units, providing sufficient samples for advanced decision-making of the phase state machine.

[0044] S4. During the read-write dual-write phase, the microservice simultaneously writes to the business table and the local message table within the source database transaction. The background task asynchronously writes from the local message table to the target database based on the phase status and updates the message status according to the retry strategy. The specific implementation is as follows: During the read-write dual-write phase, the phase state machine is in a state that allows simultaneous writes to the source database and the target database. During this phase, the microservice initiates a source database transaction in the source database for each business update. A source database transaction is a transaction unit that maintains atomicity during the execution of the same business operation on the source database. The business table is a table in the source database that records business entities such as orders, accounts, and contracts. The local message table is a table in the source database specifically used to record future write instructions to the target database.

[0045] When a microservice receives a request identified as read / write traffic by the aforementioned traffic markers, it first updates the business table according to business rules within the same source database transaction, applying the add, modify, or delete actions to the corresponding records. Simultaneously, it inserts a message record into the local message table. This message record includes at least a task identifier, a stage identifier, a business primary key, the target database table name, the operation type, a snapshot of the field values, the message status, the number of retries, and a deduplication flag. The task identifier uniquely corresponds to a single microservice database switch task, the stage identifier is consistent with the current read / write dual-write stage, and the business primary key is used to locate the same business record in the target database. The target database table name indicates the table in the target database that needs to be written to. The operation type is used to distinguish between insert, update, and delete operations. The field value snapshot is the set of values ​​of the fields involved in the updated business record. The message status is used to mark the status of the message as pending, sending, completed, or permanently failed. The retry count records the number of times the message has been attempted to be written to the target database. The deduplication flag is preferably composed of the task identifier, business primary key, and operation type, and a unique constraint is applied to the local message table. This allows idempotent control to be achieved when the same business primary key is repeatedly written to the target database under the same switching task.

[0046] Ideally, the local message table and the business table should reside in the same source database instance. The operations team should agree on the field names, lengths, and constraints, and create corresponding indexes during the deployment phase. This ensures that the insert action and the business table update action belong to the same source database transaction. When the source database transaction is successfully committed, the write to disk of the business table and the local message table is completed simultaneously, thus ensuring that the source database still maintains complete write records when the target database is temporarily unavailable.

[0047] After the source database transaction is committed, a background task running within the microservice instance or a dedicated service node scans the local message table for messages in the pending state at fixed intervals. The background task determines whether extended writes to the target database are allowed based on the stage status. When the stage status is in the read-write dual-write stage, message records are retrieved sequentially by business primary key or by insertion time. Insert, update, or delete operations corresponding to the operation type are performed in the target database. When the target database returns success, the message status of the corresponding message record is updated to completed, and the retry count is no longer increased. When the target database returns failure due to insufficient resources, lock waiting, or rule conflicts, the background task maintains the message status in pending or marks it as postponed according to the preset retry strategy, and accumulates the retry count. Before reaching the set retry limit, the retry is postponed according to the backoff interval. Once the retry count exceeds the limit, the message status is marked as permanently failed, and the task identifier, business primary key, operation type, and failure reason are recorded in the monitoring system or audit table for subsequent intervention by operations and maintenance personnel.

[0048] Background tasks are preferably configured with fixed concurrency and throttling parameters. By limiting the number of records retrieved from the local message table and the upper limit of write batches initiated to the target database in each cycle, the write scale of the target database per unit time does not exceed the capacity assessment result, thus avoiding large-scale rewrites that prolong online transaction latency. In multi-instance deployment scenarios, sharding keys and instance identifiers can be added to the local message table, allowing different instances to be responsible for specific business primary key ranges, thereby reducing lock contention.

[0049] As an alternative approach, the local message table can also be hosted by an independent message service. Before the source database transaction is committed, the microservice persists the message corresponding to the task identifier, stage identifier, business primary key, target database table name, operation type, and field value snapshot to the message service through reliable calls, and attaches the aforementioned deduplication mark. The message service ensures the order and non-duplication of message persistence. Then, an independent message consumer program reads the message and performs write to the target database according to the stage status. The overall behavior is semantically consistent with the form of the local message table located in the source database.

[0050] In a preferred embodiment, for the scenario where the order microservice migrates from a local relational source database to a cloud database target database, the scanning cycle of the background task can be set to several seconds to several tens of seconds, the retry limit can be set to several times, the backoff interval can be gradually extended according to the number of retries, and the number of write requests initiated by a single instance to the target database per second can be set to a certain percentage of the historical peak write capacity. Under this configuration, even if the source database continues to receive high-concurrency writes, the target database can gradually complete the supplementary writing through multiple cycles under controllable load.

[0051] S5. The data verification and repair unit compares the source database results and target database results in the verification task, generates repair instructions for discrepancies, and performs insert, update, and delete operations in the target database. Simultaneously, it calculates the discrepancy rate and the number of repair failures. The specific implementation is as follows: During the microservice database switching process, the data verification and repair unit is deployed as an independent service or along with the microservice. It is used to compare the results of the source database with the results of the target database and drive the data correction of the target database without changing the online business logic.

[0052] The data verification and repair unit refers to a functional module that can read verification tasks sequentially from the verification queue, identify difference records, and form a set of repair instructions. The verification queue is a persistent channel for storing verification tasks, which can be implemented in the form of message middleware topics or dedicated tables. Each verification task includes at least a task identifier, a stage identifier, a request identifier, query conditions, a set of primary keys, and a set of field values ​​corresponding to the source database results and the target database results.

[0053] A difference record refers to a situation where, under the premise of identical primary keys or unique keys, the target database is missing records, the field content is inconsistent with the source database, or the target database contains records that the source database has not returned. Repair instructions are record units that generate insert, update, and delete instructions for each difference record. The data verification and repair unit compares the source database results with the target database results in the verification task one by one according to the primary key. When a record with a missing primary key is found in the target database, an insert instruction is constructed, which includes the task identifier, stage identifier, target database table name, primary key value, and the set of field values ​​to be added. When a record exists in the target database but the field values ​​are inconsistent with the source database field values, an update instruction is constructed, which includes the task identifier, stage identifier, target database table name, primary key value, and the fields and values ​​to be overwritten. When a record exists in the target database that is not returned by the source database and meets the pre-agreed deletion rules, a delete instruction is constructed, which includes the task identifier, stage identifier, target database table name, and primary key value.

[0054] The generated repair instructions are written to the repair queue or repair table in the form of records. The repair queue and repair table use unique constraint fields to store the deduplication mark formed by the combination of task identifier, stage identifier and primary key, in order to prevent the same difference record from being repaired repeatedly.

[0055] The repair execution logic retrieves repair instructions from the repair queue in batches according to their generation time. For multiple repair instructions with the same primary key, they are sorted by generation time. When a later instruction overwrites a previous instruction, the latest instruction is retained first and the old instruction is discarded. Then, insert, update, or delete operations are executed sequentially in the target database. Successful instructions are marked as successful in the repair queue. For instructions that fail due to constraint conflicts, resource limitations, or version conflicts, the reason for failure is recorded and the number of retries is increased according to the preset retry rules. If the number of retries has not reached the upper limit, the instruction is put back into the queue. If the number of retries reaches the upper limit, the instruction is marked as permanently failed and the task identifier, stage identifier, primary key value, and failure reason are written to the audit table or monitoring system.

[0056] The data verification and repair unit counts the number of records participating in verification and the number of discrepancies within a fixed time window. The ratio of the number of discrepancies to the number of records participating in verification is recorded as the discrepancy rate. At the same time, it counts the number of repair instructions added to the repair queue, the number of repair instructions marked as permanently failed, and the number of times the target database access failed during the execution of verification queries. The discrepancy rate and the number of repair failures are used as indicators of switching quality.

[0057] The aforementioned metrics are written to the monitoring database in the form of records containing task identifiers, stage identifiers, start and end times of the time window, difference rate, number of repair failures, and number of target database access failures. Field names and units in the monitoring database are uniformly defined in the configuration center and periodically transmitted back to the configuration center and monitoring system via a fixed set of fields for use by the stage state machine in decision-making. To ensure order and idempotency, verification tasks are bound to unique identifiers when written to the verification queue, and the repair queue prevents duplicate repairs through the aforementioned deduplication flag. Only one final effective repair instruction is retained for the same primary key within the same time window.

[0058] In terms of compliance, the data verification and repair unit only accesses tables and fields that have been whitelisted. The whitelist records the table names, field names, and uses that are allowed to participate in verification and repair. Fields containing personal identification features are partially masked in the logs and audit tables. All insert, update, and delete actions performed in the target database are recorded in the audit log with task identifiers and stage identifiers so that the specific repair process can be restored during subsequent inspections.

[0059] In a preferred embodiment, for database switching of order-type microservices, the time window can be set to one hour, the difference rate threshold can be set to one-thousandth, and the number of permanent repair failures allowed within each time window can be set to a limit. When the difference rate remains below the threshold and the number of permanent repair failures is zero within several consecutive time windows, the configuration center determines that the target database data quality is stable. It can then drive the phase state machine from the read-only dual-write phase to the read-write dual-write phase or from the read-write dual-write phase to the read-switch target database phase. Engineering technicians can fully realize the comparison and repair of the source database results and the target database results by following the above field definitions and sequence control, and achieve continuous tracking and repair of data consistency during the database switching process.

[0060] S6. The configuration center aggregates difference rate, error rate, and latency indicators according to time windows to drive the stage state machine transition. When the advanced conditions are met, the next stage strategy is issued. When the indicators exceed the alarm threshold, the transition back to the safe stage is completed and a rollback strategy is issued. The specific implementation is as follows: During the microservice database switching process, the configuration center aggregates difference rate, error rate, and latency indicators according to a pre-set time window, and drives the migration of the phase state machine accordingly, thereby completing the gradual switch from the source database to the target database without interrupting business operations.

[0061] The time window refers to a fixed-duration monitoring period used to statistically analyze operational quality indicators. The observation window refers to an interval composed of several consecutive time windows, used to determine whether a stage can be advanced or needs to be rolled back. The difference rate refers to the ratio of the number of difference records to the number of records participating in the verification within the observation window. The number of difference records is statistically obtained by the aforementioned data verification and repair unit based on the verification task comparison results. The number of records participating in the verification is the total number of records included in the verification within the observation window. The error rate refers to the ratio of the number of failed target library accesses to the number of access requests within the observation window. The number of failed target library accesses and the number of access requests are recorded by the access monitoring module. The latency indicator refers to the statistical value of the target library access latency within the observation window, preferably using the high quantile value to represent the upper limit of the response time.

[0062] At the end of each time window, the configuration center reads the difference rate, error rate, and latency indicators summarized in the previous time window from the monitoring database according to the task identifier and stage identifier. It then stitches the indicators from multiple consecutive time windows together to form a snapshot of the current observation window's stage operation and compares it with the advanced conditions and alarm thresholds predefined by the switching task. The advanced conditions refer to the difference rate not exceeding the difference rate threshold, the error rate not exceeding the error rate threshold, the latency indicator not exceeding the latency threshold, and the number of time windows included in the observation window reaching a set value. The alarm thresholds refer to the upper limits that the difference rate, error rate, and latency indicators are allowed to reach.

[0063] When the configuration center detects that a snapshot of a certain phase meets the advancement conditions, it advances the phase state machine from the current phase to the next phase, generates a new policy version for the switching task, and updates the new phase identifier, phase routing matrix, experimental traffic ratio, and monitoring threshold into the policy record library. The version number is used to lock and record each policy change. Subsequently, the new switching policy is issued to the relevant microservices through the aforementioned microservice-oriented configuration interface, so that the microservices adjust their access patterns for normal traffic and experimental traffic according to the new phase identifier and phase routing matrix in the next time window.

[0064] If the configuration center finds that the difference rate, error rate, or latency indicators exceed the corresponding alarm threshold in any time window, it determines that there is an operational risk in the current stage and migrates the stage state machine back to the pre-set safe stage. The safe stage refers to the stage where no more write behavior is extended to the target database and the source database is used first. At the same time as the migration, the configuration center generates a rollback policy, restores the stage identifier to the value corresponding to the safe stage, restores the stage routing matrix to the configuration with the source database access mode as the main focus, and issues the rollback policy through the configuration interface. After receiving the rollback policy, the microservice quickly restores the access mode to the state corresponding to the safe stage, thereby completing the rollback without modifying the business code.

[0065] To ensure the controllability and idempotency of phase migration operations, the configuration center defines a minimum set of fields for the phase migration interface, including task identifier, policy version number, phase identifier, migration reason, and timestamp. The migration reason is used to record whether it is a normal advancement or an alarm rollback, and the timestamp is used to record the time when the migration occurred. When a migration request is received, the configuration center first verifies whether the task identifier exists, whether the phase identifier is consistent with the current record in the task database, whether the policy version number is not earlier than the current version, and whether the migration direction belongs to the allowed path of the phase state machine. If any condition is not met or the corresponding observation window indicator is missing in the monitoring database, the configuration center returns the agreed error identifier and keeps the current phase unchanged, thereby avoiding abnormal phase state switching due to incomplete information or external misoperation.

[0066] Through the above mechanism, the configuration center of this invention can automatically adjust the stage of the stage state machine based on the combination of difference rate, error rate and latency indicators in scenarios such as e-commerce orders, payment clearing and settlement, and power customer management that require long-term operation and smooth database migration. This enables the database switching process to have both a clear progression path and the ability to quickly fall back to a safe stage when the indicators deteriorate.

[0067] In a preferred embodiment, for a migration scenario of an e-commerce order microservice, the experimental traffic ratio can be initially set to 1% of the total number of requests in the read-only dual-write phase, the time window length can be set to one hour, and the observation window can be set to several consecutive time windows. In the read-only dual-write phase, when the difference rate remains below 0.1% for two consecutive days within the observation window, the number of permanent failures to be repaired is zero, and the high percentile value of the target database access latency is lower than the estimated upper limit, the configuration center will advance the phase state machine from the read-only dual-write phase to the read-write dual-write phase and increase the experimental traffic ratio to 10%. In the read-write dual-write phase, when the above indicators remain stable for one consecutive day within the observation window, the system will advance to the read-to-target database phase, and during off-peak hours, it will advance to the full target database phase. If the error rate or latency indicator exceeds the corresponding alarm threshold in the observation window of any phase, the configuration center will immediately migrate the phase state machine back to the safe phase and issue a rollback policy. Engineering technicians can implement the driving logic and policy version management of the phase state machine based on the above field definitions, phase conditions, and interface constraints.

[0068] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for switching between gray-scale dual-write operations in a microservice database based on traffic marking, characterized in that, include: S1. The configuration center registers the microservice database switching task, establishes a phase state machine, generates and distributes a switching strategy carrying source database and target database identifiers and a phase routing matrix. S2. The microservice entry point calculates a hash value based on the request identifier to select experimental traffic and adds a traffic tag containing the traffic type, experimental flag, and stage identifier to the request. S3. The database routing component parses the stage identifier in the traffic tag, determines the access mode according to the stage routing matrix, executes the selected access mode between the source database access mode and the source database-target database dual-path access mode on the experimental traffic, and generates a verification task. S4. During the read-write dual-write phase, the microservice writes to the business table and the local message table simultaneously within the source database transaction. The background task asynchronously writes from the local message table to the target database according to the phase status and updates the message status according to the retry strategy. S5. The data verification and repair unit compares the source database results and the target database results in the verification task, generates repair instructions for the difference records, and performs insert, update and delete operations in the target database. At the same time, it calculates the difference rate and the number of repair failures. S6. The configuration center aggregates difference rate, error rate and latency indicators according to the time window to drive the stage state machine migration. When the advanced conditions are met, the next stage strategy is issued. When the indicators exceed the alarm threshold, the migration back to the safe stage is completed and the rollback strategy is issued.

2. The microservice database canary dual-write switching method based on traffic marking according to claim 1, characterized in that, S1 includes: The configuration center registers microservice database switching tasks based on the existing network configuration repository and monitoring system, and generates a switching strategy that includes source database identifier, target database identifier, stage identifier and stage routing matrix; The stage routing matrix is ​​arranged by stage and by normal traffic and experimental traffic (read-only traffic and read-write traffic) as columns. Each unit is used to select between source database access mode and source database / target database dual access mode. The switching strategy, along with the difference rate threshold, error rate threshold, latency threshold, and dwell time threshold, is stored in the configuration center task library and distributed to the corresponding microservices through a unified interface; Before each deployment, the configuration center performs a consistency check based on the task identifier, stage status, and policy version number, and records the serial number, operator identifier, and policy content summary.

3. The microservice database canary dual-write switching method based on traffic marking according to claim 1, characterized in that, S2 include: The microservice entry point obtains the phase identifier and experimental traffic ratio from the switching strategy; Extract a request identifier for each service request, consisting of the user ID, tenant ID, and session ID; The hash value is calculated according to the pre-agreed hash rules, and the requests are divided into two categories, experimental traffic and normal traffic, according to the hash interval corresponding to the proportion of experimental traffic. The microservice entry point adds traffic tags to the requests, which include traffic type, experiment flag, and phase identifier; Write the request header fields and keep them consistent in the thread context. The database routing component selects the access mode in the stage routing matrix based on the stage identifier in the traffic tag.

4. The microservice database canary dual-write switching method based on traffic marking according to claim 1, characterized in that, S3 includes: The database routing component obtains the phase routing matrix from the switching policy and parses the traffic type, experiment flag, and phase identifier from the traffic tags; Based on the stage routing matrix, the read-only traffic and read-write traffic of normal traffic and experimental traffic are respectively determined to be one of the source database access mode and the source database and target database dual-path access mode. When performing dual-path access mode for source and target databases on experimental traffic, the source and target database results, along with the request identifier, task identifier, and stage identifier, are used to generate a verification task and write it into the verification queue.

5. The microservice database canary dual-write switching method based on traffic marking according to claim 1, characterized in that, S4 includes: During the read-write dual-write phase, for requests marked as read-write traffic, the microservice updates the business table and inserts message records into the local message table within the same source database transaction; The message record includes task identifier, stage identifier, business primary key, target database table name, operation type, field value snapshot, message status, retry count, and deduplication flag; The deduplication tag consists of a task identifier, a business primary key, and an operation type, and a unique constraint is set on the local message table to achieve idempotent control over writing the same business primary key to the target database within the same switching task.

6. The microservice database canary dual-write switching method based on traffic marking according to claim 5, characterized in that: After the source database transaction is committed, the background task scans the local message table at fixed intervals for message records in the pending state and corresponding read / write dual-write stages. Execute insert, update, and delete operations in the target database according to the insertion time sequence, corresponding to the operation type. When the target database returns a successful response, the message status is updated to "completed". When the target database returns a failed response, the number of retries is increased according to the preset retry strategy and a delayed sending is set. When the number of retries exceeds the limit, mark the message status as permanently failed and write the task identifier, business primary key, operation type and failure reason to the monitoring log. The background task limits the number of messages read from the local message table and the number of write batches executed in the target database in each cycle by using fixed concurrency and throttling parameters.

7. The microservice database canary dual-write switching method based on traffic marking according to claim 1, characterized in that, S5 include: Set up a data verification and repair unit to sequentially read verification tasks from the verification queue, which include task identifier, stage identifier, primary key set, source database result field value set, and target database result field value set. Based on the primary key, determine the difference record and generate a repair instruction carrying the task identifier, stage identifier, target database table name, primary key value and field value set, and write it to the repair queue with unique constraints on the task identifier, stage identifier and primary key value; Within a fixed time window, the number of records participating in verification and the number of discrepancy records are counted to obtain the discrepancy rate. The task identifier, stage identifier, start and end time of the time window, discrepancy rate, and number of repair failures are written into the monitoring database.

8. The microservice database canary dual-write switching method based on traffic marking according to claim 1, characterized in that, S6 include: The configuration center aggregates difference rate, error rate, and latency indicators from the monitoring database based on task and stage identifiers according to the time window to form a stage operation snapshot for the observation window; When a phase snapshot meets the preset advanced conditions, the phase state machine is migrated from the current phase to the next phase and a new policy version is generated. When any indicator exceeds the alarm threshold, the stage state machine is transitioned to a safe stage and a rollback strategy is issued.

Images