A long chain runtime configuration drift detection and consistency coordination method

Abstract

The application discloses a Chang'an chain runtime configuration drift detection and consistency coordination method, relates to the technical field of distributed system configuration management, and deploys a main controller and a query-sidecar double container in an Operator controller Pod, the query-sidecar is provided with a built-in client tool and a hierarchical cache, and the main controller obtains on-chain configuration through a localhost interface; in a Reconcile cycle, configuration drift is determined through normalization processing and hash comparison, and the source of change is identified in combination with custom resource metadata and chain-side change metadata; coordination is performed according to four synchronization modes, namely, CR forcing, chain observation, bandwidth limited CR forcing and manual arbitration; finally, the custom resource state and the Conditions state are updated, and key information is recorded, so that the problems of state ambiguity and error coverage caused by double fact sources are solved, low-delay detection and flexible coordination are realized, and the consistency of Chang'an chain runtime configuration is ensured.

Description

Technical Field

[0001] This invention relates to the field of distributed system configuration management technology, and in particular to a method for runtime configuration drift detection and consistency coordination of Chang'an Chain based on Kubernetes Operator. Background Technology

[0002] In the declarative architecture of Kubernetes, the core responsibility of the Operator is to continuously reconcile the "expected state" (the intent declared by the user in the CR (Custom Resource)) with the "actual state" (the current running state of the cluster). 'State drift' refers to the phenomenon where the actual state deviates from the expected state unexpectedly. This is typically caused by two situations: first, human intervention bypassing the Operator (e.g., directly modifying the underlying ConfigMap or Deployment); second, environmental changes (such as node failure, resource deletion by external processes, etc.) causing the system state to no longer conform to the definition.

[0003] For stateless applications or database systems, the Operator only needs to focus on the state of Kubernetes resources (such as Pods, Services, ConfigMaps, etc.), and the CR statement is the sole source of fact. However, blockchain systems have special characteristics. Taking Chang'an Chain as an example, it includes the following situations:

[0004] 1. Dual sources of truth: In addition to the network configuration declarations in Kubernetes CR, the blockchain maintains its own configuration. The on-chain configuration is also an independent source of state. Moreover, the on-chain configuration can be modified in real time and take effect immediately by sending transactions to the on-chain via the SDK or cmc command-line tool, and synchronized to all nodes. The whole process does not require restarting the nodes. However, some network parameter modifications may have been declared through CR, which causes inconsistencies.

[0005] 2. Emergency Operation and Maintenance Needs: In production environments, when emergencies such as node failures occur, operations and maintenance personnel often need to respond quickly (e.g., through the CMC command-line tool) and cannot wait for slower declarative change processes such as modifying CR or triggering CI / CD processes.

[0006] In the existing Hyperledger Fabric fabric-operator project, since operations such as adding a node to a channel (represented as a chain in Fabric) and updating a channel (such as changing channel configuration) are business layer operations, the operator does not care about or coordinate these changes, so there is no issue of dual source of fact for configuration.

[0007] However, for blockchain platforms like Chang'an Chain, where chain networking and node configuration are tightly coupled, existing Operator implementations cannot solve the following problems:

[0008] Issue 1: Ambiguous Status. When the Operator detects a discrepancy between the CR configuration and the actual on-chain configuration, it cannot determine whether the intention behind the change was a user modifying the CR to update the on-chain configuration, or an administrator directly modifying the on-chain configuration via CMC.

[0009] Question 2: Risk of incorrect overwriting. A simple Operator implementation might assume that all differences need to be corrected, leading to the incorrect overwriting of an administrator's urgent actions.

[0010] Question 3: Query performance issues. To detect the difference between the on-chain state and the CR state, the Operator needs to frequently query the on-chain configuration. Simply integrating the chain SDK will make the Operator bloated, but if a temporary query Pod or Job is created every time, it will generate huge resource overhead and latency (Pod startup time is usually 10-30 seconds), which cannot meet the needs of real-time monitoring.

[0011] Question 4: Lack of flexible governance models. Different organizations have different governance requirements for configuration management. Some require a strict GitOps model (CR is the sole source of fact), some need to retain the flexibility of traditional command-line tools, and some require manual intervention to make decisions when conflicts are detected. Existing Operator implementations do not support multiple governance models.

[0012] In summary, addressing the issues of state drift detection, change intent identification, and automatic coordination decision-making in blockchain operators when faced with dual sources of fact is a research topic of significant practical importance. Summary of the Invention

[0013] In view of this, the purpose of this invention is to propose a method for runtime configuration drift detection and consistency coordination of Chang'an Chain. This method aims to solve the problem of how to continuously detect and coordinate the differences between the CR declared configuration and the actual on-chain configuration after the Chang'an Chain network has completed initialization and entered steady-state operation.

[0014] To achieve the above-mentioned technical objectives, the technical solution adopted by this invention is as follows:

[0015] A method for runtime configuration drift detection and consistency coordination of Chang'an Chain, which manages Chang'an Chain through an Operator in a Kubernetes cluster, includes the following steps:

[0016] S1. Run an Operator in a Kubernetes cluster to manage the Chang'an Chain network. Run a main controller container and a query-sidecar container within the controller Pod of the Operator. The query-sidecar container has built-in Chang'an Chain client tools, a caching module, and an HTTP service. The main controller container calls the query-sidecar container through the localhost interface to obtain the on-chain configuration status.

[0017] S2. In each Reconcile loop, the main controller container calls the query-sidecar container to query the on-chain configuration status. The on-chain configuration status includes at least the runtime configuration of the Chang'an Chain network (including the consensus node list, trust root certificate, block parameters) and chain-side configuration change metadata. The query-sidecar container returns the query result directly when the cache is hit, and accesses the blockchain node to obtain the query result and writes it to the cache when the cache is not hit.

[0018] S3. Normalize the expected configuration of the custom resource (ChainMakerNetwork) and the actual on-chain configuration and calculate the configuration hash value. Compare the two hash values ​​to determine whether there is configuration drift. When the two hash values ​​are inconsistent, it is determined that there is configuration drift.

[0019] S4. When it is determined that there is configuration drift, perform change source identification: compare the metadata.generation of the custom resource with the status.observedGeneration of the custom resource to determine whether there is a change in the custom resource; and compare the chain-side configuration change metadata lastConfigBlockHeight and / or lastConfigSequence with the corresponding record value in the status of the custom resource to determine whether there is an on-chain configuration change, thereby outputting the change source identification result;

[0020] S5. Perform consistency coordination based on the preset synchronization mode field configSync.mode and the change source identification result, wherein the synchronization mode includes one of the following:

[0021] (1) CR Forced Mode: Create a configuration synchronization Job and execute the Chang'an Chain client tool command to synchronize the desired configuration of the custom resource to the chain;

[0022] (2) Chain observation mode: Update the status.observedConfig field of the custom resource to reflect the actual configuration on the chain;

[0023] (3) Bandwidth-limited CR forced mode: When the custom resource change is detected, a configuration synchronization job is created to synchronize the expected configuration of the custom resource to the chain; when the on-chain configuration change is detected, an alarm is issued and the drift time is recorded. Forced synchronization is not performed within the preset grace period. If configuration drift still exists after the grace period expires, a configuration synchronization job is created to roll back the on-chain configuration to the expected configuration of the custom resource.

[0024] (4) Manual arbitration mode: Set DriftDetected Condition and wait for the administrator to indicate the synchronization direction through annotation before performing the corresponding consistency coordination;

[0025] S6. After completing the consistency coordination process or the chain observation status update, update the status sub-resource of the custom resource and the Kubernetes standard Conditions status, and record the status information used to characterize the drift detection and consistency coordination results (including observedGeneration, lastConfigBlockHeight, lastConfigSequence, configHash, lastSyncTime, and drift detection status information).

[0026] As a possible implementation, further, in step S2 of this solution, the query-sidecar container sets a tiered cache validity period (TTL) for different types of configuration items, and the TTL includes at least: a first cache validity period of 10 seconds for configuration items with high frequency of change, a second cache validity period of 60 seconds for configuration items with medium frequency of change, and a third cache validity period of 300 seconds for configuration items with low frequency of change.

[0027] As a possible implementation, further, in step S1 or S2 of this solution, the query-sidecar container subscribes to the configuration change event stream of the CHAIN_CONFIG system contract of Chang'an Chain. When it receives a configuration change event, it performs active invalidation on the cache key corresponding to the configuration change event and sends a change notification to the main controller container to trigger the Reconcile loop.

[0028] As a preferred implementation option, preferably, in step S1 or S2 of this scheme, in order to initialize the cache used for on-chain queries, the query-sidecar container performs a full on-chain configuration query after restarting to obtain the current configuration and lastConfigBlockHeight and / or lastConfigSequence, and establishes an event subscription connection after initializing the cache.

[0029] As a possible implementation, further, in step S2 of this solution, the query-sidecar container is configured to access multiple blockchain nodes, and a polling strategy or a minimum latency strategy is used to select the target blockchain node to execute the on-chain configuration query.

[0030] As a possible implementation, further, in step S6 of this scheme, the drift detection status information includes at least driftDetected, driftDetectedAt, and driftDetails, where driftDetails is used to characterize the differences in configuration drift.

[0031] As a possible implementation, the normalization process in step S3 of this solution further includes at least sorting the configuration fields, removing whitespace characters, and generating normalized configuration text for hash calculation using unified serialization rules, so as to ensure that configurations with the same semantics generate the same hash value.

[0032] As a possible implementation, further, in step S4 of this scheme, when the chain-side configuration change metadata lastConfigBlockHeight cannot be obtained, the increment of lastConfigSequence is used as the basis for determining the on-chain configuration change, and the comparison of configHash is used to avoid unnecessary consistency coordination triggered when the metadata is updated but the configuration content remains unchanged.

[0033] As a possible implementation, further, in step S5 of this scheme, in the manual arbitration mode, the administrator specifies the synchronization direction by setting an annotation in the format chainmaker.io / sync-direction:CR-to-chain or chainmaker.io / sync-direction: chain-to-CR for the custom resource. The main controller container clears the annotation after performing consistency coordination according to the synchronization direction.

[0034] As a possible implementation, step S6 of this solution further includes: recording monitoring indicators, which at least include the number of configuration drifts, the configuration synchronization time, and the total number of reconciles, for monitoring and alarm purposes.

[0035] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art:

[0036] This solution improves the certainty and predictability of configuration governance by eliminating the risk of erroneous overwriting caused by ambiguity in the "source of change" status. Traditional approaches often only consider whether the "current value is consistent," making it difficult to distinguish between "declarative (CR) changes" and "imperative (on-chain CMC / SDK) changes," leading to problems such as "the controller rollback immediately after emergency measures" or "mistakenly treating legitimate on-chain changes as drift and forcibly overwriting." This invention's solution jointly tracks CR-side metadata (generation / observedGeneration) and on-chain configuration change metadata (lastConfigBlockHeight / sequence) during the arbitration process, and uses configuration content hash verification as a fallback, thereby enabling more accurate identification of the source and sequence of changes and avoiding erroneous overwriting.

[0037] In terms of synchronization modes, this solution covers governance needs across all scenarios through four synchronization modes, enabling configurable implementation of "the same system with different governance styles." This invention provides synchronization modes such as CRD (mandatory), chain (observation), CRD-with-warning (mandatory grace period), and manual (manual arbitration). This allows the system to support both the strict GitOps / CI-CD principle of "CR as the sole source of fact," and the common "chain as the sole source of fact" read-only observation window in production operations, as well as manual arbitration in scenarios with strong oversight / approval requirements. This avoids the need to repeatedly develop multiple sets of control logic for different organizations and network environments, significantly reducing the fragmented costs of the operations and maintenance system.

[0038] The grace period mechanism in this solution significantly reduces production and maintenance risks, balancing automation and emergency response. In CRD-with-warning mode, when an increase in `lastConfigBlockHeight / sequence` is detected on the chain side, the system first issues a warning and records the drift occurrence time `driftDetectedAt`. During the grace period, forced synchronization is not immediately enforced. If the user has not updated the CRD after the grace period ends, a forced rollback / synchronization is then performed, thus providing a decision-making window for emergency fault handling and preventing "emergency changes from being instantly overwritten." Test conclusion: This mechanism can reduce the risk of misoperation by more than 80%.

[0039] This solution enables low-latency and low-overhead on-chain queries. Through caching and event-driven mechanisms, it achieves faster system response and is more "chain-efficient." The solution centrally deploys the query sidecar within the controller Pod. The main controller calls via localhost, prioritizing responses from the cache. Only if the cache misses does it access the chain node, reducing the overhead of repeated queries.

[0040] Furthermore, this solution utilizes the Sidecar's ability to subscribe to the configuration change event stream of the system contract CHAIN_CONFIG. After EventHub pushes an event, the Sidecar immediately invalidates the corresponding cache key and notifies the controller to trigger Reconcile, avoiding complete reliance on timed polling. This "event-driven" mechanism reduces the configuration change awareness latency from an average of 15 seconds to 1–2 seconds, improving response speed by approximately 10 times, while also reducing the query pressure on blockchain nodes.

[0041] This solution offers a simpler and more stable drift detection logic. By employing cross-format normalization and hash comparison, it reduces the complexity and risk of omissions associated with field-by-field comparisons. Since the CR spec (YAML / JSON) and the on-chain configuration data structure may differ in field naming and arrangement, direct field-by-field comparisons are not only complex but also prone to errors. This invention first normalizes the configurations on both sides (e.g., sorting, removing spaces) before calculating and comparing hashes. This ensures that "same semantics → same hashes" and serves as a fallback check to prevent omissions caused by caching, untimely metadata updates, etc., while simultaneously reducing implementation complexity.

[0042] This solution offers strong observability and auditability, and seamlessly integrates with the native Kubernetes ecosystem. It adopts the standard Kubernetes Conditions pattern and retains event logs, allowing operations personnel to directly observe states such as "whether migration has occurred / whether synchronization has been completed" using native commands like kubectl. It also facilitates integration with standard monitoring systems and audit trails such as Prometheus, meeting production and compliance requirements.

[0043] This solution enhances robustness by supporting fallback mechanisms for "event loss / unavailability" and self-healing upon startup. When lastConfigBlockHeight cannot be obtained or event subscriptions are lost, the system can use sequence incrementing as a fallback change identifier. Furthermore, after a Sidecar restart, a full query can be performed to retrieve the current configuration and configBlockHeight / sequence, then the cache is initialized and subscriptions are restored, improving long-term operational stability. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1This describes the complete architecture of the drift detection system involved in this scheme and the relationship between its components;

[0046] Figure 2 This is the working mechanism by which the Sidecar involved in this solution evolved from the "active polling query" mode to the "event-driven + fallback query" mode;

[0047] Figure 3 This is the complete decision-making process of the arbitration engine involved in this solution and the processing of the four synchronization modes;

[0048] Figure 4 This is the complete closed loop of the Reconcile process (S01-S07) involved in this solution method. Detailed Implementation

[0049] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be particularly noted that the following embodiments are for illustrative purposes only and do not limit the scope of the invention. Similarly, the following embodiments are only some, not all, embodiments of the present invention, and all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] This embodiment is implemented after the blockchain network has completed initialization and entered a steady state. It solves the problem of continuously detecting and coordinating the differences between the CR declaration configuration and the actual on-chain configuration. In other words, this embodiment focuses on the "runtime configuration management phase" and solves the problem of "how to keep the configuration consistent after it is running".

[0051] The nature of the dual source of facts problem

[0052] The core problem addressed in this embodiment stems from the architectural characteristics of consortium blockchains like Chang'an Chain: Chang'an Chain supports real-time modification of on-chain configurations via system contracts. These changes are triggered by the CMC command-line tool (Chang'an Chain client) and take effect immediately, without requiring a node restart. This characteristic, after the introduction of Kubernetes Operator for declarative management, creates two legitimate "sources of fact," specifically as follows:

[0053] (1) Declarative Source of Facts (CR): The spec part of ChainMakerNetwork (defined for the Chang'an Chain Network) represents the "expected state" declared by the user in Kubernetes. This is the input to the Operator, conforming to the cloud-native declarative management philosophy, with all changes having a complete Git history and audit logs.

[0054] (2) Imperative Fact Source (cmc CLI): Administrators execute commands directly through the cmc tool, which modifies the real-time, truly effective "on-chain state". This is the traditional method of blockchain operation and maintenance, which is immediate and flexible, and indispensable in emergency fault handling.

[0055] The technical approach of this embodiment includes the following:

[0056] 1. Three-tier architecture design

[0057] To address the aforementioned issues, this solution proposes a three-tier configuration coordination system, which includes:

[0058] First layer: Status query layer

[0059] Deploy a Sidecar container in the Operator controller Pod to achieve low-latency, high-efficiency querying of on-chain configurations. The Sidecar container can have a built-in ChangAn Chain client (which can be a CMC tool or SDK) and a caching mechanism, providing the controller with millisecond-level state query services through the localhost interface, mainly solving the performance problem of "how to efficiently obtain on-chain state".

[0060] Second layer: Arbitration engine

[0061] Simultaneously tracking the Kubernetes resource's generation field (indicating the number of times the CR specification has been modified) and chain-side configuration change metadata helps determine the source and intent of configuration changes, primarily addressing the ambiguity of "how to distinguish the intent of a change." Chain-side configuration change metadata can come from two sources:

[0062] (1) Obtain the configuration transaction confirmation height lastConfigBlockHeight by subscribing to the event;

[0063] (2) Directly query the on-chain configuration to obtain the configuration version number sequence.

[0064] Third layer: Synchronization strategy

[0065] Following the "single source of fact" principle, it provides four configuration synchronization strategies (crd, chain, crd-with-warning, manual), mainly addressing the decision-making problem of "how to handle after detecting differences" and meeting the governance needs of different scenarios.

[0066] The query layer provides low-latency state awareness, the arbitration engine provides deterministic judgment of change intent, and the synchronization strategy provides flexible execution strategy selection. Together, they form a complete "perception-judgment-execution" closed loop.

[0067] Figure 2 This demonstrates the complete architecture of the drift detection system and the relationships between its components. The system employs a three-tier architecture, which includes the following:

[0068] (1) Operator Pod Dual-Container Architecture: Deployed in the Operator's namespace (e.g., chainmaker-system), it includes a main controller container (Network Controller) and a Query-Sidecar container. The two containers communicate via the localhost:8080 interface with millisecond-level latency, without needing to go through the Kubernetes network stack. The Sidecar has a built-in HTTPServer, caching module (distinguishing TTLs such as 60-300 seconds), and CMC tools to maintain gRPC connections to multiple blockchain nodes.

[0069] (2) Blockchain network namespace resources: including ChainMakerNetwork CR (core state tracking, including fields such as observedGeneration, lastConfigBlockHeight, lastConfigSequence, configHash, driftDetected, etc.), global ConfigMap (bc.yml) and configuration synchronization Job (executes cmc commands).

[0070] (3) Dual source of facts and event-driven: Users can declaratively change CR via kubectl or imperatively change on-chain configuration via cmc. The on-chain EventHub pushes configuration change events to the Sidecar. After parsing the events, the Sidecar actively invalidates the relevant caches and notifies the Controller to trigger Reconcile.

[0071] 2. Status query mechanism design

[0072] 2.1 Sidecar Design

[0073] Since on-chain state queries require a persistent, reusable query component, a sidecar container is considered for use in the Kubernetes ecosystem. This means that in the Operator's controller Pod, in addition to the main controller container, a query-sidecar container is deployed separately.

[0074] The main controller container is responsible for executing the Reconcile loop, state comparison, arbitration decision-making, and resource creation. The query-Sidecar container is responsible for querying the on-chain configuration state. It has a built-in CMC command-line tool and a caching module, does not participate in any decision-making logic, and maintains its functional independence.

[0075] The main controller and the query-sidecar container are in the same Pod, so the controller can communicate with the sidecar through the localhost HTTP interface (such as http: / / 127.0.0.1:8080) with a latency in the millisecond range. There is no need to configure a Service or Ingress, and the sidecar cannot be directly accessed from outside, so the security is high.

[0076] The query-sidecar can be configured to access multiple chain nodes, using either a round-robin or lowest-latency strategy to select nodes for querying. Query results are cached at multiple levels based on TTL, with different cache durations set according to the frequency of configuration changes. For example:

[0077] (1) Configurations that do not change frequently (such as trusted root certificates trust_roots): TTL is set to 300 seconds to reduce repeated queries.

[0078] (2) Configuration with medium frequency of change (e.g., consensus node list consensus.nodes): TTL is set to 60 seconds to balance consistency and performance.

[0079] (3) Configuration for high-frequency changes: TTL is set to 10 seconds to ensure timeliness.

[0080] The specific fields in the above configuration categories are just examples for reference. When implementing this solution, you can flexibly configure them according to the actual needs of the scenario.

[0081] Simultaneously, it supports two cache expiration mechanisms: event-based proactive expiration (listening to on-chain configuration events) and time-based automatic expiration. Caching can reduce most actual on-chain queries, thereby reducing the query pressure on blockchain nodes.

[0082] 2.2 Sidecar Deployment Location Decision

[0083] Regarding the placement / deployment location of the query-sidecar, this invention chooses a centralized sidecar (deployed within the controller Pod), which offers low communication latency (milliseconds), requires the fewest sidecar containers, is simple to maintain, and can support a scale of hundreds of nodes through caching. Other optional deployment schemes include:

[0084] (1) Node level: Deploy a Sidecar within each blockchain node Pod, and the controller accesses the Sidecar of each node across Pods.

[0085] (2) Independent service: Deploy the Sidecar independently as a Deployment or DaemonSet, and provide query services through Service.

[0086] When a sidecar is deployed as a standalone service, it can be horizontally scaled using HPA to support a larger number of nodes (e.g., when the number of networks exceeds 10 and the number of nodes exceeds 1000). Centralized sidecar deployments can also be seamlessly migrated to the "standalone service" approach without modifying the core arbitration logic of the controller; only the query URL needs to be switched via environment variables.

[0087] 2.3 Event-Driven Enhancement

[0088] As an optional solution, this Sidecar supports an event-driven model to further reduce query overhead and improve response speed. The Sidecar can subscribe to the event stream of the Chang'an Chain's system contract (CHAIN_CONFIG) to listen for configuration events in real time, such as consensus node change events, trust root update events, and block parameter modification events.

[0089] The event-driven process includes one or more of the following:

[0090] (1) When the on-chain configuration is changed via the cmc command (or the SDK sends a transaction), the change will trigger an event issued by the system contract.

[0091] (2) The Changan Chain EventHub actively pushes events to the Sidecar that has subscribed to the configuration change event.

[0092] (3) Sidecar parses the event content and immediately invalidates the relevant cache keys (such as the cache of the consensus node list) according to the change type.

[0093] (4) Sidecar notifies the controller of configuration changes via localhos gRPC / HTTP and other methods.

[0094] (5) The controller immediately triggers the Reconcile loop without waiting for periodic polling.

[0095] In the "event-driven" mode, the configuration change awareness latency is reduced from the polling interval (average 15 seconds) to the event push latency (1-2 seconds), improving the response speed by 10 times. Combined with caching and event-driven, the system minimizes the query pressure on blockchain nodes while ensuring low latency.

[0096] Figure 2 This demonstrates the working mechanism of Sidecar's evolution from an "active polling query" mode to an "event-driven + fallback query" mode.

[0097] The upper part shows the event-driven pattern, including the following:

[0098] (1) The on-chain EventHub proactively pushes events (such as "new consensus node") to the Sidecar when configuration changes occur;

[0099] (2) The Sidecar parses the event content and immediately invalidates the relevant cache keys;

[0100] (3) The Sidecar notifies the Controller of the configuration change via local gRPC;

[0101] (4) The controller queries the latest status and executes the arbitration logic.

[0102] The lower part shows the activation of the fallback query mechanism, including the following:

[0103] (1) When the Sidecar restarts, a full query is performed to obtain the current configuration and configBlockHeight / sequence;

[0104] (2) After initializing the cache, subscribe to the event stream and establish a connection.

[0105] 3. Arbitration Engine Design

[0106] The arbitration engine aims to resolve the ambiguity of change sources. It determines the "intent" of a configuration change by comparing metadata from two factual sources.

[0107] In reality, only the user knows their true intentions. The purpose of the arbitration engine is to make the relationship between intentions and system behavior precise and predictable, eliminating ambiguity.

[0108] 3.1 Metadata Tracking Mechanism

[0109] In the status sub-resource of ChainMakerNetwork CR, track the following key metadata:

[0110] (1) CR-side metadata:

[0111] observedGeneration: Records the metadata.generation value of the last successfully processed CR. Kubernetes automatically increments the generation field each time the CR's spec is modified, so this value can be used to determine if there are any new user changes to the CR.

[0112] lastSyncTime: Records the timestamp of the last successful synchronization of the configuration, used for auditing and troubleshooting.

[0113] (2) Chain-side metadata:

[0114] `lastConfigBlockHeight`: Records the block height at the time of the last on-chain configuration change. When Chang'an Chain modifies the configuration through the system contract, it generates a new block, and the height of this block is the identifier of the configuration change. By comparing the block heights, it can be determined whether there have been external changes to the on-chain configuration.

[0115] `lastConfigSequence`: Records the on-chain configuration version number (sequence). This field is obtained by directly querying the on-chain configuration. The Chang'an Chain automatically increments the version number after each configuration takes effect, serving as a fallback change identifier in case of no event subscription or event loss.

[0116] configHash: Records the hash value of the configuration content on the chain, used to determine whether the configuration has actually changed, avoiding unnecessary synchronization operations triggered due to metadata updates but unchanged content.

[0117] (3) Drift detection status:

[0118] driftDetected: Boolean value indicating whether the current configuration is in a drift state (CR is inconsistent with the on-chain configuration).

[0119] driftDetectedAt: Timestamp, recording the time when drift was first detected.

[0120] driftDetails: String, records a brief description of the drift, such as "The consensus node has changed from [nodeA, nodeB, nodeC] to [nodeA, nodeB, nodeD], which helps operations and maintenance personnel understand the differences."

[0121] 3.2 Logic for Determining the Source of Change

[0122] In each Reconcile cycle, the controller uses the following three criteria to determine the source of the configuration change:

[0123] (1) CR change judgment, which includes the following:

[0124] 1.1 Compare the current CR's metadata.generation with status.observedGeneration.

[0125] 1.2 If generation > observedGeneration, it means that the user has modified the spec part of CR (e.g., through kubectl edit or GitOps pipeline) with the intention of synchronizing "CR → chain".

[0126] 1.3. Source of change: User-driven declarative change.

[0127] 1.4 Expected Behavior: The Operator should create a Job to execute the cmc command and synchronize the configuration in CR to the chain.

[0128] (2) On-chain change detection, which includes the following:

[0129] 2.1. Query the on-chain configuration change metadata via the Sidecar and compare it with status.lastConfigBlockHeight / status.lastConfigSequence. If the configuration transaction confirmation height lastConfigBlockHeight increases, it indicates that a configuration update transaction has occurred on-chain; if lastConfigBlockHeight cannot be obtained, the configuration version number sequence is used as the basis for determining the change. Since the Operator updates the corresponding status fields synchronously when executing CMC commands, a separate increase in metadata can be identified as an on-chain configuration update triggered by an external command-based change.

[0130] 2.2 Source of change: Administrator-driven command-based change.

[0131] 2.3 Expected Behavior: Depending on the configured synchronization mode, it may be updating the CR to reflect the on-chain state, or forcibly rolling back after the grace period.

[0132] (3) Content hash verification

[0133] Calculate the hash value of the CR spec configuration and the hash value of the on-chain configuration. Compare the two hash values: if they are the same, the configuration content is consistent; if they are different, there is a discrepancy. Hash value comparison serves as a fallback check to prevent inconsistencies caused by cache expiration, outdated metadata, etc., from being overlooked, and also avoids the complex logic of comparing each field.

[0134] Because the CR specification uses custom configuration field names and arrangements, and adopts Kubernetes' YAML / JSON format, it differs from the data structure of on-chain configurations. To ensure that semantically identical configurations produce the same hash value, the configuration needs to be normalized before hash calculation, thereby guaranteeing correct semantic comparisons across formats.

[0135] Through the above three-layer judgment mechanism, the system can accurately identify the source and time sequence of configuration changes, completely eliminating state ambiguity.

[0136] 3.3 Status Management

[0137] This embodiment adopts the Kubernetes standard Conditions pattern for state management, which can be easily integrated with tools in the Kubernetes ecosystem and facilitate access to standard monitoring systems (such as Prometheus) and audit trails.

[0138] 4. Synchronization Mode Design

[0139] This embodiment is based on the "single source of fact" principle and provides four unified configuration synchronization modes, which are as follows:

[0140] Mode 1: CRD (CR Forced Mode)

[0141] Source of fact: CR is the only source of fact.

[0142] Working mechanism: If any on-chain configuration is detected to be inconsistent with CR, a Job is immediately created and the cmc command is executed to synchronize it to the chain.

[0143] Applicable scenarios: Strict GitOps workflow, CI / CD automated deployment, development and testing environments.

[0144] Mode 2: Chain (Chain Observation Mode)

[0145] Fact source: The chain is the only source of facts; CR is only used as a read-only observation window.

[0146] Working mechanism: When an on-chain configuration change is detected, the status.observedConfig field of CR is automatically updated to reflect the on-chain status.

[0147] Applicable scenarios: Traditional operation and maintenance mode, independent operation by multiple teams, and scenarios requiring maximum flexibility.

[0148] Mode 3: CRD-with-warning (Forced CR mode with bandwidth limits)

[0149] Source of fact: CR is the source of fact, but provides a grace period for decision-making.

[0150] Working mechanism: When an increase in CR generation is detected, it is immediately synchronized to the chain; when an increase in lastConfigBlockHeight / sequence on the chain is detected, a Warning event is first issued and a driftDetectedAt timestamp is set. Within the configured grace period (e.g., 30 minutes, which needs to be configured according to actual needs), only a warning is issued and no forced synchronization is performed. If the user does not update CR after the grace period ends, forced synchronization will be performed automatically.

[0151] Applicable scenarios: It can be used in production environments, but a balance needs to be struck between automation and emergency response capabilities.

[0152] Mode 4: Manual (Manual Arbitration Mode)

[0153] Source of fact: Human decision, not automatic selection by the system.

[0154] Working mechanism: When a configuration conflict is detected, a Condition (such as DriftDetected) is set, and a Critical-level event is issued requiring manual intervention. Administrators specify the synchronization direction by adding annotations (e.g., chainmaker.io / sync-direction: CR-to-chain or chain-to-CR). The controller executes synchronization in the specified direction and then clears the marker. For field-level differences, non-critical fields can be automatically merged, while critical fields require manual decision-making.

[0155] Applicable scenarios: Strongly regulated environments (finance, healthcare, etc.), changes requiring approval processes, and consortium blockchain scenarios with multi-party co-management.

[0156] The table below summarizes the differences between the four synchronization modes:

[0157] crd chain crd-with-warning manual Source of facts CR chain CR Human decision When CR changes Synchronize to chain neglect Synchronize to chain Waiting for annotation When the chain changes Rollback to CR Update CR status Alarms during the grace period, rollback after timeout Waiting for annotation Automation level high high Medium and high Low Emergency Operation and Maintenance Support Difference good good Okay (manual confirmation) Audit trail whole part whole whole

[0158] Figure 3 This demonstrates the complete decision-making process of the arbitration engine and the processing procedures of four synchronous modes. The main process includes:

[0159] (1) Query the on-chain status using Sidecar;

[0160] (2) Calculate the hash values ​​of the CR spec and the on-chain configuration;

[0161] (3) If the hashes are the same, update the timestamp and end; otherwise, proceed to the pattern distribution logic.

[0162] In this scheme, the processing branches for the four synchronization modes are as follows:

[0163] (a) In CRD mode, Jobs are created directly and forced to synchronize to the chain;

[0164] (b) The chain mode updates the CR state to reflect the on-chain configuration;

[0165] (c) The crd-with-warning mode determines whether to synchronize immediately or issue a warning based on the source of the change (generation vs. block height), and forces synchronization after the grace period;

[0166] (d) In manual mode, set the DriftDetected Condition and wait for the administrator to specify the processing method through annotation.

[0167] 5. Core Process

[0168] The controller performs configuration reconciliation in each Reconcile cycle according to the following seven steps:

[0169] S01 Query On-Chain Status: The query-sidecar function is called via the localhost interface to retrieve the current configuration status on the chain. The Sidecar function prioritizes returning from the cache; only if the cache is not found will it query the blockchain nodes. The query content includes: consensus node list, root certificate, block parameters, current block height, and other key configuration items.

[0170] S02 Calculate the configuration hash: Calculate the hash value of the CR spec configuration and the hash value of the on-chain configuration respectively. Before hash calculation, the configuration needs to be normalized (e.g., sorted, whitespace removed) to ensure that the same content generates the same hash value.

[0171] S03 State Consistency Check: Compare the two hash values. If the hashes are the same, it means the configuration is consistent, and proceed to step S06 to update the timestamp and end; if the hashes are different, it means there is a difference, and continue to step S04.

[0172] S04 Change Source Identification: Execute the three-layer judgment logic of the arbitration engine:

[0173] 1. Compare generation with observedGeneration to determine if CR has changed;

[0174] 2. Compare the chain-side configuration change metadata (lastConfigBlockHeight / sequence) with status.lastConfigBlockHeight / status.lastConfigSequence;

[0175] 3. Determine the source of the change by combining both factors (CR change, chain change, changes by both parties, abnormal situations);

[0176] S05 Policy Mode Distribution: Based on the configSync.mode configuration and the source of the change, the corresponding synchronization policy is executed, which includes one of the following:

[0177] (1) CRD mode: Directly create a Job and synchronize the changes to the chain.

[0178] (2) Chain mode: Update the status of CR to reflect the on-chain state.

[0179] (3) CRD-with-warning mode: If the CR changes, it will be synchronized to the chain. If the chain changes, the grace period will be checked. During the grace period, only an alarm will be issued. After the timeout, it will be forcibly synchronized to the chain.

[0180] (4) Manual mode: Set DriftDetected Condition and wait for manual annotation instructions.

[0181] S06 Status Update: Update the status sub-resource of CR (update metadata such as observedGeneration, lastConfigBlockHeight, lastConfigSequence, configHash, lastSyncTime, etc.); update the Conditions list, such as setting ConfigInSync=True or ConfigInSync=False; clear or set driftDetected related fields.

[0182] As an optional solution, this embodiment also includes the following:

[0183] S07 Monitoring Metrics Record: Records Prometheus metrics, including the following:

[0184] (1) chainmaker_config_drift_total (configures the number of drifts)

[0185] (2) chainmaker_config_sync_duration_seconds (synchronization operation time)

[0186] (3) chainmaker_reconcile_total (total number of Reconciles), etc., are used for monitoring and alarms.

[0187] Figure 4 This demonstrates the complete closed loop of the core Reconcile process (S01-S07) of this invention. The process is triggered by timed coordination, CR changes, or Sidecar event notifications. First, the on-chain state is obtained via the Sidecar and normalized hash calculation is performed. When a difference is detected, the source of the change is further identified, and different synchronization strategies (creating a configuration synchronization job, observation-only updates, grace period strategy, or manual arbitration) are distributed based on configSync.mode. Finally, the controller uniformly updates the CR's status / Conditions and records monitoring metrics, then proceeds to the next round of coordination.

[0188] In summary, the solution of this embodiment can be summarized as follows:

[0189] A method for runtime configuration drift detection and consistency coordination of Chang'an Chain, which manages Chang'an Chain through an Operator in a Kubernetes cluster, includes the following steps:

[0190] S1. Run an Operator in a Kubernetes cluster to manage the Chang'an Chain network. Run a main controller container and a query-sidecar container within the controller Pod of the Operator. The query-sidecar container has built-in Chang'an Chain client tools, a caching module, and an HTTP service. The main controller container calls the query-sidecar container through the localhost interface to obtain the on-chain configuration status.

[0191] S2. In each Reconcile loop, the main controller container calls the query-sidecar container to query the on-chain configuration status. The on-chain configuration status includes at least the consensus node list, trust root certificate, block parameters, and on-chain configuration change metadata. The query-sidecar container returns the query result directly when the cache is hit, and accesses the blockchain node to obtain the query result and writes it into the cache when the cache is not hit.

[0192] S3. Normalize the expected configuration of the custom resource (ChainMakerNetwork) and the actual on-chain configuration and calculate the configuration hash value. Compare the two hash values ​​to determine whether there is configuration drift. When the two hash values ​​are inconsistent, it is determined that there is configuration drift.

[0193] S4. When it is determined that there is configuration drift, perform change source identification: compare the metadata.generation of the custom resource with the status.observedGeneration of the custom resource to determine whether there is a change in the custom resource; and compare the chain-side configuration change metadata lastConfigBlockHeight and / or lastConfigSequence with the corresponding record value in the status of the custom resource to determine whether there is an on-chain configuration change, thereby outputting the change source identification result;

[0194] S5. Perform consistency coordination based on the preset synchronization mode field configSync.mode and the change source identification result, wherein the synchronization mode includes one of the following:

[0195] (1) CR Forced Mode: Create a configuration synchronization Job and execute the Chang'an Chain client tool command to synchronize the desired configuration of the custom resource to the chain;

[0196] (2) Chain observation mode: Update the status.observedConfig field of the custom resource to reflect the actual configuration on the chain;

[0197] (3) Bandwidth-limited CR forced mode: When the custom resource change is detected, a configuration synchronization job is created to synchronize the expected configuration of the custom resource to the chain; when the on-chain configuration change is detected, an alarm is issued and the drift time is recorded. Forced synchronization is not performed within the preset grace period. If configuration drift still exists after the grace period expires, a configuration synchronization job is created to roll back the on-chain configuration to the expected configuration of the custom resource.

[0198] (4) Manual arbitration mode: Set DriftDetected Condition and wait for the administrator to indicate the synchronization direction through annotation before performing the corresponding consistency coordination;

[0199] S6. After completing the consistency coordination process or the chain observation status update, update the status sub-resource of the custom resource and the Kubernetes standard Conditions status, and record at least the observedGeneration, lastConfigBlockHeight, lastConfigSequence, configHash, lastSyncTime and drift detection status information.

[0200] As an example of one embodiment, in step S2 of this solution, the query-sidecar container sets tiered cache validity periods (TTLs) for different types of configuration items, and the TTLs include at least: a first cache validity period of 10 seconds for configuration items that change frequently, a second cache validity period of 60 seconds for configuration items that change moderately, and a third cache validity period of 300 seconds for configuration items that change infrequently.

[0201] In step S1 or S2 of this scheme, the query-sidecar container subscribes to the configuration change event stream of the CHAIN_CONFIG system contract of Chang'an Chain. When it receives a configuration change event, it performs active invalidation on the cache key corresponding to the configuration change event and sends a change notification to the main controller container to trigger the Reconcile loop.

[0202] In step S1 or S2 of this scheme, in order to initialize the cache used for on-chain queries, the query-sidecar container performs a full on-chain configuration query after restarting to obtain the current configuration and lastConfigBlockHeight and / or lastConfigSequence, and establishes an event subscription connection after initializing the cache.

[0203] In step S2 of this scheme, the query-sidecar container is configured to access multiple blockchain nodes and uses a polling strategy or a minimum latency strategy to select the target blockchain node to execute the on-chain configuration query.

[0204] In step S6 of this scheme, the drift detection status information includes at least driftDetected, driftDetectedAt, and driftDetails, where driftDetails is used to characterize the differences in configuration drift.

[0205] The normalization process in step S3 of this scheme includes at least sorting the configuration fields, removing whitespace characters, and generating normalized configuration text for hash calculation using unified serialization rules, so as to ensure that configurations with the same semantics generate the same hash value.

[0206] In step S4 of this scheme, when the chain-side configuration change metadata lastConfigBlockHeight cannot be obtained, the increment of lastConfigSequence is used as the basis for determining the on-chain configuration change, and the comparison of configHash is used to avoid unnecessary consistency coordination triggered when the metadata is updated but the configuration content remains unchanged.

[0207] In step S5 of this scheme, in the manual arbitration mode, the administrator specifies the synchronization direction by setting an annotation in the format chainmaker.io / sync-direction:CR-to-chain or chainmaker.io / sync-direction:chain-to-CR for the custom resource. The main controller container then clears the annotation after performing consistency coordination according to the synchronization direction.

[0208] Step S6 of this solution also includes: recording monitoring metrics, which include at least the number of configuration drifts, the configuration synchronization time, and the total number of reconciles, for monitoring and alarm purposes.

[0209] In this embodiment, regarding the placement / deployment location of the query-sidecar, the above-mentioned solution of choosing a centralized sidecar (deployed within the controller Pod) is cited. This solution offers advantages such as low communication latency (milliseconds), minimal requirement of sidecar containers, simple maintenance, and the ability to support a scale of hundreds of nodes through caching. Other optional deployment solutions include:

[0210] (1) Node level: Deploy a Sidecar within each blockchain node Pod, and the controller accesses the Sidecar of each node across Pods.

[0211] (2) Independent service: Deploy the Sidecar independently as a Deployment or DaemonSet, and provide query services through Service.

[0212] When a sidecar is deployed as a standalone service, it can be horizontally scaled using HPA to support a larger number of nodes (e.g., when the number of networks exceeds 10 and the number of nodes exceeds 1000). Centralized sidecar deployments can also be seamlessly migrated to the "standalone service" approach without modifying the core arbitration logic of the controller; only the query URL needs to be switched via environment variables.

[0213] The solution in this embodiment has the advantage of good scalability, which can adapt to the evolution of consortium blockchain networks from small-scale to large-scale.

[0214] The above description is only a part of the embodiments of the present invention and does not limit the scope of protection of the present invention. Any equivalent device or equivalent process transformation made based on the content of the present invention specification and drawings, or direct or indirect application in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for runtime configuration drift detection and consistency coordination of Chang'an Chain, which manages Chang'an Chain through an Operator in a Kubernetes cluster, characterized in that, It includes the following steps: S1. Run an Operator in a Kubernetes cluster to manage the Chang'an Chain network. Within the Operator's controller Pod, run a main controller container and a query-sidecar container. The main controller container obtains the on-chain configuration status through the query-sidecar container. S2. In each Reconcile loop, the main controller container calls the query-sidecar container to query the on-chain configuration status, which includes at least the runtime configuration of the Chang'an Chain network and the chain-side configuration change metadata. S3. Normalize the expected configuration of the custom resource and the actual configuration on the chain and calculate the configuration hash value. Compare the two hash values ​​to determine whether there is configuration drift. When the hash values ​​of the two are inconsistent, it is determined that there is a configuration drift. S4. When configuration drift is detected, perform change source identification and output the change source identification result; S5. Perform consistency coordination based on the preset synchronization mode field and the change source identification result; S6. After completing the consistency coordination process, update the sub-resources and Kubernetes standard Conditions status of the custom resource, and record the status information used to characterize the drift detection and consistency coordination results.

2. The method for runtime configuration drift detection and consistency coordination of the Chang'an Chain according to claim 1, characterized in that, In step S1, the query-sidecar container has built-in Chang'an Chain client tools, a caching module, and an HTTP service; the main controller container calls the query-sidecar container through the localhost interface to obtain the on-chain configuration status; In step S2, the runtime configuration of the Chang'an Chain network includes a consensus node list, a trust root certificate, and block parameters; In step S2, the query-sidecar container directly returns the query result when the cache is hit, and accesses the blockchain node to obtain the query result and writes it into the cache when the cache is not hit. In step S4, when it is determined that there is a configuration drift, the change source identification includes: comparing the metadata.generation of the custom resource with the status.observedGeneration of the custom resource to determine whether there is a change in the custom resource; and comparing the chain-side configuration change metadata lastConfigBlockHeight and / or lastConfigSequence with the corresponding record value in the status of the custom resource to determine whether there is an on-chain configuration change, thereby outputting the change source identification result; In step S5, the synchronization mode includes one of the following: (1) CR Forced Mode: Create a configuration synchronization Job and execute the Chang'an Chain client tool command to synchronize the desired configuration of the custom resource to the chain; (2) Chain observation mode: Update the status.observedConfig field of the custom resource to reflect the actual configuration on the chain; (3) Bandwidth-limited CR forced mode: When the custom resource change is detected, a configuration synchronization job is created to synchronize the expected configuration of the custom resource to the chain; when the on-chain configuration change is detected, an alarm is issued and the drift time is recorded. Forced synchronization is not performed within the preset grace period. If configuration drift still exists after the grace period expires, a configuration synchronization job is created to roll back the on-chain configuration to the expected configuration of the custom resource. (4) Manual arbitration mode: Set DriftDetected Condition and wait for the administrator to indicate the synchronization direction through annotation before performing the corresponding consistency coordination; S6 includes: after completing the consistency coordination process or completing the chain observation state update, updating the status sub-resource of the custom resource and the Kubernetes standard Conditions state; The status information includes at least observedGeneration, lastConfigBlockHeight, lastConfigSequence, configHash, lastSyncTime, and drift detection status information.

3. The method for runtime configuration drift detection and consistency coordination of the Chang'an Chain according to claim 1 or 2, characterized in that, In step S2, the query-sidecar container sets tiered cache validity periods (TTLs) for different types of configuration items, and the TTLs include at least: a first cache validity period of 10 seconds for configuration items that change frequently, a second cache validity period of 60 seconds for configuration items that change moderately, and a third cache validity period of 300 seconds for configuration items that change infrequently.

4. The method for runtime configuration drift detection and consistency coordination of the Chang'an Chain according to claim 3, characterized in that, In step S1 or S2, the query-sidecar container subscribes to the configuration change event stream of the CHAIN_CONFIG system contract of Chang'an Chain. When it receives a configuration change event, it performs active invalidation on the cache key corresponding to the configuration change event and sends a change notification to the main controller container to trigger the Reconcile loop. In step S1 or S2, to initialize the cache used for on-chain queries, the query-sidecar container performs a full on-chain configuration query after restarting to obtain the current configuration and lastConfigBlockHeight and / or lastConfigSequence, and establishes an event subscription connection after initializing the cache.

5. The method for runtime configuration drift detection and consistency coordination of the Chang'an Chain according to claim 1 or 2, characterized in that, In step S2, the query-sidecar container is configured to access multiple blockchain nodes and uses a polling strategy or a minimum latency strategy to select the target blockchain node to execute the on-chain configuration query.

6. The method for runtime configuration drift detection and consistency coordination of the Chang'an Chain according to claim 1 or 2, characterized in that, In step S6, the drift detection status information includes at least driftDetected, driftDetectedAt, and driftDetails, where driftDetails is used to characterize the differences in configuration drift.

7. The method for runtime configuration drift detection and consistency coordination of the Chang'an Chain according to claim 1 or 2, characterized in that, The normalization process in step S3 includes at least sorting the configuration fields, removing whitespace characters, and generating normalized configuration text for hash calculation using a unified serialization rule to ensure that configurations with the same semantics generate the same hash value.

8. The method for runtime configuration drift detection and consistency coordination of the Chang'an Chain according to claim 1 or 2, characterized in that, In step S4, when the chain-side configuration change metadata lastConfigBlockHeight cannot be obtained, the increment of lastConfigSequence is used as the basis for determining the on-chain configuration change, and the comparison of configHash is used to avoid unnecessary consistency coordination triggered when the metadata is updated but the configuration content remains unchanged.

9. The method for runtime configuration drift detection and consistency coordination of the Chang'an Chain according to claim 2, characterized in that, In step S5, in the manual arbitration mode, the administrator specifies the synchronization direction by setting an annotation in the format chainmaker.io / sync-direction:CR-to-chain or chainmaker.io / sync-direction:chain-to-CR for the custom resource. The main controller container then clears the annotation after performing consistency coordination according to the synchronization direction.

10. The method for runtime configuration drift detection and consistency coordination of the Chang'an Chain according to claim 1 or 2, characterized in that, Step S6 further includes: recording monitoring metrics, which include at least the number of configuration drifts, the configuration synchronization time, and the total number of reconciles, for monitoring and alarm purposes.

Images