A routing-aware based multi-tenant certificate management method and system
By adopting a route-aware multi-tenant certificate management method, the problems of long memory consumption and fault recovery time in multi-tenant scenarios are solved, achieving efficient certificate management and elastic scaling, and reducing operation and maintenance costs and attack surface.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAMEN XINGZONG DIGITAL TECH CO LTD
- Filing Date
- 2026-05-25
- Publication Date
- 2026-07-03
AI Technical Summary
In multi-tenant scenarios, existing technologies suffer from problems such as prolonged cold start time, high resource consumption, high risk of configuration errors, expanded attack surface, and difficult operation and maintenance due to the linear increase in memory usage and number of tenants. Especially in scenarios with tens of thousands of domain names, existing solutions cannot effectively solve write conflicts and consistency issues, resulting in long fault recovery times.
We adopt a route-aware multi-tenant certificate management approach, which uses on-demand certificate distribution, certificate aggregation, and template-driven rendering mechanisms, combined with declarative filter chain automatic injection, to achieve handshake admission control and read/write layered copy architecture, thereby optimizing the certificate management process.
It significantly shortens gateway startup time, improves elastic scaling and fault recovery speed, reduces memory usage, lowers the attack surface, reduces operation and maintenance costs, and enables efficient certificate management and configuration updates.
Smart Images
Figure CN122339839A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cloud-native gateways and service meshes, and in particular to a method and system for managing multi-tenant certificates based on route awareness. Background Technology
[0002] In multi-tenant scenarios, a single gateway cluster needs to provide Hypertext Transfer Security Protocol (TLS) access for tens of thousands of tenants with custom domain names. Existing static certificate loading schemes read the configuration file and load all tenants' TLS certificates (Transport Layer Security Certificates) and private keys into memory at startup. Each tenant's domain certificate typically contains 2KB to 4KB of certificate data and 2KB to 4KB of private key data. Furthermore, the gateway process also needs to build an internal data structure for each certificate that magnifies the actual raw data by 3 to 5 times. Therefore, when the number of tenants reaches hundreds of thousands, the memory usage of a single gateway process may exceed 4GB. In other words, memory usage exhibits a linear growth relationship with the number of tenants. At this point, cold start time also increases linearly, the waiting time for parsing tenant certificate files is prolonged, leading to a decrease in the response speed of rolling updates or fault recovery, and a decrease or even interruption of service capacity. In addition, the full reload mechanism triggered by configuration changes continuously consumes system resources and increases the risk of configuration errors.
[0003] To address the aforementioned issues, existing SDS (Secret Discovery Service) on-demand certificate distribution schemes, based on the xDS protocol (x Discovery Service, a collective term for a set of dynamically configured discovery APIs used by Envoy agents, where x represents specific resources such as routes, clusters, and keys), do not load any certificates upon gateway startup. Instead, during the TLS handshake phase, the gateway specifies the domain name to be accessed based on the SNI (Server Name Indication) extension field carried in the client message. After receiving the client message, the gateway sends a request to the SDS control plane to query the certificate corresponding to that domain name, employing a fault-tolerance mechanism that returns a fallback certificate or a self-signed certificate. This allows unauthorized domain names to complete the TLS handshake and establish a connection. Therefore, attackers can obtain the gateway's certificate information through TLS handshakes initiated by arbitrarily constructed SNI values or through domain name enumeration, leading to the leakage of sensitive data. Even with the introduction of WAF (Web Application Firewall, used to filter, monitor, and block malicious HTTP traffic targeting web applications) or L4ACL (Layer 4 Access Control List, a list of rules at the transport layer that blocks traffic based on IP and port), the SNI field in the TLS handshake message still cannot be parsed, meaning it cannot distinguish between handshake requests from legitimate and illegitimate domains. Another existing automated gateway API management solution automatically issues certificates based on the ACME (Automatic Certificate Management Environment) protocol and mounts them to the gateway. This requires simultaneous support for HTTP / 1.1 and HTTP / 2 based on TCP (Transmission Control Protocol) and HTTP / 3 based on the QUIC (Quick UDP Internet Connections) protocol. Because TCP and UDP (User Datagram Protocol) have fundamentally different connection management mechanisms, and QUIC's connection ID is bound to the domain name, HTTP / 3 requires configuring independent filter chain matching rules for each domain. In real-world applications, different service modules are derived from the same tenant and configured with multiple subdomains. When the number of tenants reaches tens of thousands, the configuration file needs to maintain a corresponding number of filter chain entries. Any configuration error may lead to large-scale service failures, making operation and maintenance extremely difficult.Furthermore, in multi-replica deployment scenarios of the gateway control plane, concurrent writes can lead to various race conditions, such as overwrite writes, conflict retries, or configuration jitter. Even with the existing master election mechanism, technical problems such as low resource utilization, long fault recovery time, and high operation and maintenance costs will occur. Summary of the Invention
[0004] The purpose of this invention is to provide a route-aware handshake admission control mechanism, which jointly designs a modular architecture including an on-demand certificate distribution mechanism, a certificate aggregation mechanism, a template-driven rendering mechanism, and a declarative filter chain automatic injection mechanism. Its core is to solve the problems of write conflicts and consistency issues, low resource utilization, and long fault recovery time in existing technologies in scenarios with tens of thousands of domain names.
[0005] In a first aspect, embodiments of the present invention provide a route-aware multi-tenant certificate management method, including: Upon receiving a tenant domain name creation request, create a custom resource for the tenant domain name and initialize it. When the tenant domain controller of the primary replica listens to the creation event of the tenant domain custom resource, it reads all the seven-layer route resource templates with preset tags from the route template cache, traverses each template, reads the pre-declared subdomain prefix in the template, and concatenates the subdomain prefix with the tenant base domain name to generate a derived subdomain. Aggregate all derived subdomains under this tenant's domain into a multi-domain list of a single certificate resource and submit it to the certificate issuance controller to execute the write event of the TLS key resource; When the gateway proxy receives the server name indication extension from the client during the TLS handshake phase, it initiates an incremental discovery request to the key discovery distribution service and queries the local cache using the target domain name carried in the request as the resource name. If the query matches, the multi-domain certificate corresponding to the target domain is returned to the gateway proxy; if the query does not match, exact matching and wildcard matching are performed in the certificate index in turn; if either match is found, the TLS key resource is loaded and the multi-domain certificate is returned. When no certificate index is found, query the full set of domain names consisting of declared domain names in the route template cache and perform the handshake admission control steps: When a Layer 7 routing resource matching the target domain exists in the full domain set, the target domain is determined to be in the certificate issuance transition period, and a preset default certificate is returned; If no Layer 7 routing resource matching the target domain exists in the full set of domain names, the target domain is determined to be an unauthorized domain. An incremental discovery response is then constructed indicating that the resource corresponding to the target domain does not exist and does not need to be removed, and a handshake failure is returned.
[0006] Optionally, the step of submitting to the certificate issuance controller to execute the write event of the TLS key resource further includes: The certificate issuance controller issues multi-domain certificates covering all derived subdomains and writes the multi-domain certificates and their corresponding private keys into the TLS key resource; After the key listener captures the write event of the TLS key resource, it parses the extended fields of the multi-domain certificate, extracts all the subdomains covered by the certificate, and writes the mapping relationship between each subdomain and the TLS key resource into the certificate index; The filter chain updater subscribes to the change events of the certificate index and presets a debouncing time window, merging the change events collected within the debouncing time window into a batch processing task; When the anti-jitter time window ends, the filter chain updater reads all currently active domains from the certificate index and generates a JSON patch add operation for each active domain, pointing to the filter chain array in the gateway proxy listening configuration; the add operation includes a filter chain matching field for storing the active domain and a transport socket field configured as a QUIC protocol parameter, and the active domain is consistent with the resource name in the key discovery and distribution service; All add operations in this batch are assembled into a gateway filtering chain policy consisting of a patch array, and written to the gateway proxy all at once through the container orchestration platform's interface. When the gateway proxy detects a change event in the gateway filtering chain policy, it converts the change event into the native filtering chain configuration for each subdomain and loads it to take effect.
[0007] Optionally, the key listener writes the mapping relationship between each domain name and the TLS key resource into the certificate index, including: The derived subdomain formed by the subdomain prefix and the tenant's base domain name is written into the certificate index as an exact match entry; where the key is the domain name string of the derived subdomain and the value is the corresponding TLS key resource. The tenant's base domain name is used as a wildcard matching entry and written into the certificate index; where the key is the domain name string of the tenant's base domain name and the value is the corresponding TLS key resource.
[0008] Optionally, performing exact matching and wildcard matching in the certificate index includes: The target domain name is searched in the exact match entry. When a match is found, the TLS key resource is loaded and the multi-domain certificate is returned based on the value in the exact match entry. If no exact match is found, the tenant base domain to the left of the separator in the target domain is extracted as the parent domain, and wildcard matching entries with the parent domain as the key are searched. When a wildcard match is found, the TLS key resource is loaded and a multi-domain certificate is returned based on the value in the wildcard match; when a wildcard match is not found, a handshake admission control step is performed.
[0009] Optionally, the step of creating and initializing a custom resource for a tenant domain name includes: Deploy two or more copies; Upon startup of each replica, a key discovery and distribution service, a key listener, and a route template cache are registered and run to form a read-only path; The primary replica is elected based on the distributed lock mechanism of the container orchestration platform's lease resources, and the remaining replicas are secondary replicas; the lease resources are lightweight resources natively provided by the container orchestration platform, with primary election lease period and lease renewal interval, used to achieve mutual exclusion; The primary replica holds a distributed lock and runs a tenant domain name controller and a filter chain updater to form a write path; The read-only path execution from the copy is used in response to on-demand certificate requests.
[0010] Optionally, the method further includes: When the first primary replica experiences a process crash, node failure, or network partition, and the primary replica's lease expires, the distributed lock is automatically released, indicating that the current lease resource can be competed for by other replicas. After the primary replica's master election lease expires, the remaining replicas compete again based on the distributed lock mechanism. Within the duration of the master election lease, the replica that successfully acquires the distributed lock becomes the second primary replica and runs the tenant domain name controller and filter chain updater to take over the write path.
[0011] Optionally, the step of concatenating the subdomain prefix with the tenant's base domain name to generate a derived subdomain includes: Based on the routing rules in the Layer 7 routing resource template, render independent Layer 7 routing resources for the derived subdomain, specifically as follows: Based on the index position of each Layer 7 routing resource template in the routing template cache, the name of the tenant domain name custom resource is concatenated with the index position according to a preset format to obtain the resource name of the rendered Layer 7 routing resource. Replace the placeholder domain name field in the template with the derived subdomain, and inherit the path matching rules, request header matching rules, traffic distribution strategies and backend service references defined in the template, so that the routing patterns of the same type of business can be reused among tenants.
[0012] Secondly, embodiments of the present invention also provide a route-aware multi-tenant certificate management system, comprising: The key discovery and distribution service module is used to respond to on-demand certificate requests from gateway agents based on the incremental discovery protocol. A key listener is used to subscribe to events related to the addition, deletion, and modification of TLS key resources with preset tags in a container orchestration platform through a listening mechanism. The routing template caching module is used to subscribe to all Layer 7 routing resources in the cluster through a listening mechanism, and maintain a template list and a full set of domain names in local memory; the full set of domain names is an ordered data structure that centrally records the target domain names that have been declared by Layer 7 routing resources in the current cluster. The certificate index module is used to respond to the write event of the TLS key resource captured by the key listener, parse the extended fields of the multi-domain certificate, and write the mapping relationship between all the covered domains and the TLS key resource into the certificate index; The tenant domain controller is used to respond to changes in tenant domain resources and render routing resources and certificate resources according to the template. The filter chain updater is used to dynamically update the gateway filter chain policy based on JSON patching. The master election module is used to elect a master replica based on a distributed lock mechanism and distinguish between slave replicas. The master replica holds a distributed lock and runs a tenant domain name controller and a filter chain updater to form a write path. The slave replicas execute read-only paths to respond to on-demand certificate requests.
[0013] Thirdly, embodiments of the present invention also provide an electronic device, comprising: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any of the first aspects.
[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method as described in any of the first aspects.
[0015] The technical solution provided by this invention eliminates the need for the gateway proxy to load tenant certificates during startup. Instead, the key discovery and distribution service obtains them on demand during the TLS handshake. This transforms the gateway process's memory usage from being linearly related to the total number of tenants to being solely related to the number of active domains, significantly shortening cold start time and greatly improving the gateway's elasticity and scalability in large-scale tenant scenarios, as well as its fault recovery speed. One of the core innovations of this invention is handshake admission control based on route awareness. This moves the application-layer security boundary, which previously introduced additional interception attacks, to the TLS handshake stage. By intercepting unauthorized domains at the protocol level, it fundamentally blocks the feasibility of attacks such as server name indication scanning, domain name enumeration, and certificate sniffing. This narrows the attack surface in scenarios with tens of thousands of domains to the set of legally declared domains. Furthermore, by returning a preset default certificate, it maintains handshake continuity, preventing legitimate business interruptions due to certificate issuance delays. In addition, the embodiments of the present invention use a template-driven rendering mechanism and an aggregated multi-domain certificate mechanism to pre-maintain a small number of typical business templates and mark them with preset tags. Multiple subdomains under the same tenant are aggregated into a single multi-domain certificate and managed as a whole. This reduces the number of certificates at the subdomain level to the tenant level, significantly reducing the number of application programming interface calls and renewal overhead of the certificate issuance controller.
[0016] This invention, through a batch processing link consisting of a key listener, a filter chain updater, and a debouncing time window, and by establishing a closed-loop association between the filter chain and on-demand certificates, achieves declarative injection of an independent QUIC filter chain for each domain name. This reduces the number of gateway proxy configuration reloads from a linear correlation with the number of change events to a linear correlation with the number of debouncing windows, avoiding high-frequency gateway jitter in scenarios with tens of thousands of domains. This fundamentally solves the problem that existing solutions cannot handle HTTP / 30,000-domain configuration requirements. This invention also implements a read-write layered replica architecture. The key listener and route template caching module store the subscription resources of all replicas. When electing a new master replica, the certificate index and the full set of domain names are already up-to-date, eliminating the need for re-initialization from external storage. During failover, all active replicas provide uninterrupted read services, significantly reducing failover latency. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating a route-aware multi-tenant certificate management method provided in an embodiment of the present invention.
[0018] Figure 2 A flowchart illustrating the handshake admission control steps provided in an embodiment of the present invention.
[0019] Figure 3 This is a flowchart illustrating the workflow for generating QUIC filter chain patch files, as provided in an embodiment of the present invention.
[0020] Figure 4 This is a schematic diagram of a route-aware multi-tenant certificate management device provided in an embodiment of the present invention. Detailed Implementation
[0021] The present invention will be described in detail below through embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0022] In this invention, the terms "in one possible embodiment," "exemplary," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "in one possible embodiment," "exemplary," or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "in one possible embodiment," "exemplary," or "for example" is intended to present the relevant concepts in a specific manner.
[0023] In application scenarios with tens of thousands of tenants, existing solutions generally employ static certificate loading. The memory usage of a single process increases linearly with the number of tenants, and the cold start time also increases proportionally with the number of certificates. Furthermore, any change to a tenant's certificate triggers a full configuration reload of the gateway proxy, severely impacting business continuity. Additionally, existing general on-demand certificate distribution solutions attempt to return a fallback or self-signed certificate for any server indication extension, allowing unauthorized domains to still complete TLS handshakes. This makes them vulnerable to large-scale attacks by attackers for server name indication scanning, domain enumeration, and certificate sniffing, significantly expanding the system's attack surface.
[0024] While automated solutions based on the Automatic Certificate Management Environment Protocol can automatically issue certificates and mount them to the gateway, the same tenant often needs to derive multiple subdomains, making it impossible to meet the configuration requirements of HTTP / 3 independent filter chain matching fields for each domain. Furthermore, manual maintenance is extremely difficult in scenarios with tens of thousands of domains.
[0025] In multi-replica deployment scenarios, existing solutions typically employ a global master election mechanism, using distributed locks to elect a unique master replica. Only the master replica performs write operations, while the other replicas handle only read requests. This approach effectively avoids write conflicts, but suffers from low resource utilization and long fault recovery times. Furthermore, the system's focus on a globally mutually exclusive master election mechanism fails to provide fine-grained coordination capabilities simultaneously, making write conflicts and consistency issues key bottlenecks restricting system scalability and reliability.
[0026] To address this, this invention modularly designs a multi-tenant certificate management architecture that automatically implements multi-subdomain certificate routing and orchestration for tenants and employs a debouncing mechanism. Running on a container orchestration platform, it collaborates with gateway proxies and certificate issuance controllers. The modules are loosely coupled and collaborate through the container orchestration platform's native listening and event mechanisms, stably supporting custom domain name access for tens of thousands to hundreds of thousands of tenants in a single cluster. This fundamentally solves the technical problems faced by existing technologies in large-scale multi-tenant scenarios, such as memory bottlenecks, lack of security boundaries, high operational costs, and long fault recovery times.
[0027] Firstly, embodiments of the present invention provide a route-aware multi-tenant certificate management method, such as... Figure 1 As shown, it includes: S110: When a tenant domain name creation request is received, a custom resource for the tenant domain name is created and initialized.
[0028] Specifically, the application programming interface of the container orchestration platform receives a tenant domain name creation request from the workflow, which carries the tenant base domain name and optional configuration parameters.
[0029] Next, a TenantDomain is created in the container orchestration platform. The TenantDomain is an application programming interface object extended through the custom resource definition mechanism. It is used to centrally declare all the domain name access configurations required by the tenant at the tenant granularity, and the status field of the TenantDomain is initialized to the Pending state.
[0030] S120 When the tenant domain name controller of the primary replica listens for the creation event of the tenant domain name custom resource, it reads all the Layer 7 route resource templates with preset tags from the route template cache, traverses each template, reads the pre-declared subdomain prefix in the template, and concatenates the subdomain prefix with the tenant base domain name to generate the derived subdomain.
[0031] Specifically, the tenant domain controller running on the primary replica continuously subscribes to change events of tenant domain custom resources through the listening mechanism of the container orchestration platform. This listening mechanism is a long-connection-based resource change notification mechanism provided by the container orchestration platform.
[0032] When the creation of a new tenant domain custom resource is detected, the tenant domain controller reads all Layer 7 routing resource (HTTPRoute) templates with preset template tags (td-httproute-template=true) from the route template cache. It's important to note that these preset template tags are key-value pairs of metadata pre-attached by operations personnel to the Layer 7 routing resource templates to distinguish between general business routing resources and reusable derived templates. The route template cache pre-loads all Layer 7 routing resources in the cluster through a monitoring mechanism and indexes them in memory according to their tags.
[0033] Next, the tenant domain controller iterates through each Layer 7 routing resource template and reads the subdomain prefix declared by the preset prefix annotation (td-hostname-prefix) in the template's metadata. The preset prefix annotation is a key-value pair stored in the resource metadata field that does not participate in tag filtering but carries subdomain derivation semantics. This subdomain prefix can include business prefixes such as API, web frontend, and long-lived connections.
[0034] Then, the subdomain prefix is concatenated with the tenant's base domain name to generate a derived subdomain.
[0035] S130: Aggregate all derived subdomains under this tenant's domain name into a multi-domain list of a single certificate resource and submit it to the certificate issuance controller to execute the TLS key resource write event.
[0036] Specifically, the multi-domain list corresponds to the Subject Alternate Name extension field of an X.509 certificate, allowing a single certificate to provide authentication for multiple different domains simultaneously.
[0037] S140, when the gateway proxy receives the server name indication extension from the client during the TLS handshake phase, it initiates an incremental discovery request to the key discovery distribution service and queries the local cache using the resource name carried in the request as the target domain name.
[0038] Specifically, such as Figure 2 As shown, when initiating a TLS handshake, the client explicitly specifies the target domain name to be accessed in the client message using the Server Name Indication (SNI) extension of the TLS protocol. After receiving the client message, the gateway proxy extracts the target domain name carried in the Server Name Indication extension, uses it as the resource name to construct an incremental discovery request, and sends it to the key discovery and distribution service.
[0039] The key discovery and distribution service of this invention is applicable to the xDS protocol (x Discovery Service, a general term for a set of dynamically configured discovery APIs used by the Envoy proxy), and can also be adapted to other gateways that implement the xDS protocol, such as MOSN (Modular Open Smart Network-proxy, cloud-native network proxy software) and Pipy (Pipeline-based Proxy, programmable network proxy), or can be connected to the SSL (Secure Socket Layer) dynamically loaded interface of Nginx (reverse proxy server) through an adaptation layer. No limitation is made here.
[0040] In this embodiment of the invention, the server name indication of the client greeting message in the QUIC (Quick UDP Internet Connections) initial packet is also applicable, and can also be extended to the server name indication of ECH (Encrypted Client Hello), which is not limited here.
[0041] Next, upon receiving the request, the key discovery and distribution service first queries its local cache. This local cache is an in-process cache using the most recently active domain name as the key and the assembled incremental discovery response as the value, which can speed up the return path of the domain certificate.
[0042] S150, if the query is successful, the multi-domain certificate corresponding to the target domain name is returned to the gateway proxy; if the query is unsuccessful, exact matching and wildcard matching are performed in the certificate index in turn; if either match is successful, the TLS key resource is loaded and the multi-domain certificate is returned.
[0043] S160, when no certificate index is found, query the full set of domain names consisting of declared domain names in the route template cache, and perform the handshake admission control steps: First, when there is a Layer 7 routing resource in the full set of domain names that matches the target domain name, the target domain name is determined to be a domain name that has been legally declared but whose certificate has not yet been issued and is in the certificate issuance transition period. At this time, the preset default certificate is returned to maintain the continuity of the TLS handshake.
[0044] The default certificate can be a short-term certificate from a self-signed CA (Certificate Authority) within the cluster, a self-signed certificate generated and signed by the user, or an empty PEM (PrivacyEnhancedMail). No restrictions are imposed here.
[0045] Second, when no Layer 7 routing resource matching the target domain exists in the full set of domain names, the target domain is determined to be an unauthorized domain. An incremental discovery response is then constructed indicating that the resource corresponding to the target domain does not exist and does not need to be removed, and a handshake failure alert is returned to the client. Without introducing any additional web application firewalls, this mechanism, through the "resource not declared unavailable" semantics of the incremental discovery protocol, moves the security boundary from the application layer to the TLS handshake phase, effectively blocking attacks such as server name indication, domain name enumeration, and certificate sniffing.
[0046] In this embodiment of the invention, verifying the existence of Layer 7 routing resources can also be replaced by verifying the existence of the hostname field in the gateway interface listening resource, or the existence of the request host field matched by the virtual service and the custom tenant registry.
[0047] The technical solution provided by this invention eliminates the need for the gateway proxy to load tenant certificates during startup. Instead, the key discovery and distribution service obtains them on demand during the TLS handshake. This transforms the gateway process's memory usage from being linearly related to the total number of tenants to being solely related to the number of active domains, significantly shortening cold start time and greatly improving the gateway's elasticity and scalability in large-scale tenant scenarios, as well as its fault recovery speed. One of the core innovations of this invention is handshake admission control based on route awareness. This moves the application-layer security boundary, which previously introduced additional interception attacks, to the TLS handshake stage. By intercepting unauthorized domains at the protocol level, it fundamentally blocks the feasibility of attacks such as server name indication scanning, domain name enumeration, and certificate sniffing. This narrows the attack surface in scenarios with tens of thousands of domains to the set of legally declared domains. Furthermore, by returning a preset default certificate, it maintains handshake continuity, preventing legitimate business interruptions due to certificate issuance delays.
[0048] In addition, the embodiments of the present invention use a template-driven rendering mechanism and an aggregated multi-domain certificate mechanism to pre-maintain a small number of typical business templates and mark them with preset tags. Multiple subdomains under the same tenant are aggregated into a single multi-domain certificate and managed as a whole. This reduces the number of certificates at the subdomain level to the tenant level, significantly reducing the number of application programming interface calls and renewal overhead of the certificate issuance controller.
[0049] exist Figure 1 Based on the illustrated embodiment, as one implementation of this invention, submitting multi-domain certificates to the certificate issuance controller and executing a TLS key resource write event further includes the following steps: Step a1: The certificate issuance controller issues multi-domain certificates covering all derived subdomains and writes the multi-domain certificates and their corresponding private keys into the TLS key resource.
[0050] Specifically, the certificate issuance controller initiates a new request to the certificate authority through the Automatic Certificate Management Environment Protocol (ACCEP) and obtains the complete multi-domain certificate and its private key, which are then written into the TLS key resource.
[0051] Step a2: After the key listener captures the write event of the TLS key resource, it parses the extended fields of the multi-domain certificate, extracts all the subdomains covered by the certificate, and writes the mapping relationship between each subdomain and the TLS key resource into the certificate index.
[0052] Specifically, the key listener pre-subscribes to TLS key resources with preset tags through a listening mechanism. When the TLS key resource is written to or updated, the key listener immediately receives a notification of the relevant event, extracts the serialized X.509 certificate content from the data field, parses its extended fields to obtain all covered subdomains, and atomically updates the certificate index.
[0053] Step a3: The filter chain updater subscribes to the certificate index change events and presets a debouncing time window. The change events collected within the debouncing time window are merged into a batch processing task.
[0054] Specifically, such as Figure 3 As shown, the certificate index change events include tenant1.example.com, tenant2.example.com, ... (N events), and the anti-jitter time window is set to a preset duration T. Multiple change events occurring within the preset duration T are collected and merged into a single batch processing task, thereby avoiding gateway configuration jitter caused by frequent small-batch updates under a scale of tens of thousands of domains.
[0055] Step a4: When the debouncing time window ends, the filter chain updater reads all currently active domains from the certificate index and generates a JSON patch addition operation for each active domain. For example, the JSON patch addition operation is as follows: Figure 3 The values {op:add,path: / … / filter_chains / -,value:FC(tenant1)}, {op:add,path: / … / filter_chains / -,value:FC(tenant2)}, and {op:add,…FC(…)} generate independent filter chain matches, which are then combined into the patch file EnvoyPatchPolicy.spec.jsonPatches, pointing to the filter chain array in the gateway proxy listening configuration.
[0056] The addition operation includes a filter chain match field (filter_chain_match) used to store the active domain name and a transport socket field (transport_protocol:quic) configured as a QUIC protocol parameter, and the active domain name is consistent with the resource name in the key discovery and distribution service.
[0057] Specifically, the server name parameter in the filter chain matching field is set to the corresponding active domain name, and the certificate reference method in the transport socket field is configured to be dynamically obtained through the key discovery distribution service and the resource name of the certificate is strictly aligned with the active domain name, thereby forming a constraint closed loop between the filter chain and the on-demand certificate.
[0058] Step a5: Assemble all the add operations in this batch into a gateway filtering chain policy consisting of a patch array, and write it to the gateway proxy all at once through the container orchestration platform's interface.
[0059] Specifically, the gateway filter chain policy (EnvoyPatchPolicy CR) corresponds to the gateway patch policy resource defined in the gateway project. It supports injecting custom fragments into the core configuration of the gateway proxy in the form of a patch array, thereby enabling flexible configuration expansion without modifying the core gateway resources.
[0060] Step a6: When the gateway proxy QUIC listener receives a change event for the gateway filtering chain policy, it converts the change event into a native independent filter chain configuration for each subdomain and loads and applies it.
[0061] This invention, through a batch merging mechanism of the debouncing time window, merges multiple discrete changes to the certificate index that occur within a short period of time into a single filtering chain strategy reconstruction. This reduces the number of gateway proxy configuration reloads from being linearly related to the number of change events to being related to the debouncing time window, significantly reducing gateway jitter in scenarios with tens of thousands of domains. It also implements declarative injection of independent QUIC filtering chains for each domain, solving the problem of the lack of a standard solution for HTTP / 3 per-domain filtering chain configuration in existing solutions.
[0062] exist Figure 1 Based on the illustrated embodiment, as one implementation of the present invention, the key listener writes the mapping relationship between each domain name and TLS key resources into the certificate index.
[0063] First, the derived subdomain formed by the subdomain prefix and the tenant's base domain name is written into the certificate index as an exact match entry. Here, the key is the domain name string of the derived subdomain, and the value is the corresponding TLS key resource.
[0064] Second, the tenant's base domain name is used as a wildcard matching entry and written to the certificate index. Here, the key is the domain name string of the tenant's base domain name, and the value is the corresponding TLS key resource.
[0065] This invention maintains both exact match entries and wildcard match entries in the certificate index, enabling certificate queries to support both precise certificate returns at the subdomain level and wildcard domain fallback at the tenant level. This allows newly added subdomains within the wildcard coverage area to immediately provide HTTPS services without waiting for new certificate issuance, thus providing availability and flexibility in tenant subdomain expansion scenarios.
[0066] exist Figure 1 Based on the illustrated embodiment, as one implementation of this invention, performing exact matching and wildcard matching in the certificate index includes the following steps: Step b1: Search for the target domain name in the exact match entries. If a match is found, load the TLS key resource and return the multi-domain certificate based on the value in the exact match entries.
[0067] Step b2: If no exact match is found, extract the tenant base domain to the left of the separator in the target domain as the parent domain, and search for wildcard matching entries with the parent domain as the key.
[0068] The separator is a period (.) between domain levels, and the extraction process starts from the leftmost tag of the target domain.
[0069] Step b3: When a wildcard match is found, load the TLS key resource and return the multi-domain certificate based on the value in the wildcard match.
[0070] If no wildcard match is found, a handshake admission control step is performed.
[0071] This invention employs a two-level query sequence—first exact matching, then wildcard matching—to return the most accurate certificate when a subdomain-specific certificate already exists, and to return a valid certificate as a fallback when an exact match fails, thus avoiding unnecessary entry into the handshake admission control branch and improving the certificate query hit rate and overall system response performance.
[0072] exist Figure 1 Based on the illustrated embodiment, as one implementation of this invention, a tenant domain name custom resource is created and initialized, including the following steps: Step c1: Deploy two or more replicas to form a multi-replica control plane cluster to ensure high availability of the service.
[0073] Specifically, embodiments of the present invention can adopt the deployment mode of Kubernetes Operator (automated packaging and deployment), or they can be deployed as independent binary on virtual machines, serverless functions, and edge nodes, and connect to platform resources through API interfaces.
[0074] Step c2: When each replica starts, register and run the key discovery and distribution service, key listener, and route template cache to form a read-only path.
[0075] The read-only path does not involve any write operations to any resources in the container orchestration platform. It only undertakes functions such as listening for subscriptions, index maintenance, and on-demand certificate responses. Therefore, there are no concurrent conflicts between the replicas, and each replica can independently serve on-demand certificate requests proxied by the gateway.
[0076] Step c3: Based on the distributed lock mechanism of the container orchestration platform's lease resources, a primary replica is elected, and the remaining replicas are secondary replicas. The lease resources are lightweight resources natively provided by the container orchestration platform, with a primary election lease period and lease renewal interval, used to achieve mutual exclusion.
[0077] The distributed lock mechanism can be deployed as Kubernetes Lease (a built-in mechanism for implementing leases in control plane components), or it can be replaced by ectd distributed lock (a distributed key-value store system), Consul Session (which uses the Gossip protocol and Raft consensus algorithm to elect a leader), ZooKeeper ephemeral node (which uses the Zab algorithm to elect a leader), or Raft consensus member identity. No restrictions are imposed here.
[0078] Specifically, the primary lease term is selected as the maximum holding period for the distributed lock, and the lease renewal interval is the period during which the replica holding the lock actively renews the lease. The primary replica holds the distributed lock and, in addition to the read-only path, runs the tenant domain name controller and filter chain updater to form the write path; the secondary replica does not hold the distributed lock and only executes the read-only path to respond to on-demand certificate requests.
[0079] For example, the primary lease period can be set to 15 seconds and the lease renewal interval to 2 seconds, so that the failover time is less than the primary lease period.
[0080] This invention employs a read-write layered replica architecture, enabling all replicas on the read path to respond to on-demand certificate requests in parallel, achieving linear horizontal scaling of read capacity with the number of replicas. On the write path, a distributed lock mechanism ensures that only a single primary replica executes the write path across the entire cluster, avoiding race conditions and jitter issues caused by concurrent writes to the same resource by multiple replicas. Furthermore, this invention restricts the primary election mechanism to the write path, ensuring that the read-only path is unaffected by changes in the primary election state. Even during a primary replica switch, all secondary replicas can still provide uninterrupted read services.
[0081] exist Figure 1 Based on the illustrated embodiment, as another implementation of the present invention, the following failover steps are also included: When the first primary replica experiences a process crash, node failure, or network partition, and its primary lease expires, the distributed lock is automatically released, indicating that the current lease resource can be competed for by other replicas.
[0082] Specifically, when the first primary replica is unable to actively renew the lease resources according to the lease renewal interval due to operational abnormalities, the container orchestration platform will automatically release the lease after the primary lease expires.
[0083] After the primary replica's master election lease expires, the remaining replicas compete again based on the distributed lock mechanism. Within the duration of the master election lease, the replica that successfully acquires the distributed lock becomes the second primary replica and runs the tenant domain name controller and filter chain updater to take over the write path.
[0084] Specifically, since the key listener and route template cache have been continuously subscribing to relevant resources, their certificate index and full domain name set are already in the latest state at the moment of the second primary copy switch, without the need for re-initialization or loading from external storage.
[0085] The embodiments of the present invention strictly control the failover time within the duration of the primary lease term and provide uninterrupted read services during the failover time, which can solve the problem of state gaps during the handover period in the prior art.
[0086] exist Figure 1 Based on the illustrated embodiment, as one implementation of this invention, the subdomain prefix is concatenated with the tenant's base domain name to generate a derived subdomain. That is, based on the routing rules in the Layer 7 routing resource template, independent Layer 7 routing resources are rendered for the derived subdomain. Specifically, this includes the following steps: Step d1: Based on the index order of each Layer 7 routing resource template in the routing template cache, concatenate the name of the tenant domain name's custom resource with the index order according to a preset format to obtain the resource name of the rendered Layer 7 routing resource.
[0087] Step d2 replaces the placeholder domain name field in the template with the derived subdomain, and inherits the path matching rules, request header matching rules, traffic distribution strategies and backend service references defined in the template, allowing tenants to reuse routing patterns for the same type of business.
[0088] Specifically, similar services typically share the same routing semantics across different tenants. For example, all tenants' interface subdomains need to be routed to the interface gateway backend, and all tenants' webpage frontend subdomains need to be routed to the static resource service backend. The template integration mechanism can achieve one-time declaration and multi-tenant reuse.
[0089] Furthermore, in this embodiment of the invention, the rendered Layer 7 routing resources have a unique identifier for the corresponding tenant domain name custom resource written in the owner reference field to establish a cascading reference relationship. This allows the container orchestration platform to automatically cascade and reclaim all derived Layer 7 routing resources when the tenant domain name custom resource is deleted.
[0090] This invention, through a template-driven rendering mechanism, reduces resource management costs in multi-subdomain derivation scenarios from being linearly related to the number of subdomains to being linearly related to the number of templates. Furthermore, by utilizing cascading references, it automatically cleans up all derived resources of offline tenants, avoiding configuration drift issues.
[0091] Secondly, embodiments of the present invention also provide a route-aware multi-tenant certificate management system 40, such as... Figure 4 As shown in the diagram, dashed lines represent subscription or listening relationships, and solid lines represent write or calling relationships, including: The key discovery and distribution service module 410 is used to respond to on-demand certificate requests from the gateway agent based on the incremental discovery protocol. The certificate index module 420 is used to respond to the write event of the TLS key resource captured by the key listener 430, parse the extended fields of the multi-domain certificate, and write the mapping relationship between all the covered domains and the TLS key resource into the certificate index; Key listener 430 is used to subscribe to add, delete, and modify events of TLS key resources with preset tags in the container orchestration platform through a listening mechanism; The routing template caching module 440 is used to subscribe to all Layer 7 routing resources in the cluster through a listening mechanism and maintain a template list and a full set of domain names in local memory; the full set of domain names is an ordered data structure that centrally records the target domain names that have been declared by Layer 7 routing resources in the current cluster. Tenant Domain Controller 450 is used to respond to changes in tenant domain resources and render routing resources and certificate resources according to the template. Filter Chain Updater 460 is used to dynamically update gateway filter chain policies based on JSON patching. The master election module 470 is used to elect a master replica and distinguish slave replicas based on a distributed lock mechanism. The master replica holds a distributed lock and runs a tenant domain name controller and a filter chain updater to form a write path. The slave replicas execute a read-only path to respond to on-demand certificate requests.
[0092] Thirdly, embodiments of the present invention also provide an electronic device, comprising: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any of the first aspects.
[0093] Fourthly, embodiments of the present invention also provide a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method as described in any of the first aspects.
[0094] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.
Claims
1. A route-aware multi-tenant certificate management method, characterized in that, include: Upon receiving a tenant domain name creation request, create a custom resource for the tenant domain name and initialize it. When the tenant domain controller of the primary replica listens to the creation event of the tenant domain custom resource, it reads all the seven-layer route resource templates with preset tags from the route template cache, traverses each template, reads the pre-declared subdomain prefix in the template, and concatenates the subdomain prefix with the tenant base domain name to generate a derived subdomain. Aggregate all derived subdomains under this tenant's domain into a multi-domain list of a single certificate resource and submit it to the certificate issuance controller to execute the write event of the TLS key resource; When the gateway proxy receives the server name indication extension from the client during the TLS handshake phase, it initiates an incremental discovery request to the key discovery distribution service and queries the local cache using the target domain name carried in the request as the resource name. If the query finds a match, the multi-domain certificate corresponding to the target domain name will be returned to the gateway proxy. If the query does not find a match, perform exact matching and wildcard matching in the certificate index in turn; if either match is found, load the TLS key resource and return the multi-domain certificate. When no certificate index is found, query the full set of domain names consisting of declared domain names in the route template cache and perform the handshake admission control steps: When a Layer 7 routing resource matching the target domain exists in the full domain set, the target domain is determined to be in the certificate issuance transition period, and a preset default certificate is returned; If no Layer 7 routing resource matching the target domain exists in the full set of domain names, the target domain is determined to be an unauthorized domain. An incremental discovery response is then constructed indicating that the resource corresponding to the target domain does not exist and does not need to be removed, and a handshake failure is returned.
2. The method according to claim 1, characterized in that, The submission to the certificate issuance controller to execute the TLS key resource write event also includes: The certificate issuance controller issues multi-domain certificates covering all derived subdomains and writes the multi-domain certificates and their corresponding private keys into the TLS key resource; After the key listener captures the write event of the TLS key resource, it parses the extended fields of the multi-domain certificate, extracts all the subdomains covered by the certificate, and writes the mapping relationship between each subdomain and the TLS key resource into the certificate index; The filter chain updater subscribes to the change events of the certificate index and presets a debouncing time window, merging the change events collected within the debouncing time window into a batch processing task; When the anti-jitter time window ends, the filter chain updater reads all currently active domains from the certificate index and generates a JSON patch add operation for each active domain, pointing to the filter chain array in the gateway proxy listening configuration; the add operation includes a filter chain matching field for storing the active domain and a transport socket field configured as a QUIC protocol parameter, and the active domain is consistent with the resource name in the key discovery and distribution service; All add operations in this batch are assembled into a gateway filtering chain policy consisting of a patch array, and written to the gateway proxy all at once through the container orchestration platform's interface. When the gateway proxy detects a change event in the gateway filtering chain policy, it converts the change event into the native filtering chain configuration for each subdomain and loads it to take effect.
3. The method according to claim 2, characterized in that, The key listener writes the mapping relationship between each domain name and the TLS key resource into the certificate index, including: The derived subdomain formed by the subdomain prefix and the tenant's base domain name is written into the certificate index as an exact match entry; where the key is the domain name string of the derived subdomain and the value is the corresponding TLS key resource. The tenant's base domain name is used as a wildcard matching entry and written into the certificate index; where the key is the domain name string of the tenant's base domain name and the value is the corresponding TLS key resource.
4. The method according to claim 1, characterized in that, The certificate index performs exact matching and wildcard matching, including: The target domain name is searched in the exact match entry. When a match is found, the TLS key resource is loaded and the multi-domain certificate is returned based on the value in the exact match entry. If no exact match is found, the tenant base domain to the left of the separator in the target domain is extracted as the parent domain, and wildcard matching entries with the parent domain as the key are searched. When a wildcard match is found, the TLS key resource is loaded and a multi-domain certificate is returned based on the value in the wildcard match; when a wildcard match is not found, a handshake admission control step is performed.
5. The method according to claim 1, characterized in that, The creation and initialization of custom resources for tenant domains includes: Deploy two or more copies; Upon startup of each replica, a key discovery and distribution service, a key listener, and a route template cache are registered and run to form a read-only path; The primary replica is elected based on the distributed lock mechanism of the container orchestration platform's lease resources, and the remaining replicas are secondary replicas; the lease resources are lightweight resources natively provided by the container orchestration platform, with primary election lease period and lease renewal interval, used to achieve mutual exclusion; The primary replica holds a distributed lock and runs a tenant domain name controller and a filter chain updater to form a write path; The read-only path execution from the copy is used in response to on-demand certificate requests.
6. The method according to claim 5, characterized in that, The method further includes: When the first primary replica experiences a process crash, node failure, or network partition, and the primary replica's lease expires, the distributed lock is automatically released, indicating that the current lease resource can be competed for by other replicas. After the primary replica's master election lease expires, the remaining replicas compete again based on the distributed lock mechanism. Within the duration of the master election lease, the replica that successfully acquires the distributed lock becomes the second primary replica and runs the tenant domain name controller and filter chain updater to take over the write path.
7. The method according to claim 1, characterized in that, The step of concatenating the subdomain prefix with the tenant's base domain name to generate a derived subdomain includes: Based on the routing rules in the Layer 7 routing resource template, render independent Layer 7 routing resources for the derived subdomain, specifically as follows: Based on the index position of each Layer 7 routing resource template in the routing template cache, the name of the tenant domain name custom resource is concatenated with the index position according to a preset format to obtain the resource name of the rendered Layer 7 routing resource. Replace the placeholder domain name field in the template with the derived subdomain, and inherit the path matching rules, request header matching rules, traffic distribution strategies and backend service references defined in the template, so that the routing patterns of the same type of business can be reused among tenants.
8. A multi-tenant certificate management system based on route awareness, characterized in that, include: The key discovery and distribution service module is used to respond to on-demand certificate requests from gateway agents based on the incremental discovery protocol. The certificate index module is used to respond to the write event of the TLS key resource captured by the key listener, parse the extended fields of the multi-domain certificate, and write the mapping relationship between all the covered domains and the TLS key resource into the certificate index; A key listener is used to subscribe to events related to the addition, deletion, and modification of TLS key resources with preset tags in a container orchestration platform through a listening mechanism. The routing template caching module is used to subscribe to all Layer 7 routing resources in the cluster through a listening mechanism, and maintain a template list and a full set of domain names in local memory; the full set of domain names is an ordered data structure that centrally records the target domain names that have been declared by Layer 7 routing resources in the current cluster. The tenant domain controller is used to respond to changes in tenant domain resources and render routing resources and certificate resources according to the template. The filter chain updater is used to dynamically update the gateway filter chain policy based on JSON patching. The master election module is used to elect a master replica based on a distributed lock mechanism and distinguish between slave replicas. The master replica holds a distributed lock and runs a tenant domain name controller and a filter chain updater to form a write path. The slave replicas execute read-only paths to respond to on-demand certificate requests.
9. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1 to 7.