Agent cluster monitoring method, system and device, medium and product

Abstract

The invention discloses a proxy cluster monitoring method, system, device, medium and product, a deployer deploys a Redis server and a proxy cluster in K8S, and synchronizes the original name and original IP address of each proxy Pod in the proxy cluster to a metadatabase; an event monitor monitors the state of each proxy Pod in real time, and when the state of the proxy Pod changes, a state change event is triggered and sent to an event processing center; the event processing center updates the newest name and the newest IP address of the Pod to a metadatabase by adopting a dual-consistency algorithm according to the state change event; and executing a monitoring task on each proxy Pod by an acquisition and analysis system according to the metadata in the metadatabase. By adopting the method and the device, the real-time and automatic updating of the metadata of the proxy Pod in the K8S can be realized, and the monitoring of the proxy node is ensured.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, system, device, medium, and product for monitoring agent clusters. Background Technology

[0002] Deploying a Redis (remote dictionary service) cluster in a Kubernetes (K8S) environment (a platform for managing containerized applications) involves the Redis cluster (in Cluster mode) automatically redirecting requests for load balancing and data sharding. The Redis cluster deployed in Kubernetes uses Kubernetes Pod IPs, which are temporary internal IPs that change after a Pod restart or scaling up / down. NodePort is a way for Kubernetes to expose services externally. It maps Pods to node IPs and fixed ports, allowing external access to the Redis service.

[0003] When making requests to a Redis Cluster, Redis Cluster redirects the request directly to the target Pod's IP address instead of the NodePort. This prevents external networks from accessing the Pod's IP address within Kubernetes, leading to connection failures. To bypass the redirection and Pod IP limitations, a proxy cluster, such as Codis, needs to be deployed in front of the Redis cluster. By deploying a NodePort on top of the proxy cluster, load balancing is performed to the proxy cluster, allowing external applications to indirectly access the Redis cluster.

[0004] However, while proxy clusters solve the external access problem, they introduce new monitoring challenges. Currently, the commonly used monitoring middleware suite consists of Exporter (data collection), Prometheus (storage analysis), and Grafana (visualization). To monitor proxy Pods, the IP address of each proxy Pod must be configured in Exporter to collect data. However, in a Kubernetes environment, abnormal restarts of proxy Pods can cause IP changes, or scaling up or down of proxy Pods can increase or decrease the number of IPs. In these cases, the monitoring system needs to manually modify the Exporter configuration and restart the monitoring system. In large-scale Redis clusters with numerous proxy nodes, this approach leads to high manpower costs and difficulty in real-time synchronization updates, resulting in monitoring gaps. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, device, medium, and product for monitoring agent clusters, which can realize real-time and automatic updates of the metadata of agent Pods in the K8S environment, and ensure automated and real-time monitoring of dynamic changes in agent nodes.

[0006] To achieve the above objectives, embodiments of the present invention provide a method for monitoring a proxy cluster, comprising: In response to user terminal Redis service requests, the deployer deploys a Redis server and a proxy cluster in the Kubernetes environment, and synchronizes the original information of each proxy Pod in the proxy cluster to the metadata database; wherein, the original information includes the original name and the original IP address; The status of each proxy Pod is monitored in real time by an event listener, and a status change event is triggered and sent to the event processing center when the status of the proxy Pod changes. The event processing center updates the latest information of the proxy Pod to the metadata database using a double consistency algorithm based on the state change event; wherein the latest information includes the latest name and the latest IP address; The data collection and analysis system performs monitoring tasks on each of the agent Pods based on the metadata in the metadata database.

[0007] As an improvement to the above scheme, the step of updating the latest information of the proxy Pod to the metadata database by the event processing center based on the state change event using a double consistency algorithm includes: The event processing center uses a real-time consistency algorithm to process the state change events in real time, obtain the latest information of the agent Pod, and store it in the metadata database; The event processing center uses an eventual consistency algorithm to perform a final verification of the latest information of the proxy Pod and synchronize it to the metadata database.

[0008] As an improvement to the above solution, the event processing center uses a real-time consistency algorithm to process the state change events in real time, obtain the latest information of the proxy Pod, and store it in the metadata database, including: The event processing center acquires the state change events in real time. The concurrency controller detects whether the proxy Pod is currently processing data. If the proxy Pod is not currently processing, the event filter determines whether the proxy Pod has already processed the data. If the proxy Pod has not been processed, the mantissa allocator allocates a mantissa to the proxy Pod according to the scenario corresponding to the state change event, generates a new name in a fixed format, and stores the new name and new IP address of the proxy Pod in the metadata database.

[0009] As an improvement to the above solution, the event processing center uses an eventual consistency algorithm to perform a final verification of the latest information of the proxy Pod and synchronize it to the metadata database, including: When a timeout condition is triggered, the event processing center directly retrieves the latest information of all the agent Pods from the K8S and forces synchronization to the metadata database. The timeout condition refers to the fact that the agent cluster has not updated its metadata for a preset time threshold.

[0010] As an improvement to the above scheme, the state change events of the proxy Pod include at least one of addition, deletion, modification and abnormal restart.

[0011] As an improvement to the above solution, the data acquisition and analysis system includes a load balancer, a data collector, and an analyzer. The step of the data collection and analysis system performing monitoring tasks for each agent Pod based on the metadata in the metadata database includes: The load balancer will then... The monitoring tasks of each agent Pod are evenly distributed to the collectors; The collector collects the metadata of the agent Pod from the metadata database according to the assigned monitoring task, and then sends the metadata to the corresponding analyzer. The analyzer performs real-time calculations and analyses based on the metadata, and triggers alerts based on the calculation and analysis results.

[0012] This invention also provides a monitoring system for a proxy cluster, comprising: The deployer is used to respond to the Redis service requests of user terminals, deploy Redis servers and proxy clusters in the K8S environment, and synchronize the original information of each proxy Pod in the proxy cluster to the metadata database; wherein, the original information includes the original name and the original IP address; An event listener is used to monitor the status of each proxy Pod in real time, and to trigger a status change event and send it to the event processing center when the status of the proxy Pod changes. The event processing center is used to update the latest information of the proxy Pod to the metadata database based on the state change event using a double consistency algorithm; wherein, the latest information includes the latest name and the latest IP address; The data collection and analysis system is used to perform monitoring tasks on each of the agent Pods based on the metadata in the metadata database.

[0013] This invention also provides a monitoring device for a proxy cluster, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the monitoring method for the proxy cluster as described in any of the above embodiments.

[0014] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the agent cluster monitoring method as described in any of the above embodiments.

[0015] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the agent cluster monitoring method as described in any of the above embodiments.

[0016] Compared with existing technologies, the agent cluster monitoring method, system, device, medium, and product disclosed in this invention address the problems of IP changes due to abnormal restarts of agent pods in a K8S environment or IP increases / decreases due to scaling up or down, and the inability to accurately obtain monitoring data for agent pods. The invention uses an event listener to monitor the status of each agent pod in real time. When the status of an agent pod changes, a status change event is triggered and sent to the event processing center. A method combining real-time consistency and eventual consistency of tail mapping is designed. The event processing center updates the latest information of the agent pod to the metadata database based on the status change event. This solves the problem of IP increases / decreases due to abnormal restarts of agent pods or scaling up / down, enabling real-time and automatic updates of agent pod metadata in a K8S environment. It accurately and promptly updates agent IPs and maps them to metadata, ensuring the accuracy of IPs collected by the management component and ensuring automated and real-time monitoring of dynamic changes in agent nodes. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a method for monitoring a proxy cluster provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the architecture of the monitoring system for the agent cluster in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the double consistency algorithm in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the working principle of the mantissa distributor in an embodiment of the present invention; Figure 5 This is a flowchart illustrating the eventual consensus algorithm in an embodiment of the present invention; Figure 6 This is a schematic diagram illustrating the principle of the consistent hash algorithm in this embodiment of the invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0020] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0021] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0022] See Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating a method for monitoring a proxy cluster provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the architecture of a proxy cluster monitoring system in an embodiment of the present invention. The embodiment of the present invention provides a method for monitoring a proxy cluster. The method is applied to a proxy cluster monitoring system, which consists of a deployer, a proxy cluster, an event listener (Kubernetes watcher), an event processing center (proxy-watcher), a metadata database, and a data collection and analysis system.

[0023] The method includes steps S11 to S14: S11. In response to the user terminal's Redis service request, the deployer deploys a Redis server and a proxy cluster in the K8S environment, and synchronizes the original information of each proxy Pod in the proxy cluster to the metadata database; wherein, the original information includes the original name and the original IP address; S12. The event listener monitors the status of each proxy Pod in real time, and triggers a status change event and sends it to the event processing center when the status of the proxy Pod changes. S13. The event processing center updates the latest information of the proxy Pod to the metadata database using a double consistency algorithm based on the state change event; wherein, the latest information includes the latest name and the latest IP address; S14. The data collection and analysis system performs a monitoring task for each of the agent Pods based on the metadata in the metadata database.

[0024] It should be noted that the proxy cluster mentioned refers to a proxy cluster that can be deployed in a Kubernetes (K8S) environment to solve the redirection problem of Redis servers in Cluster mode, serving as a cache cluster proxy or a database cluster proxy. Monitoring of Redis server proxy clusters in a K8S environment has some issues, such as dynamic changes in proxy IPs or the number of proxy pods: abnormal restarts of proxy pods will change their IPs, and scaling up or down will increase or decrease IPs, causing previously configured IPs in monitoring components (such as Exporter) to become invalid and unable to obtain monitoring data; pod names cannot map to metadata: proxy pods created by K8S Deployments have random strings at the end of their names, and the suffix changes after re-creation or rebuilding, making it impossible to match them with records in the metadata database; unstable K8S events: K8S pod events (such as deletion and modification) may be triggered repeatedly or delayed, leading to misjudgments by the monitoring system and further affecting the accuracy of metadata.

[0025] To address the aforementioned issues, this invention resolves the problem of inadequate monitoring caused by inconsistent metadata after changes to proxy Pods. Regardless of whether the proxy is added, deleted, or restarted, its IP address, name, and other information can be accurately and promptly updated in the metadata database, ensuring that the monitoring component always obtains the correct proxy address. The key lies in renaming Pods with random suffixes using a fixed format with incrementing suffixes, and then using an algorithm to ensure real-time updates and final verification.

[0026] Specifically, in this embodiment of the invention, during the deployment phase, the user terminal places an order for Redis service on the cloud platform, and the deployer deploys a Redis server and a proxy cluster on K8S. The proxy cluster is middleware between the external application and the Redis server, used to forward requests, handle Redis Cluster redirection, and is also the core object of monitoring.

[0027] The agent cluster includes several agent Pods. The metadata of each agent Pod, including its original name and original IP, is stored in a metadata database. The metadata database includes a real-time cache library etcd and a persistent storage library MySQL, which is the data source for the monitoring component to obtain agent information.

[0028] During the event monitoring phase, the proxy Pod is monitored in real time using the native Kubernetes Watcher. Once the proxy Pod changes, a state change event is triggered and transmitted to the event processing center, Proxy-watcher. Preferably, the state change event of the proxy Pod includes at least one of the following: ADDED, DELETED, MODIFIED, and ERROR, representing scenarios such as a newly created proxy Pod during scaling up, a deleted proxy Pod during scaling down, a change in the state of the proxy Pod, and a failure of the proxy Pod, respectively.

[0029] During the metadata update phase, this embodiment of the invention introduces an event processing center. The event processing center, Proxy-watcher, uses a fixed format naming method with tail mapping and combines a double consistency algorithm to process state change events, updating the latest information such as the latest IP of the proxy Pod and the latest name after renaming to the metadata database, ensuring the consistency and accuracy of the metadata of the proxy Pod after changes (addition, removal, and restart).

[0030] During the monitoring execution phase, the data collection and analysis system performs monitoring tasks for each agent Pod based on the metadata in the metadata database, thereby realizing a full monitoring process of agent Pod analysis, processing, and early warning.

[0031] Using the technical means of this invention, an automatic monitoring and maintenance solution for Redis cluster proxies based on K8S Serverless auto-scaling is proposed. This solution addresses the issues of IP changes due to abnormal restarts of proxy pods in a K8S environment or IP additions / reductions caused by scaling, which prevent accurate acquisition of proxy pod monitoring data. It also addresses the randomness of pod name suffixes in K8S deployments and the instability of K8S events, which prevent mapping to the metadata database. A method combining real-time consistency and eventual consistency of suffix mapping is designed to solve the problems of IP additions / reductions caused by abnormal pod restarts or scaling. This ensures accurate and timely updates of proxy IPs and mapping to metadata, guaranteeing the accuracy of IPs collected by the management component.

[0032] As a preferred embodiment, this invention further implements the above embodiments. Step S13, namely, the event processing center updating the latest information of the proxy Pod to the metadata database according to the state change event using a double consistency algorithm, includes steps S131 to S132: S131. The event processing center uses a real-time consistency algorithm to process the state change event in real time, obtain the latest information of the agent Pod, and store it in the metadata database. S132. The event processing center uses an eventual consistency algorithm to perform a final verification of the latest information of the proxy Pod and synchronize it to the metadata database.

[0033] In this embodiment of the invention, the dual consistency algorithm includes a real-time consistency algorithm and an eventual consistency algorithm.

[0034] It's important to note that the proxy cluster is deployed in Kubernetes Serverless. Serverless's capabilities include on-demand elastic scaling without manual intervention. Therefore, dynamic scenarios exist, such as automatic scaling up / down and abnormal restarts. Kubernetes will automatically scale the number of proxy shards based on business load, i.e., increasing or decreasing the number of proxy shards. Here, shard refers to the proxy stage; one shard corresponds to one proxy Pod. Furthermore, if a proxy Pod itself malfunctions or fails, Kubernetes will automatically restart it, generating a new proxy Pod.

[0035] Kubernetes' watcher mechanism allows you to listen for pod events by registering pod labels. Event types include adding resources, deleting resources, modifying resources, resource version numbers, and exception events.

[0036] Whether scaling up or down, or restarting, the Pod name and Pod IP of newly created proxy Pods will change. This is because Pods created by Kubernetes Deployments have a random string at the end of their names (e.g., redis-proxy-deployment-545694d94-4mp9h), and after a restart or creation, the suffix (4mp9h) will be replaced with a new random string. Kubernetes Pod IPs are temporarily assigned within the cluster, and new proxy Pods will receive entirely new IPs. If the name and IP of the proxy Pod change frequently, the old information (old name, old IP) in the metadata database will become invalid. Monitoring components will still try to connect using the old information, resulting in connection failures and the inability to obtain monitoring data. Furthermore, the new Pod names are random and cannot correspond to historical records in the metadata database, leading to operational chaos. Therefore, a method is needed to monitor Pod changes and maintain the accuracy and consistency of its information in the metadata database.

[0037] To maintain consistency in proxy slicing information, all proxy Pods are reassigned names with a fixed format. Specifically, the automatically generated random suffix names from Kubernetes are discarded, and all Pods are uniformly named `redis-proxy-deployment-X`, where X is an incrementing natural number starting from zero (0, 1, 2, ..., N). The metadata database reassigns slicing names when storing slicing information, defining the following order: redis-proxy-deployment-0, redis-proxy-deployment-1, redis-proxy-deployment-2, ...... redis-proxy-deployment-N.

[0038] By employing the technical means of this invention, a fixed identifier is established so that no matter how the Pod restarts or how the IP changes, its unified name remains unchanged. The metadata database only needs to maintain the unified name and the latest IP to ensure information accuracy. Operation and maintenance personnel and monitoring systems do not need to remember random names; they can identify the agent simply by the suffix "0, 1, 2". Subsequent load balancing and task allocation can also be based on this fixed identifier.

[0039] See Figure 3 This is a flowchart illustrating the double consistency algorithm in this embodiment of the invention. By receiving state change events from Kubernetes watch, proxy-watcher ensures the accuracy of proxy Pod metadata by designing a method that combines real-time consistency and eventual consistency of tail index mapping.

[0040] Further preferably, step S131, namely, the event processing center using a real-time consistency algorithm to process the state change event in real time, obtain the latest information of the agent Pod, and store it in the metadata database, includes: The event processing center acquires the state change events in real time. The concurrency controller detects whether the proxy Pod is currently processing data. If the proxy Pod is not currently processing, the event filter determines whether the proxy Pod has already processed the data. If the proxy Pod has not been processed, the mantissa allocator allocates a mantissa to the proxy Pod according to the scenario corresponding to the state change event, generates a new name in a fixed format, and stores the new name and new IP address of the proxy Pod in the metadata database.

[0041] It should be noted that the Kubernetes API's event listening methods are not strictly synchronous and idempotent in terms of consumption. This is because Kubernetes resource objects are updated in stages in etcd. When implemented in Go, the informer in client-go will watch multiple events through the reflector.

[0042] Deleting a Pod in Kubernetes is not a one-step process, but rather involves updating the etcd database in four steps. Ultimately, the client will receive three Modify events and one Delete event, instead of a single Delete event. The Pod deletion process is as follows: 1. After a user initiates a request to delete a pod, the apiServer receives the request and updates the deletion time and other fields of the pod in etcd, such as the DeletionTimestamp and DeletionGracePeriodSeconds fields. etcd listens for changes in the pod's ResourceVersion (RV), and the client will receive the first Modify event.

[0043] 2. After the Kubernetes node (Kubelet reflector) detects the above update event, it begins deleting the pod. Upon completion, it requests the apiServer to update the pod's status information, specifically the condition and containerStatuses fields. Etcd records the status change, and the RV change causes the client to receive a second Modify event.

[0044] 3. After completing the previous step, Kubelet will send another pod deletion request to the apiServer. The apiServer will update the deletion grace period field of that pod in etcd, that is, set the DeletionGracePeriodSeconds field to 0. Etcd records state changes, and similarly, changes in RV cause the client to receive a third Modify event.

[0045] 4. Finally, the apiServer completely deletes the etcd pod record, and the client receives a Delete event.

[0046] If these events are used to update the metadata database directly, issues such as duplicate processing and concurrency conflicts may occur. If the deletion process of the same Pod is triggered by 3 Modify events and 1 Delete event, it may lead to duplicate updates of the metadata database and data corruption. If the Pod is restarted and scaled up at the same time, multiple events will be processed concurrently, which may lead to naming conflicts and IP mapping errors.

[0047] Therefore, the core objective of real-time consensus algorithms is: Deduplication: Filters out duplicate events; changes to the same Pod (deletion, addition, or restart) are processed only once. Concurrency safety: Prevent multiple events from processing the same Pod simultaneously, which could lead to data conflicts; Consistent naming: Assign fixed-format names to Pods with random tails to ensure that agent information in the metadata database is traceable.

[0048] In other words, the real-time consistency algorithm requires deduplication logic, and the business side only needs to process the deleted pod and the newly created pod once. The real-time consistency strategy first needs to ensure the concurrency safety of the algorithm, secondly, filter duplicate pod events, and finally assign a mantissa to the name of the newly created pod and update the name and IP in the metadata database.

[0049] The following sections detail the algorithms for the three modules: the concurrency controller, the event filter, and the mantissa allocator.

[0050] 1. Concurrency Controller. The concurrency controller is used to solve the problem of concurrent processing of events within the same Pod. Its goal is to control the processing frequency of identical watch events. That is, if a proxy pod is already processing an event, subsequent events will not be processed simultaneously. Events from the same proxy pod can only be processed once at a time, avoiding concurrency conflicts. The implementation principle is as follows: A global "Events Currently Being Processed Msp" is maintained: map[key]func(), where key is the Redis cluster ID + Pod name (uniquely identifying a proxy Pod). When an event arrives, the system first checks if the corresponding key exists in the Map. If it exists, it means the event is being processed by that Pod, and the process returns directly without repeating the process. If it does not exist, the key is stored in the Map, and the event processing logic (such as subsequent filtering and naming) is executed. After processing is complete, the key is deleted from the Map, releasing processing privileges.

[0051] 2. Event Filter. The event filter addresses the issue of duplicate events, ensuring idempotency. Its goal is to determine whether an event has already been processed, avoiding duplicate updates to the metadata database. The algorithm utilizes Redis's `SETNX` command to achieve idempotency, with the key being `redisId` + `podName`. Specifically, before event processing, a unique key is generated: `redisId + podName` (Redis cluster ID + original pod name, uniquely identifying the event). During the `REDis SETNX key` process, if it returns success (key does not exist), the event has not been processed, and the subsequent process continues; if it returns failure (key already exists), the event has already been processed and is discarded. After event processing is complete, this Redis key is deleted to avoid consuming space.

[0052] 3. Endnote Allocator. The endnote allocator solves the problem of random Pod names and ensures metadata consistency. Its core goal is to change the automatically generated random endnote Pod names in Kubernetes (e.g., redis-proxy-deployment-545694d94-4mp9h) into fixed-format names (redis-proxy-deployment-X, where X is a natural number from 0 to 2), allowing the metadata database to stably map Pod IPs and identities. To rename pod names, the source format is redis-proxy-deployment-xxx-xxx, with xxx-xxx being a random string from Kubernetes. The renamed pod name format is redis-proxy-deployment-X, where X is an incrementing natural number starting from zero. The endnote allocation algorithm maintains the allocation of X for various scenarios.

[0053] See Figure 4 This is a schematic diagram of the working principle of the mantissa distributor in the embodiment of the present invention, and the mantissa distribution rules of the mantissa distributor are introduced in different scenarios.

[0054] Deployment Creation: This stage is the initial generation of proxy Pods. Based on the number of proxy slices, the last digit X (incrementing from 0) is used to name new pods, such as redis-proxy-deployment-0, redis-proxy-deployment-1, redis-proxy-deployment-2, etc. The original name of the proxy Pod, the latest name after renaming, the latest IP address, and other information are stored in the MySQL database.

[0055] Shard scaling: Redis supports automatic proxy scaling of shards in the cluster. During scaling, Kubernetes deploymentreplicas directly adjusts the value, and newly added Pods are allocated a last digit of "current maximum X + 1", incrementing sequentially. As shown in the figure, scaling from three shards to five shards is achieved through proxy. The two added shards will be named redis-proxy-deployment-3 and redis-proxy-deployment-4.

[0056] Shard scaling down: Redis supports automatic proxy scaling down of shards in the cluster. During scaling up and down, Kubernetes deployment replicas directly adjust their values. Kubernetes prioritizes deleting pods with more recent creation times, and the corresponding X (the larger number) is reclaimed and stored in the deletion queue for later use. As shown in the diagram, a five-shard proxy is scaled down to a three-shard proxy. The Deployment scaling down strategy prioritizes deleting pods with more recent creation times, which reduces the number of pods redis-proxy-deployment-3 and redis-proxy-deployment-4.

[0057] Abnormal Restart: Failures or migrations can cause proxy pods to restart abnormally. The process involves the old pod crashing, a new pod being created, and the old pod's name (X) being recycled into the deletion queue. The new pod first retrieves its name (X) from the queue (ensuring the name remains unchanged), and updates the new IP address to the metadata database. As shown in the diagram, pod-0 and pod-1 restart. The mantissa allocator captures the old pod event, queries MySQL for the mantissa already allocated to that pod name based on the old pod name, and saves the mantissa to the deletion queue. The newly created pod's name (X) will be retrieved from the deletion queue. Finally, the original pod name, the renamed pod name, and the restarted pod's IP address information are stored in MySQL.

[0058] Multi-scenario hybrid: Newly created pods prioritize taking X from the deletion queue; if the queue is empty, they are allocated according to the maximum X+1. As shown in the diagram, pod-1 restarts, and pod-3 and pod-4 are added. The tail-number allocator maintains a deletion queue that stores the tail numbers of deleted pods. Newly created pods prioritize taking the tail number from this queue for naming. If the queue is empty, allocation is performed incrementally according to the maximum allocation number, which is the largest natural number that the algorithm maintains for allocating pods to this instance.

[0059] After allocating the last digit, the pod IP address from the Kubernetes Event and the newly named pod name are saved to etcd and MySQL, respectively. Then, a waiting period is initiated before resetting the proxy's processing status. This waiting period must be longer than the proxy's deduplication time. Only after resetting the proxy's processing status will subsequent proxy disaster recovery be monitored and processed.

[0060] By employing the technical means of this invention, the real-time consistency algorithm solves the inherent defects of K8S event repetition and concurrency, ensuring that changes to proxy Pods are processed correctly only once; by using a fixed naming format, the randomness of Pod names is broken, allowing the metadata database to stably track the IP and identity of the proxy; and it provides an accurate metadata foundation for subsequent monitoring task allocation and metric collection, preventing monitoring components from failing due to changes in IP and name.

[0061] Further preferably, step S132, namely, the event processing center performing a final verification of the latest information of the proxy Pod using an eventual consistency algorithm and synchronizing it to the metadata database, includes: When a timeout condition is triggered, the event processing center directly retrieves the latest information of all the agent Pods from the K8S and forces synchronization to the metadata database. The timeout condition refers to the fact that the agent cluster has not updated its metadata for a preset time threshold.

[0062] It's important to note that while real-time consistency algorithms can handle duplicate events and concurrent conflicts, respond promptly to agent changes, and maintain the accuracy and consistency of agent information, maximizing the protection against data loss in monitoring, the Kubernetes watcher mechanism is not highly reliable. Issues such as event loss, event timeouts, and Watcher disconnections and reconnections can occur, causing the real-time consistency algorithm to fail. This can lead to inconsistencies between the agent information (IP, name) in the metadata database and the actual Kubernetes data, resulting in monitoring components using outdated information to collect data and causing collection failures. Therefore, a redundant strategy is needed to ensure eventual agent consistency.

[0063] See Figure 5 This is a flowchart illustrating the eventual consistency algorithm in this embodiment of the invention. A special key, / updatingProxyPod / :redisid (redisid is the unique identifier of the Redis cluster), is defined in etcd to mark whether the proxy of this Redis cluster is updating metadata. A fixed time threshold (e.g., 30 seconds, configurable) is set to represent the longest reasonable time for proxy metadata updates. If this time is exceeded, the previous update process is considered to have failed. When the timeout condition is triggered, instead of relying on Watcher events, the K8S API is directly called to fully retrieve the latest information (IP, name, etc.) of all proxy Pods under this Redis cluster and force synchronization to the metadata database.

[0064] Specifically, when Watcher detects a Modify event in a proxy Pod of a Redis cluster (redisid=xxx), it checks in etcd whether the value of key= / updatingProxyPod / :redisid exists.

[0065] If the key does not exist, record the timestamp in etcd. The key is ` / updatingProxyPod / :redisid`, and the value is the current timestamp, indicating that the update of the proxy metadata for this cluster has started. This value is used to distinguish whether the proxy under this `redisid` is changing and the start time of the update. Simultaneously, start a separate coroutine to continuously monitor the existence time of this key. If the coroutine finds that the current time - the timestamp of the key's value > a fixed time threshold, it indicates that the previous update process may have stalled or failed. The coroutine directly calls the Kubernetes API to fetch the latest IPs, names, and other information of all proxy Pods under the `redisid=xxx` cluster, and then fully overwrites and updates ` / proxyList` (the main proxy metadata list) in etcd, ensuring that the metadata database is consistent with the actual data. If the key exists, calculate the time difference based on the current time minus the timestamp of the key's value to determine if the update has timed out. If it has not timed out, the update process is considered to be still in progress, and no operation is performed, waiting for the real-time consistency algorithm to process it. If it has timed out, the previous update process is considered to have failed, and the K8S API is directly called to fully fetch the cluster proxy information, overwrite the update to / proxyList, and force the correction of metadata.

[0066] By employing the technical means of this invention, the eventual consistency algorithm and the real-time consistency algorithm complement each other. The real-time consistency algorithm ensures real-time monitoring, while the eventual consistency algorithm ensures uninterrupted monitoring. When combined, regardless of the reliability of K8SWatcher or the frequent changes in proxy Pods, the metadata database will ultimately be consistent with reality, and the monitoring component will always obtain the correct proxy IP and name, achieving the goal of uninterrupted and data-free monitoring.

[0067] As a preferred embodiment, the present invention is further implemented based on the above embodiments, wherein the data acquisition and analysis system includes a load balancer, a data collector, and an analyzer; Step S14, namely, the monitoring task for each agent Pod performed by the collection and analysis system based on the metadata in the metadata database, includes steps S141 to S143: S141. The load balancer evenly distributes the monitoring tasks for each agent Pod to the collector. S142. The collector collects the metadata of the agent Pod from the metadata database according to the assigned monitoring task, and then sends the metadata to the corresponding analyzer. S143. The analyzer performs real-time calculations and analyses based on the metadata, and triggers an early warning based on the calculation and analysis results.

[0068] In this embodiment of the invention, the data acquisition and analysis unit consists of a load balancer, a data collector, and an analyzer. The load balancer pulls metadata from the proxy and distributes the monitoring tasks evenly across the data collector and the analyzer. After connecting to the proxy, the data collector collects metrics such as CPU usage, latency, and request count, and sends them to the analyzer for further processing. The analyzer receives the metrics from the data collector, performs calculations, analysis, and alerts.

[0069] The following section provides a detailed analysis of the load balancer, data collector, and analyzer.

[0070] The load balancer needs to monitor changes in proxy slices, collectors, and analyzers, and make real-time dynamic adjustments. It is implemented using a consistent hash algorithm and the etcd watch mechanism.

[0071] See Figure 6 This is a schematic diagram illustrating the principle of the consistent hash algorithm in this embodiment of the invention. The principle of the consistent hash algorithm is as follows: Figure 6 As shown, a ring-shaped address space is constructed, connecting all possible values ​​(0~2³²-1) end-to-end to form a ring. Locations are assigned to processing nodes (collectors or analyzers) by hashing the IP address of each collector or analyzer, resulting in a value that corresponds to a virtual node on the ring, aiming for more even distribution. Processing nodes are assigned to task objects (proxy Pods): the unique identifier (proxyId) of each proxy is hashed, also corresponding to a point on the ring, and then the nearest processing node is found clockwise; this node is the dedicated collector or analyzer for that proxy. `peer` represents the hash value of the machine node's IP address, `key` represents the object to be assigned, and the first node found clockwise is the target value. The advantage of consistent hashing is that when the number of machines changes, only a small portion of the data near that node is affected; other mappings remain unchanged. For example... Figure 6 A new peer8 has been added. Only key27 needs to be adjusted to peer8; the others remain unchanged.

[0072] etcd is a key-value database used by the system to store critical information and monitor data changes through a watch mechanism to detect node dynamics. The key key design is as follows: / proxyList / proxyId: Redis proxy metadata information, including proxyId, podName, podIp, etc.

[0073] / monitorList / podIp / proxyId: The proxy instance assigned to the collector. After the collector process runs, it obtains its own proxyId and retrieves the assigned Redis proxy from the / monitorList / podIp path in the etcd database.

[0074] ` / analyzerList / podIp / proxyId`: This is the analyzer to which the proxy instance belongs, storing the analyzer's podIp. The workflow is that after the collector finishes collecting data, it finds the podIp of the analyzer to which the `proxyId` belongs via the ` / analyzerList / podIp / ` path.

[0075] / monitoringProxy: The IP address of the data collector. The value in this key will be updated when the number of data collectors changes.

[0076] / analyzingProxy: The IP address of the analyzer. The value in this key will be updated when the number of analyzers changes.

[0077] The load balancer internally maintains two hash rings. The IP addresses of the collectors and analyzers are registered on these rings using a hash algorithm, forming virtual nodes for the target. Then, the values ​​in ` / proxyList` are traversed, with `proxyId` used as the key for hashing. The nearest virtual node is found clockwise on the hash ring, and the results are maintained in ` / monitorList / ` and ` / analyzerList`. The Redis proxy is maintained in ` / proxyList`, while the IP addresses of the collector and analyzer clusters are maintained in ` / monitoringProxy` and ` / analyzingProxy`, respectively. When the values ​​in ` / proxyList`, ` / monitoringProxy`, and ` / analyzingProxy` change, etcd's watch mechanism can detect this and re-load balancing is performed.

[0078] After the collector starts, it retrieves the Redis proxy to be processed from etcd / monitorList / podIp, and uses multiple threads to connect to the proxy and obtain metric data. Then, it retrieves the analyzer's IP address from analyzerList, uses the IP address as the key and the metric data as the value, and sends it to the proxy list. The analyzer listens to the proxy list in a blocking manner; if new metrics are received, it starts the task flow analyzer to further calculate the metrics and issue an alert.

[0079] Specifically, during the initialization and allocation phase, the load balancer reads three types of information from etcd: ` / proxyList` (all proxies), ` / monitoringProxy` (all collectors), and ` / analyzerProxy` (all analyzers). The load balancer constructs two consistent hash rings: a collector hash ring that maps all collector IPs to the ring, and an analyzer hash ring that maps all analyzer IPs to the ring. It iterates through each proxy (obtained from ` / proxyList`), hashes its `proxyId`, finds the corresponding collector on the collector ring, and stores the mapping "collector IP → proxyId" in ` / monitorList`; it hashes its `proxyId`, finds the corresponding analyzer on the analyzer ring, and stores the mapping "proxyId → analyzer IP" in ` / analyzerList`. After the collectors and analyzers start, they retrieve their tasks from ` / monitorList` and ` / analyzerList`, respectively.

[0080] During the dynamic adjustment phase when nodes change, assuming a peak business period, the Redis proxy is scaled up from 3 to 5 (2 new proxyIds are added to / proxyList). The / proxyList in etcd changes, and the load balancer is notified via the Watch mechanism. The load balancer re-executes the allocation logic of "Phase 1": only hashing the 2 newly added proxies, updating / monitorList and / analyzerList (the allocation relationship of the original 3 proxies remains unchanged). Newly added collection tasks are automatically assigned to idle collectors without manual intervention.

[0081] Similarly, if a data collector fails and goes offline ( / monitoringProxy loses one IP), the load balancer will reassign the proxies that the collector was responsible for to other data collectors and update / monitorList to ensure that monitoring of these proxies is not interrupted.

[0082] During the metric collection phase of the collector's operation, after startup, the collector retrieves the metadata (IP, proxyId, etc.) of all the proxies it is responsible for from ` / monitorList / podIp` (where `podIp` is the collector's own IP) in `etcd`. The collector starts multiple threads, connecting to multiple proxies simultaneously, and collects metrics (such as CPU utilization, average read latency, number of hot keys, etc.) using the custom command `proxy info`. The collector then finds the analyzer IP corresponding to each proxies from ` / analyzerList / proxyId` in `etcd`. The collector sends the analyzer IP as the key and the metric data as the value to the corresponding analyzer's message queue.

[0083] During the data analysis and alerting phase of the analyzer's operation, after startup, the analyzer listens to its own message queue in a blocking manner (waiting for data to be sent by the collector). Upon receiving metric data, the analyzer initiates a task flow to calculate key metrics: for example, whether CPU utilization exceeds 80% or read latency exceeds 10ms; and triggers alerts: if the metrics exceed the limits (e.g., CPU reaches 85%), an alarm is immediately pushed to the operations and maintenance system (such as Grafana). The analyzed metric data can be further stored (e.g., in Prometheus) for subsequent visualization.

[0084] The technical methods employed in this invention involve using a distributed data collection and analysis system for agent Pod monitoring. K8S Serverless can automatically scale the agent up and down based on load. A self-developed management component monitors the Redis cluster agent for faults and status, enabling automatic discovery and allocation of each newly created Redis cluster agent node to collectors and analyzers, ensuring load balancing for the management component even with a large number of Redis clusters. etcd Watch detects node changes in agents, collectors, and analyzers, automatically adjusting task allocation without manual intervention. A consistent hashing algorithm ensures even load distribution across collectors and analyzers, preventing overload of individual components. In the event of a collector or analyzer failure, only a small number of agents are affected, allowing for rapid switching to other nodes without interrupting overall monitoring. This system supports large-scale clusters and offers high scalability.

[0085] Unlike typical Redis monitoring, the Redis cluster proxy is responsible for forwarding commands to the redis-server and for tracking large values ​​and hot keys. Therefore, its monitoring metrics need to reflect its own health and downstream connectivity, as shown in Table 1 below.

[0086] Table 1

[0087] For Redis cluster proxies, the most widely used open-source cluster proxies such as Codis and Predixy do not provide direct data collection interfaces and do not consider proxy monitoring. This solution develops a proxy monitoring command for monitoring components to collect data. The proxy command is designed as `proxy info`, and the information returned by the command includes the information listed in the table above.

[0088] 127.0.0.1:6379>proxy info { "proxy_latency_avg": 0, "proxy_latency_max": 0, "proxy_latency_write": 0, "proxy_latency_read": 0, "proxy_command_count": 0, "proxy_hotkey_count": 0, "proxy_err_count": 0, "proxy_bigvalue_read": 0, "proxy_input_kbps": 0, "proxy_ops_per_sec": 0, "proxy_output_kbps": 0, "proxy_connected_clients": 1, "proxy_cpu_allocated": 8, "proxy_cpu_used_rate": 0.6242219853253138, "proxy_max_clients": 10000 } For latency statistics, read / write request type and send / receive timestamps are added to the request body. These are then written to the message queue in the response packet sent to the client, and the calculation is performed within the proxy. This method effectively alleviates the memory pressure on the proxy for latency statistics under high QPS conditions and also reduces code intervention in the native proxy system.

[0089] For hot key statistics, the most frequent n keys are counted. The method is to use the LFU algorithm to eliminate expired data and take the top n hot keys from the linked list in the LFU algorithm.

[0090] For connection statistics, the number of connections can be determined by the number of keepalives maintained in memory, the maximum number of connections can be read from the system configuration, and the connection rate is the ratio of the number of connections to the maximum number of connections.

[0091] In summary, the proxy cluster monitoring method provided in this embodiment of the invention eliminates the need to allocate NodePorts to each proxy pod, which would consume a large number of service ports, given the limited number of service ports. Furthermore, it avoids consuming excessive network resources, preventing high network loads that could impact business operations. It does not use fixed IP addresses, thus avoiding intrusion into Kubernetes and ensuring the high availability of Kubernetes. Adding or removing IPs after proxy node scaling requires no manual intervention, enabling real-time monitoring after dynamic scaling and saving significant manpower. It automatically restarts and resumes monitoring immediately after proxy failure, preventing data loss and enabling self-repair in container environments.

[0092] This invention provides a monitoring system for a proxy cluster, which includes a deployer, an event listener, an event processing center, a data collection and analysis system, and a metadata database.

[0093] The deployer is used to respond to the Redis service requirements of user terminals, deploy Redis servers and proxy clusters in the K8S environment, and synchronize the original information of each proxy Pod in the proxy cluster to the metadata database; wherein, the original information includes the original name and the original IP address; The event listener is used to monitor the status of each proxy Pod in real time, and trigger a status change event and send it to the event processing center when the status of the proxy Pod changes. The event processing center is used to update the latest information of the proxy Pod to the metadata database using a double consistency algorithm based on the state change event; wherein the latest information includes the latest name and the latest IP address; The data collection and analysis system is used to perform monitoring tasks on each of the agent Pods based on the metadata in the metadata database.

[0094] It should be noted that the agent cluster monitoring system provided in this embodiment of the invention is used to execute all the process steps of the agent cluster monitoring method in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0095] This invention also provides a monitoring device for a proxy cluster, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the monitoring method for the proxy cluster as described in any of the above embodiments.

[0096] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the agent cluster monitoring method as described in any of the above embodiments.

[0097] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the agent cluster monitoring method as described in any of the above embodiments.

[0098] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0099] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for monitoring a proxy cluster, characterized in that, include: In response to user terminal Redis service requests, the deployer deploys a Redis server and a proxy cluster in the Kubernetes environment, and synchronizes the original information of each proxy Pod in the proxy cluster to the metadata database; wherein, the original information includes the original name and the original IP address; The status of each proxy Pod is monitored in real time by an event listener, and a status change event is triggered and sent to the event processing center when the status of the proxy Pod changes. The event processing center updates the latest information of the proxy Pod to the metadata database using a double consistency algorithm based on the state change event; wherein the latest information includes the latest name and the latest IP address; The data collection and analysis system performs monitoring tasks on each of the agent Pods based on the metadata in the metadata database.

2. The monitoring method for a proxy cluster as described in claim 1, characterized in that, The process of updating the latest information of the proxy Pod to the metadata database by the event processing center based on the state change event using a double consistency algorithm includes: The event processing center uses a real-time consistency algorithm to process the state change events in real time, obtain the latest information of the agent Pod, and store it in the metadata database; The event processing center uses an eventual consistency algorithm to perform a final verification of the latest information of the proxy Pod and synchronize it to the metadata database.

3. The monitoring method for the proxy cluster as described in claim 2, characterized in that, The event processing center uses a real-time consistency algorithm to process the state change events in real time, obtain the latest information of the agent Pod, and store it in the metadata database, including: The event processing center acquires the state change events in real time. The concurrency controller detects whether the proxy Pod is currently processing data. If the proxy Pod is not currently processing, the event filter determines whether the proxy Pod has already processed the data. If the proxy Pod has not been processed, the mantissa allocator allocates a mantissa to the proxy Pod according to the scenario corresponding to the state change event, generates a new name in a fixed format, and stores the new name and new IP address of the proxy Pod in the metadata database.

4. The method for monitoring a proxy cluster as described in claim 2, characterized in that, The step of the event processing center using an eventual consistency algorithm to perform a final verification of the latest information of the proxy Pod and synchronize it to the metadata database includes: When a timeout condition is triggered, the event processing center directly retrieves the latest information of all the agent Pods from the K8S and forces synchronization to the metadata database. The timeout condition refers to the fact that the agent cluster has not updated its metadata for a preset time threshold.

5. The monitoring method for a proxy cluster as described in claim 1, characterized in that, The state change events of the agent Pod include at least one of the following: addition, deletion, modification, and abnormal restart.

6. The method for monitoring a proxy cluster as described in claim 1, characterized in that, The data acquisition and analysis system includes a load balancer, a data collector, and an analyzer. The step of the data collection and analysis system performing monitoring tasks for each agent Pod based on the metadata in the metadata database includes: The load balancer evenly distributes the monitoring tasks for each agent Pod to the collector. The collector collects the metadata of the agent Pod from the metadata database according to the assigned monitoring task, and then sends the metadata to the corresponding analyzer. The analyzer performs real-time calculations and analyses based on the metadata, and triggers alerts based on the calculation and analysis results.

7. A monitoring system for an agent cluster, characterized in that, include: The deployer is used to respond to the Redis service requests of user terminals, deploy Redis servers and proxy clusters in the K8S environment, and synchronize the original information of each proxy Pod in the proxy cluster to the metadata database; wherein, the original information includes the original name and the original IP address; An event listener is used to monitor the status of each proxy Pod in real time, and to trigger a status change event and send it to the event processing center when the status of the proxy Pod changes. The event processing center is used to update the latest information of the proxy Pod to the metadata database based on the state change event using a double consistency algorithm; wherein, the latest information includes the latest name and the latest IP address; The data collection and analysis system is used to perform monitoring tasks on each of the agent Pods based on the metadata in the metadata database.

8. A monitoring device for a proxy cluster, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the monitoring method for the agent cluster as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the monitoring method for the agent cluster as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the monitoring method for the agent cluster as described in any one of claims 1 to 6.

Images