Container cloud platform management side anomaly handling methods, devices, media, and equipment

Abstract

This invention provides a method, apparatus, medium, and device for handling anomalies on the management side of a container cloud platform. The method includes: checking whether long-lived connections exist in the container set; if long-lived connections exist, determining whether the long-lived connection will be affected by the number of retransmissions when the peer connection is abnormally disconnected; if it will be affected by the number of retransmissions, optimizing the reconnection timeout configuration to shorten the time required for the local end to detect anomalies on the peer; adding a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing and forwarding rules of the service products provided by the container set to shorten the time required for the local end to detect anomalies on the peer; and verifying the configuration effect. This invention allows the local end to promptly identify anomalies on the peer when they occur, thereby shortening the response time, facilitating rapid recovery of access to the management console, improving the high availability of the container cloud platform management side, and providing reliable assurance for the continuous operation of the container cloud platform.

Description

Technical Field

[0001] This invention relates to the field of container cloud platform technology, and in particular to a method, apparatus, medium, and equipment for handling anomalies on the management side of a container cloud platform. Background Technology

[0002] The container cloud platform can provide high-performance, scalable container application management services, support the lifecycle management of enterprise-level containerized applications, provide a one-stop cloud application empowerment platform for cloud-native applications, support heterogeneous environments such as x86, ARM and MIPS, support multiple container runtimes, multiple container networks and multiple container storage, and create the best containerized application runtime environment for users.

[0003] The container cloud platform management console is used to uniformly view and manage cloud platform products and services. It also provides multi-tenant management capabilities, allowing users to purchase, view, and use cloud platform products and services, manage service products, and monitor cloud service performance and resource utilization. Therefore, the container cloud platform management console's ability to quickly identify anomalies in the event of a failure is crucial for improving availability. Summary of the Invention

[0004] To address at least one of the above technical problems, embodiments of the present invention provide a method, apparatus, medium, and device for handling anomalies on the management side of a container cloud platform.

[0005] According to the first aspect, the container cloud platform management-side anomaly handling method provided by the embodiments of the present invention includes:

[0006] Check if there are long-lived connections in the container collection;

[0007] If a long connection exists, determine whether the long connection will be affected by the number of retransmissions when the peer disconnects abnormally; if it will be affected by the number of retransmissions, optimize the reconnection timeout configuration to shorten the time it takes for the local end to detect the abnormality of the peer.

[0008] To shorten the time required for the local end to detect anomalies on the peer end, a keep-alive detection mechanism and an abnormal instance detection and removal mechanism are added to the routing and forwarding rules of the service products provided by the container set. Specifically, the keep-alive detection mechanism sends a keep-alive probe packet to the peer end every first preset time interval. If no confirmation is received from the peer end after sending N consecutive keep-alive probe packets, the connection between the local end and the peer end is closed. The abnormal instance detection and removal mechanism closes the abnormal backend container set within a third preset time interval if M consecutive abnormal messages occur within a second preset time interval, thereby stopping the abnormal backend container set from providing services. M and N are integers greater than 1.

[0009] After optimizing the reconnection timeout configuration, the effectiveness of this method in detecting anomalies on the peer end is verified. After adding a keep-alive detection mechanism and an anomaly instance detection and removal mechanism to the routing forwarding rules, the effectiveness of this method in detecting anomalies on the peer end is verified.

[0010] In one embodiment, checking whether a long-lived connection exists in the container set includes:

[0011] Determine the node where the container set is located;

[0012] After logging into the node where the container set is located, enter the network namespace of the container set;

[0013] Check if a keep-alive connection exists in the network namespace. If it does, then the long connection exists.

[0014] In one embodiment, determining whether the long connection will be affected by the number of retransmissions when the peer connection is abnormally disconnected includes:

[0015] Determine whether a timeout limit is configured in the custom business code of the business container corresponding to the long connection;

[0016] If so, then it is determined that the long connection is not affected by the number of retransmissions;

[0017] Otherwise, determine whether the connection status of the long connection is keep-alive or retransmission in progress;

[0018] If the connection is in keep-alive state, then the long connection is determined to be unaffected by the number of retransmissions.

[0019] If the connection is in a retransmission state, determine whether the number of retransmissions for the long connection within a specific time period is constantly increasing and whether the number of retransmissions is always less than the upper limit of the number of retransmissions; if so, determine that the long connection is affected by the number of retransmissions; otherwise, determine that the long connection is not affected by the number of retransmissions.

[0020] In one embodiment, optimizing the reconnection timeout configuration includes:

[0021] An initialization container is set in the container set, and the number of retransmissions configured in the initialization container is less than the number of retransmissions configured in the service container in the same container set, so that the service container can receive the service packets retransmitted by the local end according to the number of retransmissions configured in the initialization container when it is in the retransmission state.

[0022] In one embodiment, the optimized configuration of the reconnection timeout further includes:

[0023] In the initialization container, the detection configuration information for the keep-alive state is set; wherein, the detection configuration information includes the probe detection interval time, the maximum number of probe detections, and the time to start the probe;

[0024] Specifically, the probe detection interval configured in the initialization container is less than the probe detection interval configured in the service container, the maximum number of probe detections configured in the initialization container is less than the maximum number of probe detections configured in the service container, and the probe start time configured in the initialization container is earlier than the probe start time configured in the service container, so that the service container can receive probe packets sent by the local end according to the probe configuration information configured in the initialization container in the keep-alive state.

[0025] In one embodiment, N in the keep-alive detection mechanism is the upper limit of the number of probe probes; the abnormal instance detection and removal mechanism is also configured with a maximum ratio and a minimum healthy instance ratio; wherein, the maximum ratio is the maximum ratio of the set of faulty containers that are shut down in the load balancing pool; when the set of healthy containers in the load balancing pool is less than the minimum healthy instance ratio, the abnormal instance detection and removal mechanism is disabled.

[0026] In one embodiment, after adding a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing forwarding rules, the effectiveness of verifying whether the time taken for the local end to detect abnormalities at the peer end has been shortened includes:

[0027] Determine the node where the container set is located; log in to the node where the container set is located and enter the network namespace; add an IP list discarding rule in the network namespace, wherein the IP list discarding rule discards the container set IPs of the ingress gateway;

[0028] Simulate sending multiple requests to the container set corresponding to the container set IP;

[0029] Observe the logs of the ingress gateway, and determine whether the request can be forwarded to the container set if the abnormal information occurs M times consecutively within the second preset time period based on the logs.

[0030] If not, the IP list discard rule is deleted, and the log is used to determine whether the request can be forwarded to the backend container set after the third preset time. If so, it is determined that the verification of the shortened time for the local end to detect anomalies on the other end after adding the abnormal instance detection and removal mechanism to the routing forwarding rule has passed.

[0031] According to the second aspect, the container cloud platform management-side anomaly handling device provided in the embodiments of the present invention includes:

[0032] The first inspection module is used to check whether there are long connections in the container collection;

[0033] The first judgment module is used to determine whether the long connection will be affected by the number of retransmissions when the peer is abnormally disconnected if a long connection exists; if it will be affected by the number of retransmissions, the reconnection timeout is optimized to shorten the time it takes for the local end to detect the abnormality of the peer.

[0034] A mechanism addition module is used to add a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing and forwarding rules of the service products provided by the container set, so as to shorten the time required for the local end to detect abnormalities on the peer end; wherein, the keep-alive detection mechanism sends a keep-alive probe packet to the peer end every first preset time interval, and if no confirmation information is received from the peer end after sending N consecutive keep-alive probe packets, the connection between the local end and the peer end is closed; the abnormal instance detection and removal mechanism closes the abnormal backend container set within a third preset time interval if M consecutive abnormal information occurs within a second preset time interval, so as to stop the abnormal backend container set from providing services; M and N are integers greater than 1;

[0035] The effect verification module is used to verify whether the time taken for the local end to detect anomalies on the peer end has been shortened after optimizing the reconnection timeout configuration; after adding a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing forwarding rules, the effect verification is also performed on whether the time taken for the local end to detect anomalies on the peer end has been shortened.

[0036] In one embodiment, the first inspection module is specifically used to: determine the node where the container set is located; after logging into the node where the container set is located, enter the network namespace of the container set; query the network namespace to see if there is a keep-alive type connection, and if so, the long connection exists.

[0037] In one embodiment, the process by which the first determining module determines whether the long connection will be affected by the number of retransmissions when the peer is abnormally disconnected includes: determining whether a timeout limit is configured in the custom business code of the business container corresponding to the long connection; if so, determining that the long connection is not affected by the number of retransmissions; otherwise, determining whether the connection status of the long connection is keep-alive or retransmission in progress; if it is keep-alive, determining that the long connection is not affected by the number of retransmissions; if it is retransmission in progress, determining whether the number of retransmissions of the long connection keeps increasing within a specific time period and whether the number of retransmissions is always less than the upper limit of the number of retransmissions; if so, determining that the long connection is affected by the number of retransmissions; otherwise, determining that the long connection is not affected by the number of retransmissions.

[0038] In one embodiment, the process of the first judgment module optimizing the reconnection timeout configuration includes: setting an initialization container in the container set, and the number of retransmissions configured in the initialization container is less than the number of retransmissions configured in the service container in the same container set, so that the service container can receive the service packet retransmitted by the local end according to the number of retransmissions configured in the initialization container when in retransmission state.

[0039] In one embodiment, the process of optimizing the reconnection timeout configuration by the first judgment module further includes: setting probe configuration information in the keep-alive state in the initialization container; wherein the probe configuration information includes a probe detection interval time, a maximum number of probe detections, and a probe start time; wherein the probe detection interval time configured in the initialization container is less than the probe detection interval time configured in the service container, the maximum number of probe detections configured in the initialization container is less than the maximum number of probe detections configured in the service container, and the probe start time configured in the initialization container is earlier than the probe start time configured in the service container, so that the service container can receive probe packets sent by the local end according to the probe configuration information configured in the initialization container in the keep-alive state.

[0040] In one embodiment, N in the keep-alive detection mechanism is the upper limit of the number of probe probes; the abnormal instance detection and removal mechanism is also configured with a maximum ratio and a minimum healthy instance ratio; wherein, the maximum ratio is the maximum ratio of the set of faulty containers that are shut down in the load balancing pool; when the set of healthy containers in the load balancing pool is less than the minimum healthy instance ratio, the abnormal instance detection and removal mechanism is disabled.

[0041] In one embodiment, the effect verification module is specifically used for: determining the node where the container set is located; logging into the node where the container set is located and entering the network namespace; adding an IP list discarding rule to the network namespace, wherein the IP list discarding rule discards the container set IP of the ingress gateway; simulating sending multiple requests to the container set corresponding to the container set IP; observing the logs of the ingress gateway, and determining, based on the logs, whether the request can be forwarded to the container set if M consecutive abnormal messages occur within the second preset time period; if not, deleting the IP list discarding rule, and determining, based on the logs, whether the request can be forwarded to the backend container set after the third preset time period; if so, determining that the verification of the shortened time used by the local end to detect abnormalities at the other end after adding the abnormal instance detection and removal mechanism to the routing forwarding rule has passed.

[0042] According to a third aspect, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method provided in the first aspect.

[0043] According to a fourth aspect, the computing device provided in the embodiments of the present invention includes a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method provided in the first aspect.

[0044] The container cloud platform management-side anomaly handling method, apparatus, medium, and device provided in this embodiment of the invention first checks whether there are long-lived connections in the container set. If so, it determines whether the long-lived connection will be affected by the number of retransmissions when the peer is abnormally disconnected. If it will be affected by the number of retransmissions, the reconnection timeout is optimized. A keep-alive detection mechanism and an abnormal instance detection and removal mechanism are added to the routing and forwarding rules in the service products provided by the container set. Finally, the effectiveness is verified by checking whether the time taken for the local end to detect an anomaly at the peer is shortened. Because this embodiment of the invention optimizes the reconnection timeout configuration and adds a keep-alive detection mechanism and an abnormal instance detection and removal mechanism, the local end can promptly identify the anomaly at the peer when it occurs, thereby shortening the response time, helping to quickly restore access to the management console, improving the high availability of the container cloud platform management side, and providing reliable assurance for the continuous operation of the container cloud platform. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating an anomaly handling method for the management side of a container cloud platform according to an embodiment of the present invention.

[0046] Figure 2 This is an overall flowchart of an exception handling method for the management side of a container cloud platform according to an embodiment of the present invention;

[0047] Figure 3 This is a structural block diagram of an anomaly handling device on the management side of a container cloud platform according to an embodiment of the present invention. Detailed Implementation

[0048] In a first aspect, embodiments of the present invention provide a method for handling anomalies on the management side of a container cloud platform, see [link to relevant documentation]. Figure 1 and Figure 2 The method includes the following steps S110 to S140:

[0049] S110. Check if there are long connections in the container set;

[0050] In one embodiment, checking whether there are long-lived connections in the container set in S110 may include the following steps:

[0051] Determine the node where the container set is located;

[0052] After logging into the node where the container set is located, enter the network namespace of the container set;

[0053] Check if a keep-alive connection exists in the network namespace. If it does, then the long connection exists.

[0054] The container set is referred to as a pod, and the container is referred to as a container. A container set includes one or more containers.

[0055] Among them, the keepalive type is the type that keeps the user alive.

[0056] As can be seen, in this embodiment of the invention, keep-alive type connections are treated as long connections.

[0057] Long-lived connections can be configured using S120 and S130. Both long-lived and short-lived connections can be configured using S130.

[0058] S120. If a long connection exists, determine whether the long connection will be affected by the number of retransmissions when the peer disconnects abnormally; if it will be affected by the number of retransmissions, optimize the reconnection timeout configuration to shorten the time required for the local end to detect the abnormality of the peer.

[0059] In one embodiment, determining whether the long connection will be affected by the number of retransmissions when the peer end is abnormally disconnected may include the following steps:

[0060] 1. Determine whether a timeout limit is configured in the custom business code of the business container corresponding to the long connection;

[0061] 2. If so, then it is determined that the long connection is not affected by the number of retransmissions;

[0062] 3. Otherwise, determine whether the connection status of the long connection is keep-alive or retransmission in progress;

[0063] 4. If it is in keep-alive state, then the long connection is determined to be unaffected by the number of retransmissions;

[0064] 5. If it is in the retransmission state, determine whether the number of retransmissions of the long connection within a specific time period is constantly increasing and the number of retransmissions is always less than the upper limit of the number of retransmissions; if so, determine that the long connection is affected by the number of retransmissions; otherwise, determine that the long connection is not affected by the number of retransmissions.

[0065] Understandably, if a timeout limit is configured in the custom business code of the business container, the business code will automatically detect whether to actively disconnect, which takes very little time. If no timeout limit is configured in the custom business code, it is necessary to determine whether the long connection is in a keepalive state or a retransmission state (on state). If it is in a keepalive state, a keepalive probe packet needs to be sent to check if the peer is normal. The number of probe packets sent is unrelated to the number of retransmissions and is therefore not affected by the number of retransmissions. If it is in an on state, it is determined whether the number of retransmissions of the long connection has been increasing within a certain period of time, and whether it is less than the maximum number of retransmissions. If so, it is considered to be affected by the number of retransmissions; otherwise, it is not affected by the number of retransmissions. Note that during retransmissions, business packets, not probe packets, are sent.

[0066] In one embodiment, optimizing the reconnection timeout configuration may include:

[0067] An initialization container is set in the container set, and the number of retransmissions configured in the initialization container is less than the number of retransmissions configured in the service container in the same container set, so that the service container can receive the service packets retransmitted by the local end according to the number of retransmissions configured in the initialization container when it is in the retransmission state.

[0068] As can be seen, for long connections that are affected by the number of retransmissions when the other end experiences a disconnection, an initialization container is added to the container and the container. The retransmission number in the configuration information of the initialization container is less than the retransmission number configured in the service containers in the same container set. In this way, it is not necessary to modify the configuration information of the service containers in the same container set. If the local end continuously sends service packets with the number of retransmissions configured in the initialization container to the service container of the other end in the retransmission state and does not receive an acknowledgment feedback, the long connection is closed.

[0069] For example, in the same set of containers, the retransmission count in the service container is 15, while the retransmission count in the initialization container is 8. Therefore, the local end closes the long connection if it does not receive an acknowledgment after sending 8 service packets in a row, instead of closing the long connection only after sending 15 service packets in a row. This allows the local end to close long connections that have encountered abnormalities in a timely manner.

[0070] In one embodiment, the optimized configuration of the reconnection timeout may further include:

[0071] In the initialization container, the detection configuration information for the keep-alive state is set; wherein, the detection configuration information includes the probe detection interval time, the maximum number of probe detections, and the time to start the probe;

[0072] Specifically, the probe detection interval configured in the initialization container is less than the probe detection interval configured in the service container, the maximum number of probe detections configured in the initialization container is less than the maximum number of probe detections configured in the service container, and the probe start time configured in the initialization container is earlier than the probe start time configured in the service container, so that the service container can receive probe packets sent by the local end according to the probe configuration information configured in the initialization container in the keep-alive state.

[0073] In other words, by setting the probe detection interval, maximum number of probe detections, and probe start time in the keep-alive state within the initialization container, and ensuring that the data configured in the initialization container is less than the corresponding configuration data in the service container, the local end can promptly detect anomalies on the peer in the keep-alive state and close the long-lived connection accordingly. Therefore, by configuring the initialization container, the service container is affected without needing to modify the configuration of each individual service container, making the process more convenient and reliable.

[0074] S130. Add a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing and forwarding rules of the service products provided by the container set, so as to shorten the time required for the local end to detect abnormalities at the peer end; wherein, the keep-alive detection mechanism is to send a keep-alive probe packet to the peer end every first preset time interval, and if no confirmation information is received from the peer end after sending N consecutive keep-alive probe packets, the connection between the local end and the peer end is closed; the abnormal instance detection and removal mechanism is to close the abnormal backend container set within a third preset time interval if M consecutive abnormal information occurs within a second preset time interval, so as to stop the abnormal backend container set from providing services; M and N are integers greater than 1;

[0075] In the liveness detection mechanism, N can be the upper limit of the number of probe probes.

[0076] For example, if no acknowledgment is received from the peer after sending 8 keep-alive probe packets consecutively, the connection between the local end and the peer end will be closed, with a 3-second interval between each two probe packets sent.

[0077] The abnormal instance detection and removal mechanism can also be configured with a maximum ratio and a minimum healthy instance ratio; wherein, the maximum ratio is the maximum ratio of the set of faulty containers that are shut down in the load balancer pool; when the set of healthy containers in the load balancer pool is less than the minimum healthy instance ratio, the abnormal instance detection and removal mechanism is disabled.

[0078] For example, in the abnormal instance detection and removal mechanism, if three abnormal instances occur consecutively within 10 seconds, the abnormal backend container set will be shut down within the next 5 minutes. In this way, the abnormal backend container set will stop providing services for 5 minutes and will be able to provide services again after 5 minutes.

[0079] Specifically, the exceptions can be 502, 503, or 504 errors.

[0080] A 502 error indicates a gateway timeout or gateway error. When a server acts as a gateway or proxy, it returns a 502 error if it receives an invalid response from an upstream server. This usually means that the upstream server has failed or the network connection has been interrupted, preventing it from responding to requests. Possible solutions include checking the server and proxy configurations, clearing browser cache and cookies, and checking network connectivity.

[0081] A 503 error indicates that the service is unavailable. This is usually because the server is temporarily unable to process the request, possibly due to server overload, maintenance, or upgrades. Clients can try sending the request again later or contact the website administrator for more information. Solutions include waiting a while and trying again, or contacting the website administrator.

[0082] A 504 error indicates a gateway timeout. This means that the server acting as a gateway or proxy failed to receive a timely response from the upstream or auxiliary server while attempting to execute a request. Possible solutions include checking network connectivity, clearing browser cache and cookies, and checking website status.

[0083] S140. After optimizing the reconnection timeout configuration, verify whether the time taken for the local end to detect anomalies on the peer end has been shortened; after adding a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing forwarding rules, verify whether the time taken for the local end to detect anomalies on the peer end has been shortened.

[0084] In other words, after optimizing the reconnection timeout configuration and adding a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing forwarding rules, it is necessary to verify the effect to verify whether the local end can close the long connection in time when the other end is abnormal.

[0085] In one embodiment, after adding a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing forwarding rules, the effectiveness verification of whether the time taken for the local end to detect an anomaly at the peer end is shortened may include S141 to S144:

[0086] S141. Determine the node where the container set is located; log in to the node where the container set is located and enter the network namespace; add an IP list discarding rule in the network namespace, wherein the IP list discarding rule discards the container set IP of the ingress gateway;

[0087] S142. Simulate sending multiple requests to the container set corresponding to the container set IP;

[0088] S143. Observe the logs of the ingress gateway and determine whether the request can be forwarded to the container set if the abnormal information occurs M times consecutively within the second preset time period based on the logs.

[0089] S144. If not, delete the IP list discard rule and determine whether the request can be forwarded to the backend container set after the third preset time according to the log. If so, it is confirmed that the verification of the shortened time used by the local end to detect the abnormality of the other end after adding the abnormal instance detection and removal mechanism in the routing forwarding rule has passed.

[0090] Understandably, after adding the IP list discard rule, the container set corresponding to the container set IP will not receive requests, thus causing an anomaly. At this point, multiple requests are simulated to this container set. By observing the ingress gateway logs, it is determined whether the request can be forwarded to the container set if M consecutive anomaly messages occur within the second preset time period. If the request cannot be forwarded to the container set if M consecutive anomaly messages occur within the second preset time period, it indicates that the local end has identified an anomaly on the other end, and the other end cannot provide service. Therefore, the IP list discard rule is deleted. Then, it is determined whether the request can be forwarded to the backend container set after the third preset time period. If it can, it indicates that the other end has recovered and can provide service. Thus, the effectiveness of the anomaly instance detection and removal mechanism is verified.

[0091] Understandably, the verification process for the effectiveness of the keep-alive detection mechanism and the verification process for the effectiveness of the optimized configuration for reconnection timeout are similar. After successful verification, the above methods can be used for anomaly handling to ensure that the local end can promptly identify anomalies on the remote end.

[0092] Here, "this end" refers to the container cloud platform management console.

[0093] Keepalive is an HTTP persistent connection mechanism used to maintain connections during network requests to improve efficiency. Its main difference from short connections is that short connections close immediately after receiving a response, while persistent connections remain open for extended periods, thus reducing server concurrency pressure. In Keepalive connections, retransmission typically refers to TCP heartbeat messages, which detect whether the peer has crashed or is unreachable, initiating a reconnection process.

[0094] As can be seen, in scenarios such as cloud platform management-side node server failures, services with persistent connections maintain the connection even when the peer connection is abnormally disconnected. Because these persistent connections cannot be closed promptly, the service remains unavailable for an extended period. This invention addresses this by promptly closing persistent connections, adding keep-alive detection mechanisms and abnormal instance detection and removal mechanisms, quickly identifying peer anomalies, shortening response time, facilitating rapid restoration of access to the management console, improving the high availability of the container cloud platform management side, and providing reliable assurance for the continuous operation of the container cloud platform.

[0095] Secondly, embodiments of the present invention provide an anomaly handling device for the management side of a container cloud platform, see [link to relevant documentation]. Figure 3 The device 100 includes:

[0096] The first inspection module 110 is used to check whether there are long connections in the container collection;

[0097] The first judgment module 120 is used to determine whether the long connection will be affected by the number of retransmissions when the peer is abnormally disconnected if a long connection exists; if it will be affected by the number of retransmissions, the reconnection timeout is optimized to shorten the time it takes for the local end to detect the abnormality of the peer.

[0098] A mechanism addition module 130 is used to add a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing and forwarding rules of the service products provided by the container set, so as to shorten the time required for the local end to detect abnormalities at the peer end; wherein, the keep-alive detection mechanism sends a keep-alive probe packet to the peer end every first preset time interval, and if no confirmation information is received from the peer end after sending N consecutive keep-alive probe packets, the connection between the local end and the peer end is closed; the abnormal instance detection and removal mechanism closes the abnormal backend container set within a third preset time interval if M consecutive abnormal information occurs within a second preset time interval, so as to stop the abnormal backend container set from providing services; M and N are integers greater than 1;

[0099] The effect verification module 140 is used to verify whether the time taken for the local end to detect anomalies on the peer end has been shortened after optimizing the reconnection timeout configuration; and to verify whether the time taken for the local end to detect anomalies on the peer end has been shortened after adding a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing forwarding rules.

[0100] In one embodiment, the first inspection module is specifically used to: determine the node where the container set is located; after logging into the node where the container set is located, enter the network namespace of the container set; query the network namespace to see if there is a keep-alive type connection, and if so, the long connection exists.

[0101] In one embodiment, the process by which the first determining module determines whether the long connection will be affected by the number of retransmissions when the peer is abnormally disconnected includes: determining whether a timeout limit is configured in the custom business code of the business container corresponding to the long connection; if so, determining that the long connection is not affected by the number of retransmissions; otherwise, determining whether the connection status of the long connection is keep-alive or retransmission in progress; if it is keep-alive, determining that the long connection is not affected by the number of retransmissions; if it is retransmission in progress, determining whether the number of retransmissions of the long connection keeps increasing within a specific time period and whether the number of retransmissions is always less than the upper limit of the number of retransmissions; if so, determining that the long connection is affected by the number of retransmissions; otherwise, determining that the long connection is not affected by the number of retransmissions.

[0102] In one embodiment, the process of the first judgment module optimizing the reconnection timeout configuration includes: setting an initialization container in the container set, and the number of retransmissions configured in the initialization container is less than the number of retransmissions configured in the service container in the same container set, so that the service container can receive the service packet retransmitted by the local end according to the number of retransmissions configured in the initialization container when in retransmission state.

[0103] In one embodiment, the process of optimizing the reconnection timeout configuration by the first judgment module further includes: setting probe configuration information in the keep-alive state in the initialization container; wherein the probe configuration information includes a probe detection interval time, a maximum number of probe detections, and a probe start time; wherein the probe detection interval time configured in the initialization container is less than the probe detection interval time configured in the service container, the maximum number of probe detections configured in the initialization container is less than the maximum number of probe detections configured in the service container, and the probe start time configured in the initialization container is earlier than the probe start time configured in the service container, so that the service container can receive probe packets sent by the local end according to the probe configuration information configured in the initialization container in the keep-alive state.

[0104] In one embodiment, N in the keep-alive detection mechanism is the upper limit of the number of probe probes; the abnormal instance detection and removal mechanism is also configured with a maximum ratio and a minimum healthy instance ratio; wherein, the maximum ratio is the maximum ratio of the set of faulty containers that are shut down in the load balancing pool; when the set of healthy containers in the load balancing pool is less than the minimum healthy instance ratio, the abnormal instance detection and removal mechanism is disabled.

[0105] In one embodiment, the effect verification module is specifically used for: determining the node where the container set is located; logging into the node where the container set is located and entering the network namespace; adding an IP list discarding rule to the network namespace, wherein the IP list discarding rule discards the container set IP of the ingress gateway; simulating sending multiple requests to the container set corresponding to the container set IP; observing the logs of the ingress gateway, and determining, based on the logs, whether the request can be forwarded to the container set if M consecutive abnormal messages occur within the second preset time period; if not, deleting the IP list discarding rule, and determining, based on the logs, whether the request can be forwarded to the backend container set after the third preset time period; if so, determining that the verification of the shortened time used by the local end to detect abnormalities at the other end after adding the abnormal instance detection and removal mechanism to the routing forwarding rule has passed.

[0106] It is understood that explanations, specific implementation methods, beneficial effects, examples, etc. of the contents of the apparatus provided in the embodiments of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0107] Thirdly, embodiments of the present invention provide a computer-readable medium storing computer instructions, which, when executed by a processor, cause the processor to perform the method provided in the first aspect.

[0108] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0109] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0110] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0111] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0112] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the function of any of the above embodiments.

[0113] It is understood that explanations, specific implementation methods, beneficial effects, examples, etc. of the contents in the computer-readable medium provided in the embodiments of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0114] Fourthly, one embodiment of this specification provides a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method of any embodiment of the specification.

[0115] It is understood that explanations, specific implementation methods, beneficial effects, examples, etc. of the computing device provided in the embodiments of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0116] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0117] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, widgets, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0118] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for handling anomalies on the management side of a container cloud platform, characterized in that, include: Check if there are long-lived connections in the container collection; If a long connection exists, determine whether the long connection will be affected by the number of retransmissions when the peer disconnects abnormally; if it will be affected by the number of retransmissions, optimize the reconnection timeout configuration to shorten the time it takes for the local end to detect the abnormality of the peer. To shorten the time required for the local end to detect anomalies on the peer end, a keep-alive detection mechanism and an abnormal instance detection and removal mechanism are added to the routing and forwarding rules of the service products provided by the container set. Specifically, the keep-alive detection mechanism sends a keep-alive probe packet to the peer end every first preset time interval. If no confirmation is received from the peer end after sending N consecutive keep-alive probe packets, the connection between the local end and the peer end is closed. The abnormal instance detection and removal mechanism closes the abnormal backend container set within a third preset time interval if M consecutive abnormal messages occur within a second preset time interval, thereby stopping the abnormal backend container set from providing services. M and N are integers greater than 1. After optimizing the reconnection timeout configuration, the effectiveness of this optimization was verified to see if the time taken for the local end to detect anomalies on the peer end was reduced. After adding a keep-alive detection mechanism and an anomaly instance detection and removal mechanism to the routing forwarding rules, the effectiveness of this optimization was verified to see if the time taken for the local end to detect anomalies on the peer end was reduced. The optimization configuration for reconnection timeout includes: An initialization container is set in the container set, and the number of retransmissions configured in the initialization container is less than the number of retransmissions configured in the service container in the same container set, so that the service container can receive the service packets retransmitted by the local end according to the number of retransmissions configured in the initialization container when it is in the retransmission state.

2. The method according to claim 1, characterized in that, The check for the presence of long-lived connections in the container set includes: Determine the node where the container set is located; After logging into the node where the container set is located, enter the network namespace of the container set; Check if a keep-alive connection exists in the network namespace. If it does, then the long connection exists.

3. The method according to claim 1, characterized in that, The determination of whether the long connection will be affected by the number of retransmissions when the peer end is abnormally disconnected includes: Determine whether a timeout limit is configured in the custom business code of the business container corresponding to the long connection; If so, then it is determined that the long connection is not affected by the number of retransmissions; Otherwise, determine whether the connection status of the long connection is keep-alive or retransmission in progress; If the connection is in keep-alive state, then the long connection is determined to be unaffected by the number of retransmissions. If the connection is in a retransmission state, determine whether the number of retransmissions for the long connection within a specific time period is constantly increasing and whether the number of retransmissions is always less than the upper limit of the number of retransmissions; if so, determine that the long connection is affected by the number of retransmissions; otherwise, determine that the long connection is not affected by the number of retransmissions.

4. The method according to claim 3, characterized in that, The optimization configuration for reconnection timeout also includes: In the initialization container, the detection configuration information for the keep-alive state is set; wherein, the detection configuration information includes the probe detection interval time, the maximum number of probe detections, and the time to start the probe; Specifically, the probe detection interval configured in the initialization container is less than the probe detection interval configured in the service container, the maximum number of probe detections configured in the initialization container is less than the maximum number of probe detections configured in the service container, and the probe start time configured in the initialization container is earlier than the probe start time configured in the service container, so that the service container can receive probe packets sent by the local end according to the probe configuration information configured in the initialization container in the keep-alive state.

5. The method according to claim 4, characterized in that, In the keep-alive detection mechanism, N is the upper limit of the number of probe probes; the abnormal instance detection and removal mechanism is also configured with a maximum ratio and a minimum healthy instance ratio; wherein, the maximum ratio is the maximum ratio of the set of faulty containers that are shut down in the load balancer pool; when the set of healthy containers in the load balancer pool is less than the minimum healthy instance ratio, the abnormal instance detection and removal mechanism is disabled.

6. The method according to claim 1, characterized in that, After adding a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing forwarding rules, the effectiveness of this mechanism in detecting abnormalities at the peer end is verified, including: Determine the node where the container set is located; log in to the node where the container set is located and enter the network namespace; add an IP list discarding rule in the network namespace, wherein the IP list discarding rule discards the container set IPs of the ingress gateway; Simulate sending multiple requests to the container set corresponding to the container set IP; Observe the logs of the ingress gateway, and determine whether the request can be forwarded to the container set if the abnormal information occurs M times consecutively within the second preset time period based on the logs. If not, the IP list discard rule is deleted, and the log is used to determine whether the request can be forwarded to the backend container set after the third preset time. If so, it is determined that the verification of the shortened time for the local end to detect anomalies on the other end after adding the abnormal instance detection and removal mechanism to the routing forwarding rule has passed.

7. An anomaly handling device for the management side of a container cloud platform, characterized in that, include: The first inspection module is used to check whether there are long connections in the container collection; The first judgment module is used to determine whether the long connection will be affected by the number of retransmissions when the peer is abnormally disconnected if a long connection exists; if it will be affected by the number of retransmissions, the reconnection timeout is optimized to shorten the time it takes for the local end to detect the abnormality of the peer. A mechanism addition module is used to add a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing and forwarding rules of the service products provided by the container set, so as to shorten the time required for the local end to detect abnormalities on the peer end; wherein, the keep-alive detection mechanism sends a keep-alive probe packet to the peer end every first preset time interval, and if no confirmation information is received from the peer end after sending N consecutive keep-alive probe packets, the connection between the local end and the peer end is closed; the abnormal instance detection and removal mechanism closes the abnormal backend container set within a third preset time interval if M consecutive abnormal information occurs within a second preset time interval, so as to stop the abnormal backend container set from providing services; M and N are integers greater than 1; The effect verification module is used to verify whether the time taken for the local end to detect anomalies on the peer end has been shortened after optimizing the reconnection timeout configuration; and to verify whether the time taken for the local end to detect anomalies on the peer end has been shortened after adding a keep-alive detection mechanism and an abnormal instance detection and removal mechanism to the routing forwarding rules. The optimization configuration for reconnection timeout includes: An initialization container is set in the container set, and the number of retransmissions configured in the initialization container is less than the number of retransmissions configured in the service container in the same container set, so that the service container can receive the service packets retransmitted by the local end according to the number of retransmissions configured in the initialization container when it is in the retransmission state.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed in a computer, causes the computer to perform the method described in any one of claims 1 to 6.

9. A computing device, characterized in that, The method includes a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method described in any one of claims 1 to 6.

Images