Implementation method and system for cloud native platform event collection

Abstract

The invention relates to the technical field of cloud native, in particular to an implementation method and system for cloud native platform event collection, and the method comprises the following steps: collecting cloud native platform key events through a collection device; analyzing the original event information acquired by the acquisition device, and pushing the original event information to the screening device; screening the event information analyzed by the analysis device according to a rule; pushing the screened event information to a specified receiving device through a pushing device according to a user demand; the method has the beneficial effects that the event information operated by the cloud native platform is collected in real time and pushed to the user, so that the user is helped to quickly position a root source, solve problems and recover services, or discover abnormities and avoid faults as early as possible; mTTR (average failure recovery time) is reduced, and QoS (quality of service) and SLA (service level of service) indexes are comprehensively improved; multiple collection sources can be configured, events of multiple cloud native platforms can be collected at the same time, and repeated deployment is not needed.

Description

Technical Field

[0001] The present invention relates to the field of cloud native technology, and in particular to a method and system for implementing event collection on a cloud native platform. Background Art

[0002] The cloud native platform is a full-stack cloud platform based on container technology, integrated with Kubernetes, and deeply supports DevOps and microservice governance. The cloud native platform can help users simplify deployment, monitoring, operation and maintenance, and other container application lifecycle management tasks, while providing management and operation capabilities for DevOps pipelines and microservice governance. It helps users to achieve containerization transformation of traditional applications and rapid launch of new applications, thereby reducing costs and improving efficiency for digital transformation. It is an inevitable trend for cloud native platforms to become a necessity for enterprise digital infrastructure. Not only the Internet industry, but also various industries such as manufacturing, energy, medical care, and education are embracing cloud native technology.

[0003] The cloud native platform is an automated platform that helps cloud native applications develop and deliver. It is composed of a series of related platform capabilities, including application resource management, unified service management, configuration management, security management, measurement control, and observability. As cloud native covers more and more industries, the scale of use is getting larger and larger, and the impact of failures is increasing, it is crucial to ensure the stable operation of the cloud native platform. For users, every time they create a resource, in addition to seeing its final state, they also want to see the resource execution process and the steps that have been taken in the middle. This feedback information is very important for debugging. Some tasks will fail or get stuck at a certain step. With this information, the problem can be accurately located. The cloud native platform has the gene of self-healing of faults, and problems in the platform operation may be automatically repaired. However, when a serious fault occurs and the platform cannot heal itself, how the platform manager / maintainer can grasp the fault situation in the first time and take relevant measures to repair it is a challenge currently faced.

[0004] Currently, cloud native platform health status monitoring generally adopts the solution of collecting platform logs for analysis. This solution collects logs regularly and persists them to the database, and then analyzes the platform health status through intelligent algorithms. The current solution has the following problems:

[0005] Low efficiency: The cloud-native platform generates a large number of logs. If they are analyzed one by one, a large amount of computing resources are required, and the efficiency is low, the latency is high, and fault information cannot be pushed to users in real time.

[0006] Insufficient accuracy: It is impossible to judge the operation status of resource objects only through logs. For example, after a pod stops due to a failure, the pod will no longer generate logs, making it impossible to judge whether the pod is abnormal through logs. Summary of the invention

[0007] The purpose of the present invention is to provide a method and system for implementing event collection on a cloud native platform to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: a method for implementing event collection of a cloud native platform, the method comprising the following steps:

[0009] Collect key events of the cloud native platform through collection devices;

[0010] Analyze the original event information collected by the collection device and push it to the screening device;

[0011] Filtering the event information analyzed by the analyzing device according to the rules;

[0012] Push the filtered event information to the designated receiving device through the push device according to user needs;

[0013] Users can view event information generated by the cloud native platform in real time from the receiving device.

[0014] Preferably, the specific operations for collecting key events of the cloud native platform include:

[0015] The key event information of the cloud native platform is collected through the collection device, and the event stream is transmitted to the parsing device. The normal or abnormal events generated by the current Kubernetes cluster are monitored in real time through the event interface kubeClient.CoreV1().Events opened by Kubernetes, the underlying container orchestration and scheduling system of the cloud native platform. By monitoring this interface, events of the specified namespace or all namespaces are obtained. The event information includes event type, event occurrence time, event object, event name, event details, event cause, and event metadata; event types are divided into two types: warning and normal.

[0016] Preferably, the specific operations of parsing the original event information include:

[0017] The event information collected by the collection device is in string format. To facilitate the processing flow of subsequent devices, the strings pushed from the collection device are processed one by one through the code program and parsed into a fixed format structure; after each event information is parsed, the event structure is pushed to the next step process screening device for processing.

[0018] Preferably, the specific operations of filtering event information according to the rules include:

[0019] Read filter rules from configuration files or database;

[0020] Traverse the event structures pushed by the parsing device one by one and match them according to the screening rules;

[0021] If the event structure meets the filtering rules, add it to the collection; otherwise, discard it;

[0022] When the collection reaches a certain number or exceeds a certain time, the structure in the collection is pushed to the next step.

[0023] Preferably, the push device has multiple push source object modules built in, and each push module has its corresponding push logic written in the form of a plug-in. If other channels need to receive push messages in the future, the configuration information of the receiving device is obtained from the program startup parameters, and the built-in push module is matched for push according to the user's configuration.

[0024] A cloud native platform event collection implementation system, the system is composed of a collection device, an analysis device, a screening device, a push device and a receiving device;

[0025] A collection device is used to collect key events of the cloud native platform;

[0026] The parsing device parses the original event information collected by the collecting device and pushes it to the screening device;

[0027] A screening device, screening the event information parsed by the parsing device according to the rules;

[0028] A push device pushes the filtered event information to a designated receiving device through the push device according to user needs;

[0029] Receiving device: Users can view event information generated by the cloud native platform in real time from the receiving device.

[0030] Preferably, the collection device collects key event information of the cloud native platform through the collection device, and transmits the event stream to the parsing device, and monitors the normal or abnormal events generated by the current Kubernetes cluster in real time through the event interface kubeClient.CoreV1().Events opened by Kubernetes, the underlying container orchestration and scheduling system of the cloud native platform, and obtains events of the specified namespace or all namespaces by monitoring the interface. The event information includes event type, event occurrence time, event object, event name, event details, event cause, and event meta-information; event types are divided into two types: warning and normal.

[0031] Preferably, the event information collected by the analysis device and the collection device is in a string format. To facilitate the processing flow of subsequent devices, the strings pushed from the collection device are processed one by one through the code program and parsed into a structure in a fixed format; after each event information is parsed, the event structure is pushed to the next step process screening device for processing.

[0032] Preferably, the screening device reads screening rules from a configuration file or a database;

[0033] Traverse the event structures pushed by the parsing device one by one and match them according to the screening rules;

[0034] If the event structure meets the filtering rules, add it to the collection; otherwise, discard it;

[0035] When the collection reaches a certain number or exceeds a certain time, the structure in the collection is pushed to the next step.

[0036] Preferably, the push device has multiple push source object modules built in, and the corresponding push logic is written in each push module, which exists in the form of a plug-in. If other channels need to receive push messages in the future, the configuration information of the receiving device is obtained from the program startup parameters, and the built-in push module is matched for push according to the user's configuration.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The implementation method and system for cloud native platform event collection proposed in the present invention collect event information of cloud native platform operation in real time and push it to users, so as to help users quickly locate the root cause, solve the problem, restore the service, or discover anomalies and avoid failures as early as possible; reduce MTTR (mean time to recovery) and comprehensively improve service QoS (quality of service) and SLA (service level) indicators; multiple collection sources can be configured to collect events of multiple cloud native platforms at the same time without repeated deployment; multiple receiving sources can be configured to support simultaneous push to multiple receiving devices to meet different usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Embodiment 1

[0042] See also Figure 1 The present invention provides a technical solution: a method for implementing event collection of a cloud native platform, the method comprising the following steps:

[0043] The key events of the cloud native platform are collected through the collection device; the key event information of the cloud native platform is collected through the collection device, and the event stream is transmitted to the parsing device, and the normal or abnormal events generated by the current Kubernetes cluster are monitored in real time through the event interface kubeClient.CoreV1().Events opened by Kubernetes, the underlying container orchestration and scheduling system of the cloud native platform, and the events of the specified namespace or all namespaces are obtained by monitoring the interface. The event information includes event type, event occurrence time, event object, event name, event details, event cause, and event meta-information; event types are divided into 2 types: warning and normal.

[0044] The original event information collected by the collection device is parsed and pushed to the screening device; the event information collected by the collection device is in a string format. To facilitate the processing flow of subsequent devices, the strings pushed from the collection device are processed one by one through the code program and parsed into a fixed format structure; after each event information is parsed, the event structure is pushed to the next step process screening device for processing.

[0045] Filtering the event information analyzed by the analysis device according to the rules; reading the filtering rules from the configuration file or the database;

[0046] Traverse the event structures pushed by the parsing device one by one and match them according to the screening rules;

[0047] If the event structure meets the filtering rules, add it to the collection; otherwise, discard it;

[0048] When the collection reaches a certain number or exceeds a certain time, the structure in the collection is pushed to the next step.

[0049] The filtered event information is pushed to the designated receiving device through the push device according to user needs; the push device has multiple built-in push source object modules, and each push module has its corresponding push logic written in the form of a plug-in. If other channels need to receive push messages later; the configuration information of the receiving device is obtained from the program startup parameters, and the built-in push module is matched for push according to the user's configuration.

[0050] Users can view event information generated by the cloud native platform in real time from the receiving device.

[0051] Embodiment 2

[0052] Based on the first embodiment, a cloud native platform event collection implementation system is proposed, the system is composed of a collection device, a parsing device, a screening device, a push device and a receiving device;

[0053] 1. Collection device

[0054] The collection device is a device for collecting key events of the cloud native platform. The main function of the collection device is to collect key event information of the cloud native platform and transmit the event stream to the parsing device. Through the event interface kubeClient.CoreV1().Events opened by Kubernetes, the underlying container orchestration and scheduling system of the cloud native platform, normal or abnormal events generated by the current Kubernetes cluster can be monitored in real time. By monitoring this interface, events of the specified namespace or all namespaces (namespace is a logical isolation resource of Kubernetes, used to control multi-tenant usage permissions) are obtained. The event information includes event type, event occurrence time, event object, event name, event details, event cause, event meta information, etc.; event types are divided into two types: warning and normal. Warning events indicate that the state of generating this event is generated in an unexpected state, and Normal events indicate the expected state to be reached, which is consistent with the current state. Take the creation of a pod life cycle as an example. When creating a Pod, the Pod will first enter the Pending state, waiting for the image to be pulled. When the image is pulled and passes the health check, the Pod's state changes to Running. At this time, a Normal event will be generated. If a Pod fails during operation due to OOM or other reasons and enters a Failed state, which is unexpected, a Warning event will be generated in Kubernetes. In this scenario, if we can monitor the generation of events, we can check some problems that are easily overlooked by resource monitoring in a timely manner. Event object resources include pod / node / kubelet / custom objects, etc.

[0055] The event collection device supports multiple collection sources, that is, it can collect event information from multiple cloud native platforms at the same time. The specific steps are:

[0056] The source configuration information in the configuration file or the program startup parameters is read through program code, and the source configuration information is added to the source collection; the source configuration information includes the IP address, account, password or token of the Kubernetes cluster, and the account needs to be granted query permissions for the Kubernetes cluster event collection interface.

[0057] By reading the source configuration file, determine the number of Kubernetes in the source configuration, start a thread for each Kubernetes cluster, connect to the Kubernetes cluster, and perform event collection tasks;

[0058] Each collection task thread first connects to the Kubernetes cluster through IP, user, password and other information, calls the kubeClient.CoreV1().Events interface, and loads the data returned by the interface into memory;

[0059] Push the event information in memory to the parsing device.

[0060] The event collection device only performs event collection operations, does not perform any processing or screening on the collected event information, and directly submits it to the event analysis device for processing.

[0061] 2. Analysis device

[0062] The analyzing device analyzes the original event information collected by the collecting device and pushes it to the screening device.

[0063] The event information collected by the collection device is in a string format. To facilitate the processing flow of subsequent devices, the strings pushed from the collection device are processed one by one through the code program and parsed into a fixed-format structure.

[0064] Take the golang language as an example:

[0065]

[0066]

[0067] Cluster: Cluster name

[0068] Namespace: The namespace where the event object is generated.

[0069] Kind: the bound event type, for example: Node / Pod / Namespace, etc.

[0070] Name: Event name.

[0071] Type: event level, Normal / Warning.

[0072] Reason: The reason why the event occurred.

[0073] Message: Event information details.

[0074] After each event information is parsed, the event structure is pushed to the next step process screening device for processing.

[0075] 3. Screening device

[0076] The main function of the screening device is to screen the event information parsed by the parsing device according to certain rules. The cloud native platform generates a large amount of event information. Users can use the screening device to filter out event information that does not need to be paid attention to according to the screening rules, thereby retaining the corresponding event information for the next step of processing based on the event importance level or the resource object of concern, thereby improving the overall efficiency of the system and reducing the resource loss of the program itself.

[0077] The filtering dimensions include the cluster where the event is located, the namespace where the event is located, the resource type that generates the event, and the level of the event.

[0078] The filtering rules can be configured in the configuration file or in the database. The events in the cloud native platform include various event objects, such as pod, deployment, node, kubelet, crd (custom object), etc. The event types are divided into normal and warning;

[0079] Event screening process:

[0080] Read filter rules from configuration files or database;

[0081] Traverse the event structures pushed by the parsing device one by one and match them according to the screening rules;

[0082] If the event structure meets the filtering rules, add it to the collection; otherwise, discard it;

[0083] When the collection reaches a certain number or exceeds a certain time, the structure in the collection is pushed to the next step.

[0084] 4. Pushing device

[0085] The push device pushes the filtered event information to the designated receiving device according to the user's needs. The main functions of the push device include data packaging and push.

[0086] The push device has built-in multiple push source object modules, such as DingTalk, WeChat, database, webhook, etc. The corresponding push logic is written in each push module and exists in the form of a plug-in. If other channels need to receive push in the future, a plug-in can be added to the push device. The advantage of this model is that it is highly scalable.

[0087] The configuration information of the receiving device is obtained from the program startup parameters, and the built-in push module is matched for push according to the user's configuration.

[0088] 5. Receiving device

[0089] The receiving device, as the next step of the push device, is a device that receives or stores event information, such as WeChat, email, etc. Users can view the event information generated by the cloud native platform in real time from the receiving device.

[0090] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for implementing event collection on a cloud native platform, characterized in that: The method comprises the following steps: Collect key events of the cloud native platform through collection devices; Analyze the original event information collected by the collection device and push it to the screening device; Filtering the event information analyzed by the analyzing device according to the rules; Push the filtered event information to the designated receiving device through the push device according to user needs; Users can view event information generated by the cloud native platform in real time from the receiving device.

2. According to claim 1, a method for implementing event collection of a cloud native platform is characterized in that: The specific operations for collecting key events of the cloud native platform include: The key event information of the cloud native platform is collected through the collection device, and the event stream is transmitted to the parsing device. The normal or abnormal events generated by the current Kubernetes cluster are monitored in real time through the event interface kubeClient.CoreV1().Events opened by Kubernetes, the underlying container orchestration and scheduling system of the cloud native platform. By monitoring this interface, events of the specified namespace or all namespaces are obtained. The event information includes event type, event occurrence time, event object, event name, event details, event cause, and event meta-information; event types are divided into two types: warning and normal.

3. According to claim 1, a method for implementing event collection of a cloud native platform is characterized in that: The specific operations for parsing the original event information include: The event information collected by the collection device is in string format. To facilitate the processing flow of subsequent devices, the strings pushed from the collection device are processed one by one through the code program and parsed into a fixed format structure; after each event information is parsed, the event structure is pushed to the next step process screening device for processing.

4. The method for implementing cloud native platform event collection according to claim 1, characterized in that: The specific operations for filtering event information according to rules include: Read filter rules from configuration files or database; Traverse the event structures pushed by the parsing device one by one and match them according to the screening rules; If the event structure meets the filtering rules, add it to the collection; otherwise, discard it; When the collection reaches a certain number or exceeds a certain time, the structure in the collection is pushed to the next step.

5. The method for implementing cloud native platform event collection according to claim 1, characterized in that: The push device has built-in push source object modules. Each push module has its corresponding push logic written in it, which exists in the form of a plug-in. If other channels need to receive push messages in the future, the configuration information of the receiving device is obtained from the program startup parameters, and the built-in push module is matched for push according to the user's configuration.

6. A system for implementing cloud native platform event collection according to the method for implementing cloud native platform event collection according to any one of claims 1 to 5, characterized in that: The system is composed of a collection device, an analysis device, a screening device, a pushing device and a receiving device; A collection device is used to collect key events of the cloud native platform; The parsing device parses the original event information collected by the collecting device and pushes it to the screening device; A screening device, screening the event information parsed by the parsing device according to the rules; A push device pushes the filtered event information to a designated receiving device through the push device according to user needs; Receiving device: Users can view event information generated by the cloud native platform in real time from the receiving device.

7. A cloud native platform event collection implementation system according to claim 6, characterized in that: The collection device collects key event information of the cloud native platform through the collection device, and transmits the event stream to the parsing device, and monitors the normal or abnormal events generated by the current Kubernetes cluster in real time through the event interface kubeClient.CoreV1().Events opened by the underlying container orchestration and scheduling system Kubernetes of the cloud native platform. By monitoring the interface, the event of the specified namespace or all namespaces is obtained, and the event information includes event type, event occurrence time, event object, event name, event details, event cause, and event meta information; Event types are divided into two types: warning and normal.

8. The cloud native platform event collection implementation system according to claim 6, characterized in that: The event information collected by the analysis device and the collection device is in a string format. To facilitate the processing flow of the subsequent device, the strings pushed from the collection device are processed one by one by the code program and parsed into a structure with a fixed format; After each event information is parsed, the event structure is pushed to the next step process screening device for processing.

9. A cloud native platform event collection implementation system according to claim 6, characterized in that: The screening device reads screening rules from a configuration file or a database; Traverse the event structures pushed by the parsing device one by one and match them according to the screening rules; If the event structure meets the filtering rules, add it to the collection; otherwise, discard it; When the collection reaches a certain number or exceeds a certain time, the structure in the collection is pushed to the next step.

10. The cloud native platform event collection implementation system according to claim 6, characterized in that: The push device has multiple push source object modules built in. Each push module has its corresponding push logic written in it, which exists in the form of a plug-in. If other channels need to receive push messages in the future, the configuration information of the receiving device is obtained from the program startup parameters, and the built-in push module is matched for push according to the user's configuration.

Images