Application log management method and system based on cloud native scene

Abstract

The invention provides an application log management method and system based on a cloud native scene. According to the method, firstly, log classification standards are set based on the requirements of different application programs deployed by a Kubernetes cluster, then an output path is configured based on the classification standards, and different types of logs are recorded into respective corresponding files respectively, so that subsequent management and analysis of the application logs are facilitated, and the efficiency of the application logs is improved. And searching in a large number of mixed logs is not needed. The method comprises the following steps of: using a local shared storage volume EmptyDir mounted on a temporary directory of each application Pod to share a log directory between a main application container and a log agent container Sedar, running a Logging Agent through the log agent container Sedar, monitoring a log file in the local shared storage volume, and sending the log file to a remote storage so as to be convenient to retrieve. According to the method, complicated middleware and configuration are not needed, so that the management efficiency of the application logs can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to an application log management method and system based on a cloud native scenario. Background Art

[0002] In a modern microservices architecture system, the management of application logs plays a crucial role and is an important link to ensure the stable operation of application programs, conduct fault troubleshooting, and meet compliance requirements.

[0003] Currently, traditional log management methods usually rely on complex middleware and configurations. At the same time, in order to achieve log collection and forwarding, specialized log collection middleware may also be used. These middleware each undertake different tasks and jointly constitute the infrastructure for log management.

[0004] However, the existing technology relies on middleware and configurations for log management, and requires multi-faceted configurations at the entire system architecture level to ensure that logs can be safely and efficiently transmitted between various nodes. At the same time, each middleware has its own complex configuration parameters and options, which will lead to low management efficiency of application logs. Summary of the Invention

[0005] An embodiment of the present invention provides an application log management method based on a cloud native scenario, which can improve the management efficiency of application logs. The method includes:

[0006] A1: Determine a log classification standard according to the requirements of the application programs deployed in the Kubernetes cluster. Among them, at least one application Pod is configured for each application program, and each of the application Pods includes a log proxy container Sidercar and at least one main application container Application Container;

[0007] A2: Configure the output paths of different categories of log data according to the log classification standard. Among them, the output path is the local shared storage EmptyDir corresponding to the temporary directory mounted on each of the application Pods;

[0008] A3: Based on the output path, store the log data generated by the operation of each of the main application containers Application Container in the local shared storage EmptyDir. Among them, the log data includes at least one of error log data ErrLog Data, warning log data WarnLog Data, and information log data InfoLog Data;

[0009] A4: Run the Logging Agent using the log proxy container Sidercar, and read the log data in the local shared storage EmptyDir based on the configuration mapping of the log proxy container Sidercar, where the configuration mapping is ConfigMap Logging Agent;

[0010] A5: Use the fluent-bit of the log proxy container Sidercar to send the read log data to the remote storage Remote Elastic Search to retrieve the target log data.

[0011] Preferably,

[0012] The application requirements include: different running states and event types of the application.

[0013] Preferably,

[0014] After A3 and before A4, it further includes:

[0015] Create a configuration mapping, where the configuration mapping is the read configuration information of LoggingAgent in the log proxy container Sidercar, and the read configuration information includes at least one of the file path of the log data, the format of the log data, and the read frequency;

[0016] A4 includes:

[0017] Run the Logging Agent using the log proxy container Sidercar, and mount the configuration mapping into the log proxy container Sidercar;

[0018] Based on the Logging Agent and the configuration mapping, write the log data to the local shared storage EmptyDir and the standard output;

[0019] Use the tail command to redirect the log data file, and execute the kubectl logs command to read the log data in the local shared storage EmptyDir.

[0020] Preferably,

[0021] After A5, it further includes:

[0022] Configure a health check probe in each application Pod, where the health check pointer is used to determine the status of the log data file;

[0023] Based on the health check probe, determine whether the storage status of the log data file is abnormal, where the storage status includes whether the log data file is generated according to a preset time and whether the log data file exists in the local shared storage EmptyDir;

[0024] Determine whether the reading status of the log data file is abnormal, where the reading status includes whether the log proxy container Sidercar can normally read the log data file;

[0025] Determine whether the file size status is abnormal, where the file size status includes whether the size of the log data file exceeds a preset threshold;

[0026] Using the timestamp of the log data file, determine whether the update status is abnormal, where the update status includes whether new log data is written into the log data file;

[0027] When the health check probe detects that at least one of the storage status, the reading status, the file size status, and the update status is abnormal, generate an alarm information prompt.

[0028] In a second aspect, an embodiment of the present invention provides an application log management system based on a cloud native scenario. The system includes:

[0029] A determination module, configured to determine a log classification standard according to the requirements of the application program deployed in the Kubernetes cluster, where each application program corresponds to at least one configured application Pod, and each of the application Pods includes a log proxy container Sidercar and at least one main application container Application Container; A configuration module, configured to configure the output paths of different categories of log data according to the log classification standard determined by the determination module, where the output paths are local shared storages EmptyDirs corresponding to the temporary directories mounted on each of the application Pods;

[0030] A local storage module, which is used to store the log data generated by each main application container running based on the output path configured by the configuration module in the local shared storage EmptyDir, where the log data includes at least one of error log data ErrLog Data, warning log data WarnLog Data, and information log data InfoLog Data; A reading module, which is used to run Logging Agent by using the log proxy container Sidercar, and read the log data in the local shared storage EmptyDir corresponding to the local storage module based on the configuration mapping of the log proxy container Sidercar, where the configuration mapping is ConfigMapLogging Agent;

[0031] A remote storage module, which is used to send the log data read by the reading module to the remote storage Remote Elastic Search by using fluent-bit of the log proxy container Sidercar to retrieve the target log data.

[0032] Preferably,

[0033] The determination module is further used to determine the log classification standard according to the different running states and event types of the applications deployed in the Kubernetes cluster.

[0034] Preferably,

[0035] After the local storage module and before the reading module, it further includes:

[0036] A creation module, which is used to create a configuration mapping, where the configuration mapping is the reading configuration information of Logging Agent in the log proxy container Sidercar, and the reading configuration information includes at least one of the file path of the log data, the format of the log data, and the reading frequency;

[0037] The reading module is further used to execute:

[0038] Run Logging Agent by using the log proxy container Sidercar, and mount the configuration mapping into the log proxy container Sidercar;

[0039] Write the log data into the local shared storage EmptyDir and the standard output based on the Logging Agent and the reading configuration information;

[0040] Use the tail command to redirect the log data file, and execute the kubectl logs command to read the log data in the local shared storage EmptyDir.

[0041] Preferably,

[0042] After the remote storage module, it further includes: an inspection module;

[0043] The inspection module is used to execute:

[0044] Configure a health check probe in each of the application Pods, where the health check pointer is used to determine the status of the log data file;

[0045] Based on the health check probe, determine whether the storage status of the log data file is abnormal, where the storage status includes whether the log data file is generated according to a preset time and whether the log data file exists in the local shared storage EmptyDir;

[0046] Determine whether the reading status of the log data file is abnormal, where the reading status includes whether the log proxy container Sidercar can normally read the log data file;

[0047] Determine whether the file size status is abnormal, where the file size status includes whether the size of the log data file exceeds a preset threshold;

[0048] Use the timestamp of the log data file to determine whether the update status is abnormal, where the update status includes whether new log data is written into the log data file;

[0049] When the health check probe detects that at least one of the storage status, the reading status, the file size status, and the update status is abnormal, generate an alarm information prompt.

[0050] In a third aspect, an embodiment of the present invention provides an application log management system based on a cloud native scenario, including: at least one memory and at least one processor;

[0051] The at least one memory is used to store machine-readable programs;

[0052] The at least one processor is used to call the machine-readable program and execute any of the methods in the first aspect.

[0053] A computer-readable medium, characterized in that computer instructions are stored on the computer-readable medium, and when the computer instructions are executed by a processor, the processor executes any of the methods in the first aspect.

[0054] An embodiment of the present invention provides an application log management method and system based on a cloud-native scenario. First, the method sets the log classification criteria based on the requirements of different application programs deployed in the Kubernetes cluster, and then configures the output path based on the classification criteria to record different types of logs into their respective corresponding files, which is convenient for subsequent management and analysis of application logs and eliminates the need to search through a large number of mixed logs. To achieve better performance, a local shared storage volume EmptyDir mounted on the temporary directory of each application Pod is used to share the log directory between the main application container Application Container and the log proxy container Sidecar. The Logging Agent is run through the log proxy container Sidercar to monitor the log files in the local shared storage volume and send them to the remote storage for retrieval. Compared with the prior art, the present invention can collect and manage the logs of specified applications running on the Kubernetes cluster in a cluster-level-logging manner, with the advantages of simple deployment, high flexibility, no intrusion into the application itself, and low transformation cost. Compared with the node-level Logging Agent, this method supports more refined control for multi-tenant scenarios based on the log proxy container Sidecar, without the need for complex middleware and configuration, thereby improving the management efficiency of application logs. Brief Description of the Drawings

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0056] Figure 1 It is a flowchart of an application log management method based on a cloud-native scenario provided by an embodiment of the present invention;

[0057] Figure 2 It is a flowchart of another application log management method based on a cloud-native scenario provided by an embodiment of the present invention;

[0058] Figure 3 It is a schematic diagram of an application log management system based on a cloud-native scenario provided by an embodiment of the present invention;

[0059] Figure 4 It is a schematic diagram of another application log management system based on a cloud-native scenario provided by an embodiment of the present invention;

[0060] Figure 5It is a schematic diagram of another application log management system based on cloud native scenarios provided by an embodiment of the present invention. Detailed implementation manners

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0062] As Figure 1 shown, an embodiment of the present invention provides an application log management method based on cloud native scenarios. The method may include the following steps:

[0063] Step 101: Determine a log classification standard according to the requirements of the application programs deployed in the Kubernetes cluster. Each application program corresponds to at least one configured application Pod, and each application Pod includes a log proxy container Sidercar and at least one main application container Application Container.

[0064] Step 102: Configure the output paths of different types of log data according to the log classification standard. The output path is the local shared storage EmptyDir corresponding to the temporary directory mounted on each application Pod.

[0065] Step 103: Based on the output path, store the log data generated by the operation of each main application container Application Container in the local shared storage EmptyDir. The log data includes at least one of error log data ErrLogData, warning log data WarnLog Data, and information log data InfoLog Data.

[0066] Step 104: Run Logging Agent using the log proxy container Sidercar, and read the log data in the local shared storage EmptyDir based on the configuration mapping of the log proxy container Sidercar. The configuration mapping is ConfigMap Logging Agent.

[0067] Step 105: Use fluent-bit of the log proxy container Sidercar to send the read log data to the remote storage Remote Elastic Search to retrieve the target log data.

[0068] In an embodiment of the present invention, an application log management method based on a cloud-native scenario is provided. The method first sets the log classification criteria based on the requirements of different application programs deployed in the Kubernetes cluster, and then configures the output path based on the classification criteria, recording different types of logs into their respective corresponding files, which is convenient for subsequent management and analysis of application logs and does not require searching in a large number of mixed logs. To have better performance, a local shared storage volume EmptyDir mounted on the temporary directory of each application Pod is used to share the log directory between the main application container Application Container and the log proxy container Sidecar. The Logging Agent is run through the log proxy container Sidercar to monitor the log files in the local shared storage volume and send them to a remote storage for retrieval. Compared with the prior art, the present invention can collect and manage the logs of specified applications running on the Kubernetes cluster in the manner of cluster-level-logging, and has the advantages of simple deployment, high flexibility, no invasiveness to the application itself, and low transformation cost. Compared with the node-level Logging Agent, this method supports more refined control for multi-tenant scenarios based on the log proxy container Sidecar, without complex middleware and configuration, thus improving the management efficiency of application logs.

[0069] In an embodiment of the present invention, to determine the log classification criteria, the application program requirements in the above embodiment include different running states and event types of the application program.

[0070] In an embodiment of the present invention, through the preset classification criteria, different types of logs can be respectively recorded into their respective corresponding files according to the classification criteria, which is convenient for subsequent management and analysis of application logs, so there is no need to search in a large number of mixed logs, and the management efficiency of application logs can be further improved.

[0071] In an embodiment of the present invention, to achieve resource sharing between the log proxy container Sidercar and the main application container Application Container, after the above A3 and before the A4 in the above embodiment, the following is further included:

[0072] Create a configuration map, where the configuration map is the reading configuration information of the LoggingAgent in the log proxy container Sidercar, and the reading configuration information includes at least one of the file path of the log data, the format of the log data, and the reading frequency;

[0073] The A4 includes:

[0074] Run the Logging Agent using the log proxy container Sidercar and mount the configuration map into the log proxy container Sidercar;

[0075] Based on the Logging Agent and the configuration map, write the log data to the local shared storage EmptyDir and the standard output;

[0076] Use the tail command to redirect the log data file and execute the kubectl logs command to read the log data in the local shared storage EmptyDir.

[0077] In an embodiment of the present invention, when the log proxy container Sidecar is started, it is necessary to configure the Logging Agent to specify information such as the path of the log file to be monitored, the processing rules for log data, and the target address of the remote storage. The Logging Agent continuously monitors the log files in the shared storage volume. Once new log data is generated, it processes the log according to the configured rules and then sends the processed log data to the remote storage. In this way, the log files of the application can be efficiently sent to the remote storage for subsequent analysis and management. The log proxy container Sidecar writes the log to both the local shared storage EmptyDir and the standard output. By simply modifying the container startup command and redirecting the log data file using the tail command, the kubectl logs function can be ensured to be normal, facilitating users to use the kubectl logs command to view the application logs without explicitly specifying the container name with the -c parameter.

[0078] In order to periodically check the status of the log file to ensure the stability and reliability of log collection, in an embodiment of the present invention, after the above-mentioned A5, it further includes:

[0079] Configure a health check probe in each application Pod, where the health check pointer is used to determine the status of the log data file;

[0080] Based on the health check probe, determine whether the storage status of the log data file is abnormal, where the storage status includes whether the log data file is generated according to a preset time and whether the log data file exists in the local shared storage EmptyDir;

[0081] Determine whether the reading status of the log data file is abnormal, where the reading status includes whether the log proxy container Sidercar can normally read the log data file;

[0082] Determine whether the file size status is abnormal, where the file size status includes whether the size of the log data file exceeds a preset threshold;

[0083] Use the timestamp of the log data file to determine whether the update status is abnormal, where the update status includes whether there is new log data written in the log data file;

[0084] When the health check probe detects that at least one of the storage status, the read status, the file size status, and the update status is abnormal, generate an alarm information prompt.

[0085] In the embodiments of the present invention, a health check probe can be deployed in each application Pod to regularly check the status of the log file to ensure the stability and reliability of log collection. For example, it can be checked and determined whether the storage status, the read status, the file size status, and the update status of the log data file are in abnormal states, and an alarm information prompt is generated when there is an abnormal state so that the operation and maintenance personnel can timely maintain the system.

[0086] Such as Figure 2 shown, in order to more clearly illustrate the technical solution and advantages of the present invention, the embodiments of the present invention provide a method for managing application logs based on a cloud-native scenario, which will be described in detail below. Specifically, it can include the following steps:

[0087] Step 201: Determine the log classification standard according to the different running states and event types of the application programs deployed in the Kubernetes cluster. Each application program corresponds to at least one configured application Pod, and each application Pod includes a log proxy container Sidercar and at least one main application container Application Container;

[0088] Specifically, Kubernetes, as a leading container orchestration platform, provides flexible management capabilities. The processing system for application Pods and their container logs in the Kubernetes cluster is completely decoupled from the life cycles of containers, application Pods, and Nodes, ensuring that application logs can still be normally obtained whether a container is abnormal, an application Pod is deleted or evicted, or even a cluster node fails.

[0089] Step 202: Configure the output paths of different categories of log data according to the log classification standard, where the output path is the local shared storage EmptyDir corresponding to the temporary directory mounted on each application Pod;

[0090] Specifically, to achieve better performance, a memory-type temporary directory can be adopted, which will consume the memory resource quota of the application Pod. The lifecycle of this temporary volume is the same as that of the application Pod, and when the application Pod is deleted, this temporary volume will also be cleared.

[0091] For example, the classification criteria can be Info / Warn / Error, etc. The app container is the actual application container, and in this container, the application logs of the specified level are output to the app.log file on the temporary volume mounted on the application Pod.

[0092] Step 203: Based on the output path, store the log data generated by each main application container Application Container in the local shared storage EmptyDir, where the log data includes at least one of error log data ErrLogData, warning log data WarnLog Data, and information log data InfoLog Data;

[0093] Step 204: Create a config map, where the config map is the reading configuration information of LoggingAgent in the log proxy container Sidercar, and the reading configuration information includes at least one of the file path of the log data, the format of the log data, and the reading frequency;

[0094] Step 205: Run Logging Agent using the log proxy container Sidercar and mount the config map into the log proxy container Sidercar;

[0095] Step 206: Based on Logging Agent and the config map, write the log data to the local shared storage EmptyDir and the standard output;

[0096] Step 207: Use the tail command to redirect the log data file and execute the kubectl logs command to read the log data in the local shared storage EmptyDir.

[0097] Step 208: Use fluent-bit of the log proxy container Sidercar to send the read log data to the remote storage Remote Elastic Search to retrieve the target log data;

[0098] Specifically, fluent-bit of the container Sidercar sends the application log files to a remote storage, and the corresponding configuration file fluent-bit.conf is mounted into the application Pod in the form of a ConfigMap for the container to use. Since the container Sidecar shares the Volume with the main application container, the additional storage overhead of fluent-bit in the container Sidecar is very low. At the same time, under the condition of ensuring high performance, Fluent Bit occupies very few resources (for example, CPU occupancy is 0.02 cores, memory occupancy is 8 - 30MB, and network bandwidth is 50KB / s); running fluent-bit in the same application Pod as the main application container through a container Sidecar has higher flexibility than deploying the Logging Agent on the Kubernetes cluster nodes and better supports multi-tenant scenarios.

[0099] Step 209: Configure a health check probe in each application Pod, where the health check pointer is used to determine the status of the log data file;

[0100] Specifically, in the configuration file of the application Pod (usually in YAML format), livenessProbe (liveness probe) and readinessProbe (readiness probe) can be added to check the status of the log file. The health check probe can periodically check the status of the log file by executing a Shell script. For example, the configuration can be set to start the health check 5 seconds after the container starts and perform a health check every 10 seconds.

[0101] Step 210: Based on the health check probe, determine whether the storage status of the log data file is abnormal, where the storage status includes whether the log data file is generated according to a preset time and whether the log data file exists in the local shared storage EmptyDir;

[0102] Step 211: Determine whether the reading status of the log data file is abnormal, where the reading status includes whether the log proxy container Sidercar can normally read the log data file;

[0103] Step 212: Determine whether the file size status is abnormal, where the file size status includes whether the size of the log data file exceeds a preset threshold;

[0104] Step 213: Use the timestamp of the log data file to determine whether the update status is abnormal, where the update status includes whether new log data is written into the log data file;

[0105] Step 214: When the health check probe detects an abnormality in at least one of the storage status, read status, file size status, and update status, generate an alarm message for prompt.

[0106] As Figure 3 shown, an embodiment of the present invention provides an application log management system based on a cloud-native scenario. The system includes:

[0107] A determination module 301, configured to determine a log classification standard according to the requirements of the application program deployed in the Kubernetes cluster. Each application program corresponds to at least one application Pod configured, and each of the application Pods includes a log proxy container Sidercar and at least one main application container Application Container;

[0108] A configuration module 302, configured to configure the output paths of different types of log data according to the log classification standard determined by the determination module 301. The output path is the local shared storage EmptyDir corresponding to the temporary directory mounted on each of the application Pods;

[0109] A local storage module 303, configured to store the log data generated by the operation of each of the main application containers Application Container in the local shared storage EmptyDir based on the output path configured by the configuration module 302. The log data includes at least one of error log data ErrLog Data, warning log data WarnLogData, and information log data InfoLog Data;

[0110] A reading module 304, configured to run a Logging Agent by using the log proxy container Sidercar, and read the log data in the local shared storage EmptyDir corresponding to the local storage module 303 based on the configuration mapping of the log proxy container Sidercar. The configuration mapping is ConfigMap Logging Agent;

[0111] A remote storage module 305, configured to send the log data read by the reading module 304 to a remote storage Remote Elastic Search by using the fluent-bit of the log proxy container Sidercar to retrieve target log data.

[0112] As Figure 3As shown, in the embodiment of the present invention, the determining module 301 is further configured to determine a log classification criterion according to different running states and event types of the applications deployed in the Kubernetes cluster.

[0113] Based on Figure 3 An application log management system based on the cloud native scenario as shown in Figure 4 As shown, in the embodiment of the present invention, after the local storage module 303 and before the reading module 304, it further includes:

[0114] A creating module 306, configured to create a configuration map, where the configuration map is the reading configuration information of the Logging Agent in the log proxy container Sidercar, and the reading configuration information includes at least one of the file path of the log data, the format of the log data, and the reading frequency;

[0115] The reading module 304 is further configured to execute:

[0116] Run the Logging Agent using the log proxy container Sidercar, and mount the configuration map into the log proxy container Sidercar;

[0117] Based on the Logging Agent and the configuration map, write the log data into the local shared storage EmptyDir and the standard output;

[0118] Use the tail command to redirect the log data file, and execute the kubectl logs command to read the log data in the local shared storage EmptyDir.

[0119] Based on Figure 4 A log management system based on the cloud native scenario as shown in Figure 5 As shown, after the remote storage module 305, it further includes: an inspection module 307;

[0120] The inspection module 307 is configured to execute:

[0121] Configure a health check probe in each application Pod, where the health check pointer is used to determine the status of the log data file;

[0122] Based on the health check probe, determine whether the storage status of the log data file is abnormal, where the storage status includes whether the log data file is generated according to a preset time and whether the log data file exists in the local shared storage EmptyDir;

[0123] Determine whether the reading status of the log data file is abnormal, where the reading status includes whether the log proxy container Sidercar can normally read the log data file;

[0124] Determine whether the file size status is abnormal, where the file size status includes whether the size of the log data file exceeds a preset threshold;

[0125] Use the timestamp of the log data file to determine whether the update status is abnormal, where the update status includes whether new log data is written into the log data file;

[0126] When the health check probe detects that at least one of the storage status, the reading status, the file size status, and the update status is abnormal, generate an alarm information prompt.

[0127] It can be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on an application log management system based on a cloud native scenario. In other embodiments of the present invention, an application log management system in a cloud native scenario may include more or fewer components than those illustrated, or combine certain components, or split certain components, or arrange different components. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0128] Regarding the information interaction, execution process, etc. between the various units in the above device, since it is based on the same concept as the method embodiments of the present invention, the specific content can be referred to the description in the method embodiments of the present invention and will not be elaborated here.

[0129] The embodiments of the present invention also provide an application log management system based on a cloud native scenario, including: at least one memory and at least one processor;

[0130] At least one memory for storing machine-readable programs;

[0131] At least one processor for calling the machine-readable program and executing an application log management method in any embodiment of the present invention.

[0132] The embodiments of the present invention also provide a computer-readable medium, on which computer instructions are stored. When the computer instructions are executed by a processor, the processor executes an application log management method in any embodiment of the present invention.

[0133] Specifically, a system or device equipped with a storage medium can be provided. On this storage medium, software program codes for implementing the functions of any one of the above embodiments are stored, and the computer (or CPU or MPU) of the system or device is made to read and execute the program codes stored in the storage medium.

[0134] In this case, the program codes read from the storage medium itself can implement the functions of any one of the above embodiments. Therefore, the program codes and the storage medium storing the program codes constitute a part of the present invention.

[0135] Examples of the storage medium for providing the program codes include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program codes can be downloaded from a server computer via a communication network.

[0136] Furthermore, it should be clear that not only can the functions of any one of the above embodiments be realized by executing the program codes read by the computer, but also by making the operating system or the like operating on the computer complete part or all of the actual operations based on the instructions of the program codes.

[0137] In addition, it can be understood that the program codes read from the storage medium are written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer. Subsequently, based on the instructions of the program codes, the CPU or the like installed on the expansion board or the expansion unit executes part and all of the actual operations, thereby realizing the functions of any one of the above embodiments.

[0138] Each embodiment of the present invention has at least the following beneficial effects:

[0139] 1. In an embodiment of the present invention, an application log management method based on a cloud-native scenario is provided. First, the log classification criteria are set according to the requirements of different application programs deployed in the Kubernetes cluster, and then the output path is configured based on the classification criteria, and different types of logs are respectively recorded in their corresponding files, which is convenient for subsequent management and analysis of application logs and does not require searching in a large number of mixed logs. At the same time, the local shared storage volume EmptyDir mounted on the temporary directory of each application Pod is used to share the log directory between the main application container Application Container and the log proxy container Sidecar. The LoggingAgent is run through the log proxy container Sidercar to monitor the log files in the local shared storage volume and send them to the remote storage for easy retrieval. Compared with the prior art, the present invention can collect and manage the logs of specified applications running on the Kubernetes cluster in the way of cluster-level-logging, and has the advantages of simple deployment, high flexibility, no invasiveness to the application itself, and low transformation cost. Compared with the node-level Logging Agent, this method supports more fine-grained control for multi-tenant scenarios based on the log proxy container Sidecar, without complex middleware and configuration, thus improving the management efficiency of application logs;

[0140] 2. In an embodiment of the present invention, through the preset classification criteria, different types of logs can be respectively recorded in their corresponding files according to the classification criteria, which is convenient for subsequent management and analysis of application logs and does not require searching in a large number of mixed logs, thereby further improving the management efficiency of application logs;

[0141] 3. In an embodiment of the present invention, when the log proxy container Sidecar is started, it is necessary to configure the Logging Agent to specify information such as the path of the log file to be monitored, the processing rules of log data, and the target address of the remote storage. The Logging Agent continuously monitors the log files in the shared storage volume. Once new log data is generated, it processes the logs according to the configured rules and then sends the processed log data to the remote storage. In this way, the log files of the application can be efficiently sent to the remote storage for subsequent analysis and management. The application container writes the logs to both the local shared storage EmptyDir and the standard output at the same time. By simply modifying the container startup command and redirecting the log data file through the tail command, the kubectl logs function can be ensured to be normal, which is convenient for users to view the application logs using the kubectl logs command without explicitly specifying the container name with the -c parameter.

[0142] It should be noted that not all steps and modules in the above-mentioned processes and system structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structures described in the above embodiments can be physical structures or logical structures, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices can be jointly implemented.

[0143] In the above embodiments, the hardware units can be implemented mechanically or electrically. For example, a hardware unit can include permanent dedicated circuits or logic (such as dedicated processors, FPGAs or ASICs) to perform corresponding operations. The hardware unit can also include programmable logic or circuits (such as general-purpose processors or other programmable processors), which can be temporarily set by software to perform corresponding operations. The specific implementation method (mechanical method, or dedicated permanent circuit, or temporarily set circuit) can be determined based on cost and time considerations.

[0144] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the scope of protection of the present invention.

Claims

1. An application log management method based on cloud native scenarios, characterized in that The method includes: A1: Determine the log classification criteria according to the application requirements deployed in the Kubernetes cluster. Each application corresponds to at least one configured application Pod, and each of the application Pods includes a log proxy container Sidercar and at least one main application container Application Container; A2: Configure the output paths for different categories of log data according to the log classification criteria. The output path is the local shared storage EmptyDir corresponding to the temporary directory mounted on each of the application Pods; A3: Based on the output path, store the log data generated by the operation of each main application container Application Container in the local shared storage EmptyDir. The log data includes at least one of error log data ErrLog Data, warning log data WarnLog Data, and information log data InfoLog Data; A4: Run Logging Agent using the log proxy container Sidercar. Based on the configuration mapping of the log proxy container Sidercar, read the log data in the local shared storage EmptyDir. The configuration mapping is ConfigMap Logging Agent; A5: Use the fluent-bit of the log proxy container Sidercar to send the read log data to the remote storage Remote Elastic Search to retrieve the target log data.

2. The method according to claim 1, wherein: The application requirements include different running states and event types of the application.

3. The method according to claim 1, wherein: After A3 and before A4, it further includes: Create a configuration mapping, where the configuration mapping is the reading configuration information of Logging Agent in the log proxy container Sidercar. The reading configuration information includes at least one of the file path of the log data, the format of the log data, and the reading frequency; A4 includes: Run Logging Agent using the log proxy container Sidercar and mount the configuration mapping into the log proxy container Sidercar; Based on Logging Agent and the configuration mapping, write the log data to the local shared storage EmptyDir and the standard output; Use the tail command to redirect the log data file and execute the kubectl logs command to read the log data in the local shared storage EmptyDir.

4. The method according to any one of claims 1-3, wherein: After A5, it further includes: Configure a health check probe in each of the application Pods, where the health check pointer is used to determine the status of the log data file; Based on the health check probe, determine whether the storage status of the log data file is abnormal, where the storage status includes whether the log data file is generated according to a preset time and whether the log data file exists in the local shared storage EmptyDir; Determine whether the reading status of the log data file is abnormal, where the reading status includes whether the log proxy container Sidercar can normally read the log data file; Determine whether the file size status is abnormal, where the file size status includes whether the size of the log data file exceeds a preset threshold; Use the timestamp of the log data file to determine whether the update status is abnormal, where the update status includes whether new log data is written into the log data file; When the health check probe detects that at least one of the storage status, the reading status, the file size status, and the update status is abnormal, generate an alarm information prompt.

5. An application log management system based on cloud native scenarios, characterized in that, The system includes: A determination module, configured to determine a log classification standard according to the application requirements deployed in the Kubernetes cluster, where each application corresponds to at least one configured application Pod, and each of the application Pods includes a log proxy container Sidercar and at least one main application container Application Container; A configuration module, configured to configure the output paths of different types of log data according to the log classification standard determined by the determination module, where the output path is the local shared storage EmptyDir corresponding to the temporary directory mounted on each of the application Pods; A local storage module, configured to store the log data generated by the operation of each main application container Application Container in the local shared storage EmptyDir based on the output path configured by the configuration module, where the log data includes at least one of error log data ErrLog Data, warning log data WarnLog Data, and information log data InfoLog Data; A reading module, configured to run a Logging Agent using the log proxy container Sidercar, and read the log data in the local shared storage EmptyDir corresponding to the local storage module based on the configuration mapping of the log proxy container Sidercar, where the configuration mapping is ConfigMap Logging Agent; A remote storage module, configured to send the log data read by the reading module to a remote storage Remote Elastic Search using the fluent-bit of the log proxy container Sidercar to retrieve target log data.

6. The system according to claim 5, wherein the determining module is further configured to determine a log classification criterion according to different running states and event types of the applications deployed in the Kubernetes cluster.

7. The system according to claim 5, wherein after the local storage module and before the reading module, it further includes: a creating module, configured to create a configuration map, where the configuration map is the reading configuration information of the Logging Agent in the log proxy container Sidercar, and the reading configuration information includes at least one of the file path of the log data, the format of the log data, and the reading frequency; the reading module is further configured to perform: running the Logging Agent by using the log proxy container Sidercar and mounting the configuration map into the log proxy container Sidercar; writing the log data into the local shared storage EmptyDir and the standard output based on the Logging Agent and the configuration map; redirecting the log data file by using the tail command and executing the kubectl logs command to read the log data in the local shared storage EmptyDir.

8. The system according to any one of claims 5-7, wherein after the remote storage module, it further includes: an inspection module; the inspection module is configured to perform: configuring a health check probe in each of the application Pods, where the health check pointer is used to determine the status of the log data file; determining whether the storage status of the log data file is abnormal based on the health check probe, where the storage status includes whether the log data file is generated according to a preset time and whether the log data file exists in the local shared storage EmptyDir; determining whether the reading status of the log data file is abnormal, where the reading status includes whether the log proxy container Sidercar can normally read the log data file; determining whether the file size status is abnormal, where the file size status includes whether the size of the log data file exceeds a preset threshold; determining whether the update status is abnormal by using the timestamp of the log data file, where the update status includes whether new log data is written into the log data file; generating an alarm information prompt when the health check probe detects that at least one of the storage status, the reading status, the file size status, and the update status is abnormal.

9. An application log management system based on cloud native scenarios, characterized in that, It includes: at least one memory and at least one processor; the at least one memory is used to store a machine-readable program; the at least one processor is used to call the machine-readable program and execute the method according to any one of claims 1 to 4.

10. A computer-readable medium, characterized in that, A computer instruction is stored on the computer-readable medium, and when the computer instruction is executed by a processor, the processor executes the method according to any one of claims 1 to 4.

Images