Container log processing method and device based on kubernetes local storage volume

By modifying the host log directory permissions in the container and persisting the logs to the kubernetes cluster, the complex problem of log directory management in the existing technology is solved, and the automated management of log directory and efficient collection and storage are realized.

CN118939378APending Publication Date: 2024-11-12CHINA UNITED NETWORK COMM GRP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411064773.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

During the container log processing of existing kubernetes local storage volumes, the creation and permission modification of the service log directory need to be done manually on the host, increasing the operation and maintenance complexity and workload.

Method used

Modify the second log directory permission of the host through the container to the ordinary user permission, and write the logs of the application service to the first log directory of the container, and then persist to the second log directory of the host, and use kubernetes to maintain each host in a unified manner. The service log directory on the service log directory to realize operation and maintenance automation.

Benefits of technology

It reduces the complexity and workload of operation and maintenance, realizes the automated management of service log directories, and improves the efficiency of log collection and storage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939378A_ABST
    Figure CN118939378A_ABST
Patent Text Reader

Abstract

The invention discloses a kubernetes local storage volume-based container log processing method and device, and the method comprises the steps: writing a log in a container with an application service into a first log directory of the container, and persisting the log in the first log directory to a second log directory of a host machine; the permission of the second log catalogue is modified into the normal user permission through the container; collecting a log of the application service of the common user right from a second log directory of the host machine through a log collection service configured on the host machine; through the log collection service, the collected log of the application service with the common user permission is sent to a log index service corresponding to the kubernetes cluster, and the log index service corresponds to each host machine and is used for querying the log of the application service with the common user permission. According to the method and the device, the service log directories on the host machines can be uniformly maintained by the kubernetes, so that the operation and maintenance automation of the service log directories on the host machines is realized, and the operation and maintenance complexity and workload are reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of network communication technology, and in particular to a container log processing method and device based on a Kubernetes local storage volume. Background Art

[0002] When an application service (referred to as "application service") runs in a container environment, Kubernetes is usually used to schedule and deploy a service pod containing a single service container. Multiple pods of an application service may be scheduled to multiple hosts in a Kubernetes cluster.

[0003] In order to collect, filter, store, and query application service logs, the hostPath local storage volume provided by Kubernetes is usually used to mount the log directory of the host machine of the Kubernetes cluster to the log directory of the application service in the container, and then send it to the log processing service (such as logstash) through the log collection service (such as filebeat) deployed on the host machine. After filtering, it is stored in the log indexing service (such as elasticsearch), and finally the log of the application service is viewed through the log visualization tool (such as kibana).

[0004] However, in the existing Kubernetes local storage volume container log processing process, the creation of the service log directory and the modification of the service log directory permissions are manually performed on the host before the container runs. In this way, the service log directory needs to be manually maintained on each host, which increases the complexity and workload of operation and maintenance. Summary of the invention

[0005] The technical problem to be solved by the present application is to provide a container log processing method and device based on Kubernetes local storage volume in response to the above-mentioned deficiencies in the prior art. Using this method, Kubernetes uniformly maintains the service log directory on each host machine, thereby realizing the operation and maintenance automation of the service log directory, thereby reducing the complexity and workload of operation and maintenance.

[0006] In a first aspect, the present application provides a container log processing method based on a kubernetes local storage volume, the method comprising:

[0007] S1. Write the logs in the container with the application service to the first log directory of the container, and persist the logs in the first log directory to the second log directory of the host; wherein the permissions of the second log directory are modified to ordinary user permissions by the container;

[0008] S2. Collect logs of application services with ordinary user permissions from the second log directory of the host through the log collection service configured on the host;

[0009] S3. The collected logs of the application service with common user rights are sent to the log indexing service of the kubernetes cluster through the log collection service. The log indexing service corresponds to each host machine and is used to query the logs of the application service with common user rights.

[0010] In some embodiments, step S1 specifically includes:

[0011] S11. Obtain a second log directory on the host; wherein the permissions of the second log directory are special user permissions;

[0012] S12. Mount the second log directory into the container, so as to modify the permissions of the second log directory of the host machine to ordinary user permissions through the container;

[0013] S13. Run an application service with ordinary user privileges in the container and write the logs in the container to the first log directory;

[0014] S14. Persist the logs in the first log directory to the second log directory with modified permissions.

[0015] In some embodiments, step S12 specifically includes:

[0016] Mounting the second log directory to the initial container of the application service through the local storage volume;

[0017] By the initial container, the permission of the second log directory is changed from special user permission to ordinary user permission.

[0018] In some embodiments, step S13 specifically includes:

[0019] Destroying the initial container, and starting an application service running with ordinary user permissions in a service container; the service container and the initial container have the same first log directory;

[0020] Use the local storage volume to mount the second log directory of the host machine with modified permissions to the first log directory of the service container;

[0021] Write the service container's logs to the first log directory.

[0022] In some embodiments, step S11 specifically includes:

[0023] Determine whether there is a second log directory corresponding to the application service on the host machine;

[0024] If yes, directly obtain the second log directory;

[0025] If not, a second log directory with special user permissions is created on the host machine.

[0026] In some embodiments, the system environment stage of the application service includes a development environment and a test environment, and the second log directory with special user permissions is created on the host machine, specifically including:

[0027] According to the system environment stage of the application service, a second log directory including an environment identifier and an application service name is created on the host machine; the environment identifier includes a development identifier and a test identifier.

[0028] In some embodiments, the host machine is also provided with a log collection service, and step S2 specifically includes:

[0029] According to the environment identifier in the second log directory of the host machine, third log directories of the log collection service corresponding to different environment identifiers are respectively configured on the host machine;

[0030] According to the environment identifier in the third log directory of the log collection service, the log of the application service with common user authority is collected from the second log directory of the corresponding host machine through the log collection service.

[0031] In some embodiments, step S3 specifically includes:

[0032] Sending a log processing request to the log filtering service through the log collection service; the log processing request carries the log of the application service with ordinary user rights and the third log directory;

[0033] Through the log filtering service and according to the environment identifier and application service name in the third log directory, the log of the application service is sent to the corresponding fourth log directory in the log indexing service, so as to query the log of the application service corresponding to the system environment stage where the application service is located through the log indexing service.

[0034] In some embodiments, before step S11, the method further includes:

[0035] Create deployment files for application services and determine the number of deployment files;

[0036] Schedule each deployment file to the corresponding number of hosts.

[0037] In a second aspect, the present application provides a container log processing device based on a kubernetes local storage volume, the device comprising:

[0038] A writing module, which is configured to write the logs in the container with the application service to the first log directory of the container, and persist the logs in the first log directory of the container to the second log directory on the host; wherein the permissions of the second log directory are modified to ordinary user permissions by the container;

[0039] A collection module, connected to the writing module, configured to collect logs of application services with ordinary user permissions from a second log directory of the host machine through a log collection service configured on the host machine;

[0040] The sending module is connected to the collection module and is configured to send the collected logs of the application service with ordinary user permissions to the log indexing service of the kubernetes cluster through the log collection service. The log indexing service corresponds to each host machine and is used to query the logs of the application service with ordinary user permissions.

[0041] The container log processing method and device based on the Kubernetes local storage volume provided in the present application modify the permissions of the second log directory of the host machine through the container, write the log of the application service into the first log directory of the container, and then persist it in the second log directory of the host machine. Then, after the log of the application service is collected and stored, the second log directory (service log directory) on each host machine can be uniformly maintained by Kubernetes, so as to realize the operation and maintenance automation of the service log directory on the host machine and reduce the complexity and workload of operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] Figure 1 A flow chart of a container log processing method based on a kubernetes local storage volume provided in an embodiment of the present application;

[0044] Figure 2 A flowchart of an application embodiment of a container log processing method based on a kubernetes local storage volume provided in an embodiment of the present application;

[0045] Figure 3 Another flowchart of an application embodiment of the container log processing method based on the Kubernetes local storage volume provided in the embodiment of the present application;

[0046] Figure 4 Another flowchart of an application embodiment of the container log processing method based on the Kubernetes local storage volume provided in the embodiment of the present application;

[0047] Figure 5 Another flowchart of an application embodiment of the container log processing method based on the Kubernetes local storage volume provided in the embodiment of the present application;

[0048] Figure 6 Another flowchart of an application embodiment of the container log processing method based on the Kubernetes local storage volume provided in the embodiment of the present application;

[0049] Figure 7 A schematic diagram of the structure of a container log processing device based on a kubernetes local storage volume provided in an embodiment of the present application;

[0050] Figure 8 Another structural schematic diagram of a container log processing device based on a Kubernetes local storage volume provided in an embodiment of the present application.

[0051] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the technical solution of the present application, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments and drawings described herein are only used to explain the present application, rather than to limit the present application.

[0053] It can be understood that, in the absence of conflict, the various embodiments in the present application and the various features in the embodiments can be combined with each other.

[0054] It can be understood that, for the convenience of description, the drawings of the present application only show the parts related to the present application, while the parts unrelated to the present application are not shown in the drawings.

[0055] It can be understood that each unit and module involved in the embodiments of the present application may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0056] It can be understood that the terms "first", "second", etc. in the embodiments of the present application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects.

[0057] It is understandable that, in the absence of conflict, the functions and steps marked in the flowcharts and block diagrams of the present application may occur in an order different from that marked in the drawings.

[0058] It is understood that the flowcharts and block diagrams of the present application illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present application. Among them, each box in the flowchart or block diagram may represent a unit, module, program segment, code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or by a combination of hardware and computer instructions.

[0059] It can be understood that the units and modules involved in the embodiments of the present application can be implemented by software or hardware, for example, the units and modules can be located in a processor.

[0060] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0061] In the related art, during the processing of the container log of the kubernetes local storage volume, the hostPath local storage volume provided by kubernetes is used to mount the log directory (second log directory) created by the host of the kubernetes cluster to the log directory (first log directory) of the application service in the container, so as to persist the log content in the log directory of the application service in the container to the second log directory of the host. However, in the prior art, the second log directory of the host is manually created on the host before the container runs, and the owner and group of the created second log directory are both root, and ordinary non-privileged users have no permission to write. If the application service of an ordinary non-privileged user uses the second log directory, it is also necessary to manually modify the permissions of the second log directory on each host so that the service running user has read and write permissions. This increases the complexity and workload of operation and maintenance.

[0062] The inventor has found through research that during the operation of the container, the container modifies the permissions of the second log directory of the host machine, completes the initialization before the application service with ordinary user permissions runs in the container and persists the log of the application service to the second log directory of the host machine, and can realize the operation and maintenance automation of the service log directory by uniformly maintaining the service log directory on each host machine through kubernetes, thereby reducing the complexity and workload of operation and maintenance. Then, through the collection log service of the host machine, the logs of the application service container persisted in the second log directory of the host machine are collected and sent to the log index service of the kubernetes cluster, so that the logs of the application service with ordinary user permissions can be queried.

[0063] Based on this, in order to solve the problems of the prior art, the present invention proposes a container log processing method based on Kubernetes local storage volume, in which Kubernetes uniformly maintains the service log directory on each host machine, realizes the operation and maintenance automation of the service log directory, and thus reduces the complexity and workload of operation and maintenance.

[0064] Embodiment 1:

[0065] like Figure 1 As shown, the present application provides a container log processing method based on kubernetes local storage volume, the method comprising steps S1 to S3:

[0066] Step S1. Write the logs in the container with the application service into the first log directory of the container, and persist the logs in the first log directory to the second log directory of the host; wherein the permissions of the second log directory are modified to ordinary user permissions through the container.

[0067] Here, the kubernetes cluster integrates a certain number of hosts, so that kubernetes can schedule the hosts to run application services according to the resource load of each host. Specifically, when kubernetes runs multiple application services, it will deploy a preset number of pods for each application service, where each pod contains a main container and auxiliary containers. Then, according to the resource load of the host, kubernetes can schedule each pod to the corresponding number of hosts, so as to run the pod on the host and run the corresponding application service in the pod's container.

[0068] It should also be noted that the number of hosts integrated into a kubernetes cluster can range from a small deployment of a few nodes (hosts) to a large cluster of thousands of nodes, which can be considered according to business needs. For example, a small development environment and test environment may have only 3-5 nodes, while an enterprise-level cluster in a production environment may contain hundreds to thousands of nodes.

[0069] The application service runs in the container, and the running process generates logs of the application service. The logs in the container can be written to the first log directory, for example, the first log directory can be " / home / app / logs".

[0070] The second log directory can contain the application service name. For example, if the application service name is "backend-portal", the second log directory of the host machine can be " / data / logs / backend-portal". It should be noted that after the logs in the first log directory are persisted to the second log directory of the host machine, the container can be rebuilt or deleted, and the logs of the application service can still be retained in the second log directory of the host machine.

[0071] The permissions of the second log directory are common user permissions, that is, non-privileged users. For example, by setting the owner and group of the second log directory to be iot, the permissions of the second log directory are defined as common user permissions.

[0072] During implementation, the auxiliary container (or initial container) can be configured to enable the container to modify the permissions of the second log directory of the host machine; in addition, the preprocessing of modifying the permissions of the second log directory can be configured in the container running the application service to enable the container to modify the permissions of the second log directory of the host machine. In this way, the logs of the application service with ordinary user permissions can be persisted to the second log directory of the host machine. After the container is rebuilt or deleted, the logs of the application service with ordinary user permissions are still retained in the second log directory of the host machine.

[0073] It should also be noted that the permission of the second log directory is the permission of ordinary users, which is applicable not only to the persistence of logs of application services with ordinary user permissions, but also to the persistence of logs of application services with special user permissions.

[0074] Step S2: Collect logs of application services with common user privileges from a second log directory of the host machine through a log collection service configured on the host machine.

[0075] Here, a log collection service, such as filebeat, is installed on each host. The log collection directory is configured according to the application service name of the second log directory. For example, if the second log directory is " / data / logs / backend-portal", the directory in the log index service can be " / data / logs / backend-portal / application.log".

[0076] During implementation, a log collection service is installed on each host, and the logs of each application service with ordinary user permissions can be collected from the second log directory of each host in the kubernetes cluster. In this way, the logs of application services with ordinary user permissions running on all hosts in the kubernetes cluster can be collected.

[0077] Step S3. Through the log collection service, the collected logs of the application service with ordinary user permissions are sent to the log indexing service of the kubernetes cluster. The log indexing service corresponds to each host machine and is used to query the logs of the application service with ordinary user permissions.

[0078] Here, the log collection service sends the collected logs of the application services with common user permissions and the directory (file path) of the application service logs to the log indexing service.

[0079] The log indexing service, such as elasticsearch, is used to store the logs of all application services running on each host of the kubernetes cluster. Specifically, the log indexing service can store the logs of application services in different directories (in the service index) according to the application service name. For example, if the application service name is "backend-portal", the directory in the log indexing service can be " / data / logs / backend-portal".

[0080] In some other implementations, by calling the API interface of a log visualization tool (such as kibana), all application service index names (such as test-backend-portal) on the log index service can be obtained. By calling the API interface of the log visualization tool, a corresponding index pattern (such as test-backend-portal) can be automatically created according to the obtained application service index name (such as test-backend-portal). Through the exploration module of the log visualization tool, the logs of the application service corresponding to the application service name can be quickly queried by filtering the index pattern.

[0081] In this embodiment, the permissions of the second log directory of the host machine are modified through the container, and the logs of the application service are written to the first log directory of the container and then persisted to the second log directory of the host machine. Then, after the logs of the application service are collected and stored, kubernetes can uniformly maintain the second log directory (service log directory) on each host machine, realize the operation and maintenance automation of the service log directory on the host machine, and reduce the complexity and workload of operation and maintenance.

[0082] In some embodiments, step S1 specifically includes steps S11 to S14:

[0083] Step S11. Obtain a second log directory on the host machine; wherein the permission of the second log directory is a special user permission.

[0084] Here, the permissions of the second log directory on the host machine obtained are special user permissions, such as a system administrator account with the highest level of permissions. For example, the second log directory obtained is defined as a special user permission by setting the owner and group of the second log directory to be root. It can be understood that the second log directory with special user permissions is a restriction on the permissions of the second log directory, which can improve the security of the second log directory of the host machine.

[0085] Step S12: Mount the second log directory into the container, so as to modify the permission of the second log directory of the host machine to the permission of a common user through the container.

[0086] Here, mounting the second log directory into the container may be performed by mounting the second log directory of the host into the container through the hostPath local storage volume, so that the container shares the data resources in the second log directory of the host.

[0087] During implementation, the second log directory whose owner and group are both root can be modified into a second log directory whose owner and group are both iot through the container, so as to implement the modification of the second log directory permission by the container.

[0088] Step S13: Run an application service with ordinary user privileges in the container, and write the logs in the container into a first log directory.

[0089] Here, the container runs an application service with ordinary user permissions, and generates application service logs in the container. The generated application service logs are written to the first log directory. For example, if the container is configured with the first log directory " / home / app / logs", the generated application service logs with ordinary user permissions are written to " / home / app / logs".

[0090] Step S14: Persist the logs in the first log directory to the second log directory with modified permissions.

[0091] Here, the second log directory with modified permissions supports the persistence of logs of application services with ordinary user permissions. It is understandable that if the permissions of the second log directory are special user permissions, for application services with ordinary user permissions, the logs of the application services will fail to be written to the second log directory.

[0092] In this embodiment, by obtaining the second log directory, mounting the second log directory, modifying the permissions of the second log directory by the container, and persisting the logs of the application service, Kubernetes can uniformly maintain the second log directory on each host machine and modify the permissions of the second log directory, thereby realizing the operation and maintenance automation of the service log directory, thereby reducing the complexity and workload of operation and maintenance.

[0093] In some embodiments, step S12 specifically includes steps S121 to S122:

[0094] Step S121: Mount the second log directory to the initial container of the application service through the local storage volume.

[0095] Here, the initial container may be configured with a first log directory, for example, " / home / app / logs". During implementation, the second log directory is mounted to the first log directory of the initial container of the application service through the local storage volume hostPath.

[0096] Step S122: The initial container modifies the permission of the second log directory from special user permission to common user permission.

[0097] Here, the permission of the second log directory is modified from special user permission to common user permission through the initial container. After the permission of the second log directory is modified in the initial container, it can be rebuilt or deleted.

[0098] In this embodiment, by configuring the initial container and modifying the permissions of the second log directory, the subsequent application service container can be started in a controllable and expected environment, thereby providing support for improving the stability and reliability of the application program.

[0099] In some embodiments, step S13 specifically includes steps S131 to S133:

[0100] Step S131: destroy the initial container, and start the application service running with ordinary user permissions in the service container; the service container and the initial container have the same first log directory;

[0101] Step S132. Mount the second log directory of the host machine with modified permissions to the first log directory of the service container through the local storage volume;

[0102] Step S133: Write the log of the service container into the first log directory.

[0103] The service container and the initial container have the same first log directory. For example, the initial container may be configured with a first log directory " / home / app / logs", and the service container also has a first log directory " / home / app / logs".

[0104] In this embodiment, by running the application service with ordinary user permissions, mounting the second log directory, and writing the log of the application service, the application service with ordinary user permissions can be run in the service container, so that the service container can be started in a controllable and expected environment, which can improve the stability and reliability of the application.

[0105] In some embodiments, step S11 specifically includes steps S111 to S113:

[0106] Step S111: Determine whether there is a second log directory corresponding to the application service on the host machine.

[0107] Here, the second log directory contains the application service name of each application service. During implementation, it is determined whether there is a second log directory corresponding to the application service based on the application service name in the second log directory. For example, to identify each segment in the second log directory (for example, " / data / logs / backend-portal"), the " / " in the second log directory can be used as the basis for division, and the information of each segment ("data", "logs", "backend-portal") and the information at the location of the application service name ("backend-portal") can be determined respectively.

[0108] Step S112. If yes, directly obtain the second log directory;

[0109] Step S113: If not, create a second log directory with special user permissions on the host machine.

[0110] During implementation, the service running in the initial container can create the second log directory on the host, or other containers deployed by Kubernetes can create the second log directory. In addition, other methods can be used, such as manually creating a second log directory with special user permissions on the host.

[0111] A second log directory with special user permissions is created on the host machine, which can be applied to application services with special user permissions or application services with ordinary user permissions. Any application service can create a second log directory with special user permissions on the host machine to improve the security of the host machine.

[0112] In this embodiment, two methods of obtaining the second log directory are determined by judging whether the second log directory corresponding to the application service exists. Directly obtaining the second log directory can simplify the log processing steps, and creating a second log directory with special user permissions can improve the security of the host machine.

[0113] In some embodiments, the system environment stage of the application service includes a development environment and a test environment, and the second log directory with special user permissions is created on the host machine, specifically including:

[0114] According to the system environment stage of the application service, a second log directory including an environment identifier and an application service name is created on the host machine; the environment identifier includes a development identifier and a test identifier.

[0115] Here, the system environment of the application service may be a development environment and / or a test environment. For different system environments of the application service, when creating the second log directory on the host machine, an identification may be made to filter the logs of the application service in a specific system environment for subsequent queries.

[0116] During implementation, the environment identifier can be "test" to indicate that the system environment is a test environment, and "dev" to indicate that the system environment is a development environment. And the environment identifier is configured to the preset position of the second log directory. For example, the environment identifier can be placed in the first part of the application service name in the second log directory. For example, the second log directory for the test environment stage of the application service "backend-portal" can be configured as " / data / logs / test / backend-portal".

[0117] In this embodiment, by configuring the environment identifier and the application service name in the second log directory, support is provided for subsequent query and screening of logs of application services in a specific system environment.

[0118] In some embodiments, the host machine is also provided with a log collection service, and step S2 specifically includes steps S21 to S22:

[0119] Step S21. According to the environment identifier in the second log directory of the host machine, third log directories of the log collection service corresponding to different environment identifiers are respectively configured on the host machine.

[0120] Here, each segment in the second log directory (for example, " / data / logs / test / backend-portal") is identified, and the information of each segment ("data", "logs", "backend-portal"), as well as the environment identifier ("test") and the information at the location of the application service name ("backend-portal") are determined respectively.

[0121] According to the environment identifier in the second log directory, determine the environment identifier of the third log directory of the log collection service, and configure the third log directory of the log collection service. For example, if the environment identifier in the second log directory is "test", then the environment identifier in the third log directory to be configured is also "test", and the application service name is configured in the third log directory, for example, the application service name can be configured to the latter part of the environment identifier. For example, the configured third log directory of the log collection service can be " / data / logs / test / backend-portal / application.log".

[0122] Step S22: According to the environment identifier in the third log directory of the log collection service, the log collection service collects the logs of the application service with ordinary user privileges from the second log directory of the corresponding host machine.

[0123] Here, a mapping relationship between the third log directory of the log collection service and the second log directory of the host is constructed, and the logs of the application services with ordinary user permissions are collected through the log collection service. It can be understood that the log collection service collects the logs of the application service from the second log directory with the same environment identifier and the same application service name according to the environment identifier and the application service name in each third log directory.

[0124] In this embodiment, by configuring the third log directory and collecting logs of application services with common user privileges, it is possible to classify and collect logs of application services according to the system environment, so as to facilitate subsequent query and screening of logs of application services in a specific system environment.

[0125] In some embodiments, step S3 specifically includes steps S31 to S32:

[0126] Step S31: Send a log processing request to the log filtering service through the log collection service; the log processing request carries the log of the application service with ordinary user rights and the third log directory.

[0127] Here, the log collection service sends the environment identifier of each application service log, the third log directory (log file path) and the specific log content of the application service to the log processing request.

[0128] After receiving the log processing request, the log collection service uses the filter plug-in to identify the application service name according to the third log directory. In some embodiments, after receiving the log processing request, the log collection service uses the filter plug-in to identify the application service name and the environment identifier according to the third log directory.

[0129] Step S32. Through the log filtering service and according to the environment identifier and application service name in the third log directory, the log of the application service is sent to the corresponding fourth log directory in the log indexing service, so as to query the log of the application service corresponding to the system environment stage where the application service is located through the log indexing service.

[0130] Here, a mapping relationship between the fourth log directory of the log indexing service and the third log directory of the log collection service is constructed, and the log of the application service is sent to the corresponding fourth log directory in the log indexing service through the log filtering service. It can be understood that the log filtering service sends the log of each application service to the fourth log directory with the same environment identifier and the same application service name according to the environment identifier and the application service name in each third log directory.

[0131] In this embodiment, by sending log processing requests, log filtering, and sending application service logs, the application service logs can be stored in a log indexing service, so that the application service logs corresponding to the system environment stage where the application service is located can be quickly queried through the log indexing service, thereby improving the efficiency of application service log queries.

[0132] In some embodiments, before step S11, the method further includes:

[0133] Create deployment files for application services and determine the number of deployment files;

[0134] Schedule each deployment file to the corresponding number of hosts.

[0135] Here, create a kubernetes deployment service deployment file (such as backend-portal.yaml), configure the container image used by the application service, startup parameters, and the number of deployment files (number of copies).

[0136] The correspondence between the deployment file and the host machine can be 1:1, that is, each deployment file is dispatched to a different host machine to run the deployment file of the application service on each host machine. It is also possible to dispatch some deployment files to the same host machine according to the load situation, and execute these deployment files on the host machine in chronological order.

[0137] In this embodiment, a quantitative relationship between the application service and the host machine is established by creating and scheduling the deployment file of the application service, so as to run the application service with ordinary user permissions on the host machine and persist the log of the application service with ordinary user permissions on the host machine.

[0138] Embodiment 2:

[0139] The specific implementation process of the above method is described below in conjunction with specific application examples. Figure 2 As shown, this embodiment includes steps S100 to S400:

[0140] Step S100. Automatically create a dedicated log directory (second log directory) for each service (application service) on the host machine through the initial container, and adjust the permissions of the dedicated log directory to be readable and writable by the service running user (ordinary user permissions).

[0141] Specifically, Figure 3 As shown, step S100 includes steps S110 to S170:

[0142] Step S110. Create a kubernetes deployment service deployment file (such as backend-portal.yaml), configure the container image used by the service, startup parameters, number of replicas and other general configurations, and configure:

[0143] 1.1. In the initContainers initial container, mount the service-specific log directory on the host (such as / data / logs / test / backend-portal) into the initial container through the kubernetes hostPath local storage volume, and modify the permissions of the service-specific log directory mounted in the initial container to make it readable and writable by the running user in the service container (such as ordinary non-privileged user iot).

[0144] It should be noted that a kubernetes cluster usually deploys multiple environments of an application system (such as development environment and test environment). The test environment where the application system is located is marked with test, and the development environment where the application system is located is marked with dev.

[0145] 1.2. In the service container, mount the service-specific log directory on the host (such as / data / logs / test / backend-portal) to the log directory in the service container (such as / home / app / logs) through the kubernetes hostPath local storage volume. The service-specific log directory path on the host contains the system environment identifier (such as test) and the service name (application service name, such as backend-portal).

[0146] Step S120. Apply the kubernetes deployment service deployment file (such as backend-portal.yaml). kubernetes will automatically schedule the service pod to the corresponding number (number of replicas) of hosts according to the number of replicas configured in the deployment file and the resource load of each host. According to the configuration in step 1.1, if there is no service-specific log directory on the host, kubernetes will automatically create a corresponding service-specific log directory; the owner and group of the created service-specific log directory are both root, and ordinary non-privileged users have no permission to write.

[0147] Step S130. Kubernetes continues to mount the service-specific log directory on the host (such as / data / logs / test / backend-portal) into the initial container (such as / home / app / logs) according to the configuration in step 1.1.

[0148] Step S140. Kubernetes continues to follow the configuration in step 1.1 and modifies the permissions of the service-specific log directory mounted in the initial container to allow the running user in the service container (ordinary user permissions, such as ordinary non-privileged user iot) to read and write.

[0149] Step S150. After the initial container completes the task configured in step 1.1, Kubernetes will destroy the initial container and start the service container;

[0150] Step S160. Kubernetes mounts the service-specific log directory on the host (such as / data / logs / test / backend-portal) to the log directory in the service container (the first log directory, such as / home / app / logs) according to the configuration in step 1.2.

[0151] Step S170. The logs in the service container are written to the log directory (such as / home / app / logs) and persisted to the dedicated log directory on the host (such as / data / logs / test / backend-portal).

[0152] Step S200: The log collection service sends the log path (third log directory) and the log environment identifier to the log processing service (log filtering service).

[0153] Specifically, Figure 4 As shown, step S200 includes steps S210 to S230:

[0154] Step S210. Install a log collection service (such as filebeat) on each host machine, and configure the environment identifier (such as test) to which the log belongs according to the environment identifier (such as test) contained in the collected log path (such as / data / logs / test / * / *.log).

[0155] Step S220. The log collection service collects logs of all services in all environments according to the configured log collection directory (such as / data / logs / test / * / *.log, / data / logs / dev / * / *.log).

[0156] Step S230: The log collection service sends the environment identifier (such as test), log file path (third log directory, such as / data / logs / test / backend-portal / application.log) and specific log content of each log to the log processing service.

[0157] Step S300: The log processing service classifies the service logs according to the system environment and service name and stores them in the log search server (log indexing service).

[0158] Specifically, Figure 5 As shown, step S300 includes steps S310 to S330: Step S310. Configure a log processing service (such as logstash), use a filtering plug-in, and identify the service name (such as backend-portal) to which the log belongs according to the log file path (such as / data / logs / test / backend-portal / application.log) contained in the received log data.

[0159] Step S320: The log processing service continues to format the log content according to the filtering plug-in.

[0160] Step S330. The log processing service writes the formatted log content into the corresponding service index (such as test-backend-portal) on the log search server (such as elasticsearch) according to the environment identifier (such as test) and service name (such as backend-portal) to which the log belongs.

[0161] It is understandable that by classifying and storing the service logs in the log search server according to the system environment and service name, it is possible to easily and quickly filter the logs of specific services in a specific environment on the log visualization tool.

[0162] Step S400: Automatically create an index pattern according to the service name on the log visualization tool to improve log processing efficiency.

[0163] Specifically, Figure 6 As shown, step S400 includes steps S410 to S430: Step S410. By calling the API interface of a log visualization tool (such as kibana), all service index names (such as test-backend-portal) on the log search server can be obtained.

[0164] Step S420. By calling the API interface of the log visualization tool, the corresponding index pattern (such as test-backend-portal) can be automatically created according to the service index name (such as test-backend-portal) obtained in the previous step; the above two steps are automated through scripts, and then executed regularly through scheduled tasks.

[0165] Step S430. In the exploration module of the log visualization tool, the service (backend-portal) log of the corresponding environment (test) can be quickly queried by filtering the index pattern (such as test-backend-portal).

[0166] It is understandable that the index pattern is automatically created by the service name on the log visualization tool, and the logs can be queried through the exploration module of the log visualization tool without manually configuring the index pattern.

[0167] In this embodiment, a dedicated log directory is automatically created for each service on the host machine, and the exclusive log directory permissions are adjusted through the container to be readable and writable by ordinary service operation users. Kubernetes can uniformly maintain the second log directory (service log directory) on each host machine, realize the operation and maintenance automation of the service log directory on the host machine, and reduce the complexity and workload of operation and maintenance. In addition, this embodiment can also realize the automatic collection, processing, and classified storage of service logs according to the environment and service name on the premise of improving the security of users running Kubernetes service containers, thereby improving the efficiency of log queries.

[0168] Embodiment 3:

[0169] like Figure 7 As shown, the present application provides a container log processing device based on a kubernetes local storage volume, and the device 100 includes:

[0170] A writing module 11 is configured to write the logs in the container with the application service into the first log directory of the container, and persist the logs in the first log directory of the container to the second log directory on the host; wherein the permissions of the second log directory are modified to ordinary user permissions by the container;

[0171] The collection module 12 is connected to the writing module 11 and is configured to collect logs of application services with ordinary user permissions from a second log directory of the host machine through a log collection service configured on the host machine;

[0172] The sending module 13 is connected to the collection module 12, and is configured to send the collected logs of the application service with ordinary user permissions to the log indexing service of the kubernetes cluster through the log collection service. The log indexing service corresponds to each host machine and is used to query the logs of the application service with ordinary user permissions.

[0173] In some embodiments, the writing module 11 includes:

[0174] An acquisition unit, configured to acquire a second log directory on the host machine; wherein the permission of the second log directory is a special user permission;

[0175] A mounting unit, connected to the acquisition unit, configured to mount the second log directory into the container so as to modify the permission of the second log directory of the host machine to the permission of a common user through the container;

[0176] a writing unit, connected to the mounting unit, configured to run an application service with ordinary user permissions in the container, and write the logs in the container into a first log directory;

[0177] The persistence unit is connected to the writing unit and is configured to persist the logs in the first log directory to the second log directory with modified permissions.

[0178] In some embodiments, the mounting unit is further used to mount the second log directory into an initial container of the application service through a local storage volume; and by the initial container, modify the permission of the second log directory from special user permission to ordinary user permission.

[0179] In some embodiments, the writing unit is also used to destroy the initial container and start the application service running with ordinary user permissions in the service container; the service container and the initial container have the same first log directory; through the local storage volume, the second log directory of the host machine with modified permissions is mounted to the first log directory of the service container; and the log of the service container is written to the first log directory.

[0180] In some embodiments, the acquisition unit is further used to determine whether there is a second log directory corresponding to the application service on the host machine; if so, directly acquire the second log directory; if not, create a second log directory with special user permissions on the host machine.

[0181] In some embodiments, the system environment stage of the application service includes a development environment and a test environment, and the step of creating a second log directory with special user permissions on the host machine performed by the acquisition unit specifically includes:

[0182] According to the system environment stage of the application service, a second log directory including an environment identifier and an application service name is created on the host machine; the environment identifier includes a development identifier and a test identifier.

[0183] In some embodiments, a log collection service is also provided on the host machine, and the collection module is also used to configure a third log directory of the log collection service corresponding to different environment identifiers on the host machine according to the environment identifier in the second log directory of the host machine; according to the environment identifier in the third log directory of the log collection service, the logs of the application service with ordinary user permissions are collected from the second log directory of the corresponding host machine through the log collection service.

[0184] In some embodiments, the sending module is also used to send a log processing request to the log filtering service through the log collection service; the log processing request carries the log of the application service with ordinary user permissions and the third log directory; through the log filtering service, and based on the environment identifier and the application service name in the third log directory, the log of the application service is sent to the corresponding fourth log directory in the log indexing service, so as to query the log of the application service corresponding to the system environment stage where the application service is located through the log indexing service.

[0185] In some embodiments, the device also includes a deployment module connected to the writing module, which is configured to create a deployment file for the application service, determine the number of deployment files; and schedule each deployment file to a corresponding number of host machines.

[0186] It should be noted that this embodiment is a corresponding device of the container log processing method based on the kubernetes local storage volume in the above-mentioned embodiment 1. The method in embodiment 1 can be implemented by using this device. The specific implementation method can refer to the description in the above-mentioned embodiment 1, and this embodiment will not be repeated here.

[0187] Figure 8 Another structural diagram of a container log processing device based on Kubernetes local storage volume provided by this application is as follows Figure 8As shown, the present application provides a container log processing device based on a Kubernetes local storage volume, including a memory 41 and a processor 42, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the container log processing method based on the Kubernetes local storage volume in the above-mentioned embodiments of the present application.

[0188] The memory is connected to the processor, the memory may be a flash memory or a read-only memory or other memory, and the processor may be a central processing unit or a single-chip microcomputer.

[0189] In some embodiments, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the container log processing method based on the Kubernetes local storage volume in the above-mentioned embodiments of the present application is implemented.

[0190] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0191] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. A container log processing method based on kubernetes local storage volume, characterized in that: The method comprises: S1. Write the logs in the container with the application service to the first log directory of the container, and persist the logs in the first log directory to the second log directory of the host; wherein the permissions of the second log directory are modified to ordinary user permissions by the container; S2. Collect logs of application services with ordinary user permissions from the second log directory of the host through the log collection service configured on the host; S3. The collected logs of the application service with common user rights are sent to the log indexing service of the kubernetes cluster through the log collection service. The log indexing service corresponds to each host machine and is used to query the logs of the application service with common user rights.

2. According to claim 1, the container log processing method based on the kubernetes local storage volume is characterized in that: Step S1 specifically includes: S11. Obtain a second log directory on the host; wherein the permissions of the second log directory are special user permissions; S12. Mount the second log directory into the container, so as to modify the permissions of the second log directory of the host machine to ordinary user permissions through the container; S13. Run an application service with ordinary user privileges in the container and write the logs in the container to the first log directory; S14. Persist the logs in the first log directory to the second log directory with modified permissions.

3. According to claim 2, the container log processing method based on the kubernetes local storage volume is characterized in that: Step S12 specifically includes: Mounting the second log directory to the initial container of the application service through the local storage volume; By the initial container, the permission of the second log directory is changed from special user permission to ordinary user permission.

4. According to claim 3, the container log processing method based on the kubernetes local storage volume is characterized in that: Step S13 specifically includes: Destroying the initial container, and starting an application service running with ordinary user permissions in a service container; the service container and the initial container have the same first log directory; Use the local storage volume to mount the second log directory of the host machine with modified permissions to the first log directory of the service container; Write the service container's logs to the first log directory.

5. According to claim 2, the container log processing method based on the kubernetes local storage volume is characterized in that: Step S11 specifically includes: Determine whether there is a second log directory corresponding to the application service on the host machine; If yes, directly obtain the second log directory; If not, a second log directory with special user permissions is created on the host machine.

6. The container log processing method based on kubernetes local storage volume according to claim 5 is characterized in that: The system environment stage of the application service includes a development environment and a test environment. The second log directory with special user permissions is created on the host machine, specifically including: According to the system environment stage of the application service, a second log directory including an environment identifier and an application service name is created on the host machine; the environment identifier includes a development identifier and a test identifier.

7. The container log processing method based on kubernetes local storage volume according to claim 6 is characterized in that: The host machine is also provided with a log collection service, and step S2 specifically includes: According to the environment identifier in the second log directory of the host machine, third log directories of the log collection service corresponding to different environment identifiers are respectively configured on the host machine; According to the environment identifier in the third log directory of the log collection service, the log of the application service with common user authority is collected from the second log directory of the corresponding host machine through the log collection service.

8. The container log processing method based on kubernetes local storage volume according to claim 7 is characterized in that: Step S3 specifically includes: Sending a log processing request to the log filtering service through the log collection service; the log processing request carries the log of the application service with ordinary user rights and the third log directory; Through the log filtering service and according to the environment identifier and application service name in the third log directory, the log of the application service is sent to the corresponding fourth log directory in the log indexing service, so as to query the log of the application service corresponding to the system environment stage where the application service is located through the log indexing service.

9. The container log processing method based on the kubernetes local storage volume according to any one of claims 2 to 7, characterized in that: Before step S11, the method further includes: Create deployment files for application services and determine the number of deployment files; Schedule each deployment file to the corresponding number of hosts.

10. A container log processing device based on kubernetes local storage volume, characterized in that: The device comprises: A writing module, which is configured to write the logs in the container with the application service to the first log directory of the container, and persist the logs in the first log directory of the container to the second log directory on the host; wherein the permissions of the second log directory are modified to ordinary user permissions by the container; A collection module, connected to the writing module, configured to collect logs of application services with ordinary user permissions from a second log directory of the host machine through a log collection service configured on the host machine; The sending module is connected to the collection module and is configured to send the collected logs of the application service with ordinary user permissions to the log indexing service of the kubernetes cluster through the log collection service. The log indexing service corresponds to each host machine and is used to query the logs of the application service with ordinary user permissions.