Log collection method and device

Abstract

A log collection method and device are used to more effectively and flexibly collect logs in a container set in a K8S environment during the process of container set management and service platform project construction. In the present application, the method includes: according to the resource file of the target container set, obtaining the collection path of the target container set from the container runtime of the host machine, the collection path includes the storage path of the file log to be collected in the target container set in the host machine; according to the collection path of the target container set, collecting the logs in the target container set.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a log collection method and device. Background Art

[0002] Kubernetes (K8S for short) is an open source application for managing pods on multiple hosts in a cloud platform. K8S can be used to quickly and effectively respond to user needs, deploy, and expand user applications.

[0003] In the K8S environment, container sets can be dynamically increased or decreased. For example, after the application is updated or the host fails and causes the container set to be migrated, the container set name will change, the old container set will be destroyed, and a new container set will be created. For another example, the container set has the characteristics of elastic scaling, and the number of container sets can also be increased or decreased accordingly.

[0004] Therefore, in the process of building a container set management and service platform project, how to collect application logs in the container set in the K8S environment is an urgent problem that needs to be solved. Summary of the invention

[0005] The present application provides a log collection method and device for more effectively and flexibly collecting application logs in a container set in a K8S environment during the process of container set management and service platform project construction.

[0006] In a first aspect, the present application provides a log collection method, comprising:

[0007] According to the resource file of the target container set, the collection path of the target container set is obtained from the container runtime of the host machine, wherein the collection path includes the storage path of the file logs to be collected in the target container set in the host machine; according to the collection path of the target container set, the logs in the target container set are collected.

[0008] In the above technical solution, according to the resource file of the target container set, the collection path of the target container set is obtained from the container runtime, the collection path is for a target container set, and the collection path includes the storage path of the file log to be collected in the host machine, so that the logs in the target container set can be accurately and efficiently obtained from the storage path of the host machine based on the collection path.

[0009] In a possible implementation, according to the resource file of the target container set, the collection path of the target container set is obtained from the container runtime of the host machine, including: obtaining a target container set identifier corresponding to the target container set and a target volume (data volume) identifier corresponding to the file log to be collected from the resource file of the target container set; according to the target container set identifier, obtaining multiple corresponding relationships of the target container set from the container runtime, the corresponding relationships including a volume identifier and a host machine path corresponding to the volume identifier; according to the target volume identifier and the multiple corresponding relationships, determining the target host machine path corresponding to the target volume identifier as the storage path of the file log to be collected in the host machine.

[0010] In the above technical solution, the target container set identifier and the target volume identifier can be obtained from the resource file of the target container set, and then based on the target container set identifier and the target volume identifier, the target host path can be obtained from the container runtime. The target host path is specifically the path in the host machine mapped from the target container set path of the file log to be collected. In this way, the file log to be collected can be obtained from the host machine according to the target host path.

[0011] In a possible implementation, the resource file includes a target container set path of the file log to be collected in the target container set and a corresponding relationship between the target volume identifier; before obtaining the collection path of the target container set from the container runtime of the host machine according to the resource file of the target container set, the method further includes: creating a target container set in the host machine according to the resource file of the target container set, wherein the target volume corresponding to the target volume identifier is used to map the target container set path to the target host machine path; and storing the target host machine path and the target volume identifier in correspondence in the container runtime.

[0012] In the above technical solution, when creating a target container set, the container runtime can map the target container set path to the target host path in the host according to the correspondence between the target container set path and the target volume identifier in the resource file, and record the correspondence between the target volume identifier and the target host path in the container runtime. In this way, the target host path can be obtained from the container runtime according to the target volume identifier, and then the file log to be collected can be obtained from the host according to the target host path.

[0013] In a possible implementation, before obtaining the collection path of the target container set from the container runtime of the host machine according to the resource file of the target container set, the method further includes: obtaining a resource file sent by a user, where the resource file includes a target container set path in the target container set defined by the user and containing file logs to be collected; generating a target volume corresponding to the target container set path; and writing the target volume identifier into the resource file.

[0014] In the above technical solution, after obtaining the resource file sent by the user, the corresponding target volume can be generated according to the target container set path of the file log to be collected defined by the user in the resource file, and then the target volume identifier is written into the resource file, and the target volume corresponds to the target container set path. In this way, when creating a container set at container runtime, the target container set path can be mapped to the target host path based on the target volume.

[0015] In a possible implementation, collecting logs in the target container set according to a collection path of the target container set includes: generating a configuration file for the target container set according to the collection path of the target container set; updating an existing configuration file in a hot update manner according to the configuration file; and collecting logs in the target container set by using the updated configuration file.

[0016] In the above technical solution, a configuration file for the target container set can be generated according to the collection path of the target container set. The configuration file can be an update of an existing configuration file by hot updating without interrupting the current collection task, which helps to improve the efficiency of log collection.

[0017] In a possible implementation, the collection path also includes a storage path of a standard output log in the target container set in the host machine; collecting the logs in the target container set according to the collection path of the target container set includes: acquiring the file log according to the storage path of the file log to be collected in the host machine; acquiring the standard output log according to the storage path of the standard output log in the host machine; and acquiring the collection result according to the file log and the standard output log.

[0018] In the above technical solution, the storage path of the file log to be collected in the host machine can be obtained, and the file log of the target container set can be collected based on the storage path; of course, the storage path of the standard output log in the host machine can also be obtained, and the standard output log of the target container set can be collected based on the storage path, so that the logs of the application running in the target container set can be obtained more comprehensively.

[0019] In a possible implementation, obtaining a collection result according to the file log and the standard output log includes: obtaining a log identifier of the log to be stored; selecting the log to be stored from the file log and the standard output log according to the log identifier of the log to be stored, and using the log to be stored as the collection result.

[0020] In the above technical solution, based on user needs, the logs to be stored can be obtained from the file logs and the standard output logs, and then the logs to be stored can be persistently stored.

[0021] In a possible implementation, it further includes: generating index information of the collection result based on the field in the metadata information of the target container set. In the above technical solution, index information of the collection result can also be generated, so as to help users find the logs of the target container set more conveniently.

[0022] In a second aspect, the present application provides a log collection device, including:

[0023] The processing module is used to obtain the collection path of the target container set from the container runtime of the host machine according to the resource file of the target container set, wherein the collection path includes the storage path of the file logs to be collected in the target container set in the host machine; the collection module is used to collect the logs in the target container set according to the collection path of the target container set.

[0024] In a possible implementation, the processing module is specifically used to: obtain a target container set identifier corresponding to the target container set and a target volume identifier corresponding to the file log to be collected from a resource file of the target container set; obtain multiple corresponding relationships of the target container set from the container runtime according to the target container set identifier, wherein the corresponding relationships include a volume identifier and a host path corresponding to the volume identifier; and determine a target host path corresponding to the target volume identifier according to the target volume identifier and the multiple corresponding relationships, so as to serve as a storage path of the file log to be collected in the host.

[0025] In a possible implementation, the resource file includes a target container set path of the file log to be collected in the target container set and a corresponding relationship between the target volume identifier; the processing module is further used to: before the processing module obtains the collection path of the target container set from the container runtime of the host machine according to the resource file of the target container set, create a target container set in the host machine according to the resource file of the target container set, wherein the target volume corresponding to the target volume identifier is used to map the target container set path to the target host machine path; and store the target host machine path and the target volume identifier in the container runtime in correspondence.

[0026] In a possible implementation, before obtaining the collection path of the target container set from the container runtime of the host machine according to the resource file of the target container set, the processing module is further used to obtain a resource file sent by a user, where the resource file includes a target container set path in the target container set defined by the user and containing file logs to be collected; generate a target volume corresponding to the target container set path; and write the target volume identifier into the resource file.

[0027] In a possible implementation, the collection module is specifically used to: generate a configuration file for the target container set according to the collection path of the target container set; update the existing configuration file by hot update according to the configuration file; and collect logs in the target container set by using the updated configuration file.

[0028] In a possible implementation, the collection path also includes a storage path of the standard output log of the target container set in the host machine; the collection module is specifically used to: obtain the file log according to the storage path of the file log to be collected in the host machine; obtain the standard output log according to the storage path of the standard output log in the host machine; obtain the collection result according to the file log and the standard output log.

[0029] In a possible implementation, the collection module is specifically used to: obtain a log identifier of the log to be stored; select the log to be stored from the file log and the standard output log according to the log identifier of the log to be stored, and use the log to be stored as the collection result.

[0030] In a possible implementation manner, the collection module is further configured to generate index information of the collection result based on the fields in the metadata information of the target container set.

[0031] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program or instructions are stored. When the computer program or instructions are executed by a computing device, the computing device executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0032] In a fourth aspect, the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a computing device, the method in the above-mentioned first aspect or any possible implementation manner of the first aspect is implemented.

[0033] In a fifth aspect, the present application provides a computing device, comprising a processor, the processor being connected to a memory, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory, so that the computing device implements the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0034] The technical effects that can be achieved in any of the second to fifth aspects mentioned above can refer to the description of the beneficial effects in the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of a system architecture provided for this application;

[0036] Figure 2 A schematic diagram of a path mapping method provided in this application;

[0037] Figure 3 A flow chart of a log collection method provided by this application;

[0038] Figure 4 A flowchart of another log collection method provided by this application;

[0039] Figure 5 A schematic diagram of the structure of a log collection device provided by this application;

[0040] Figure 6 A schematic diagram of the structure of another log collection device provided in the present application. DETAILED DESCRIPTION

[0041] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0042] K8S is an open source platform for automated deployment, expansion, and maintenance of container sets (pods). In the K8S environment, the deployed container set can be a Docker container set, which is decoupled from the host machine through the isolation feature of the container set, so that the service does not need to rely on the host machine to run and does not affect the host machine. K8S can be used to manage all container sets in the service, such as creating, running, restarting or deleting container sets.

[0043] like Figure 1 A system architecture provided for this application includes a client, a K8S cluster, and a log storage backend. The K8S cluster may include an API server (apiserver), a scheduling server, and multiple hosts.

[0044] Exemplarily, the client can publish a deployment request to the API server, and then the scheduling server reads the deployment request from the API server, and determines a host that meets the deployment conditions from multiple hosts based on the deployment request, and deploys the corresponding container set on the host, where the container set may include one or more containers, each of which runs an application. Furthermore, the log storage backend can obtain the logs generated by the application running in the container set through the log collection tool, and then store the collected logs persistently.

[0045] Typically, before a log collection tool collects logs from a container set, the staff needs to pre-configure a corresponding configuration file for the log collection tool, where the configuration file may include collection rules, such as log collection method, path, and collection target information. The collection tool can then collect application logs from the container set based on the configuration file.

[0046] However, in the K8S environment, container sets can be dynamically increased or decreased. For example, after the application is updated or the host fails and causes the container set to be migrated, the name of the container set will change, the old container set will be destroyed, and a new container set will be created. For another example, the container set has the characteristics of elastic scaling, and the number of container sets can also be increased or decreased accordingly.

[0047] In this way, when the container set changes, the log collection tool cannot perceive the dynamic changes of the container set, resulting in the log collection tool being unable to effectively collect logs in the container set.

[0048] Based on this, the present application provides a log collection method and device for more effectively and flexibly collecting logs in container sets for a K8S environment during container set management and service platform project construction.

[0049] It is noted in advance that the present application may deploy an agent on the host machine, and the agent may be used to collect logs of the running of the application in the container set deployed on the host machine, and push the collected logs to the log storage backend.

[0050] The following is an explanation of the present application embodiment based on two parts: container set creation and log collection:

[0051] 1. Container Set Creation

[0052] It is noted that the application logs of the container set may include standard output logs and file logs.

[0053] The standard output logs, namely, stdout and stderr of the container set, can optionally have output paths of the standard output logs of the multiple container sets uniformly mapped to a fixed path (or fixed directory) on the host machine.

[0054] File logs are logs that are output by applications to specified files. The log path inside the container set is not fixed and varies depending on the application. For example, the log paths corresponding to the tomcat application and the nginx application are inconsistent.

[0055] Since the Agent is deployed on the host node, the log output paths of all container sets deployed on the host need to be uniformly mapped to the host path. For example, the standard output log of the container set is mapped to the fixed path of the host (such as / var / lib / docker / containers). In this way, the Agent on the host can uniformly collect the standard output log.

[0056] Furthermore, considering that the path of the file log in the container set is not fixed, the present application provides a method for mapping the path of the file log in the container set to the path in the host machine. For details, see Figure 2 The process is shown as an example:

[0057] Step 201: The client obtains a resource file of a container set to be created.

[0058] In the present application, the user can write a resource file corresponding to the container set to be created in the interface displayed by the client, and accordingly, the client can generate the resource file of the container set to be created based on the user instruction.

[0059] The resource file may be in yaml format, xml format or json format.

[0060] When the user needs to obtain the logs of the application running in the container set to be created, a declarative configuration can be performed in advance in the resource file of the container set to be created.

[0061] Specifically, the resource file may include a declarative configuration, which may include log information of one or more file logs to be collected, wherein the log information of each file log to be collected includes a log identifier of the file log to be collected, and a path of the corresponding file log to be collected inside the container set to be created (which may be referred to as a container set path), and the container set path may be customized by the user based on the application. The container set path may also be referred to as a file log path in the container set, or a file log directory in the container set, a container set directory, etc.

[0062] Exemplarily, the declarative configuration may be included in the annotation of the yaml resource file. Annotation is a function provided by K8S that can attach some metadata information to resource objects in the K8S environment to facilitate subsequent application controllers to obtain and process them.

[0063] The following is an example of an annotation in a YAML resource file provided by this application:

[0064] Table 1

[0065]

[0066] The annotation includes multiple container identifiers, each of which corresponds to log information of the file log to be collected in the respective container. Specifically, each log information includes a log identifier and a container set path.

[0067] For example, if the container ID is "c0", there are two log messages corresponding to the container ID "c0":

[0068] The first log information includes the file log identifier "1.log" and the container set path " / logdir1", which means that the file log corresponding to the file log identifier "1.log" in the container set "c0" needs to be collected, and the container set path of the file log in the container set "c0" is " / logdir1";

[0069] The second log information includes the file log identifier "2.log" and the container set path " / logdir2", which means that the file log corresponding to the file log identifier "2.log" in the container set "c0" needs to be collected, and the container set path of the file log in the container set "c0" is " / logdir2".

[0070] The rest are similar and will not be repeated here.

[0071] In step 202, the client defines a volume (data volume) corresponding to each file log to be collected according to the log information of one or more file logs to be collected in the resource file, and then writes the identifier of each volume into the resource file of the container set to be created.

[0072] Specifically, the client generates volumes corresponding to the one or more file logs to be collected; and then writes the identifier of the volume into the annotation of the resource file of the container set to be created.

[0073] In combination with the example in Table 1 above, the resource file annotation includes three log information, so the client can generate three volumes, and the three volumes are in a one-to-one correspondence with the three log information. The client can write the identifiers of the three volumes "volume1", "volume2", and "volume3" into the annotation accordingly.

[0074] See the bold part in Table 2 below, where the first log information in the container set "c0" corresponds to "volume1", that is, the correspondence between the log path " / logdir1" in the container set and the volume identifier "volume1";

[0075] The second log information in the container set "c0" corresponds to "volume2", that is, the log path " / logdir2" in the container set corresponds to the volume identifier "volume2".

[0076] Table 2

[0077]

[0078] Before defining one or more volumes corresponding to the container set to be created, the client can first determine whether there are file logs to be collected in the container set to be created, that is, whether the resource file of the container set to be created is configured with log information of the file logs to be collected. If it is determined that the log information of the file logs to be collected is configured, the corresponding volume can be defined based on the log information of each file log to be collected.

[0079] The volume can be of emptydir type. Emptydir type volumes are created when the container set is assigned to the host. K8S will automatically assign a host path on the host, so there is no need to specify the corresponding path (i.e. directory file) on the host. Emptydir type volumes will eventually be mapped to a non-fixed path on the host.

[0080] In step 203, the client publishes the resource files of the container set to be created in the K8S cluster, so that the host creates the container set according to the resource files of the container set to be created.

[0081] After completing the above preprocessing, the client can generate a deployment request for the container set, and then publish the deployment request in the K8S cluster. Correspondingly, the K8S cluster selects a host based on the deployment request. The selected host completes the creation of the container set based on the deployment request. The deployment request may include the resource files of the container set to be created, which further include the above defined annotations.

[0082] Specifically, the host machine includes a container runtime, which can be understood as a process running on the host machine. In a specific implementation, the container runtime can be a docker daemon.

[0083] The container runtime can create a container set on the host based on the deployment request. Furthermore, the container runtime can perform mounting operations based on multiple volume identifiers recorded in the deployment request and annotation. Specifically, each volume identifier in the annotation can be mounted to a path on the host (which can be called a host path or a mount path), that is, one volume identifier corresponds to one host path. Accordingly, the container runtime can record the correspondence between the volume identifier and the host path, which can be understood as mounting information.

[0084] When running applications in the created container set, the container runtime can store the file logs generated by the application running to the location corresponding to the host path according to the correspondence between the volume identifier and the host path.

[0085] It should be noted that after the container set is created, the created container set is obtained. If the file logs in the created container set need to be collected, the created container set can be called the target container set. Accordingly, the above container set creation process can also be considered as the target container set creation process.

[0086] 2. Log Collection

[0087] The Agent deployed on the host machine can monitor the event of creating a container set in real time through the monitoring (watch) mechanism of the container runtime and the API server in the K8S cluster. It can be understood that after the above-mentioned container set to be created is created by the container runtime of the host machine, the event can be monitored by the Agent on the host machine.

[0088] After the Agent on the host monitors the event of creating a container set through the API server, it can determine whether the container set is the target container set. Specifically, if the resource file corresponding to the container set includes the log information of the log to be collected, then this container set can be considered as the target container set.

[0089] Reference Figure 3 A flow chart of a log collection method is shown as an example:

[0090] Step 301: The agent obtains the volume information and metadata information of the target container set from the resource file of the target container set. Specifically, the agent may obtain the volume information and metadata information of the target container set from the API server.

[0091] The volume information of the target container set may include the target log identifier of the file log to be collected in the target container set, the target container set path in the target container set, the target volume identifier, etc. For details, see the above Figure 2 The metadata information of the target container set may include the target container set identifier (ID), the target container set name, the namespace to which it belongs, the target container set IP address, etc.

[0092] Step 302: The Agent obtains the collection path of the target container set from the container runtime of the host machine according to the volume information and metadata information of the target container set.

[0093] In a possible implementation, the collection path of the target container set may include a host path corresponding to the file log to be collected, that is, a storage path of the file log to be collected in the host.

[0094] The following explanation shows that the Agent obtains the target host path corresponding to the file logs to be collected in the target container set from the container runtime of the host machine based on the volume information and metadata information of the target container set:

[0095] Specifically, the container runtime records multiple corresponding relationships of each container set in the multiple container sets, wherein each corresponding relationship includes a volume identifier and a host path corresponding to the volume identifier.

[0096] For details, please refer to the example in Table 3. Three corresponding relationships of container set 1 are recorded in the container runtime. The three corresponding relationships are volume identifier 11 and host path 11, volume identifier 12 and host path 12, and volume identifier 13 and host path 13. The container runtime also records a corresponding relationship of container set 2, namely, volume identifier 2 and host path 2.

[0097] Table 3

[0098] Container Set Identifier Volume ID Host path Container set ID 1 volume logo 11 Host path 11 Container set ID 1 Volume ID 12 Host path 12 Container set ID 1 Volume ID 13 Host path 13 Container set identifier 2 Volume ID 2 Host path 2

[0099] The agent can obtain multiple correspondences of the target container set from the container runtime according to the target container set identifier in the metadata information, and then obtain the target host path corresponding to the target volume identifier from the multiple correspondences according to the target volume identifier in the volume information. The target host path is used to store the file log corresponding to the volume identifier, that is, as the storage path of the file log to be collected in the host.

[0100] In combination with the example in Table 3, for example, the target container set identifier is container set identifier 1, and the target volume identifier is volume identifier 12, then the Agent can determine that the target host path is host path 12.

[0101] In addition, the collection path of the target container set may also include the host path corresponding to the standard output log, that is, the storage path of the standard output log in the host. Specifically, the output path of the standard output log of the container set can be uniformly mapped to a fixed path on the host. The agent can obtain the host path corresponding to the standard output log from the container runtime of the host according to the container set identifier in the metadata information.

[0102] Exemplarily, the information obtained by the Agent can be seen in the example in Table 4 below: the host path where the standard output log of the target container set is located is "LogPath: / var / lib / docker / containers / xxx / xxx-json.log";

[0103] The target container set path (containerFilePath) is " / logdir1 / 1.log", and the target host path (hostRealPath) is " / xxx / volume1 / 1.log", wherein " / logdir1 / 1.log" and " / xxx / volume1 / 1.log" have a mapping relationship.

[0104] Table 4

[0105]

[0106]

[0107] Step 303: The Agent generates a configuration file for the target container set according to the collection path of the target container set.

[0108] In a possible implementation, the agent may generate a configuration file for the target container set according to the host path corresponding to the file log to be collected in the target container set (ie, the target host path) and the host path corresponding to the standard output log.

[0109] This configuration file can be implemented through the filebeat open source tool, the fluentd open source tool, or other open source tools.

[0110] Step 304: The Agent collects logs in the target container set according to the configuration file to obtain collection results.

[0111] Specifically, the Agent may include a log monitoring module and a log collection module, wherein the log monitoring module may complete the above steps 301 to 303 , and the log collection module may complete the above steps 304 .

[0112] Furthermore, the configuration file generated by the log monitoring module can be read by the log collection module. The log collection module loads the new configuration file by hot update, and realizes dynamic collection of the target container centralized logs to obtain the collection results of the target container centralized logs.

[0113] Optionally, the log monitoring module and the log collection module can be deployed in a container set of the host machine.

[0114] Step 305: The Agent stores the collection results in the log storage backend.

[0115] Optionally, the Agent may be configured with a log identifier of a log to be stored in the log storage backend, and then obtain the log corresponding to the log identifier from the collection result according to the log identifier of the log to be stored, and store the log in the log storage backend.

[0116] In addition, the Agent can also store the fields in the metadata information of the target container set to the log storage backend, such as storing one or more fields in the target container set identifier, target container set name, namespace, and target container set IP to the log storage backend. Thus, the log storage backend can generate indexed logs based on these fields and the logs from the Agent, thereby facilitating subsequent log retrieval. Among them, the fields in the metadata information can also be used as additional information in the above-mentioned configuration file and configured in the log collection module.

[0117] Optionally, if the Agent listens to the deletion event of the target container set, the configuration file of the target container set is deleted. Specifically, if the Agent listens to the deletion event of the target container set through the monitoring mechanism of the API server in the K8S cluster, the configuration file of the target container set can be garbage collected to ensure the effectiveness of log collection.

[0118] Based on the above content and the same concept, Figure 4 A flow chart of another log collection method provided in the present application, wherein the method may involve a log collection device, and the log collection device may include a log preprocessing module, a log monitoring module, a log collection module, and a log storage module.

[0119] Based on the modules included in the device, the method flow in this application is explained as follows:

[0120] The log preprocessing module can obtain the resource file sent by the user, which includes a declarative configuration, and the declarative configuration specifically includes / logdir1 / 1.log. The log preprocessing module can generate a volume corresponding to / logdir1 / 1.log according to the declarative configuration, and record the identifier of the volume (expressed as volume1) in the resource file. The log preprocessing module can publish the resource file with volume1 recorded in the API server.

[0121] The container runtime can create a container set in the host based on the resource files in the API server. The container runtime records the volume mounting information on the host, that is, the correspondence between volume1 and the host path hostpath1; and records the correspondence between the standard output log stdout and the host path hostpath2.

[0122] The log monitoring module can monitor the creation event of the container set in the API server, and then obtain the metadata information corresponding to the container set from the API server. And the log monitoring module can obtain the correspondence between volume1 and the host path hostpath1 from the container runtime, and the correspondence between the standard output log stdout and the host path hostpath2.

[0123] Furthermore, the log monitoring module generates a configuration file for the container set, which may include the collection configuration of the file log (including the correspondence between volume1 and the host path hostpath1), the collection configuration of the standard output (the correspondence between the standard output log stdout and the host path hostpath2), and additional information (including the container set metadata information). The log monitoring module may transmit the newly generated configuration file to the log collection module.

[0124] The log collection module performs hot update according to the acquired configuration file, thereby collecting the centralized logs of the container based on the configuration file after hot update to obtain the collection result. The log collection module outputs the collection result to the log storage module, and the log storage module stores the collection result persistently.

[0125] In the above technical solution, according to the resource file of the target container set, the collection path of the target container set is obtained from the container runtime, the collection path is for a target container set, and the collection path includes the storage path of the file log to be collected in the host machine, so that the logs in the target container set can be accurately and efficiently obtained from the storage path of the host machine based on the collection path.

[0126] Based on the above content and the same idea, Figure 5 and Figure 6 This is a schematic diagram of the structure of a possible log collection device provided by the present application. These devices can be used to implement the functions in the above method embodiments, and thus can also achieve the beneficial effects possessed by the above method embodiments.

[0127] like Figure 5 As shown, the device includes a processing module 501 and a collection module 502;

[0128] In a possible implementation, the processing module 501 is used to obtain a collection path of the target container set from a container runtime of a host machine according to a resource file of the target container set, wherein the collection path includes a storage path of file logs to be collected in the target container set in the host machine; and the collection module 502 is used to collect logs in the target container set according to the collection path of the target container set.

[0129] In a possible implementation, the processing module 501 is specifically used to: obtain a target container set identifier corresponding to the target container set and a target volume identifier corresponding to the file log to be collected from a resource file of the target container set; obtain multiple corresponding relationships of the target container set from the container runtime according to the target container set identifier, wherein the corresponding relationships include a volume identifier and a host path corresponding to the volume identifier; and determine, according to the target volume identifier and the multiple corresponding relationships, a target host path corresponding to the target volume identifier as a storage path of the file log to be collected in the host.

[0130] In a possible implementation, the resource file includes a target container set path of the file log to be collected in the target container set and a corresponding relationship between the target volume identifier; before the processing module 501 obtains the collection path of the target container set from the container runtime of the host machine according to the resource file of the target container set, it is also used to: create a target container set in the host machine according to the resource file of the target container set, wherein the target volume corresponding to the target volume identifier is used to map the target container set path to the target host machine path; and store the target host machine path and the target volume identifier in the container runtime in correspondence.

[0131] In a possible implementation, the processing module 501 is further used to, before obtaining the collection path of the target container set from the container runtime of the host machine according to the resource file of the target container set, obtain a resource file sent by a user, where the resource file includes a target container set path of the file log to be collected in the target container set defined by the user; generate a target volume corresponding to the target container set path; and write the target volume identifier into the resource file.

[0132] In a possible implementation, the collection module 502 is specifically used to: generate a configuration file for the target container set according to the collection path of the target container set; update the existing configuration file by hot update according to the configuration file; and collect logs in the target container set by using the updated configuration file.

[0133] In a possible implementation, the collection path also includes the storage path of the standard output log of the target container set in the host machine; the collection module 502 is specifically used to: obtain the file log according to the storage path of the file log to be collected in the host machine; obtain the standard output log according to the storage path of the standard output log in the host machine; obtain the collection result according to the file log and the standard output log.

[0134] In a possible implementation, the collection module 502 is specifically used to: obtain a log identifier of the log to be stored; select the log to be stored from the file log and the standard output log according to the log identifier of the log to be stored, and use the log to be stored as the collection result.

[0135] In a possible implementation, the collection module 502 is further configured to generate index information of the collection result based on the fields in the metadata information of the target container set.

[0136] like Figure 6 The device 600 provided in the embodiment of the present application is shown. Figure 6 The device shown can be Figure 5 A hardware circuit implementation of the device shown in the figure. The device can be applied to the flowchart shown above to perform the functions in the above method embodiment.

[0137] For ease of explanation, Figure 6 Only the main components of the device are shown.

[0138] Figure 6 The device 600 shown includes a communication interface 610, a processor 620 and a memory 630, wherein the memory 630 is used to store program instructions and / or data. The processor 620 may operate in conjunction with the memory 630. The processor 620 may execute program instructions stored in the memory 630. When the instructions or programs stored in the memory 630 are executed, the processor 620 is used to execute the operations performed by the processing module 501 in the above embodiment, and the communication interface 610 is used to execute the operations performed by the acquisition module 502 in the above embodiment.

[0139] The memory 630 is coupled to the processor 620. The coupling in the embodiment of the present application is an indirect coupling or communication connection between devices, units or modules, which can be electrical, mechanical or other forms, for information exchange between devices, units or modules. At least one of the memories 630 may be included in the processor 620.

[0140] In the embodiment of the present application, the communication interface may be a transceiver, a circuit, a bus, a module or other types of communication interfaces. In the embodiment of the present application, when the communication interface is a transceiver, the transceiver may include an independent receiver, an independent transmitter, or a transceiver with integrated transceiver functions, or a communication interface.

[0141] The device 600 may further include a communication line 640. The communication interface 610, the processor 620, and the memory 630 may be interconnected via the communication line 640; the communication line 640 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication line 640 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0142] Based on the above content and the same concept, the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed by a computing device, the computing device performs the functions in the above method embodiments.

[0143] Based on the above content and the same concept, the present application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a computing device, the functions in the above method embodiments are performed.

[0144] Based on the above content and the same concept, the present application provides a computing device, which includes a processor, the processor is connected to a memory, the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that the computing device performs the functions in the above method embodiments.

[0145] It is understood that the various numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application. The size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic.

[0146] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of protection of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A log collection method, characterized in that: include: According to the resource file of the target container set, a collection path of the target container set is acquired from the container runtime of the host machine, wherein the collection path includes a storage path of the file log to be collected in the target container set in the host machine; Collecting logs in the target container set according to the collection path of the target container set; The acquiring the collection path of the target container set from the container runtime of the host machine according to the resource file of the target container set includes: Acquire a target container set identifier corresponding to the target container set and a target data volume identifier corresponding to the file log to be collected from a resource file of the target container set; According to the target container set identifier, multiple corresponding relationships of the target container set are acquired from the container runtime, where the corresponding relationships include a volume identifier and a host path corresponding to the volume identifier; According to the target volume identifier and the multiple corresponding relationships, a target host machine path corresponding to the target volume identifier is determined to serve as a storage path of the file log to be collected in the host machine.

2. The method according to claim 1, characterized in that The resource file includes the target container set path of the file log to be collected in the target container set and the corresponding relationship between the target volume identifier; Before obtaining the collection path of the target container set from the container runtime of the host machine according to the resource file of the target container set, the method further includes: According to the resource file of the target container set, the target container set is created in the host machine, wherein the target volume corresponding to the target volume identifier is used to map the target container set path to the target host machine path; The target host path and the target volume identifier are stored in the container runtime in correspondence.

3. The method according to claim 2, characterized in that Before obtaining the collection path of the target container set from the container runtime of the host machine according to the resource file of the target container set, the method further includes: Acquire a resource file sent by a user, wherein the resource file includes a target container set path defined by the user and containing the file logs to be collected in the target container set; Generate the target volume corresponding to the target container set path; The target volume identifier is written into the resource file.

4. The method according to claim 1, characterized in that The collecting logs in the target container set according to the collection path of the target container set includes: generating a configuration file for the target container set according to the collection path of the target container set; According to the configuration file, the existing configuration file is updated by hot update; And the logs in the target container set are collected through the updated configuration file.

5. The method according to any one of claims 1 to 4, characterized in that: The collection path also includes a storage path of the target container centralized standard output log in the host machine; The collecting logs in the target container set according to the collection path of the target container set includes: Acquire the file log according to the storage path of the file log to be collected in the host machine; Obtain the standard output log according to the storage path of the standard output log in the host machine; The collection result is obtained according to the file log and the standard output log.

6. The method according to claim 5, characterized in that The obtaining of the collection result according to the file log and the standard output log includes: Get the log ID of the log to be stored; According to the log identifier of the log to be stored, the log to be stored is selected from the file log and the standard output log, and the log to be stored is used as the collection result.

7. The method according to claim 5, characterized in that Also includes: Based on the fields in the metadata information of the target container set, index information of the collection result is generated.

8. A log collection device, characterized in that: Comprising means for executing the method as claimed in any one of claims 1 to 7.

9. A computer program product, characterized in that The computer program product comprises a computer program or instructions, and when the computer program or instructions are executed by a computing device, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction, and when the computer program or instruction is executed by a computing device, the method according to any one of claims 1 to 7 is implemented.

11. A computing device, characterized in that: The computer comprises a processor connected to a memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the computing device executes the method according to any one of claims 1 to 7.

Images