Log collection method, electronic equipment and storage medium
By mapping the logs in the container group to the shared directory of the virtual machine node, collecting and posting source processing logs, the challenge of log collection in containerized deployment is solved, unified management and monitoring of logs is realized, and log traceability is supported.
Patent Information
- Application Number
- CN202411603347.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-11
- Publication Date
- 2025-03-04
AI Technical Summary
In containerized deployment mode, how to collect logs from each container group to achieve unified management and monitoring of logs.
By mapping the log to be collected in the target container group to the shared directory of the virtual machine node, the mapped log is obtained and the mapped log is collected from the shared directory. According to the container group identification and mapping log of the target container group, the source processing is performed to obtain the first source log, and finally, the collection log of the target application system is obtained based on the first source log.
It realizes log collection in containerized deployment, realizes unified log management and monitoring, and supports log traceability.
Smart Images
Figure CN119557047A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of network security technology, and in particular to a log collection method, electronic device and storage medium. Background Art
[0002] With the gradual popularization of cloud native technology, large enterprises such as banks are piloting container platforms and containerized deployment of applications, which brings new challenges to the collection and monitoring of application logs. In the traditional virtual machine deployment mode, a log collection tool is deployed on each of the multiple virtual machines deployed by each application system, and the logs can be collected to the message buffer queue for subsequent processing by the monitoring platform. However, in the containerized deployment mode, the application system is no longer deployed on a virtual machine, but in a container group. A cluster contains multiple virtual machine nodes, and each virtual machine node can deploy multiple container groups of application systems.
[0003] Therefore, it is necessary to study how to collect logs from each container group. Summary of the invention
[0004] The log collection method, electronic device, and storage medium provided in the embodiments of the present application are used to achieve the effect of collecting logs from each container group.
[0005] In a first aspect, an embodiment of the present application provides a log collection method, including:
[0006] Map the logs to be collected in the target container group to the shared directory of the virtual machine node to obtain the mapping logs. The target container group is the container group of the target application system, and the target container group is located in the virtual machine node.
[0007] Collect mapping logs from the shared directory;
[0008] Obtain the first post source log according to the container group identifier and mapping log of the target container group;
[0009] According to the first source log, the collection log of the target application system is obtained.
[0010] In a possible implementation, obtaining the first post source log according to the container group identifier of the target container group and the mapping log includes:
[0011] According to the mapping log, obtain the first file path of the mapping log;
[0012] According to the container group directory in the first file path, obtain the container group identifier of the mapping log, where the container group directory is used to identify the target container group where the mapping log is located;
[0013] According to the container group identifier, the mapping log is processed to obtain a first source log.
[0014] In a possible implementation, collecting mapping logs from a shared directory includes:
[0015] Determine the collection probe in the virtual machine node. The collection probe is a Flume probe.
[0016] According to the collection probe, the mapping log to be identified is obtained from the virtual machine node;
[0017] The mapping log is determined according to the log name of each mapping log to be identified, wherein the log name includes the identifier of the target application system.
[0018] In a possible implementation, the logs to be collected in the target container group are mapped to a shared directory of the virtual machine node to obtain the mapping logs, including:
[0019] Determine the shared directory in the virtual machine node, and mount the shared directory in the target container group;
[0020] When the target container group performs log mapping, the container group directory in the shared directory is determined, and the container group directory is the directory corresponding to the target container group;
[0021] Map the logs to be collected to the container group directory to obtain the mapping logs.
[0022] In a possible implementation, the logs to be collected are mapped to the container group directory to obtain the mapping logs, including:
[0023] Determine the log types to be collected, which include flow logs, monitoring logs, and business operation logs;
[0024] According to the log type, determine the log type directory in the container group directory, where the log type directory is a directory corresponding to the log type;
[0025] Name the logs to be collected according to the identifier of the target application system;
[0026] Map the named logs to be collected to the log type directory to obtain the mapped logs.
[0027] In a possible implementation, obtaining the collection log of the target application system according to the first post source log includes:
[0028] Determine a second file path of the first post source log;
[0029] Determine the log type of the first post source log according to the second file path;
[0030] Determining a message queue group corresponding to a log type of a target application system in a message buffer queue;
[0031] The first post source log is stored in the message queue group to obtain the collection log of the target application system.
[0032] In a possible implementation, the method further includes:
[0033] Get collection logs from message queue groups;
[0034] According to the log parsing rules of the target application system, the collected logs are parsed to obtain parsed logs;
[0035] Based on the parsed logs, the monitoring indicators of the target application system are obtained, including transaction volume, response time and success rate;
[0036] Monitor the target application system according to the monitoring indicators.
[0037] In a possible implementation, the method further includes:
[0038] Obtain the region information of the virtual machine node and the deployment unit information of the first post source log, where the region information includes the region and availability zone;
[0039] Obtaining a second post source log according to the area information, the deployment unit information, and the first post source log;
[0040] According to the second source log, the collection log of the target application system is obtained.
[0041] In a second aspect, an embodiment of the present application provides a log collection device, including:
[0042] A mapping module is used to map the logs to be collected in the target container group to the shared directory of the virtual machine node to obtain mapping logs. The target container group is the container group of the target application system, and the target container group is located in the virtual machine node.
[0043] The collection module is used to collect mapping logs from the shared directory;
[0044] A paste source module, used to obtain a first paste source log according to the container group identifier of the target container group and the mapping log;
[0045] The obtaining module is used to obtain the collection log of the target application system according to the first post source log.
[0046] According to the mapping log, obtain the first file path of the mapping log;
[0047] According to the container group directory in the first file path, obtain the container group identifier of the mapping log, where the container group directory is used to identify the target container group where the mapping log is located;
[0048] According to the container group identifier, the mapping log is processed to obtain a first source log.
[0049] In a possible implementation, the acquisition module is further used for:
[0050] Determine the collection probe in the virtual machine node. The collection probe is a Flume probe.
[0051] According to the collection probe, the mapping log to be identified is obtained from the virtual machine node;
[0052] The mapping log is determined according to the log name of each mapping log to be identified, wherein the log name includes the identifier of the target application system.
[0053] In a possible implementation, the mapping module is further used to:
[0054] Determine the shared directory in the virtual machine node, and mount the shared directory in the target container group;
[0055] When the target container group performs log mapping, the container group directory in the shared directory is determined, and the container group directory is the directory corresponding to the target container group;
[0056] Map the logs to be collected to the container group directory to obtain the mapping logs.
[0057] In a possible implementation, the mapping module is further used to:
[0058] Determine the log types to be collected, which include flow logs, monitoring logs, and business operation logs;
[0059] According to the log type, determine the log type directory in the container group directory, where the log type directory is a directory corresponding to the log type;
[0060] Name the logs to be collected according to the identifier of the target application system;
[0061] Map the named logs to be collected to the log type directory to obtain the mapped logs.
[0062] In a possible implementation, the module is further used to:
[0063] Determine a second file path of the first post source log;
[0064] Determine the log type of the first post source log according to the second file path;
[0065] Determining a message queue group corresponding to a log type of a target application system in a message buffer queue;
[0066] The first post source log is stored in the message queue group to obtain the collection log of the target application system.
[0067] In a third aspect, an embodiment of the present application provides a log collection device, including: a memory, a processor;
[0068] Memory stores computer-executable instructions;
[0069] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.
[0070] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.
[0071] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.
[0072] The log collection method, electronic device, and storage medium provided in the embodiments of the present application establish a shared directory, map the logs to be collected in the target container group to the shared directory of the virtual machine node, and thus facilitate unified management of the logs of the container group. The mapped logs are then collected from the shared directory, and the first source log is obtained based on the container group identifier of the target container group corresponding to the mapped log and the mapped log. Finally, the collection log of the target application system is obtained based on the first source log, thereby achieving the log collection purpose of containerized deployment and the effect of log tracing. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0074] Figure 1 A schematic diagram of the scenario of the log collection method provided in this application;
[0075] Figure 2 Schematic diagram of the log collection method provided for this application Figure 1 ;
[0076] Figure 3 Schematic diagram of the log collection method provided for this application Figure 2 ;
[0077] Figure 4 A schematic diagram of the structure of the log collection method provided by this application;
[0078] Figure 5 A schematic diagram of the structure of the log collection device provided by this application;
[0079] Figure 6 A schematic diagram of the structure of the log collection device provided for this application.
[0080] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0081] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0082] First, the terms involved in this application are explained:
[0083] Application system: refers to a system that provides certain functions or services to the outside world. A bank transaction will be completed through the mutual calls of multiple application systems. For distributed architecture application systems, a system can contain multiple modules (microservices).
[0084] Pod: refers to the smallest unit of K8s scheduling. A pod contains a group of containers. Pod is equivalent to a logical host, and each pod has its own IP address.
[0085] Flume: refers to a highly available, highly reliable distributed massive log collection, aggregation, and transmission system provided by Cloudera.
[0086] Message buffer queue: When used for large amounts of data, the message is placed in the queue first and processed after a certain delay to avoid dragging down subsequent processing logic.
[0087] Transaction log: refers to a non-functional log for monitoring online transactions. It records the entry and exit time, execution status and other information of a transaction scenario on each application system on the path to reflect the health status of each system on the transaction path.
[0088] Monitoring log: refers to the log that records the application system's own operating information (abnormalities, errors), which is used to quickly locate application system problems and troubleshoot the causes of errors.
[0089] Business operation log: refers to the business operation log that records the business information related to the transaction to reflect the relevant business attributes of the transaction. Each application can record business attribute information such as merchant type, card type, transaction type, province, etc. according to its own business characteristics, which is convenient for statistics and analysis from the perspective of business operations.
[0090] Region: refers to the city where the data center is located, such as Beijing, Shanghai, Hefei, etc.
[0091] Availability Zone: refers to different data centers in the same region, such as Beijing Haiying, Beijing Heishanhu, etc.
[0092] Figure 1 A schematic diagram of a scenario of the log collection method provided for this application, such as Figure 1 As shown, the specific application scenario of the present application includes an application system, a virtual machine node, a container group and a server, wherein the application system includes multiple container groups, and the multiple container groups are deployed in different virtual machine nodes. The server is used to map the logs to be collected in the target container group to the shared directory of the virtual machine node to obtain the mapping logs, the target container group is the container group of the target application system, and the target container group is located in the virtual machine node; collect the mapping log from the shared directory; obtain the first paste source log according to the container group identifier and the mapping log of the target container group; obtain the collection log of the target application system according to the first paste source log.
[0093] Combined with the above scenarios, it can be seen that in the existing technology, with the gradual popularization of cloud native technology, large enterprises such as banks are piloting container platforms and containerized deployment of applications, which brings new challenges to the collection and monitoring of application logs. In the traditional virtual machine deployment mode, a log collection tool is deployed on each of the multiple virtual machines deployed by each application system, and the logs can be collected to the message buffer queue for subsequent processing by the monitoring platform. However, in the containerized deployment mode, the application system is no longer deployed on the virtual machine, but deployed in a container group. A cluster contains multiple virtual machine nodes, and each virtual machine node can deploy multiple container groups of application systems. Therefore, it is necessary to study how to collect the logs printed by the application from each container group to the monitoring platform, so as to provide transaction indicator monitoring, link monitoring and log monitoring functions for the application.
[0094] The log collection method provided by the present application maps the logs to be collected in the target container group to the shared directory of the virtual machine node to obtain the mapping logs, and then collects the mapping logs from the shared directory, and obtains the first source log according to the container group identifier and the mapping log of the target container group; and obtains the collection log of the target application system according to the first source log, thereby realizing log collection in the case of containerized deployment.
[0095] It should be noted that the log collection method and device of the present application can be used in the computer field, and can also be used in any field other than the computer field. The application field of the log collection method and device of the present application is not limited.
[0096] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0097] Figure 2 Schematic diagram of the log collection method provided for this application Figure 1 ,like Figure 2 As shown, the method includes:
[0098] S201. Map the logs to be collected in the target container group to the shared directory of the virtual machine node to obtain mapping logs. The target container group is a container group of the target application system, and the target container group is located in the virtual machine node.
[0099] In this embodiment, the target application system for which date collection is required is first determined, and then the target container group of the target application system and the virtual machine node where the target container group is located are determined. Then, a shared directory, which may also be called a shared volume, is established in the virtual machine node, and the log mapping of the target container group is saved in the shared directory to obtain the mapping log. By establishing a shared directory, on the one hand, it is convenient to uniformly manage the logs of the container group, and on the other hand, it is convenient for the server to filter the required logs from the shared directory.
[0100] S202: Collect mapping logs from a shared directory.
[0101] In this embodiment, in order to analyze the target application system, it is necessary to obtain the mapping log stored in the shared directory. Specifically, the mapping log can be collected from the shared directory through a collection probe.
[0102] S203: For each mapping log, obtain a first posting source log according to the container group identifier of the target container group corresponding to the mapping log and the mapping log.
[0103] In this embodiment, after the required mapping log is collected, it is also necessary to collect the container group identifier of the target container group to identify the container group source of the mapping log, and then the first source log is obtained by adding the container group identifier of the target container group to the field of the mapping log, so as to identify the container group that generates the mapping log.
[0104] S204: Obtain the collection log of the target application system according to the first source log.
[0105] In this embodiment, after the mapping log is converted into the first post source log, the first post source log is used as the collection log.
[0106] The log collection method provided in the embodiment of the present application maps the logs to be collected in the target container group to the shared directory of the virtual machine node by establishing a shared directory, so as to facilitate unified management of the logs of the container group, collect the mapping log from the shared directory, and obtain the first source log according to the container group identifier and the mapping log of the target container group corresponding to the mapping log, and finally obtain the collection log of the target application system according to the first source log, so as to achieve the log collection purpose of containerized deployment and the effect of log tracing.
[0107] Figure 3 Schematic diagram of the log collection method provided for this application Figure 2 ,like Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, the log collection method is described in detail, and the method includes:
[0108] S301: Map the logs to be collected in the target container group to the shared directory of the virtual machine node to obtain the mapping logs.
[0109] S302: Collect mapping logs from a shared directory.
[0110] Specifically, collecting mapping logs from a shared directory may include:
[0111] Determine the collection probe in the virtual machine node. The collection probe is a Flume probe.
[0112] According to the collection probe, the mapping log to be identified is obtained from the virtual machine node;
[0113] The mapping log is determined according to the log name of each mapping log to be identified, wherein the log name includes the identifier of the target application system.
[0114] In this embodiment, it can be determined whether there is a collection probe in the virtual machine node. If there is no collection probe in the virtual machine node, a new collection probe is created in the virtual machine node, and the collection probe can be a Flume probe. Then, the mapping log to be identified in the shared directory is obtained through the collection probe, and the target application system corresponding to the log to be identified is determined by naming the log of the mapping log to be identified, and then the mapping log stored in the shared directory by the target container group is determined. If there are multiple mapping logs stored in the shared directory by the target container group, the naming of the mapping log can also include time to distinguish the mapping logs of the same container group.
[0115] Specifically, the logs to be collected in the target container group are mapped to the shared directory of the virtual machine node to obtain the mapping logs, which may include:
[0116] Determine the shared directory in the virtual machine node, and mount the shared directory in the target container group;
[0117] When the target container group performs log mapping, the container group directory in the shared directory is determined, and the container group directory is the directory corresponding to the target container group;
[0118] Map the logs to be collected to the container group directory to obtain the mapping logs.
[0119] In this embodiment, it is first necessary to determine whether there is a shared directory in the virtual machine node. If there is no shared directory in the virtual machine node, a shared directory is established in the virtual machine node and the shared directory is mounted in the target container group to facilitate log mapping by the target container group. When the target container group performs log mapping, it is necessary to determine whether there is a container group directory in the shared directory. The container group directory corresponds to the target container group and is used to store the mapping log of the target container group. If there is no container group directory in the shared directory, it is necessary to establish a container group directory in the shared directory. The target container group maps the log to the container group directory for further classification of the log. Among them, the container group directory may include the identifier of the container group.
[0120] Specifically, the logs to be collected are mapped to the container group directory to obtain the mapping logs, which may include:
[0121] Determine the log types to be collected, which include flow logs, monitoring logs, and business operation logs;
[0122] According to the log type, determine the log type directory in the container group directory, where the log type directory is a directory corresponding to the log type;
[0123] Name the logs to be collected according to the identifier of the target application system;
[0124] Map the named logs to be collected to the log type directory to obtain the mapped logs.
[0125] In this embodiment, the log also includes different log types, and the log types may include flow logs, monitoring logs, and business operation logs. In order to further distinguish different log types, when performing log mapping on the target container group, a log type directory may be established in the container group directory to facilitate classifying the logs of the target container group according to the log type. When collecting logs from the shared directory, log collection may be performed separately according to the log type, thereby increasing the efficiency and flexibility of log collection.
[0126] S303: According to the mapping log, obtain the first file path of the mapping log.
[0127] In this embodiment, the directories and file names of the mapping logs can be obtained through the first file path of the mapping log. An example of the first file path is: virtual machine node:\shared directory\target container group directory\running log directory\target application system mapping log.
[0128] S304: Obtain a container group identifier of the mapping log according to the container group directory in the first file path, where the container group directory is used to identify a target container group where the mapping log is located.
[0129] In this embodiment, in order to obtain the container group identifier from the first file path, the container group identifier may be used as the name of the container group directory, so that the server can read the field in the first file path to obtain the container group identifier.
[0130] S305: Perform source processing on the mapping log according to the container group identifier to obtain a first source log.
[0131] In this embodiment, source pasting processing may refer to adding a container group identifier in the mapping log. For example, assuming that the mapping log has 10 fields, source pasting processing is to add certain fields on the basis of the mapping log, such as adding the container group to which the log belongs, the available zone, etc., to supplement the log information for subsequent monitoring.
[0132] S306: Obtain the collection log of the target application system according to the first post source log.
[0133] Specifically, obtaining the collection log of the target application system according to the first post source log may include:
[0134] Determine a second file path of the first post source log;
[0135] Determine the log type of the first post source log according to the second file path;
[0136] Determine a message queue group corresponding to a log type of a target application system in a message buffer queue;
[0137] The first post source log is stored in the message queue group to obtain the collection log of the target application system.
[0138] In this embodiment, the second file path may be the same as or different from the first file path, as long as the second file path includes the log type of the first source log. For example, after the mapping log is processed, the storage location of the first source log is changed, but the first source log is still stored according to the log type. After obtaining the log type of the first source log, the message queue group corresponding to the log type of the target application system is determined in the message buffer queue. For example, assuming that the log type of the first source log is a running log, the target application system group is searched in the message buffer queue. If not, the target application system group is established, and then a running log group is established in the target application system group, and then the first source log is stored in the running log group.
[0139] In some implementations, the present application may further include:
[0140] Get collection logs from message queue groups;
[0141] According to the log parsing rules of the target application system, the collected logs are parsed to obtain parsed logs;
[0142] Based on the parsed logs, the monitoring indicators of the application system are obtained, including transaction volume, response time and success rate;
[0143] Monitor the target application system according to the monitoring indicators.
[0144] In this embodiment, after the collection log is stored in the message buffer queue, the collection log needs to be analyzed, so the collection log needs to be obtained from the message queue group. For example, the collection log can be read from the message queue group through the monitoring platform. After obtaining the collection log, the preset log parsing rules of the target application system are obtained, and then the collection log is parsed according to the log parsing rules to obtain the parsed log, and then the monitoring index of the target application system is obtained according to the parsed log. Finally, the target application system is monitored according to the monitoring index. If the monitoring index is in the alarm range, the abnormal log is located, and the container group corresponding to the abnormal log is determined according to the container group identifier of the abnormal log, providing support information for troubleshooting.
[0145] In some implementations, the present application may further include:
[0146] Obtain the region information of the virtual machine node and the deployment unit information of the first post source log, where the region information includes the region and availability zone;
[0147] Obtaining a second post source log according to the area information, the deployment unit information, and the first post source log;
[0148] According to the second source log, the collection log of the target application system is obtained.
[0149] In this embodiment, the region information of the virtual machine node and the deployment unit information of the first source log can also be pasted into the first source log to obtain a second source log, so that the region information and deployment unit information can be located according to the second source log, which facilitates multi-dimensional analysis of the collected logs and provides multi-dimensional monitoring purposes.
[0150] Among them, the method for obtaining the deployment unit information may include: when parsing the collection log, the server connects to the CMDB configuration management system, obtains the deployment unit information according to the deployment group ID and available zone information, and further sources the unit field to the first source log.
[0151] For example, Figure 4 The structural diagram of the log collection method provided by this application is as follows: Figure 4 As shown, the structure includes a K8S cluster, Kakfa (message buffer queue) and a monitoring platform, wherein the K8S cluster includes multiple nodes (virtual machine nodes), with node1 and node2 as examples in the figure, each node including pod1-podN (container group 1-container group N), a shared volume (or shared directory) and a Flume pod (collection probe); Kakfa includes multiple topic1-topicN (message queue group 1-message queue group N); the monitoring platform includes parsing, computing, storage and display modules.
[0152] Each application system packages the application version into an image and deploys it in a pod through the K8S platform, forming a Figure 4 pod1-podN on node1 and node2
[0153] When the pod of the application system is started, the shared volume of the node is mounted to the pod in the form of hostpath (host path), so that the logs in the container are mapped to the node for persistence and the Flume pod can collect logs from the shared volume. When mapping directories, the mapped paths and log file names must be defined in accordance with the specifications: the three types of logs should be stored in the flow (flow log), monitor (monitoring log), and operation (business operation log) directories under the logs (log) directory, respectively, and named {application system identifier}.log, where the application system identifier should be all capitalized. For example, T-POMP is an application system.
[0154] On each node, a Flume pod is deployed in DaemonSet (a process deployment method). The Flume pod can identify the application system information (such as T-POMP) to which the log belongs from the node shared volume mapped by each pod, and deliver the collected logs to the corresponding topic of Kakfa (each log type of each application system in Kafka is divided into different topics).
[0155] When pod maps logs in pod to node shared volumes, it is necessary to add a podname (container group identifier) level in the directory. After Flume pod collects logs, it extracts the podname field from the path and pastes the source into the collected original log to identify which pod the log comes from, so as to facilitate subsequent monitoring and location of problems.
[0156] Source processing based on region and availability zone: Flume pod further adds source region and availability zone information to the original log according to the region and availability zone of the node where the probe is deployed (the region and availability zone information is pre-written in the Flume configuration file) so that the monitoring platform can subsequently distinguish which availability zone the log comes from and calculate the monitoring indicators in different availability zones.
[0157] The log parsing module of the monitoring platform consumes the collected logs from different topics of Kakfa, and parses the logs into structured data according to the log parsing rules configured in different application systems. When parsing the logs, the CMDB configuration management system is connected to obtain the deployment unit information based on the deployment group ID and availability zone information, and further pastes the unit field into the log.
[0158] The calculation module of the monitoring platform calculates the transaction volume, response time, and success rate of different application systems based on the parsed logs (the calculation indicators are mainly based on the flow logs) to provide monitoring and alarm related functions. The calculated indicators can be counted by different dimensions such as region, availability zone, unit, podname, etc.
[0159] The log collection method provided by the embodiment of the present application maps the logs to be collected in the target container group to the shared directory of the virtual machine node to obtain the mapping logs, collects the mapping logs from the shared directory, obtains the first file path of the mapping logs according to the mapping logs, obtains the container group identifier of the mapping logs according to the container group directory in the first file path, the container group directory is used to identify the target container group where the mapping logs are located, performs source processing on the mapping logs according to the container group identifier, obtains the first source logs, and finally obtains the collection logs of the target application system according to the first source logs. The purpose of log collection for containerized deployment is achieved, and the effect of log tracing is achieved.
[0160] Figure 5 A schematic diagram of the structure of the log collection device provided in this application, such as Figure 5 As shown, the log collection device 50 provided in this embodiment includes:
[0161] A mapping module 501 is used to map the logs to be collected in the target container group to the shared directory of the virtual machine node to obtain mapping logs. The target container group is the container group of the target application system, and the target container group is located in the virtual machine node.
[0162] A collection module 502 is used to collect mapping logs from a shared directory;
[0163] A post source module 503, configured to obtain a first post source log according to the container group identifier of the target container group and the mapping log;
[0164] The obtaining module 504 is used to obtain the collection log of the target application system according to the first post source log.
[0165] In a possible implementation, the source module 503 is further used for:
[0166] According to the mapping log, obtain the first file path of the mapping log;
[0167] According to the container group directory in the first file path, obtain the container group identifier of the mapping log, where the container group directory is used to identify the target container group where the mapping log is located;
[0168] According to the container group identifier, the mapping log is processed to obtain a first source log.
[0169] In a possible implementation, the collection module 502 is further configured to:
[0170] Determine the collection probe in the virtual machine node. The collection probe is a Flume probe.
[0171] According to the collection probe, the mapping log to be identified is obtained from the virtual machine node;
[0172] The mapping log is determined according to the log name of each mapping log to be identified, wherein the log name includes the identifier of the target application system.
[0173] In a possible implementation, the mapping module 501 is further configured to:
[0174] Determine the shared directory in the virtual machine node, and mount the shared directory in the target container group;
[0175] When the target container group performs log mapping, the container group directory in the shared directory is determined, and the container group directory is the directory corresponding to the target container group;
[0176] Map the logs to be collected to the container group directory to obtain the mapping logs.
[0177] In a possible implementation, the mapping module 501 is further configured to:
[0178] Determine the log types to be collected, which include flow logs, monitoring logs, and business operation logs;
[0179] According to the log type, determine the log type directory in the container group directory, where the log type directory is a directory corresponding to the log type;
[0180] Name the logs to be collected according to the identifier of the target application system;
[0181] Map the named logs to be collected to the log type directory to obtain the mapped logs.
[0182] In a possible implementation, the obtaining module 504 is further used for:
[0183] Determine a second file path of the first post source log;
[0184] Determine the log type of the first post source log according to the second file path;
[0185] Determining a message queue group corresponding to a log type of a target application system in a message buffer queue;
[0186] The first post source log is stored in the message queue group to obtain the collection log of the target application system.
[0187] The log collection device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.
[0188] Figure 6 This is a schematic diagram of the structure of the log collection device provided by this application. Figure 6 As shown, the electronic device 60 provided in this embodiment includes: at least one processor 601 and a memory 602. Optionally, the device 60 also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.
[0189] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.
[0190] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0191] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0192] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.
[0193] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.
[0194] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0195] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.
[0196] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.
[0197] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0198] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.
[0199] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0200] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0201] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0202] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0203] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A log collection method, characterized in that: include: Mapping the logs to be collected in the target container group to the shared directory of the virtual machine node to obtain mapping logs, wherein the target container group is a container group of the target application system and the target container group is located in the virtual machine node; Collect the mapping log from the shared directory; Obtaining a first post source log according to the container group identifier of the target container group and the mapping log; According to the first post source log, a collection log of the target application system is obtained.
2. The method according to claim 1, characterized in that The obtaining the first posting source log according to the container group identifier of the target container group and the mapping log includes: According to the mapping log, obtaining a first file path of the mapping log; Acquire a container group identifier of the mapping log according to the container group directory in the first file path, where the container group directory is used to identify a target container group where the mapping log is located; According to the container group identifier, the mapping log is processed as a source to obtain a first source log.
3. The method according to any one of claims 1-2, characterized in that: The collecting the mapping log from the shared directory includes: Determine a collection probe in the virtual machine node, where the collection probe is a Flume probe; According to the collection probe, obtaining the mapping log to be identified from the virtual machine node; The mapping log is determined according to the log name of each mapping log to be identified, wherein the log name includes the identifier of the target application system.
4. The method according to claim 3, characterized in that The step of mapping the logs to be collected in the target container group to the shared directory of the virtual machine node to obtain the mapping logs includes: Determine a shared directory in the virtual machine node, wherein the shared directory is mounted in the target container group; When the target container group performs log mapping, determining a container group directory in the shared directory, the container group directory being a directory corresponding to the target container group; The log to be collected is mapped to the container group directory to obtain a mapping log.
5. The method according to claim 4, characterized in that The step of mapping the log to be collected to the container group directory to obtain a mapping log includes: Determine the log type of the log to be collected, the log type includes a running log, a monitoring log, and a business operation log; According to the log type, determine a log type directory in the container group directory, the log type directory being a directory corresponding to the log type; Naming the log to be collected according to the identifier of the target application system; The named log to be collected is mapped to the log type directory to obtain a mapping log.
6. The method according to any one of claims 1-2, characterized in that: The step of obtaining the collection log of the target application system according to the first post source log includes: Determine a second file path of the first post source log; Determine the log type of the first post source log according to the second file path; Determining, in a message buffer queue, a message queue group corresponding to the log type of the target application system; The first post source log is stored in the message queue group to obtain the collection log of the target application system.
7. The method according to any one of claims 1-2, characterized in that: The method further comprises: Obtaining the collection log from the message queue group; Parsing the collected log according to the log parsing rule of the target application system to obtain a parsed log; Obtaining monitoring indicators of the target application system according to the parsed log, wherein the monitoring indicators include transaction volume, response time and success rate; The target application system is monitored according to the monitoring indicators.
8. The method according to any one of claims 1-2, characterized in that: The method further comprises: Obtaining the region information of the virtual machine node and the deployment unit information of the first post source log, wherein the region information includes a region and an availability zone; Obtaining a second post source log according to the area information, the deployment unit information and the first post source log; According to the second post source log, the collection log of the target application system is obtained.
9. A log collection device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.