Container log collection management method and device, equipment and storage medium

Abstract

The application relates to the technical field of data processing, and discloses a container log collection management method, device and equipment and a storage medium. The method comprises the following steps: determining container log data according to a log path of a container, collecting logs of the container according to the container log data, and obtaining original container logs; performing desensitization processing on the original container logs to obtain initial log data; generating target log data according to the initial log data and a log configuration file corresponding to the initial log data; storing the target log data into a message middleware, classifying the target log data, and obtaining container logs of different types; and calling a log storage request according to the container logs of different types, and writing log information in the log storage request into a corresponding preset log file. The application supports container-based collection and satisfies different collection modes, reduces the complexity of log query, and reduces the operation and maintenance threshold.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a container log collection and management method, apparatus, device, and storage medium. Background Technology

[0002] With the increasing trend of system containerization, from simply moving to the cloud to containerization, the original log access and collection methods based on cloud hosts can no longer meet the needs of container logs.

[0003] Currently, some products and technologies in the industry can solve the log querying and management problems of small platforms and small cluster systems. There are also unified log collection and management platforms based on non-container systems on cloud servers. However, there is still no container-based solution or tool that can support log querying and management across multiple platforms and multiple different middleware clusters. This situation results in a high barrier to entry for beginners, complex integration, and is not conducive to quickly locating production problems. Summary of the Invention

[0004] The main objective of this invention is to reduce the complexity of log queries and lower the operational threshold by supporting container-based methods and meeting different data collection requirements.

[0005] The first aspect of this invention provides a container log collection and management method, comprising: determining container log data according to the container's log path; collecting logs from the container according to the container log data to obtain raw container logs; performing de-identification processing on the raw container logs to obtain initial log data; generating target log data according to the initial log data and a log configuration file corresponding to the initial log data; storing the target log data in a message middleware and classifying the target log data to obtain different types of container logs; retrieving log storage requests according to the different types of container logs, and writing the log information in the log storage requests to corresponding preset log files.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the step of determining container log data based on the container's log path and collecting logs from the container based on the container log data to obtain raw container logs includes: determining container log data based on the container's log path; matching corresponding log parsing rules from a preset log collection template library based on the container log data; generating configuration parameters corresponding to the container log data based on the log parsing rules; and collecting logs from the container based on the configuration parameters to obtain raw container logs.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the step of de-identifying the original container logs to obtain initial log data includes: obtaining the identifier ID of the original container logs, and obtaining the log de-identification rule corresponding to the original container logs from preset log de-identification configuration parameters based on the identifier ID; and de-identifying the original container logs according to the log de-identification rule to obtain initial log data.

[0008] Optionally, in a third implementation of the first aspect of the present invention, generating target log data based on the initial log data and the log configuration file corresponding to the initial log data includes: determining whether the initial log data matches the business feature field in the preset log configuration file; if so, storing the successfully matched initial log data to generate the target log data.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, storing the target log data in a message middleware and classifying the target log data to obtain different types of container logs includes: storing the logs in a message middleware cluster according to the identifier ID of the container logs; determining the attribute information corresponding to the target log data; and classifying the target log data according to the attribute information to obtain different types of container logs.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the step of writing the log information in the log storage request to the corresponding preset log file includes: parsing the log storage request to obtain the log information carried in the log storage request; clustering the log information according to a preset business type to obtain log information clusters of the same business type; setting a log folder according to the business type, and storing the log information clusters in the corresponding log folders.

[0011] A second aspect of the present invention provides a container log collection and management device, comprising: a collection module, configured to determine container log data according to the container's log path, and collect logs from the container according to the container log data to obtain raw container logs; a desensitization module, configured to desensitize the raw container logs to obtain initial log data; a generation module, configured to generate target log data according to the initial log data and a log configuration file corresponding to the initial log data; a classification module, configured to store the target log data in a message middleware and classify the target log data to obtain different types of container logs; and a writing module, configured to retrieve log storage requests according to the different types of container logs and write the log information in the log storage requests to corresponding preset log files.

[0012] Optionally, in a first implementation of the second aspect of the present invention, the acquisition module is specifically used for: determining container log data based on the container's log path; matching corresponding log parsing rules from a preset log acquisition template library based on the container log data; generating configuration parameters corresponding to the container log data based on the log parsing rules; and acquiring logs from the container based on the configuration parameters to obtain the original container logs.

[0013] Optionally, in a second implementation of the second aspect of the present invention, the desensitization module is specifically used to: obtain the identifier ID of the original container log, and obtain the log desensitization rule corresponding to the original container log from the preset log desensitization configuration parameters based on the identifier ID; and perform desensitization processing on the original container log according to the log desensitization rule to obtain initial log data.

[0014] Optionally, in a third implementation of the second aspect of the present invention, the generation module is specifically used to: determine whether the initial log data matches the business feature field in the preset log configuration file; if so, store the successfully matched initial log data and generate target log data.

[0015] Optionally, in a fourth implementation of the second aspect of the present invention, the classification module includes: a storage unit, configured to store the log in a message middleware cluster according to the identifier ID of the container log; a determination unit, configured to determine the attribute information corresponding to the target log data; and a classification unit, configured to classify the target log data according to the attribute information to obtain different types of container logs.

[0016] Optionally, in a fifth implementation of the second aspect of the present invention, the writing module is specifically used to: parse the log storage request to obtain the log information carried in the log storage request; cluster the log information according to a preset business type to obtain log information clusters of the same business type; set a log folder according to the business type, and store the log information clusters in the corresponding log folders.

[0017] A third aspect of the present invention provides a container log collection and management device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line;

[0018] The at least one processor invokes the instructions in the memory to cause the container log collection and management device to execute the various steps of the container log collection and management method described above.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the various steps of the container log collection and management method described above.

[0020] The technical solution provided by this invention determines container log data based on the container's log path, collects logs from the container based on the container log data to obtain raw container logs, performs anonymization processing on the raw container logs to obtain initial log data, generates target log data based on the initial log data and the corresponding log configuration file, stores the target log data in a message middleware, and classifies the target log data to obtain different types of container logs, retrieves log storage requests based on different types of container logs, and writes the log information in the log storage requests to the corresponding preset log files. This invention reduces the complexity of log querying and lowers the operational threshold by supporting container-based methods and satisfying different collection methods. Attached Figure Description

[0021] Figure 1 A schematic diagram of the first embodiment of the container log collection and management method provided by the present invention;

[0022] Figure 2 A schematic diagram of a second embodiment of the container log collection and management method provided by the present invention;

[0023] Figure 3 A schematic diagram of the third embodiment of the container log collection and management method provided by the present invention;

[0024] Figure 4 A schematic diagram of the first embodiment of the container log collection and management device provided by the present invention;

[0025] Figure 5 A schematic diagram of a second embodiment of the container log collection and management device provided by the present invention;

[0026] Figure 6 This is a schematic diagram of an embodiment of the container log collection and management device provided by the present invention. Detailed Implementation

[0027] This invention provides a container log collection and management method, apparatus, device, and storage medium. The technical solution of this invention first determines container log data based on the container's log path, then collects logs from the container based on the container log data to obtain raw container logs; the raw container logs are then anonymized to obtain initial log data; target log data is generated based on the initial log data and the corresponding log configuration file; the target log data is stored in a message middleware and classified to obtain different types of container logs; log storage requests are retrieved according to different types of container logs, and the log information in the log storage requests is written to the corresponding preset log files. This invention reduces the complexity of log queries and lowers the operational threshold by supporting container-based methods and satisfying different collection methods.

[0028] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the container log collection and management method in this invention includes:

[0030] 101. Determine the container log data based on the container's log path, and collect logs from the container based on the container log data to obtain the raw container logs;

[0031] In this embodiment, container log data may include: container ID, type, name, log path, etc.

[0032] Specifically, when creating a new container, the new container is used as the target container; the log path is determined based on the target container's ID; the target container's ID, container information, and log path are combined to obtain the container log data.

[0033] In this embodiment, based on the container log data, the corresponding log parsing rules are matched from a preset log collection template library. This log collection template library primarily stores log parsing rules for various types of containers.

[0034] 102. De-identify the original container logs to obtain the initial log data;

[0035] In this embodiment, the log anonymization configuration parameters are obtained from the log configuration center and include the log configuration type and the log anonymization specification corresponding to each log configuration type. The log configuration center is a management center that manages all log-related configurations through permission management. The log configuration center includes log anonymization configuration parameters, a central configuration policy file, and a sensitive data policy file. The log anonymization configuration parameters can be obtained directly from the log configuration center, or by comparing the local log anonymization configuration parameters with a non-local (i.e., online or platform) central configuration policy file, downloading the latest version of the central configuration policy file and overwriting the original. Maintenance personnel only need to modify the central configuration policy file to update all log anonymization configuration parameters.

[0036] Specifically, the process of desensitizing the original container logs in conjunction with the identifier ID is as follows: based on the identifier ID, a masking code generation algorithm is used to generate a masking code corresponding to the identifier ID; the generated masking code is then masked according to the corresponding desensitization rules to generate the desensitized log file.

[0037] 103. Generate target log data based on the initial log data and the corresponding log configuration file;

[0038] In this embodiment, it is determined whether the initial log data matches the business feature fields in the preset log configuration file. Specifically, by determining whether the initial log data in the initial log file matches the business feature fields in the preset log configuration file, the initial log data that successfully matches (i.e., corresponds to the business feature fields) in the initial log file is extracted. It is then determined whether the creation time of the target log data is within the current collection period in the log configuration file. If so, the target log data is output when the current collection period ends.

[0039] 104. Store the target log data in the message middleware and classify the target log data to obtain different types of container logs;

[0040] In this embodiment, logs are stored in the message middleware cluster based on their identifier IDs; the attribute information corresponding to the target log data is determined; and the target log data is classified according to the attribute information to obtain different types of container logs. Specifically, when the log access layer obtains logs from the business system, it sends the obtained logs to the log buffer layer; during the process of storing logs in the message middleware cluster, the log level identifier can be determined, and logs belonging to the same level are stored in a specific storage area within the message middleware cluster.

[0041] The logs include transaction logs and business logs. Attribute information is uniquely identifying the log, and can be represented by strings, which may include letters and / or numbers. For example, the attribute information for a transaction log may be a transaction serial number, and the attribute information for a business log may be a business tag. In this embodiment, different types of logs can be understood as logs corresponding to different transactions, and container logs corresponding to different businesses.

[0042] 105. Retrieve log storage requests based on different types of container logs, and write the log information in the log storage requests to the corresponding preset log files.

[0043] In this embodiment, the log storage request is parsed to obtain the log information carried in the log storage request; the log information is clustered according to the preset business type to obtain log information clusters of the same business type; a log folder is set according to the business type, and the log information clusters are stored in the corresponding log folders.

[0044] Specifically, corresponding preset log files can be pre-created in the log server according to the business types existing in the application system, so that log information of the same business type is stored in the same preset log file. The log server can receive log storage requests sent by business servers, where the log storage request includes log information and business name.

[0045] In this embodiment, the log server determines the business type of the log storage request based on the business name in the request. For example, order processing corresponds to the order business, and nickname modification corresponds to the user management business. Then, the log storage request is distributed to the corresponding log storage queue according to the business type. The log server retrieves the log information from the log storage request from each log storage queue through a pre-created thread and writes it to the corresponding preset log file. This allows log information of the same business type to be stored in the same log file, enabling unified collection and management of logs from distributed business servers.

[0046] In this embodiment of the invention, container log data is determined based on the container's log path; logs are collected from the container based on the container log data to obtain raw container logs; the raw container logs are anonymized to obtain initial log data; target log data is generated based on the initial log data and the corresponding log configuration file; the target log data is stored in a message middleware and classified to obtain different types of container logs; log storage requests are retrieved according to different types of container logs, and the log information in the log storage requests is written to the corresponding preset log files. This invention reduces the complexity of log queries and lowers the operational threshold by supporting container-based methods and satisfying different collection methods.

[0047] Please see Figure 2 The second embodiment of the container log collection and management method in this invention includes:

[0048] 201. Determine the container log data based on the container's log path;

[0049] In this embodiment, container log data may include: container ID, type, name, log path, etc.

[0050] Specifically, when creating a new container, the new container is used as the target container; the log path is determined based on the target container's ID; the target container's ID, container information, and log path are combined to obtain the container log data.

[0051] In this embodiment, when a new container is created, the new container is used as the target container. The log path is determined based on the target container's ID. The target container's ID, container information, and log path are combined to obtain the container log data. In other words, this optional solution mainly achieves the dynamic generation of corresponding configuration files for newly created containers by automatically generating the corresponding configuration files.

[0052] In this embodiment, a list of all containers is obtained, and the containers in the list are used as target containers. The corresponding log path is determined based on the ID of each target container. The ID, container information, and log path of each target container are combined to obtain the corresponding container log information. In other words, this embodiment enables the automatic generation of corresponding configuration files for multiple containers.

[0053] 202. Based on the container log data, match the corresponding log parsing rules from the preset log collection template library;

[0054] In this embodiment, the log collection template library mainly stores log parsing rules for various types of containers. It can be that log parsing rules for different types of containers are summarized during daily use to obtain this log collection template library, and then the log parsing rules corresponding to the current target container are determined by matching the types.

[0055] Therefore, in this embodiment, the log parsing rules can be matched with the type in the container log information.

[0056] 203. Generate configuration parameters corresponding to the container log data according to the log parsing rules;

[0057] In this embodiment, the configuration file mainly specifies the parsing method used to obtain the corresponding log data. Therefore, once the log parsing rules are determined, the corresponding configuration parameters can be generated. Furthermore, this step can generate the corresponding configuration parameters based on the format of the data collection service.

[0058] 204. Collect container logs according to the configuration parameters to obtain the raw container logs;

[0059] In this embodiment, the fluentd service is used to collect logs from the target container according to the configuration file. Specifically, the fluentd service is used to collect logs from the target container according to the configuration file. Fluentd is a log management tool that simplifies log collection, processing, and storage, eliminating the need for technicians to maintain and write special log processing scripts.

[0060] Furthermore, this embodiment may also include restarting the fluentd service when the configuration file is updated. That is, the new configuration file can only be applied whenever the fluentd service restarts, enabling dynamic management of the log collection process and improving log collection efficiency.

[0061] 205. Obtain the identifier ID of the original container log, and obtain the log desensing rule corresponding to the original container log from the preset log desensing configuration parameters based on the identifier ID;

[0062] In this embodiment, the log anonymization configuration parameters are obtained from the log configuration center and include the log configuration type and the log anonymization specification corresponding to each log configuration type. The log configuration center is a management center that manages all log-related configurations through permission management. The log configuration center includes log anonymization configuration parameters, a central configuration policy file, and a sensitive data policy file. The log anonymization configuration parameters can be obtained directly from the log configuration center, or by comparing the local log anonymization configuration parameters with a non-local (i.e., online or platform) central configuration policy file, downloading the latest version of the central configuration policy file and overwriting the original. Maintenance personnel only need to modify the central configuration policy file to update all log anonymization configuration parameters.

[0063] Furthermore, the log de-identification configuration parameters include multiple log configuration types and log de-identification specifications corresponding one-to-one with each log configuration type. The log configuration type is the type of various business in the application, such as application type, risk control type, customer type, etc. The log de-identification specification is the masking algorithm, de-identification type, and de-identification rules set for the corresponding log type.

[0064] 206. According to the log de-identification rules, perform de-identification processing on the original container logs to obtain the initial log data;

[0065] In this embodiment, the process of desensitizing the original container logs using the identifier ID involves generating a masking code corresponding to the identifier ID using a masking code generation algorithm, masking the generated masking code according to the corresponding desensitization rules, and generating the desensitized log file. The log desensitization specification includes the masking code generation algorithm, desensitization types, and desensitization rules corresponding to each desensitization type. For example, desensitization types include ID card desensitization types, email desensitization types, and bank card desensitization types, etc. The desensitization rules include: for the ID card desensitization type, "do not display the ID card number, use a masking code for desensitization"; for the email desensitization type, "use a masking code to desensitize both the username and domain name of the email address (e.g., abcdefg@company.com uses a masking code "X" to convert it to "XXXXXXX@XXXXXXX.com")"; for the bank card desensitization type, "display the last four digits of the bank card number, and use a masking code to desensitize other characters," etc.

[0066] The masking code generation algorithm can be set according to requirements. For example, the masking code generation algorithm can be a method that converts the identifier ID into ASCII code, then accumulates and averages the values ​​to obtain the mean value, and then maps the number of identifier IDs and the mean value to a masking code corresponding to a graphic or character. After using the masking code generation algorithm to generate the de-identified log file, the file is stored locally and printed for subsequent operations by relevant organizations.

[0067] 207. Determine whether the initial log data matches the business characteristic fields in the preset log configuration file;

[0068] In this embodiment, by determining whether the initial log data in the initial log file matches the business feature fields in the preset log configuration file, the initial log data that matches successfully (i.e., corresponds to the business feature fields) in the initial log file is extracted and stored according to the preset data storage format in the log configuration file to obtain the target log data for subsequent log data preprocessing, thereby improving the efficiency and accuracy of subsequent data preprocessing.

[0069] 208. If so, store the successfully matched initial log data and generate the target log data;

[0070] In this embodiment, it is determined whether the creation time of the target log data is within the current collection period in the log configuration file. If so, the target log data is output when the current collection period ends.

[0071] Optionally, the monitoring interval can be set to 10 minutes. When triggered, the target log data (including rolling logs) of each corresponding level in the log configuration file will be calculated, and the number of logs collected and sent and the total number of logs will be registered according to historical records and output to subsequent nodes in the distributed system.

[0072] Furthermore, it is determined whether the difference in the amount of the target log data in two adjacent current collection periods in the log configuration file exceeds the fluctuation threshold. If so, the target log data is cached and a log anomaly alarm signal is output.

[0073] 209. Store the target log data in the message middleware and classify the target log data to obtain different types of container logs;

[0074] 210. Retrieve log storage requests based on different types of container logs, and write the log information in the log storage requests to the corresponding preset log files.

[0075] Steps 209-210 in this embodiment are similar to steps 104-105 in the first embodiment, and will not be described again here.

[0076] In this embodiment of the invention, container log data is determined based on the container's log path; logs are collected from the container based on the container log data to obtain raw container logs; the raw container logs are anonymized to obtain initial log data; target log data is generated based on the initial log data and the corresponding log configuration file; the target log data is stored in a message middleware and classified to obtain different types of container logs; log storage requests are retrieved according to different types of container logs, and the log information in the log storage requests is written to the corresponding preset log files. This invention reduces the complexity of log queries and lowers the operational threshold by supporting container-based methods and satisfying different collection methods.

[0077] Please see Figure 3 The third embodiment of the container log collection and management method in this invention includes:

[0078] 301. Determine the container log data based on the container's log path, and collect logs from the container based on the container log data to obtain the raw container logs;

[0079] 302. De-identify the original container logs to obtain the initial log data;

[0080] 303. Generate target log data based on the initial log data and the log configuration file corresponding to the initial log data;

[0081] 304. Store the logs in the message middleware cluster according to the container log identifier ID;

[0082] In this embodiment, when the log access layer obtains logs from the business system, it sends the obtained logs to the log buffer layer. The log buffer layer contains a message middleware cluster. The log access layer, based on the API (Application Programming Interface) provided by the message middleware, sends the obtained logs to the message middleware cluster via HTTP (Hypertext Transfer Protocol) and stores the obtained logs in the message middleware cluster. After the message middleware cluster in the log buffer layer stores the logs, the log processing layer retrieves the logs from the message middleware cluster, classifies the retrieved logs, and obtains different types of logs. It should be noted that, to facilitate horizontal scaling of the message middleware cluster, the architecture of the message middleware cluster is set to a cross-IDC distributed architecture. Through the message middleware cluster, a throughput of 20,000 messages per second can be achieved on a regular server, realizing high throughput. To save storage space in the message middleware cluster, the log storage time in the message middleware cluster is set to a preset time period. When the storage time of logs in the message middleware cluster exceeds this preset time period, the message middleware cluster deletes the stored logs.

[0083] 305. Determine the attribute information corresponding to the target log data;

[0084] In this embodiment, during the process of storing logs to the message middleware cluster, the log's level identifier can be determined, and logs belonging to the same level can be stored in a specific storage area within the message middleware cluster. It should be noted that the business system has already added a level identifier to the log when generating it. Therefore, during the process of determining the log level, the level of the log can be determined through the level identifier contained within it.

[0085] The log processing layer retrieves logs stored in the message middleware cluster. The log processing layer consists of two sub-components: Log-Importer and the log analysis engine Log-Analyzer. When the Log-Importer in the log processing layer retrieves logs, it determines the corresponding attribute information for those logs.

[0086] 306. Based on the attribute information, classify the target log data to obtain different types of container logs;

[0087] In this embodiment, logs are categorized based on their attribute information to obtain different types of logs. These logs include transaction logs and business logs. Attribute information is uniquely identifying the log and can be represented by strings, which may include letters and / or numbers. For example, the attribute information for a transaction log may be a transaction serial number, and the attribute information for a business log may be a business tag. In this embodiment, different types of logs can be understood as logs corresponding to different transactions and container logs corresponding to different businesses.

[0088] The Log-Importer and Log-Analyzer components in the log processing layer can retrieve the required logs from the corresponding storage areas in the message middleware cluster when obtaining logs from the log buffer layer, based on business needs. For example, if the function of Log-Importer is to store all logs obtained by the log access layer, then Log-Importer will monitor all storage areas of the message middleware cluster to obtain all logs stored in the message middleware cluster; if Log-Analyzer needs to analyze the log information of the Nginx proxy, it only needs to monitor the storage area where the corresponding logs of the Nginx proxy system are located.

[0089] Furthermore, before classifying the logs, a cleaning operation can be performed to remove unnecessary fields from the logs.

[0090] 307. Parse the log storage request to obtain the log information carried in the log storage request;

[0091] In this embodiment, corresponding preset log files can be pre-created in the log server according to the business types existing in the application system, so that log information of the same business type is stored in the same preset log file. For example, if an e-commerce application system includes order business and user management business, log files order_daily.log and user_manage_daily.log can be created in the log server to store relevant logs from the order business and user management business, respectively.

[0092] The log server can receive log storage requests sent by the business servers. These requests include log information and the business name. Specifically, if end users A and B each send an order submission request to the server, the server's load balancer, according to a pre-agreed strategy, forwards user A's request to business server O1 in the order module for processing, and user B's request to business server O2 in the same module. During order processing, business servers O1 and O2 will each generate one or more corresponding log entries.

[0093] At this point, the interface for writing disk log information provided by the log server can be accessed remotely in real time through the intranet network (i.e., local area network), thereby initiating a log storage request. The input parameters of the disk log information writing interface include not only the log content, but also the business name and the node name of the business server. That is, the log storage request includes log information (log information includes log content and the node name of the business server) and the business name.

[0094] 308. Cluster log information according to preset business types to obtain log information clusters of the same business type;

[0095] In this embodiment, the log server determines the business type of the log storage request based on the business name in the log storage request. For example, order processing corresponds to the order business, and nickname modification corresponds to the user management business. Then, the log storage request is distributed to the corresponding log storage queue according to the business type.

[0096] It should be noted that the log server establishes its own independent log storage queues in memory for different business types. Log storage requests are distributed to the corresponding log storage queues according to the order in which they are received from each business server. The log storage queues, i.e. message queues, are established because message queues have functions such as loose coupling, reliable delivery, broadcasting, flow control, and eventual consistency. By using log storage queues, log storage requests can be processed asynchronously, thereby alleviating the pressure on the system.

[0097] 309. Set up log folders according to business types and store log information clusters in the corresponding log folders.

[0098] In this embodiment, the log server retrieves log information from the log storage request from each log storage queue through a pre-created thread and writes it to the corresponding preset log file. This allows log information of the same business type to be stored in the same log file, enabling unified collection and management of logs in distributed business servers. This allows operation and maintenance developers to only search in the corresponding log file when troubleshooting problems, compared to the prior art which requires sequentially viewing the logs of all business server nodes of the corresponding business module.

[0099] Steps 301-303 in this embodiment are similar to steps 101-103 in the first embodiment, and will not be described again here.

[0100] In this embodiment of the invention, container log data is determined based on the container's log path; logs are collected from the container based on the container log data to obtain raw container logs; the raw container logs are anonymized to obtain initial log data; target log data is generated based on the initial log data and the corresponding log configuration file; the target log data is stored in a message middleware and classified to obtain different types of container logs; log storage requests are retrieved according to different types of container logs, and the log information in the log storage requests is written to the corresponding preset log files. This invention reduces the complexity of log queries and lowers the operational threshold by supporting container-based methods and satisfying different collection methods.

[0101] The container log collection and management method in the embodiments of the present invention has been described above. The container log collection and management device in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 4 The first embodiment of the container log collection and management device in this invention includes:

[0102] The acquisition module 401 is used to determine the container log data according to the container's log path, and to collect the container logs according to the container log data to obtain the raw container logs;

[0103] The desensitization module 402 is used to desensitize the original container logs to obtain initial log data;

[0104] The generation module 403 is used to generate target log data based on the initial log data and the log configuration file corresponding to the initial log data;

[0105] The classification module 404 is used to store the target log data in the message middleware and classify the target log data to obtain different types of container logs;

[0106] The writing module 405 is used to retrieve log storage requests according to the different types of container logs, and write the log information in the log storage requests to the corresponding preset log files.

[0107] In this embodiment of the invention, container log data is determined based on the container's log path; logs are collected from the container based on the container log data to obtain raw container logs; the raw container logs are anonymized to obtain initial log data; target log data is generated based on the initial log data and the corresponding log configuration file; the target log data is stored in a message middleware and classified to obtain different types of container logs; log storage requests are retrieved according to different types of container logs, and the log information in the log storage requests is written to the corresponding preset log files. This invention reduces the complexity of log queries and lowers the operational threshold by supporting container-based methods and satisfying different collection methods.

[0108] Please see Figure 5 The second embodiment of the container log collection and management device in this invention specifically includes:

[0109] The acquisition module 401 is used to determine the container log data according to the container's log path, and to collect the container logs according to the container log data to obtain the raw container logs;

[0110] The desensitization module 402 is used to desensitize the original container logs to obtain initial log data;

[0111] The generation module 403 is used to generate target log data based on the initial log data and the log configuration file corresponding to the initial log data;

[0112] The classification module 404 is used to store the target log data in the message middleware and classify the target log data to obtain different types of container logs;

[0113] The writing module 405 is used to retrieve log storage requests according to the different types of container logs, and write the log information in the log storage requests to the corresponding preset log files.

[0114] In this embodiment, the acquisition module 401 is specifically used for:

[0115] Determine the container log data based on the container's log path;

[0116] Based on the container log data, match the corresponding log parsing rules from the preset log collection template library;

[0117] Based on the log parsing rules, configuration parameters corresponding to the container log data are generated;

[0118] Logs are collected from the container according to the configuration parameters to obtain the raw container logs.

[0119] In this embodiment, the desensitization module 402 is specifically used for:

[0120] Obtain the identifier ID of the original container log, and obtain the log desensitization rule corresponding to the original container log from the preset log desensitization configuration parameters based on the identifier ID;

[0121] According to the log desensitization rules, the original container logs are desensitized to obtain initial log data.

[0122] In this embodiment, the generation module 403 is specifically used for:

[0123] Determine whether the initial log data matches the business feature fields in the preset log configuration file;

[0124] If so, the initial log data that matches successfully will be stored, and the target log data will be generated.

[0125] In this embodiment, the classification module 404 includes:

[0126] Storage unit 4041 is used to store the logs in the message middleware cluster according to the identifier ID of the container logs;

[0127] Determining unit 4042 is used to determine the attribute information corresponding to the target log data;

[0128] The classification unit 4043 is used to classify the target log data according to the attribute information to obtain different types of container logs.

[0129] In this embodiment, the writing module 405 is specifically used for:

[0130] The log storage request is parsed to obtain the log information carried in the log storage request;

[0131] The log information is clustered according to a preset business type to obtain log information clusters of the same business type;

[0132] Set up a log folder according to the business type, and store the log information cluster in the corresponding log folder.

[0133] above Figure 4 and Figure 5 The container log collection and management device in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The container log collection and management device in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0134] Figure 6This is a schematic diagram of the structure of a container log collection and management device 600 provided in an embodiment of the present invention. The container log collection and management device 600 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 610 (e.g., one or more processors) and a memory 620, and one or more storage media 630 (e.g., one or more mass storage devices) for storing application programs 633 or data 632. The memory 620 and storage media 630 can be temporary or persistent storage. The program stored in the storage media 630 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the container log collection and management device 600. Furthermore, the processor 610 may be configured to communicate with the storage media 630 and execute a series of instruction operations on the storage media 630 on the container log collection and management device 600 to implement the steps of the container log collection and management method provided in the above-described method embodiments.

[0135] The container log collection and management device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input / output interfaces 660, and / or one or more operating systems 631, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 6 The illustrated container log collection and management device structure does not constitute a limitation on the container log collection and management device provided in this application. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0136] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the above-described container log collection and management method.

[0137] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0138] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0139] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for collecting and managing container logs, characterized in that, The container log collection and management method includes: The container log data is determined based on the container's log path, and the container logs are collected based on the container log data to obtain the raw container logs. The original container logs are anonymized to obtain initial log data; The initial log data is matched with the business feature fields in the preset log configuration file to obtain the successfully matched initial log data. It is then determined whether the creation time of the successfully matched initial log data is within the current collection period of the log configuration file. If the creation time is within the current collection period, the target log data is output when the current collection period ends. The target log data is stored in a message middleware and then classified to obtain different types of container logs. Based on the different types of container logs, retrieve log storage requests and write the log information in the log storage requests to the corresponding preset log files.

2. The container log collection and management method according to claim 1, characterized in that, The process of determining container log data based on the container's log path, and collecting logs from the container based on the container log data to obtain raw container logs includes: Determine the container log data based on the container's log path; Based on the container log data, match the corresponding log parsing rules from the preset log collection template library; Based on the log parsing rules, configuration parameters corresponding to the container log data are generated; Logs are collected from the container according to the configuration parameters to obtain the raw container logs.

3. The container log collection and management method according to claim 1, characterized in that, The process of anonymizing the original container logs to obtain initial log data includes: Obtain the identifier ID of the original container log, and obtain the log desensitization rule corresponding to the original container log from the preset log desensitization configuration parameters based on the identifier ID; According to the log desensitization rules, the original container logs are desensitized to obtain initial log data.

4. The container log collection and management method according to claim 1, characterized in that, The process involves storing the target log data in a message middleware and classifying the target log data to obtain different types of container logs, including: Based on the identifier ID of the container log, the log is stored in the message middleware cluster; Determine the attribute information corresponding to the target log data; Based on the attribute information, the target log data is classified to obtain different types of container logs.

5. The container log collection and management method according to claim 1, characterized in that, The step of writing the log information in the log storage request to the corresponding preset log file includes: The log storage request is parsed to obtain the log information carried in the log storage request; The log information is clustered according to a preset business type to obtain log information clusters of the same business type; Set up a log folder according to the business type, and store the log information cluster in the corresponding log folder.

6. A container log collection and management device, characterized in that, The container log collection and management device includes: The acquisition module is used to determine the container log data based on the container's log path, and to collect the container logs based on the container log data to obtain the raw container logs. The de-identification module is used to de-identify the original container logs to obtain initial log data; The generation module is used to match the initial log data with the business feature fields in the preset log configuration file to obtain the successfully matched initial log data, and to determine whether the creation time of the successfully matched initial log data is within the current collection period of the log configuration file. If the creation time is within the current collection period, the target log data is output when the current collection period ends. The classification module is used to store the target log data in the message middleware and classify the target log data to obtain different types of container logs; The writing module is used to retrieve log storage requests based on the different types of container logs, and write the log information in the log storage requests to the corresponding preset log files.

7. The container log collection and management device according to claim 6, characterized in that, The acquisition module is specifically used for: Determine the container log data based on the container's log path; Based on the container log data, match the corresponding log parsing rules from the preset log collection template library; Based on the log parsing rules, configuration parameters corresponding to the container log data are generated; Logs are collected from the container according to the configuration parameters to obtain the raw container logs.

8. A container log collection and management device, characterized in that, The container log collection and management device includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the instructions in the memory to cause the container log collection and management device to perform the steps of the container log collection and management method as described in any one of claims 1-5.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the container log collection and management method as described in any one of claims 1-5.

Images