Unified management method and device supporting multi-source logs

Abstract

The invention discloses a unified management method and device supporting multi-source logs, and relates to the technical field of log data processing, and the method comprises the steps: obtaining logs through a log collection tool deployed in each node in a container arrangement engine environment; the nodes are used by the resource affiliation unit; the log collection tool is used for collecting a log of each node in a dynamic configuration form; writing the logs of the same resource affiliation unit into the same position according to resource affiliation unit identifiers carried by the logs to form a log index list of each resource affiliation unit; based on stream calculation, judging whether each log index list reaches a rolling condition or not by utilizing a preset rolling condition; when any log index list reaches a rolling condition, deleting out-of-date logs in the log index list, and updating the log index list; the preset rolling condition comprises a preset storage capacity and a limit of storage time. According to the invention, the collection efficiency and management efficiency of the multi-source logs can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of log data processing, and in particular to a unified management method and device supporting multi-source logs. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.

[0003] Currently, enterprise system logs usually come from virtual machines, containers, different services, and multiple formats, forming a complex ecosystem of multi-source logs. This puts higher demands on log management and requires the system to support a variety of log formats, paths, and efficient collection of different services.

[0004] The currently widely used log management solutions have the following shortcomings in multi-source log processing: First, existing tools cannot achieve effective unified management when processing diverse logs from virtual machines and containers, resulting in inefficient log collection and analysis, and difficulty in meeting rapidly changing business needs. Second, traditional solutions often lack flexible dynamic configuration capabilities. When services expand or log sources change, manual configuration adjustments are often required, affecting the adaptability and scalability of the system. In addition, in the face of large-scale log volumes, existing solutions often encounter bottlenecks in performance and storage management, resulting in log processing delays and waste of resources. Summary of the invention

[0005] The embodiment of the present invention provides a unified management method supporting multi-source logs, which is used to effectively integrate logs from different sources, improve the collection efficiency and management efficiency of multi-source logs, and improve log storage performance. The method includes:

[0006] Logs are obtained by a log collection tool deployed on each node in a container orchestration engine environment; the container orchestration engine environment includes multiple nodes; the nodes are used by a resource attribution unit; the log collection tool is used to: collect logs of each node in a dynamic configuration form; the dynamic configuration form includes dynamically executing log collection according to changes in log sources; the log carries a resource attribution unit identifier;

[0007] According to the resource belonging unit identifier carried in the log, the logs of the same resource belonging unit are written to the same location to form a log index list of each resource belonging unit;

[0008] Based on stream computing, the preset rolling conditions are used to determine whether each log index list meets the rolling conditions; when any log index list meets the rolling conditions, the expired logs in the log index list are deleted and the log index list is updated; the preset rolling conditions include preset storage capacity and storage time limits.

[0009] The embodiment of the present invention also provides a unified management device supporting multi-source logs, which is used to effectively integrate logs from different sources, improve the collection efficiency and management efficiency of multi-source logs, and improve log storage performance. The device includes:

[0010] The log collection module is used to: deploy a log collection tool on each node in the container orchestration engine environment; the container orchestration engine environment includes multiple nodes; the nodes are used by the resource attribution unit; the log collection tool is used to: collect the logs of each node through a dynamic configuration form, and transmit the logs to the log collection processing module; the dynamic configuration form includes dynamically executing log collection according to changes in the log source; the log carries the resource attribution unit identifier;

[0011] The log collection and processing module is used to: receive the logs transmitted by the log collection module; write the logs of the same resource belonging unit to the same location according to the resource belonging unit identifier carried by the log, and form a log index list of each resource belonging unit;

[0012] The log storage processing module is used to: based on stream computing, obtain the log index list of all resource attribution units from the log collection processing module; determine whether each log index list meets the rolling condition according to the preset rolling condition; when any log index list meets the rolling condition, delete the expired logs in the log index list and update the log index list; the preset rolling condition includes the preset storage capacity and storage time limit.

[0013] An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned unified management method supporting multi-source logs when executing the computer program.

[0014] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the unified management method supporting multi-source logs is implemented.

[0015] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the unified management method supporting multi-source logs is implemented.

[0016] In the embodiment of the present invention, a multi-source log collection method based on dynamic configuration is adopted, which can dynamically adjust the log collection configuration in each node system, support automatic identification and collection of multiple log formats, and improve the collection efficiency of multi-source logs; write the logs of the same resource attribution unit to the same location to form a log index list of each resource attribution unit, and use preset rolling conditions to process the log index list, which can provide standardized and personalized log storage configuration solutions for different services and products, ensure compatibility and adaptability in different scenarios, effectively integrate logs from different sources, improve the collection and management efficiency of multi-source logs, and improve log storage performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0018] Figure 1 A schematic diagram of a unified management method supporting multi-source logs in an embodiment of the present invention;

[0019] Figure 2 A specific example diagram of a unified management method supporting multi-source logs in an embodiment of the present invention;

[0020] Figure 3 FIG. 4 is another specific example diagram of a unified management method supporting multi-source logs in an embodiment of the present invention;

[0021] Figure 4 FIG. 4 is another specific example diagram of a unified management method supporting multi-source logs in an embodiment of the present invention;

[0022] Figure 5 Schematic diagram of a unified management device supporting multi-source logs in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0024] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second", etc. are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order.

[0025] The acquisition, transmission, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0026] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0027] With the widespread application of virtualization and containerization technologies, the sources of enterprise system logs are becoming increasingly complex. Log collection not only involves virtual machines and container environments, but also requires processing a large number of logs generated by different formats, paths, and different services.

[0028] Existing log management systems face the following technical problems in a multi-source environment:

[0029] 1. Unified management is difficult: Logs generated from different sources (such as virtual machines and containers, different services) often have diverse formats, paths, and structures. Existing tools lack unified management capabilities, resulting in low efficiency in log collection and processing.

[0030] 2. Insufficient dynamic expansion capabilities: When services are expanded or log sources change, traditional tools often require manual reconfiguration, which lacks flexibility and affects the scalability and adaptability of the system.

[0031] 3. Insufficient performance in processing large-volume logs: When processing large-scale logs, existing solutions often cannot meet the needs of efficient storage, resulting in system performance bottlenecks and affecting overall business operations.

[0032] The embodiment of the present invention aims to provide a unified management method supporting multi-source logs, and solves the above-mentioned technical problems by improving the log collection, processing and storage mechanism. From data collection, storage to visual display, the entire process is completed on one platform, simplifying the configuration process of log collection, processing, etc., and achieving high efficiency and flexibility of log management.

[0033] Figure 1 FIG. 1 is a flow chart of a unified management method supporting multi-source logs in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0034] Step 101, obtaining logs through a log collection tool deployed on each node in a container orchestration engine environment; the container orchestration engine environment includes multiple nodes; the nodes are used by a resource attribution unit; the log collection tool is used to: collect logs of each node through a dynamic configuration form; the dynamic configuration form includes dynamically executing log collection according to changes in log sources; the log carries a resource attribution unit identifier;

[0035] Step 102: write the logs of the same resource belonging unit to the same location according to the resource belonging unit identifier carried in the log, so as to form a log index list of each resource belonging unit;

[0036] Step 103: Based on stream computing, use preset rolling conditions to determine whether each log index list meets the rolling conditions; when any log index list meets the rolling conditions, delete the expired logs in the log index list and update the log index list; the preset rolling conditions include preset storage capacity and storage time limits.

[0037] Below Figure 1 The unified management method that supports multi-source logs is explained in detail.

[0038] During implementation, a log collection tool is deployed in advance on each node in the container orchestration engine environment. The log collection tool is used to collect logs of each node through dynamic configuration.

[0039] Deploy the log collection tool in the container orchestration engine environment (for example, through the DaemonSet mode). Each node runs an instance of the log collection tool, which is responsible for collecting logs of all components in the node. Logs are output to the host's file system by mounting and collected uniformly by the log collection tool. A component can be a unit inside a container. A container can contain one or more components. Components can also run inside a virtual machine.

[0040] The log collection tool in the embodiment of the present invention is a lightweight log collection tool, which is used to read log files from data sources such as file systems or containers and transmit them to target systems or target locations. The lightweight log collection tool supports multiple configuration modes, can adapt to the requirements of different log formats and paths, and realizes flexible log collection management through dynamic configuration and multiple modular functions.

[0041] The resource belonging unit is, for example, a tenant, a project, etc. The log carries a resource belonging unit identifier, a component identifier, etc. The service of the resource belonging unit is implemented through multiple components.

[0042] In one embodiment, the node includes multiple containers and / or virtual machines, and the log collection tool includes a configuration file; wherein different types of configuration files are pre-configured in the container and / or virtual machine; the types include YAML files and key-value pair type files.

[0043] For example, a YAML configuration file is configured in a virtual machine, and a Configmap type configuration file is configured in a container. The Configmap type configuration file mainly stores configuration information in key-value pairs.

[0044] In one embodiment, the log collection tool is specifically used to: when the log source changes, obtain new log source information through a custom script and utilize the process monitoring and file system monitoring functions provided by the operating system, and update the configuration file according to the new log source information.

[0045] In the embodiments of the present invention, in virtual machines and container environments, log sources may change with the deployment, expansion or change of services. Through dynamic configuration, the basic configuration of the log collection tool is guaranteed to be stable when it is started, while allowing the log collection strategy to be flexibly adjusted during runtime, so that the system can adapt to dynamically changing log sources, especially suitable for complex scenarios with multiple environments and multiple components running concurrently.

[0046] During implementation, the log collection tool is configured to collect logs of different components and services, and report the collected logs in batch form to the target system, such as a designated system or platform that implements a unified management method that supports multi-source logs. Fine-grained log management is achieved based on the system or platform.

[0047] In order to optimize the processing logic of data before storage and reduce the burden of subsequent queries, in one embodiment, before writing logs of the same resource belonging unit to the same location according to the resource belonging unit identifier carried by the log, the method may also include: segmenting the log according to a preset personalized preprocessing program to obtain a processed standardized log; the preset personalized preprocessing program is used to perform one of filtering, renaming fields, data cleaning, date conversion or any combination of segmentation processing on the log;

[0048] Writing the logs of the same resource belonging unit into the same location according to the resource belonging unit identifier carried in the logs may include: writing the logs of the same resource belonging unit into the same location according to the resource belonging unit identifier carried in the processed standardized logs.

[0049] The embodiment of the present invention provides a set of clear log storage and format specifications, requiring the service output to conform to the standard log format and directory. The log collection tool collection and log processing segmentation are controlled by a unified dynamic configuration file configuration, and the log is segmented into standardized fields and stored in the target index (log.component) corresponding to each component. This simplifies the configuration of the log collection tool and also enhances the efficiency of log query and analysis.

[0050] In actual applications, enterprises usually use some open source products for log processing. However, since the log storage path and format of open source products are difficult to modify, the embodiments of the present invention provide a personalized log collection tool configuration solution to improve open source products. For example, for virtual machines, a new YAML configuration file is added, and for containers, a new Configmap type configuration file is added, and a preset personalized preprocessor is added to split the log field. Flexible adjustments can be made to different log formats and path requirements without large-scale modifications to its internal log system. Maximum compatibility with multiple log sources reduces integration costs.

[0051] In specific implementation, the log storage logic for log collection tools is as follows:

[0052] (1) Virtual machine log storage: Logs in the virtual machine are stored in the index of the corresponding component by date. For example, the log is written into the index named log.component.yymmdd, where component represents the specific component name and yymmdd represents the date of the log.

[0053] (2) Container log storage: For container log processing, the embodiment of the present invention implements flexible log management through a log collection processing module and a log storage processing module. According to the component labels in the container pod, the log storage requirements of the service are distinguished. Based on the preset storage capacity and storage time limits, the log index is rolled over and stored. Whenever the log file meets the set conditions, a new log index is created and expired logs are deleted, thereby ensuring efficient storage management. Among them, the log collection processing module mainly implements the content related to step 102, and the log storage processing module mainly implements the content related to step 103.

[0054] In the unified log management method of the embodiment of the present invention, the log collection processing module and the log storage processing module are two core contents, which are respectively responsible for the rolling and quota management of container log storage, ensuring efficient use of storage resources and guaranteeing system availability.

[0055] Figure 2 FIG. 1 is a specific example diagram of a unified management method supporting multi-source logs in an embodiment of the present invention. Figure 2As shown, for example, multiple logs generated by a virtual machine: / data / log / component1 / service1 / *log, etc., are stored in a unified directory, and components (component) and services (service) are reflected in the directory. Multiple log collection tools collect logs of each virtual machine respectively, and report and store them in batches through ngnix (Nginx can process network traffic inside and outside the cluster, and help implement service access control, load balancing, security protection and other functions). For example, pod1.log and pod2.log generated by container jobs pod1 and pod2 can be processed by a preset personalized preprocessor and output in a standard format, in which component, service, tenant ID, and project ID labels are pre-annotated in the container orchestration engine environment. Reference Figure 2 , pod1.log and pod2.log are collected by two log collection tools respectively, and processed by the LB load balancer and the log collection processing module to realize batch storage, query and log display in the cluster. The log storage processing module performs index rolling and cleaning. Among them, the log rolling form can be distinguished by labels. Among them, this method can be implemented based on the server in the cluster to realize massive data storage, high-speed query and complex query analysis.

[0056] During implementation, the log collection tool reports the logs to the log collection and processing module, which supports pre-labeling of tenant, project ID and other information based on tags.

[0057] When step 102 is implemented specifically, the log collection and processing module is mainly responsible for processing the real-time writing of logs. Its core functions include: for batch logs, judging the log type and selecting different storage methods; creating initial indexes and write aliases for new projects or tenants, and managing query aliases for logs across dates to ensure query optimization; ensuring that the index in the key-value storage database (for example, Redis) is synchronized with the actual state of the storage location in the actual cluster, and quickly retrieving the alias mapping relationship between log writing and query through the key-value storage database.

[0058] In a preferred embodiment, according to the resource belonging unit identifier carried by the log, the logs of the same resource belonging unit are written to the same location to form a log index list of each resource belonging unit, which may include:

[0059] Use the resource belonging unit identifier carried in the log to match the pre-stored specified resource belonging unit identifier;

[0060] Creating an alias for the log of the successfully matched resource attribution unit; wherein the correspondence between the alias and the original name of the log is stored in a key-value pair storage database;

[0061] Based on the logs created with the alias, the logs of the same resource belonging unit are written to the same location to form a log index list of each resource belonging unit.

[0062] Figure 3 FIG. 2 is another specific example diagram of a unified management method supporting multi-source logs in an embodiment of the present invention. Figure 3 As shown, the processing is as follows:

[0063] (1) Initialization:

[0064] During initialization, the log collection and processing module will Figure 2 The cluster requests the write aliases of all flow logs and loads them into memory. At the same time, the alias information in Redis is checked to ensure that the cache in Redis is consistent with the status in the actual cluster. If there is any inconsistency, the log collection and processing module will update Redis.

[0065] (2) Batch log writing:

[0066] When batch logs are written, the log collection and processing module processes the logs:

[0067] 1) Determine whether it is a flow log (by using labels, for example, whether the specified resource attribution unit identifier exists in the pre-stored data). If it is a normal log, write it directly to the cluster.

[0068] 2) If it is a flow log, the log collection and processing module will select the corresponding index based on the tenant and project identifiers in the log, as well as the date of the log, and write the log. When a new tenant or project is written for the first time, an initial write index and alias will be created. For logs that span dates, the log collection and processing module will maintain date aliases in Redis and ensure that logs can be quickly located by these aliases when querying.

[0069] Figure 3 The Java virtual machine buffer is also included, which is mainly used to improve the efficiency of data processing by reducing frequent direct read and write operations.

[0070] When step 103 is specifically implemented, the log storage processing module mainly implements two scheduled tasks: an index rolling task and an index cleaning task. Through periodic inspection, it is responsible for executing the log storage rolling operation and the index cleaning operation respectively.

[0071] In one embodiment, when any log index list reaches a rolling condition, the expired logs in the log index list are deleted and the log index list is updated, which may include: when any log index list reaches a rolling condition, the expired logs in the log index list are deleted, a new alias is written for the remaining logs in the log index list, and the correspondence between the new alias and the original name is stored and updated in a key-value pair storage database.

[0072] During implementation, taking the streaming job logs of each tenant and project as an example, the task scrolls or deletes the log index list every 5 minutes (configurable and modifiable).

[0073] Index rolling task: This task regularly rolls the log index based on the storage size, number of documents, storage days, etc. to ensure that each log index list does not expand excessively. At the same time, it creates new indexes and maintains alias relationships to ensure query efficiency and storage consistency. The main workflow is as follows:

[0074] (1) Get the log index list of all current tenants and projects;

[0075] (2) Perform log rolling operations: Determine whether each log index list meets the rolling conditions (such as the maximum storage size, the maximum number of documents, or the storage time). These threshold information can be obtained from the Redis cache or the set configuration file. The thresholds include but are not limited to three types: max_size: rolling based on index size, such as 5GB; max_docs: rolling based on the number of documents, such as 1,000 documents; max_age: rolling based on time, such as 5 days.

[0076] When the index meets the rolling conditions, the log storage processing module generates a new index and performs the following operations:

[0077] Index alias update: Create a query alias for the new index and maintain the relationship between the log write alias and the query alias;

[0078] Write alias: log.flowlogtag.tenantid_projectid;

[0079] Can write to real index;

[0080] Query alias: log.flowlogtag.tenantid_projectid.date;

[0081] Persistent alias relationship: The correspondence between the written alias and the real index is saved in Redis for subsequent quick access and query.

[0082] Index cleanup task: This scheduled task regularly deletes redundant old indexes based on the project storage limit and disk usage to ensure reasonable allocation of storage resources and avoid storage space exhaustion. The process is:

[0083] Project index limit: If the number or size of indexes for a project exceeds the specified limit (such as a single project limit of 5GB), the log storage processing module will calculate the number of indexes that need to be retained and delete the excess indexes.

[0084] Disk cleanup task: When the cluster disk usage reaches a critical value (configurable), the log storage processing module calculates the size of the disk space that needs to be cleaned up and deletes excess log indexes to ensure cluster availability.

[0085] Figure 4 FIG. 2 is another specific example diagram of a unified management method supporting multi-source logs in an embodiment of the present invention. Figure 4 As shown, the log collection processing module executes processing such as sending batch requests to write data and write aliases to form a log index list for each resource belonging unit. The log storage processing module performs index scrolling and index cleaning on the log index list. When the index reaches the scrolling threshold, the scrolling logic is executed to assign query aliases and write aliases to the new index.

[0086] Through the collaborative work of the two services of the log collection processing module in step 102 and the log storage processing module in step 103, the log storage solution of the embodiment of the present invention realizes dynamic adaptation to different log rolling strategies, ensuring efficient use of log storage. The log storage processing module is responsible for regular rolling and cleaning of logs to avoid excessive consumption of storage resources while maintaining the normal operation of the system. The log collection processing module provides fast log query capabilities through log diversion writing and alias management, and supports multiple storage strategies based on date and size. The availability and manageability of log data are enhanced.

[0087] In one embodiment, after updating the log index list, the method may further include: providing a log query interface; the log query interface is used to: query logs by alias, and / or query logs by date.

[0088] For example, for logs stored by date, an interface is provided to support query by field and date. For logs stored by tenant or project, query aliases are used for log query, and the corresponding index is queried according to the time range of the queried log. For example, to query from 12:00:00 on a certain day to 10:00:00 on a certain day, the indexes corresponding to the two aliases log.flowlogtag.xx_xx.a certain day 12:00:00 and log.flowlogtag.xx_xx.a certain day 10:00:00 are queried.

[0089] In one embodiment, the method is implemented by a first master node in a container orchestration engine environment; the container orchestration engine environment also includes a second standby node, and the second standby node serves as a backup of the first master node.

[0090] In the embodiment of the present invention, the platform side and the tenant side share the same log query service, and two nodes are deployed in the production environment to achieve high availability and ensure that the service can still operate normally when a single node fails.

[0091] The embodiment of the present invention supports log management on the platform side and the tenant side, and implements permission isolation. Within the service, the platform side and the tenant side are isolated through strict permission control to ensure clear permission boundaries for different users or systems, thereby improving security and manageability.

[0092] In the embodiment of the present invention, the cluster is deployed independently. Since the log query service depends on the cluster, it consumes a lot of memory and CPU, and when data writing surges or large-scale queries occur, it is easy to cause cluster resource shortage, thereby affecting the overall availability of the system. To avoid this situation, in the production environment, independent clusters are deployed on the platform side and the tenant side respectively. Even if an abnormality occurs on either the platform side or the tenant side, the services of each other can still maintain normal operation, avoid mutual influence, and ensure the stability and isolation of the system.

[0093] In summary, the embodiments of the present invention have the following advantages:

[0094] 1. Multi-source log collection mechanism based on dynamic configuration: Ensure that in different environments such as virtual machines and containers, the log collection configuration can be dynamically adjusted through the log collection tool, and support automatic identification and collection of multiple log formats.

[0095] 2. Personalized configuration and standardized support for log storage: Provide standardized and personalized log storage configuration solutions for different services and products to ensure compatibility and adaptability in different scenarios.

[0096] 3. Intelligent management of log rolling mechanism: By configuring the rolling threshold, an intelligent rolling storage mechanism based on storage capacity and storage time is implemented, and dynamic adjustment and efficient management of rolling configuration are achieved based on Redis.

[0097] 4. Optimization of log storage space and query performance: Implement dynamic control of project storage space and optimize log query efficiency to ensure efficient storage and query performance of log data.

[0098] 5. Authority isolation and high availability design between the platform and tenant sides: Ensure that the log services of the platform and tenants are deployed independently in the architecture to avoid global impact caused by abnormalities on either side and achieve high reliability of log management services.

[0099] The embodiment of the present invention also provides a unified management device supporting multiple source logs, as described in the following embodiment. Since the principle of solving the problem by the device is similar to that of the unified management method supporting multiple source logs, the implementation of the device can refer to the implementation of the unified management method supporting multiple source logs, and the repeated parts will not be repeated.

[0100] Figure 5 FIG. 1 is a schematic diagram of a unified management device supporting multi-source logs in an embodiment of the present invention. Figure 5As shown, the device 500 includes: a log collection module 501, a log collection processing module 502, and a log storage processing module 503.

[0101] The log collection module 501 is used to: deploy a log collection tool on each node in the container orchestration engine environment; the container orchestration engine environment includes multiple nodes; the nodes are used by the resource attribution unit; the log collection tool is used to: collect the logs of each node through a dynamic configuration form, and transmit the logs to the log collection processing module; the dynamic configuration form includes dynamically executing log collection according to changes in the log source; the log carries the resource attribution unit identifier;

[0102] The log collection processing module 502 is used to: receive the logs transmitted by the log collection module; write the logs of the same resource belonging unit to the same location according to the resource belonging unit identifier carried by the log, and form a log index list of each resource belonging unit;

[0103] The log storage processing module 503 is used to: based on stream computing, obtain the log index list of all resource attribution units from the log collection processing module; determine whether each log index list meets the scrolling condition according to the preset scrolling condition; when any log index list meets the scrolling condition, delete the expired logs in the log index list and update the log index list; the preset scrolling condition includes the preset storage capacity and storage time limit.

[0104] In one embodiment, the node includes multiple containers and / or virtual machines, and the log collection tool includes a configuration file; wherein different types of configuration files are pre-configured in the container and / or virtual machine; the types include YAML files and key-value pair type files.

[0105] In one embodiment, the log collection tool is specifically used to: when the log source changes, obtain new log source information through a custom script and utilize the process monitoring and file system monitoring functions provided by the operating system, and update the configuration file according to the new log source information.

[0106] In one embodiment, the apparatus 500 further includes:

[0107] The log segmentation processing module is used to segment the logs according to the resource belonging unit identifier carried by the logs before the log collection processing module 502 writes the logs of the same resource belonging unit to the same location, according to a preset personalized preprocessing program, to obtain a processed standardized log; the preset personalized preprocessing program is used to perform one of filtering, renaming fields, data cleaning, date conversion or any combination of segmentation processing on the logs;

[0108] The log collection and processing module 502 is specifically used to write logs of the same resource belonging unit into the same location according to the resource belonging unit identifier carried in the processed standardized log.

[0109] In one embodiment, the log collection and processing module 502 is specifically used to:

[0110] Use the resource belonging unit identifier carried in the log to match the pre-stored specified resource belonging unit identifier;

[0111] Creating an alias for the log of the successfully matched resource attribution unit; wherein the correspondence between the alias and the original name of the log is stored in a key-value pair storage database;

[0112] Based on the logs of the created aliases, the logs of the same resource belonging unit are written to the same location to form a log index list of each resource belonging unit.

[0113] In one embodiment, the log storage processing module 503 is specifically used to: when any log index list reaches the rolling condition, delete the expired logs in the log index list, write new aliases for the remaining logs in the log index list, and store the corresponding relationship between the new alias and the original name in the key-value pair storage database for update.

[0114] In one embodiment, the apparatus 500 may further include:

[0115] The log query module is used to provide a log query interface after the log storage processing module 503 updates the log index list; the log query interface is used to query logs by alias and / or by date.

[0116] In one embodiment, the method is implemented by a first master node in a container orchestration engine environment; the container orchestration engine environment also includes a second standby node, and the second standby node serves as a backup of the first master node.

[0117] An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned unified management method supporting multi-source logs when executing the computer program.

[0118] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the unified management method supporting multi-source logs is implemented.

[0119] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the unified management method supporting multi-source logs is implemented.

[0120] The embodiment of the present invention realizes the unified collection and dynamic management of logs in complex environments such as virtual machines and containers by constructing a unified management method and device that supports multi-source logs. Log collection tools are used for log collection, and standardized log storage specifications are provided. At the same time, personalized configuration solutions are customized for open source products that are difficult to modify the log format. Not only does it improve the flexibility and compatibility of log collection, but it also ensures that the logs of various services can be stored and processed in an orderly manner.

[0121] The log collection and processing module and the log storage processing module are further used to perform intelligent scrolling and storage optimization of the logs, and the storage strategy can be automatically adjusted according to the storage size, number of days or number of documents based on actual needs. The efficiency of log management is effectively improved, and the limitations of a single storage method are avoided. High availability and isolation on the platform side and the tenant side are achieved through a distributed architecture, ensuring the stable operation of the system. Ultimately, the embodiment of the present invention implements full-link closed-loop management from log collection to storage and query, greatly improving the scalability and reliability of the system.

[0122] This solution is suitable for unified collection and dynamic management of logs in complex environments such as virtual machines and containers. It provides a full-process one-stop development solution and is widely applicable to various log collection and management needs.

[0123] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0127] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A unified management method supporting multi-source logs, characterized in that: include: The logs are obtained by using a log collection tool deployed on each node in the container orchestration engine environment; the container orchestration engine environment includes multiple nodes; The node is used by the resource attribution unit; the log collection tool is used to collect the log of each node through a dynamic configuration form; the dynamic configuration form includes dynamically executing log collection according to the change of the log source; the log carries the resource attribution unit identifier; According to the resource belonging unit identifier carried by the log, the logs of the same resource belonging unit are written to the same location to form a log index list of each resource belonging unit; Based on stream computing, use the preset rolling conditions to determine whether each log index list meets the rolling conditions; When any log index list reaches a rolling condition, the expired logs in the log index list are deleted and the log index list is updated; the preset rolling condition includes a preset storage capacity and storage time limit.

2. The method according to claim 1, characterized in that The node includes multiple containers and / or virtual machines, and the log collection tool includes a configuration file; wherein different types of configuration files are pre-configured in the container and / or virtual machine; the types include YAML files and key-value pair type files.

3. The method according to claim 2, characterized in that The log collection tool is specifically used for: when the log source changes, obtaining new log source information through a custom script and utilizing the process monitoring and file system monitoring functions provided by the operating system, and updating the configuration file according to the new log source information.

4. The method according to claim 1, characterized in that Before writing logs of the same resource belonging unit to the same location according to the resource belonging unit identifier carried in the log, the method further includes: According to a preset personalized preprocessing program, the log is segmented to obtain a processed standardized log; the preset personalized preprocessing program is used to perform one or any combination of segmentation processing on the log, such as filtering, renaming fields, data cleaning, and date conversion; According to the resource unit identifier carried in the log, logs of the same resource unit are written to the same location, including: According to the resource belonging unit identifier carried by the processed standardized log, the logs of the same resource belonging unit are written to the same location.

5. The method according to claim 1, characterized in that According to the resource belonging unit identifier carried in the log, the logs of the same resource belonging unit are written to the same location to form a log index list of each resource belonging unit, including: Use the resource belonging unit identifier carried in the log to match the pre-stored specified resource belonging unit identifier; Creating an alias for the log of the successfully matched resource attribution unit; wherein the correspondence between the alias and the original name of the log is stored in a key-value pair storage database; Based on the logs of the created aliases, the logs of the same resource belonging unit are written to the same location to form a log index list of each resource belonging unit.

6. The method according to claim 5, characterized in that When any log index list reaches the rolling condition, the expired logs in the log index list are deleted and the log index list is updated, including: When any log index list reaches the rolling condition, the expired logs in the log index list are deleted, new aliases are written for the remaining logs in the log index list, and the corresponding relationship between the new alias and the original name is stored and updated in the key-value pair storage database.

7. The method according to claim 5 or 6, characterized in that After updating the log index list, it also includes: Provides a log query interface; the log query interface is used to: query logs by alias and / or query logs by date.

8. The method according to claim 1, characterized in that The method is implemented by a first master node in a container orchestration engine environment; the container orchestration engine environment also includes a second standby node, and the second standby node serves as a backup of the first master node.

9. A unified management device supporting multi-source logs, characterized in that: include: Log collection module, log collection processing module, log storage processing module; The log collection module is used to: deploy a log collection tool on each node in the container orchestration engine environment; the container orchestration engine environment includes multiple nodes; the nodes are used by the resource attribution unit; the log collection tool is used to: collect the logs of each node through a dynamic configuration form, and transmit the logs to the log collection processing module; the dynamic configuration form includes dynamically executing log collection according to changes in the log source; the log carries the resource attribution unit identifier; The log collection and processing module is used to: receive the logs transmitted by the log collection module; write the logs of the same resource belonging unit to the same location according to the resource belonging unit identifier carried by the log, and form a log index list of each resource belonging unit; The log storage processing module is used to: obtain the log index list of all resource belonging units from the log collection processing module based on stream computing; Determine whether each log index list meets the scrolling condition according to the preset scrolling condition; When any log index list reaches a rolling condition, the expired logs in the log index list are deleted and the log index list is updated; the preset rolling condition includes a preset storage capacity and storage time limit.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images