Method, device and storage medium for automatic log collection

Abstract

The application provides a log automatic collection method, device and storage medium, and relates to the technical field of containers; the method is applied to a container orchestrator and comprises the following steps: storing a preset log abstract file as a custom resource; acquiring preset log collection general parameters and a preset log logic file, wherein the log collection general parameters comprise log tool information; taking the log collection general parameters as parameters of the log abstract file, and creating a log service object according to the parameters of the log abstract file; creating a log collection service according to the log service object and the log logic file; and collecting logs by calling a log tool corresponding to the log tool information through the log collection service according to log collection logic in the log logic file. The log automatic collection method, device and storage medium provided in the application can adapt to log collection requirements of various application scenarios and can be quickly deployed.

Description

Technical Field

[0001] The embodiments of the present invention relate to, but are not limited to, the field of container technology, and particularly to a method, device and storage medium for automatic log collection. Background Technology

[0002] Traditional logging tools typically operate in two modes: independent remote log servers or local log services. The former, often commercial logging software, requires a separate log server deployment for configuration and management, with logging services implemented through clients deployed at both the source and target ends. The latter, mostly open-source tools, are deployed at either the source or the local log server. Both methods rely on manual maintenance and are not well-suited to the logging needs of container orchestrators (such as Kubernetes environments). Existing container orchestrators' built-in log collection services either depend on the platform's built-in collection tools, meaning the tools and services are strongly correlated and cannot adapt to customized collection requirements; or, while some container orchestrators allow for customized log collection services created based on the specific application, deployment is difficult. Therefore, existing log collection methods based on container orchestrators struggle to quickly deploy and adapt to diverse application scenarios and collection needs. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] This invention provides a method, device, and storage medium for automatic log collection, which can adapt to the log collection needs of various application scenarios and can be deployed quickly.

[0005] In a first aspect, embodiments of the present invention provide a method for automatic log collection, applied to a container orchestrator, comprising:

[0006] Store the pre-defined log abstract file as a custom resource;

[0007] Obtain preset general log collection parameters and preset log logic files, wherein the general log collection parameters include log tool information;

[0008] The general parameters for log collection are used as parameters for the log abstract file, and a log service object is created based on the parameters of the log abstract file.

[0009] Create a log collection service based on the log service object and the log logic file;

[0010] Based on the log collection logic in the log logic file, the log collection service calls the log tool corresponding to the log tool information to collect logs.

[0011] According to some embodiments of the first aspect of the present invention, the log collection general parameters include a configuration file, the configuration file being used to store the collection target, and the log abstract file including the target server;

[0012] The step of collecting logs by calling the log tool corresponding to the log tool information through the log collection service according to the log collection logic in the log logic file includes:

[0013] The configuration file is extracted from the log service object through the log collection service, and the first log collection directory is determined from the configuration file;

[0014] The target server is determined from the log service object through the log collection service;

[0015] The log collection service invokes the log tool to collect log files in the first log collection directory under the target server.

[0016] According to some embodiments of the first aspect of the present invention, the general parameters for log collection further include a log monitoring directory;

[0017] The step of collecting logs by calling the log tool corresponding to the log tool information through the log collection service according to the log collection logic in the log logic file further includes:

[0018] The log monitoring directory is extracted from the log service object through the log collection service;

[0019] The second log collection directory is extracted from the log monitoring directory according to a preset cycle;

[0020] Compare the currently extracted second log collection directory with the first log collection directory;

[0021] When the second log collection directory has a new path relative to the first log collection directory, the log collection service adds the new path to the configuration file, thereby updating the first log collection directory.

[0022] Restart the log collection service so that the log tool can collect logs based on the first log collection directory that has been updated in the configuration file.

[0023] According to some embodiments of the first aspect of the present invention, the step of calling the log tool corresponding to the log tool information through the log collection service to perform log collection based on the log collection logic in the log logic file further includes:

[0024] When the newly added path indicates a newly added application, the newly added application is added to the configuration file through the log collection service.

[0025] According to some embodiments of the first aspect of the present invention, adding the new application to the storage collection target through the log collection service includes:

[0026] The application name of the newly added application is extracted from the newly added path using the log collection service.

[0027] The application name is used as a parameter of the service extraction interface of the container orchestrator to obtain application information;

[0028] The application information is added to the configuration file through the log collection service.

[0029] According to some embodiments of the first aspect of the present invention, the method further includes, prior to log collection:

[0030] The verification log is obtained by calling the log tool through the log collection service.

[0031] The verification log is parsed to obtain the first log pattern;

[0032] Compare the first log pattern with the second log model preset by the container orchestrator;

[0033] The comparison result is updated to the resource status parameters of the log service object, so that the container orchestrator can monitor the collection status of the log service object according to the resource status parameters.

[0034] According to some embodiments of the first aspect of the present invention, after log collection, the method further includes:

[0035] Statistical analysis of the network traffic used for the log collection;

[0036] The network traffic is compared with a preset threshold value;

[0037] When the network traffic exceeds the threshold, the concurrency of the logging tool is reduced according to a preset adjustment algorithm.

[0038] Secondly, the present invention provides a device with automatic log collection function, comprising:

[0039] The resource definition module is used to store a preset log abstract file as a custom resource;

[0040] The acquisition module is used to acquire preset log collection general parameters and preset log logic files, wherein the log collection general parameters include log tool information;

[0041] The object creation module is used to take the general parameters of log collection as parameters of the log abstract file, and create a log service object according to the parameters of the log abstract file;

[0042] The service creation module is used to create a log collection service based on the log service object and the log logic file;

[0043] The running module is used to collect logs by calling the log tool corresponding to the log tool information through the log collection service, based on the log collection logic in the log logic file.

[0044] Thirdly, embodiments of the present invention also provide an electronic device, including: at least one processor, and a memory communicatively connected to the at least one processor; wherein the memory stores instructions which are executed by the at least one processor to cause the at least one processor to implement the automatic log collection method as described in any one of the first aspects when executing the instructions.

[0045] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for performing the automatic log collection method described in any one of the first aspects.

[0046] The above embodiments of the present invention have at least the following beneficial effects: By deploying the log abstract file as a custom resource, log tools required for actual application scenarios can be instantiated using the log abstract file as general parameters for log collection, thus obtaining a log service object. Simultaneously, the operation of the log service object is managed through a custom log logic file, achieving decoupling between the log collection service and the log tools and deployment. Therefore, when the application scenario or collection tool changes, only the general collection parameters and log collection logic need to be adjusted to create the log collection service. The deployment of the log collection service is simpler and more universal, and can meet the settings of different collection needs. Therefore, the automatic log collection method, device, and storage medium of the embodiments of the present invention can adapt to the log collection needs of various application scenarios and can be deployed quickly.

[0047] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0048] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0049] Figure 1 This is a flowchart illustrating the automatic log collection method according to an embodiment of the present invention;

[0050] Figure 2 This is an example of a log abstract file according to an embodiment of the present invention;

[0051] Figure 3 This is an example of a log service object in an embodiment of the present invention;

[0052] Figure 4 This is a flowchart illustrating the process of maintaining the collection target in the automatic log collection method according to an embodiment of the present invention;

[0053] Figure 5 This is a flowchart illustrating the automatic log pattern recognition process in the automatic log collection method of this invention.

[0054] Figure 6 This is a schematic diagram of a device with automatic log collection function according to an embodiment of the present invention;

[0055] Figure 7 This is a hardware schematic diagram of an electronic device for an application log automatic collection method according to an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0058] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0059] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0060] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0061] The following is an explanation of some terms used in this invention.

[0062] A container orchestrator can work in any environment that uses containers. It helps you deploy the same application in multiple environments without having to rewrite it.

[0063] Kubernetes, also known as K8S or Kube, is a type of container orchestrator. It is an open-source container orchestrator technology used to automate the deployment, scaling, and management of containerized applications; it simplifies the deployment and management of microservice architecture applications by creating an abstraction layer on top of the cluster.

[0064] Operators are based on a combination of two concepts in Kubernetes: custom resources and custom controllers. They allow engineers to write custom controllers for applications based on their unique domain logic. Custom resources are resource objects not natively provided by Kubernetes.

[0065] CRD, short for CustomResourceDefinition, allows for the secondary development of custom resources that extend the Kubernetes API. Through CRD, we can add new resource types to the Kubernetes API without modifying the Kubernetes source code or creating a custom API server.

[0066] EFK is currently the most popular distributed logging framework, which mainly implements log collection, storage, and analysis. It can be combined with Docker containers to collect Docker console logs.

[0067] Traditional logging tools typically operate in two modes: independent remote log servers or local log services. The former, often commercial logging software, requires independent deployment of a log server for configuration and management, with logging services implemented through clients deployed at both the source and target ends. The latter, mostly open-source tools, are deployed at either the source or the local log server. Both methods rely on manual maintenance. With technological advancements, cloud-native services, offering container orchestration, self-healing, and deployment management, are widely used. However, current log collection methods are not well-suited to these needs. For example, in a Kubernetes environment, users primarily use two modes: managed mode (purchasing off-the-shelf Kubernetes service from a cloud provider and entrusting the entire platform to them for maintenance, with the user merely using application computing power) and self-managed mode (maintaining their own Kubernetes platform on bare-metal servers or cloud-provided computing resources). While the former eliminates the need for dedicated log collection tools, its log collection service is entirely dependent on the platform's own supported tools, making it unsuitable for various application scenarios. The latter is more suitable for enterprises to build their own private cloud-native services. Its advantage lies in the ability to build cloud services tailored to their specific application characteristics while retaining the advantages of a private cloud. However, its disadvantage is that auxiliary services necessary for production environments, such as logging, need to be partially or entirely built independently, making deployment in container orchestration more difficult. Therefore, the latter method is more challenging to deploy, and existing log collection methods are insufficient for quickly deploying and adapting to the diverse application scenarios and log collection needs. Based on this, this invention provides a method, device, and storage medium for automatic log collection, which can adapt to the log collection needs of various application scenarios and enable rapid deployment.

[0068] Firstly, referring to Figure 1 As shown, an automatic log collection method according to an embodiment of the present invention, applied to a container orchestrator, includes:

[0069] Step S100: Store the preset log abstract file as a custom resource.

[0070] It should be noted that after storing the log abstract file in the directory of the custom resources, it can be used as a custom resource by the container orchestrator for the instantiation of subsequent service objects.

[0071] It should be noted that the log abstract file is abstracted based on log collection behavior and includes information about the backup software, backup type, and backup resources. These backup resources correspond to both local and K8S api-resource resources. The backup software and backup resources are mandatory parameters for the custom resources corresponding to the log abstract file. The backup resources can be determined through configuration files or direct parameters; this embodiment of the invention does not impose any restrictions on this.

[0072] For example, refer to Figure 2 As shown, taking the Kubernetes (K8S) platform as an example, a log abstraction file `loghavstdef.yaml` is created on the K8S platform based on a CRD. Here, "haevest" corresponds to the backup software information, "watchpath" is the log monitoring directory information used to monitor changes in the log collection directory, and "configfile" contains the configuration information for the logging tool, such as the log path. "remoteserver" is the server where the logs to be collected reside; one or more servers can be set. "netlimit" is the traffic limit parameter for the logging tool. "haevest," "watchpath," and "configfile" are required parameters for the log abstraction file; the remaining parameters are optional.

[0073] For example, in some other embodiments, the configuration file in the log abstract file loghavstdef.yaml can also be replaced with the log directory to be collected. Preferably, embodiments of the present invention use configuration files to set up custom resources.

[0074] Step S200: Obtain preset log collection general parameters and preset log logic file, wherein the log collection general parameters include log tool information.

[0075] It should be noted that the log logic file is used to manage the log tool's operation behavior during the log collection process, such as when to collect logs, which logs need to be collected, or traffic limits.

[0076] Step S300: Use the general parameters for log collection as parameters for the log abstract file, and create a log service object based on the parameters of the log abstract file.

[0077] It should be noted that, with Figure 2For example, after creating `loghavstdef.yaml` on the cluster, the `loghavest` service, which actually performs data collection, can be created. Based on the backup tools pre-supported in `loghavstdef`, images of the supported tools are created. Subsequently, Kubernetes will use these images to create service containers under the control of the corresponding Operator, based on the information of the custom resources, thus obtaining the log service object. For example, using... Figure 3 For example, the filebeat log service object is created.

[0078] Step S400: Create a log collection service based on the log service object and the log logic file.

[0079] It should be noted that after writing the log logic file loghavest-Operator based on the backup logic, the container orchestrator plugins such as Operator will create a log collection service based on the log collection general parameters in the log service object, so that the log collection service can run according to the backup logic.

[0080] Step S500: Based on the log collection logic in the log logic file, call the log tool corresponding to the log tool information through the log collection service to collect logs.

[0081] It should be noted that the log collection logic is used to configure the log collection service to call the log tool to collect logs.

[0082] Therefore, by deploying the log abstract file as a custom resource, the logging tools required for actual application scenarios can be instantiated using the log abstract file as general parameters for log collection, resulting in a log service object. Simultaneously, the operation of the log service object is managed through a custom log logic file, achieving decoupling between the log collection service, logging tools, and deployment. In this way, when the application scenario or collection tool changes, only the general collection parameters and log collection logic need to be adjusted to create the log collection service. The deployment of the log collection service is simpler and more universal, and can meet the settings of different collection needs. Therefore, the automatic log collection method, device, and storage medium of this invention can adapt to the log collection needs of various application scenarios and can be deployed quickly.

[0083] It should be noted that the log collection service and the log service object are in one-to-one correspondence. In some embodiments, multiple log service objects will be instantiated. Multiple log service objects can share the same log collection logic, or they can be configured with their own log collection logic. Those skilled in the art can make any combination and settings according to actual needs. In this regard, the embodiments of the present invention do not impose any restrictions.

[0084] Understandably, common parameters for log collection include configuration files, which store the collection targets, and log abstract files, which include the target server.

[0085] Reference Figure 4 As shown, step S500 involves collecting logs by calling the corresponding log tool through the log collection service, based on the log collection logic in the log logic file. This includes:

[0086] Step S510: Extract the configuration file from the log service object through the log collection service and determine the first log collection directory from the configuration file.

[0087] It's worth noting that recording the first log collection directory for the files to be collected via a configuration file allows for more efficient management of log collection directories when there are multiple collection objects. Compared to directly inputting log paths in general parameters or using wildcards, the configuration file method is more flexible, and when the collection directory is changed, new log collection can be implemented without recreating the log collection service. Therefore, this approach makes log collection more flexible and better adaptable to a wider range of application scenarios.

[0088] It should be noted that the general log collection parameters correspond one-to-one with the input parameters in the log abstract file. The general log collection parameters include at least the required parameters from the log abstract file. In practical applications, users can personalize log collection settings by entering the values ​​of optional parameters from the log abstract file into the general log collection parameters as needed. Optional parameters include... Figure 2 As shown, this includes network traffic limits, target servers, etc.

[0089] Step S520: Determine the target server from the log service object through the log collection service.

[0090] It should be noted that the log service object is instantiated based on the general parameters of log collection, therefore, the target server can be extracted from the log service object.

[0091] Step S530: Call the log tool through the log collection service to collect log files in the first log collection directory of the target server.

[0092] It should be noted that for the same log collection service, there may be multiple target servers or only one target server; this embodiment of the invention does not impose any restrictions on this. For example, refer to... Figure 3As shown, when creating the log service object, two target servers are set, namely kafka01 and kafka02. Then, during log collection, log files will be collected from the first log collection directory of kafka01 and kafka02 respectively.

[0093] Understandably, common parameters for log collection also include the log monitoring directory; see reference. Figure 4 As shown, step S500, based on the log collection logic in the log logic file, calls the log tool corresponding to the log tool information through the log collection service to collect logs, and also includes:

[0094] Step S540: Extract the log monitoring directory from the log service object through the log collection service.

[0095] Step S550: Extract the second log collection directory from the log monitoring directory according to the preset cycle.

[0096] It should be noted that the log monitoring directory is acquired periodically, such as on a daily or monthly basis. In some embodiments, when the application undergoes deployment changes, step S550 can be triggered to actively extract the log monitoring directory, and a second log collection directory can be extracted in conjunction with the periodicity to ensure that changes to each path under the log monitoring directory can be monitored. This embodiment of the invention does not impose limitations on this. Specifically, those skilled in the art can set relevant rules according to actual application scenarios.

[0097] Step S560: Compare the currently extracted second log collection directory with the first log collection directory.

[0098] Step S570: When the second log collection directory has a new path relative to the first log collection directory, the new path is added to the configuration file through the log collection service, so that the first log collection directory is updated.

[0099] It should be noted that in some embodiments, if a path exists in the first log collection directory but not in the second log collection directory, the missing path can be deleted from the configuration file to improve the efficiency of path parsing in the configuration file. This embodiment of the invention does not impose limitations on this; preferably, in this embodiment, the operation is performed only on newly added paths.

[0100] Step S580: Restart the log collection service so that the log tool can collect logs according to the first log collection directory updated in the configuration file.

[0101] It should be noted that after the log collection service is restarted, when the log collection service is run again, the log collection service will call the log tool and reread the first log collection directory recorded in the configuration file, thereby enabling automatic collection of newly added logs.

[0102] For example, taking the K8S and Operator combined application log collection service as an example, the Operator first retrieves the spec.configfile of the log service object loghavest01 and reads the pre-existing filebeat configuration file harvest01.yaml in the configmap according to the configuration file of the loghavest01 application. It updates the log-path in harvest01.yaml according to the log path scanned in step S560 (in some embodiments, the corresponding application tag under the path will also be updated). Then, it starts the demonset type filebeat collector application on K8S. Subsequently, it periodically compares whether there are any new collection paths. If so, it updates the configmap and restarts the filebeat application according to the above logic, thereby reducing the probability of frequently maintaining the collection target manually.

[0103] It should be noted that, since the collection of data is carried out using a log monitoring directory, the log generation subdirectory pattern is agreed upon when building or integrating Kubernetes as watchpath / [application name:systemname] / . Based on the application name under watchpath (which refers to the application in the Kubernetes context, corresponding to a certain deployment type resource instance such as deployment\statefulset on the cluster), the systemname and log path can be extracted through the log monitoring directory, stored in the database, and finally updated in the configuration file.

[0104] Understandably, S500, based on the log collection logic in the log logic file, calls the corresponding log tool through the log collection service to collect logs. It also includes: when a new path indicates a new application, the new application is added to the configuration file through the log collection service.

[0105] It should be noted that in some embodiments, since application modules deployed in the container orchestrator are added with each version iteration, a new path corresponding to the application module will be added to the directory of the corresponding application. In other embodiments, when a new application is deployed in the container orchestrator, a directory related to that application will be created. By recording applications in the configuration file, logs can be collected on a per-application-category basis, achieving better log management.

[0106] Understandably, adding new applications to the storage collection target via the log collection service includes: extracting the application name of the new application from the new path via the log collection service; using the application name as a parameter of the service extraction interface of the container orchestrator to obtain application information; and adding the application information to the configuration file via the log collection service.

[0107] It should be noted that since container orchestrators can manage all applications deployed on them, application information can be extracted through container orchestrators. For example, taking Kubernetes as an example, a request is made to the Kubernetes API server with the application name as a parameter to obtain application data, and the application middleware type corresponding to the application data is extracted, thereby obtaining the application tag as application information and storing it in the configuration file.

[0108] Understandably, in container orchestrators, because EFK requires the log pattern (log format) to match the EFK-configured recognition pattern, manual verification is required before deployment to ensure successful pattern recognition. Based on this, a recognition function is added: when scanning and collecting data, a portion of the logs is first matched against a pre-stored recognition pattern. Therefore, before performing log collection, refer to... Figure 5 As shown, the method also includes:

[0109] Step S610: Call the log tool through the log collection service to obtain the verification log.

[0110] It should be noted that the log verification is used to extract the log format of the file to be collected. Before collecting logs, a small number of logs can be collected from the path of each application using a log collection tool to verify the format.

[0111] Step S620: Perform pattern parsing on the verification log to obtain the first log pattern.

[0112] Step S630: Compare the first log model with the second log model preset by the container orchestrator.

[0113] It should be noted that the second log mode is the log format supported by default in EFK.

[0114] Step S640: Update the comparison result to the resource status parameters of the log service object, so that the container orchestrator can monitor the collection status of the log service object according to the resource status parameters.

[0115] It should be noted that by updating the comparison results to the resource status parameters, automatic checks can be achieved by monitoring the Loghavest resource status. For example, if a mismatch is displayed in the resource status parameters, it indicates that the first and second log modes do not match. In this case, manual intervention can be performed by comparing the first and second log modes in the comparison results.

[0116] Understandably, after log collection, the method also includes: counting the network traffic used for log collection; comparing the network traffic with a preset threshold; and reducing the concurrency of the logging tool according to a preset adjustment algorithm when the network traffic exceeds the threshold.

[0117] It should be noted that the level of concurrency affects network traffic; therefore, the amount of traffic used during log collection can be changed by adjusting the level of concurrency.

[0118] It should be noted that the threshold value can be set based on a real-time estimate of historical traffic usage. Alternatively, it can be simply set as a percentage of the traffic limit; this embodiment of the invention does not impose any restrictions on this.

[0119] For example, the Operator checks the network traffic of the host machine through the K8S metrics server. If the network traffic exceeds 70%, it adjusts the collector concurrency number bulk_max_size,worker in the filebeat configuration harvest01.yaml to control the concurrency of the collector and reduce the probability of the application being affected by log traffic.

[0120] The method of this invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and so on. This invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0121] Below, refer to Figures 1 to 5 Taking the Kubernetes platform as an example, the log collection method is as follows:

[0122] The operator listens for the creation of loghavest01, reads sepc.watchpath and filebeat01.yaml specified by spec.configmap from it, and periodically scans the directories under spec.watchpath (where watchpath is a subdirectory pattern for log generation agreed upon when building Kubernetes or integrating it, and its specific pattern is watchpath / [application name: systemname] / ).

[0123] Based on the application name under watchpath (in the Kubernetes context, this refers to an application instance of deployment type such as deployment or statefulset on the cluster), extract the systemname and log path, and store them in the database. Then, using the application name as a parameter, request the Kubernetes API server to retrieve application data, extract the application middleware type from the application data, and obtain the application tag. The Operator, based on the loghavest01 application's configuration file, first retrieves the spec.configfile from loghavest01 and reads the pre-existing filebeat configuration file harvest01.yaml in the configmap. It updates the log-path and application tag in harvest01.yaml according to the previously scanned log path. Then, it starts the demonset type filebeat collector application on Kubernetes. Subsequently, it periodically compares whether there are any newly added collection paths; if so, it updates the systemname and log path according to the above logic. The filebeat application is restarted and its `onfigmap` function is executed to reduce the probability of frequent manual maintenance of the collection targets. Simultaneously, to ensure consistency between the log format required by EFK and the recognition mode configured in EFK, and to reduce the number of manual reviews, a recognition function is added to the Operator. When scanning for access collection, a portion of the logs is first retrieved for pattern recognition to obtain the first log pattern, which is then matched with a pre-stored second log pattern. If this fails, the Operator updates the `unmach` field of `loghavest01`. Subsequent automatic checks can be achieved by monitoring the status of the `loghavest` resource (corresponding to the `unmach` field). Furthermore, the Operator checks and reads the host machine's network traffic through the Kubernetes metrics server. If the network traffic exceeds 70%, the concurrency of the collectors in the `harvest01.yaml` file in `filebeat` is adjusted (`bulk_max_size,worker`) to control the concurrency of the collectors in the log tool, reducing the probability of log traffic affecting the application itself.

[0124] Therefore, based on the above embodiments, using the CDR+OPRERATOR approach to host the backup service to the K8S cluster enables self-hosting of the log tool. Simultaneously, by adopting the K8S officially supported platform plugin development model, the log collection service is decoupled from specific log collection tools and deployments. Users can select the tools, log content, and destination server they need based on the log analysis platform. This offers the following benefits to users: 1. Reduced usage costs and learning burden; 2. Decoupling the log collection service from its specific implementation, facilitating subsequent replacement and migration. For service maintenance, it offers the following benefits: 1. Containerized backup service facilitates upgrades and migrations; 2. One-time build, repeated use, reducing maintenance costs; 3. Simple daily operations, such as fault restarts, traffic limiting, and maintenance of collection objects, can be automatically completed by K8S. For the platform, it enjoys the low-cost advantages brought by container orchestration.

[0125] Secondly, referring to Figure 6 As shown, a device with automatic log collection function according to the present invention includes:

[0126] Resource definition module 100 is used to store a preset log abstract file as a custom resource;

[0127] The acquisition module 200 is used to acquire preset log collection general parameters and preset log logic files, wherein the log collection general parameters include log tools;

[0128] The object creation module 300 is used to take the common parameters of log collection as parameters of the log abstract file and create a log service object based on the parameters of the log abstract file.

[0129] Service creation module 400 is used to create a log collection service based on the log service object and the log logic file;

[0130] The running module 500 is used to collect logs by calling the corresponding log tool through the log collection service based on the log collection logic in the log logic file.

[0131] It should be noted that, in some embodiments, the device with automatic log collection function also includes a collection target maintenance module 600. The collection target maintenance module 600 is used to periodically compare the log collection paths in the log monitoring directory set in the log service object to identify whether there are any newly added file paths in the log monitoring directory, and save the newly added file paths in the configuration file in the log service object used to store the collection target paths.

[0132] It should be noted that in some embodiments, the device with automatic log collection function also includes a pattern recognition module. The pattern recognition module is used to automatically identify the pattern of the log file to be collected, determine whether its format is consistent with the format in the default log framework, and update the comparison result in the log service object. In this way, when the log collection service is automatically managed, the hosting platform can issue an early warning by monitoring the comparison result in real time, thereby improving the efficiency of log deployment.

[0133] It should be noted that in some embodiments, devices with automatic log collection functionality also include a traffic monitoring module. This module monitors and compares the network traffic used during log collection with a preset traffic threshold. When the traffic threshold is exceeded, the number of concurrent log collectors is reduced, thus lowering the network traffic consumed by log collection. Conversely, when the current network traffic is significantly lower than the preset traffic threshold, the number of concurrent log collectors is increased, thereby rapidly improving log collection speed. Therefore, by adjusting the current network traffic based on the comparison results, bandwidth can be fully utilized without affecting the normal operation of the application.

[0134] Thirdly, according to embodiments of the present invention, an electronic device includes:

[0135] At least one processor, and,

[0136] A memory that is communicatively connected to at least one processor; wherein,

[0137] The memory stores instructions that are executed by at least one processor to enable the automatic log collection method described in the embodiments of the present invention when the at least one processor executes the instructions.

[0138] It should be noted that since the electronic device of the third aspect performs the steps in the automatic log collection method of the first aspect, it has all the beneficial effects of the first aspect.

[0139] The following is combined Figure 7 The hardware structure of the computer device is described in detail. This electronic device includes: a processor 710, a memory 720, an input / output interface 730, a communication interface 740, and a bus 750.

[0140] The processor 710 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0141] The memory 720 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 720 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 720 and called by the processor 710 to execute the training method of the model in the embodiments of this disclosure.

[0142] The input / output interface 730 is used to implement information input and output;

[0143] The communication interface 740 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and the bus 750 is used to transmit information between the various components of the device (such as processor 710, memory 720, input / output interface 730 and communication interface 740).

[0144] The processor 710, memory 720, input / output interface 730 and communication interface 740 are connected to each other within the device via bus 750.

[0145] For example, in some embodiments, the processor 710 executes steps S100 to S500 of the automatic log collection method of the first aspect. In other embodiments, the processor 710 executes steps S100 to S500 and steps S510 to S580 of the automatic log collection method of the first aspect. In still other embodiments, the processor 710 executes steps S100 to S500, steps S510 to S580, and steps S610 to S640 of the automatic log collection method of the first aspect. In still other embodiments, the processor 710 executes all steps of the automatic log collection method of the first aspect.

[0146] Fourthly, according to embodiments of the present invention, a storage medium is provided, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the log automatic collection method of the present disclosure embodiments.

[0147] It should be noted that, since the computer-readable storage medium of the fourth aspect performs the steps in the method of automatic log collection of the first aspect, it has all the beneficial effects of the first aspect.

[0148] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0149] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0150] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0151] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0152] The terms “comprising” and “having” and any variations thereof used in this specification are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product or device.

[0153] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0154] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A method for automatic log collection, applied to a container orchestrator, characterized in that, include: Store the pre-defined log abstract file as a custom resource; Obtain preset general log collection parameters and preset log logic files, wherein the general log collection parameters include log tool information, configuration files and log monitoring directories; The general parameters for log collection are used as parameters for the log abstract file, and a log service object is created based on the parameters of the log abstract file. Create a log collection service based on the log service object and the log logic file; The verification log is obtained by calling the log tool through the log collection service. The verification log is parsed to obtain the first log pattern; Compare the first log pattern with the second log model preset by the container orchestrator; The comparison result is updated to the resource status parameters of the log service object, so that the container orchestrator can monitor the collection status of the log service object according to the resource status parameters. Based on the log collection logic in the log logic file, the log collection service calls the log tool corresponding to the log tool information to collect logs. The log collection logic is used to set the operation behavior of the log collection service calling the log tool to collect logs, and the operation behavior includes: Collect log files in the first log collection directory recorded in the configuration file; Update the newly added path of the second log collection directory relative to the first log collection directory in the log monitoring directory to the first log collection directory, and collect the logs corresponding to the newly added path in the first log collection directory; Wherein, when the newly added path indicates a newly added application, the configuration file contains application information corresponding to the application name of the newly added application extracted by the container orchestrator.

2. The method for automatic log collection according to claim 1, characterized in that, When the general parameters for log collection include a configuration file, the configuration file is used to store the collection target, and the log abstract file includes the target server; The step of collecting logs by calling the log tool corresponding to the log tool information through the log collection service according to the log collection logic in the log logic file includes: The configuration file is extracted from the log service object through the log collection service, and the first log collection directory is determined from the configuration file; The target server is determined from the log service object through the log collection service; The log collection service invokes the log tool to collect log files in the first log collection directory under the target server.

3. The method for automatic log collection according to claim 2, characterized in that, When the general parameters for log collection also include a log monitoring directory; the step of calling the log tool corresponding to the log tool information through the log collection service to perform log collection based on the log collection logic in the log logic file further includes: The log monitoring directory is extracted from the log service object through the log collection service; The second log collection directory is extracted from the log monitoring directory according to a preset cycle; Compare the currently extracted second log collection directory with the first log collection directory; When the second log collection directory has a new path relative to the first log collection directory, the log collection service adds the new path to the configuration file, thereby updating the first log collection directory. Restart the log collection service so that the log tool can collect logs based on the first log collection directory that has been updated in the configuration file.

4. The method for automatic log collection according to claim 3, characterized in that, The step of collecting logs by calling the log tool corresponding to the log tool information through the log collection service according to the log collection logic in the log logic file further includes: When the newly added path indicates that a new application has been added, the application name of the newly added application is extracted from the newly added path through the log collection service; The application name is used as a parameter of the service extraction interface of the container orchestrator to obtain application information; The application information is added to the configuration file through the log collection service.

5. The method for automatic log collection according to claim 1, characterized in that, After log collection, the method further includes: Statistical analysis of the network traffic used for the log collection; The network traffic is compared with a preset threshold value; When the network traffic exceeds the threshold, the concurrency of the logging tool is reduced according to a preset adjustment algorithm.

6. A device with automatic log collection function, characterized in that, include: The resource definition module is used to store a preset log abstract file as a custom resource; The acquisition module is used to acquire preset log collection general parameters and preset log logic files. The log collection general parameters include log tool information, configuration files, and log monitoring directories. The object creation module is used to take the general parameters of log collection as parameters of the log abstract file, and create a log service object according to the parameters of the log abstract file; The service creation module is used to create a log collection service based on the log service object and the log logic file; The execution module is used to call the logging tool through the log collection service to obtain verification logs; perform pattern parsing on the verification logs to obtain a first log pattern; compare the first log pattern with a second log model preset by the container orchestrator; update the comparison result to the resource status parameters of the log service object, so that the container orchestrator monitors the collection status of the log service object according to the resource status parameters, and calls the logging tool corresponding to the logging tool information through the log collection service to collect logs according to the log collection logic in the log logic file. The log collection logic is used to set the operation behavior of the log collection service calling the log tool to collect logs, and the operation behavior includes: Collect log files in the first log collection directory recorded in the configuration file; Update the newly added path of the second log collection directory relative to the first log collection directory in the log monitoring directory to the first log collection directory, and collect the logs corresponding to the newly added path in the first log collection directory; Wherein, when the newly added path indicates a newly added application, the configuration file contains application information corresponding to the application name of the newly added application extracted by the container orchestrator.

7. An electronic device, characterized in that, include: At least one processor, and, A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that are executed by at least one processor to enable the automatic log collection method as described in any one of claims 1 to 5 when the at least one processor executes the instructions.

8. A computer-readable storage medium, characterized in that, The system stores computer-executable instructions for performing the automatic log collection method as described in at least any one of claims 1 to 5.

Images