Log collection method and device, electronic equipment and computer program product

Abstract

The invention discloses a log collection method and device, electronic equipment and a computer program product. The method comprises the following steps: according to a preset time interval, polling and monitoring a first log generation rate of generating an application log by each container group in a to-be-collected object, and obtaining a second log generation rate of generating the application log by the to-be-collected object; under the condition that the second log generation rate exceeds a preset rate threshold value, multiple container groups in the to-be-collected object are divided into a first container group set and a second container group set, the total rate of application logs generated by the multiple container groups in the first container group set is the third log generation rate, and the total rate of application logs generated by the multiple container groups in the second container group set is the third log generation rate; the generation rate of the third log does not exceed a preset rate threshold; adopting a management mode to collect application logs generated by a plurality of container groups in the first container group set; and collecting application logs generated by the plurality of container groups in the second container group set by adopting a side vehicle mode. The technical problem of low log collection efficiency is solved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a log collection method, device, electronic equipment and computer program product. Background Art

[0002] In a complete information system, the log system plays a vital role. It can record all the behaviors generated by the system and output them according to the set specifications or formats. Application logs can be used to troubleshoot the system, optimize system performance, or adjust the system behavior based on this information. In the security field, application logs are also one of the most important functions in security auditing.

[0003] Container products have quickly become popular around the world due to their good isolation, high portability, low resource usage, and fast startup. Kubernetes (k8s) is a cross-host cluster container scheduling platform that can realize the automated deployment, expansion, and operation of containers. Container technology has brought a lot of convenience to people in the deployment and delivery links, but it has brought many new challenges in the field of log processing. It mainly includes how to create a solution for persistent log retention to solve the problem of short life cycle of container applications. At the same time, it can also help us understand the system operation status globally, quickly locate the problem site, and restore the context.

[0004] Therefore, a reasonable container log collection solution is particularly important. At present, the following solutions are commonly used to collect k8s container logs:

[0005] Solution 1: Collect k8s container logs through DaemonSet mode, that is, run a collection client on each node. This deployment method communicates with the docker.sock or containerd.sock on the node when the container is running to discover all containers on the node, obtain the standard output path and storage path of the container from the returned container information, and mount the host path to collect data. This method can collect logs of all Pods running on this node.

[0006] If solution 1 is adopted, only one collection container needs to be deployed on each node, which can save resources and is non-invasive. However, for Job containers, which have a high frequency of addition and deletion and a short life cycle, the k8s container discovery mechanism depends on communication with containerd.sock, and containerd.sock cannot obtain container creation and destruction events in real time. It can only query the container status through a polling mechanism (polling interval is 1s). In this case, the container may be destroyed before it is discovered and log data cannot be collected. At the same time, since one collection container is responsible for collecting all pod logs on the node, it is easy to reach the bottleneck of throughput.

[0007] Solution 2: Collect k8s container logs through SideCar mode. That is, run a collection client in each Pod in the form of a container. This method uses the feature that containers in the same Pod of k8s can share storage volumes to collect the logs of the Pod.

[0008] If solution 2 is adopted, the collected log directory needs to be mounted and shared with the collection container, so the standard output of the container cannot be collected directly, and only the logs output to the file can be collected. Since each Pod is bound to a collection container, resource consumption is large, and the meta information of the container cannot be collected automatically, and needs to be exposed to the collection container through environment variables and other methods. At the same time, because it is bound to the Pod, it is highly invasive, and when the agent crashes, it will affect the business container.

[0009] Solution 3: Use a combination of Sidecar and DaemonSet to collect logs. Use Sidecar to collect application logs and DaemonSet to collect system logs. Configure a DaemonSet workload on each node server to collect system logs on the corresponding node server; configure a Sidecar on each Pod to collect Pod logs.

[0010] If solution three is adopted, the system logs of the node server and the logs of the container application can be collected at the same time, but it is still invasive to the application; when the throughput of the host log is too large, it is easy to cause a system bottleneck.

[0011] In summary, the current container log collection solution can only collect logs for pre-configured collection methods, but cannot reasonably utilize the resource consumption of collecting logs. In addition, when the host log throughput is too large to cause a system bottleneck, it is impossible to accurately complete the log collection, resulting in low log collection efficiency.

[0012] Currently, no effective solution has been proposed to address the above-mentioned problem of low log collection efficiency. Summary of the invention

[0013] Embodiments of the present invention provide a log collection method, device, electronic device, and computer program product to at least solve the technical problem of low log collection efficiency.

[0014] According to one aspect of an embodiment of the present invention, a log collection method is provided, comprising: at a preset time interval, polling and monitoring a first log generation rate of application logs generated by each container group in an object to be collected, to obtain a second log generation rate of the application logs generated by the object to be collected, wherein the object to be collected includes multiple container groups; when the second log generation rate exceeds a preset rate threshold, dividing the multiple container groups in the object to be collected into a first container group set and a second container group set, wherein the total rate at which the multiple container groups in the first container group set generate the application logs is a third log generation rate, and the third log generation rate does not exceed the preset rate threshold; collecting the application logs generated by the multiple container groups in the first container group set in a management mode, wherein the management mode is used to run a first collection container on each computing node of the object to be collected, and the first collection container is used to collect the application logs generated by all container groups in the same computing node; collecting the application logs generated by the multiple container groups in the second container group set in a sidecar mode, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect the application logs generated by the container group.

[0015] Optionally, at a preset time interval, polling and monitoring a first log generation rate of application logs generated by each container group in the object to be collected to obtain a second log generation rate of the application logs generated by the object to be collected includes: detecting a controller type of a controller corresponding to each container group in the object to be collected, wherein the controller type includes at least a first preset type and a second preset type, the container group corresponding to the controller of the first preset type is a first type of container group, and the container group corresponding to the controller of the second preset type is a second type of container group; using the management mode to collect the application logs generated by multiple first type container groups; at a preset time interval, polling and monitoring a rate at which multiple second type container groups generate the application logs to obtain the first log generation rate, and counting the sum of the first log generation rates to obtain the second log generation rate.

[0016] Optionally, after adopting the management mode to collect the application logs generated by multiple container groups in the first container group set, the method further includes: monitoring whether there is a newly added container application in the object to be collected, wherein the newly added container application is used to generate multiple replica container groups, and the replica container group is a replica of the container group existing in the object to be collected; when the newly added container application exists in the object to be collected, determining a fourth log generation rate of the application logs generated by the object to be collected, wherein the fourth log generation rate is the sum of the third log generation rate and a fifth log generation rate of the application logs generated by multiple replica container groups; when the fourth log generation rate exceeds the preset rate threshold, adopting the sidecar mode to collect the application logs generated by the replica container group; when the fourth log generation rate does not exceed the preset rate threshold, adopting the management mode to collect the application logs generated by the replica container group.

[0017] Optionally, after collecting the application logs generated by multiple container groups in the first container group set in management mode, the method further includes: monitoring a first rate change of the third log generation rate, wherein the first rate change is used to indicate a first duration for which the third log generation rate exceeds the preset rate threshold; when the first duration exceeds the first duration threshold, selecting multiple container groups from the first container group set to obtain a first container group subset, wherein the difference between the third log generation rate and a sixth log generation rate for generating the application logs by multiple container groups in the first container group subset does not exceed the preset rate threshold; and collecting the application logs generated by multiple container groups in the first container group subset in sidecar mode.

[0018] Optionally, when the first duration exceeds a first duration threshold, selecting multiple container groups from the first container group set to obtain a first container group subset includes: arranging the seventh log generation rate of the application log generated by each container group in the first container group set in descending order to obtain a first rate queue; when the first duration exceeds the first duration threshold, intercepting multiple container groups from the tail of the first rate queue to obtain the first container group subset, wherein the sum of the seventh log generation rate of the application log generated by each intercepted container group is the sixth log generation rate.

[0019] Optionally, after collecting the application logs generated by multiple container groups in the first container group set in management mode, the method further includes: monitoring a second rate change of the third log generation rate, and whether there is a situation in which the application logs are collected in the sidecar mode in the object to be collected, wherein the second rate change is used to indicate that the third log generation rate does not exceed the second duration of the preset rate threshold; when the second duration exceeds the second duration threshold, and there is a situation in which the application logs are collected in the sidecar mode in the object to be collected, multiple container groups are selected from the second container group set to obtain a second container group subset, wherein the sum of the third log generation rate and the eighth log generation rate of the application logs generated by the multiple container groups in the second container group subset does not exceed the preset rate threshold; and collecting the application logs generated by the multiple container groups in the second container group subset in management mode.

[0020] Optionally, when the second duration exceeds a second duration threshold and there are objects to be collected that use the sidecar mode to collect the application logs, multiple container groups are selected from the second container group set to obtain a second container group subset, including: arranging the ninth log generation rate of the application logs generated by each of the container groups in the second container group set in order from large to small to obtain a second rate queue; when the second duration exceeds the second duration threshold and there are objects to be collected that use the sidecar mode to collect the application logs, multiple container groups are intercepted from the head of the second rate queue to obtain the second container group subset, wherein the sum of the ninth log generation rate of the application logs generated by each of the intercepted container groups is the eighth log generation rate.

[0021] According to another aspect of an embodiment of the present invention, a log collection device is further provided, comprising: a monitoring module, configured to poll and monitor, at a preset time interval, a first log generation rate of application logs generated by each container group in an object to be collected, to obtain a second log generation rate of the application logs generated by the object to be collected, wherein the object to be collected includes a plurality of the container groups; a division module, configured to divide the plurality of the container groups in the object to be collected into a first container group set and a second container group set when the second log generation rate exceeds a preset rate threshold, wherein the total rate at which the plurality of container groups in the first container group set generate the application logs is a third log generation rate, and the third log generation rate is The rate does not exceed the preset rate threshold; a first collection module, used to collect the application logs generated by multiple container groups in the first container group set by adopting a management mode, wherein the management mode is used to run a first collection container on each computing node of the object to be collected, and the first collection container is used to collect the application logs generated by all container groups in the same computing node; a second collection module, used to collect the application logs generated by multiple container groups in the second container group set by adopting a sidecar mode, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect the application logs generated by the container group.

[0022] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the log collection method through the computer program.

[0023] According to another aspect of an embodiment of the present invention, a computer program product is further provided, including computer instructions, and when the computer instructions are executed by a processor, the steps of the log collection method are implemented.

[0024] In an embodiment of the present invention, a first log generation rate of application logs generated by each container group in an object to be collected is polled and monitored at a preset time interval to obtain a second log generation rate of application logs generated by the object to be collected, wherein the object to be collected includes multiple container groups; when the second log generation rate exceeds a preset rate threshold, the multiple container groups in the object to be collected are divided into a first container group set and a second container group set, wherein the total rate at which the multiple container groups in the first container group set generate application logs is a third log generation rate, and the third log generation rate does not exceed the preset rate threshold; the application logs generated by the multiple container groups in the first container group set are collected using a management mode, wherein the management mode is used A first collection container is run on each computing node of the object to be collected, and the first collection container is used to collect application logs generated by all container groups in the same computing node; a sidecar mode is used to collect application logs generated by multiple container groups in a second container group set, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect application logs generated by the container group. Therefore, according to the log generation rate of the application logs generated by the object to be collected, the management mode and the sidecar mode can be flexibly combined, thereby achieving the technical effect of ensuring the efficiency and accuracy of log collection without wasting collection resources, thereby solving the technical problem of low log collection efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 is a flow chart of a log collection method according to an embodiment of the present invention;

[0027] Figure 2 is a schematic diagram of a container log collection method capable of automatically adapting to a log generation rate according to an embodiment of the present invention;

[0028] Figure 3 is a schematic diagram of a log collection device according to an embodiment of the present invention;

[0029] Figure 4 It is a structural block diagram of a computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0033] k8s (Kubernetes, k8s for short): is an open source containerized application management platform for multiple hosts in the cloud platform. The goal of Kubernetes is to make the deployment of containerized applications simple and efficient. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance.

[0034] Pod (container group): It is the smallest unit that can be created and managed in the k8s system. It is the smallest resource object model created or deployed by the user in the resource object model. It is also the resource object for running containerized applications on k8s.

[0035] DaemonSet (management mode): It ensures that each node in the cluster runs the same set of Pods and automatically creates corresponding Pods for newly added nodes based on the node status. When a node is removed, the corresponding Pods can also be deleted. This mode is implemented through a controller, which is actually a daemon process. The type of this controller is DaemonSet.

[0036] SideCar mode: Without changing the original container function, other containers are added under the same Pod to add corresponding functions; when the SideCar method is used to collect container logs, a SideCar log agent container is run in a Pod to collect the logs generated by the main container of the Pod.

[0037] According to an embodiment of the present invention, a log collection method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] Figure 1 is a flow chart of a log collection method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0039] Step S102, at a preset time interval, polling and monitoring a first log generation rate of application logs generated by each container group in the object to be collected, to obtain a second log generation rate of application logs generated by the object to be collected, wherein the object to be collected includes multiple container groups;

[0040] Step S104: when the second log generation rate exceeds the preset rate threshold, multiple container groups in the object to be collected are divided into a first container group set and a second container group set, wherein the total rate at which multiple container groups in the first container group set generate application logs is a third log generation rate, and the third log generation rate does not exceed the preset rate threshold;

[0041] Step S106, collecting application logs generated by multiple container groups in the first container group set in a management mode, wherein the management mode is used to run a first collection container on each computing node of the object to be collected, and the first collection container is used to collect application logs generated by all container groups in the same computing node;

[0042] Step S108, using the sidecar mode to collect application logs generated by multiple container groups in the second container group set, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect application logs generated by the container group.

[0043] In an embodiment of the present invention, a first log generation rate of application logs generated by each container group in an object to be collected is polled and monitored at a preset time interval to obtain a second log generation rate of application logs generated by the object to be collected, wherein the object to be collected includes multiple container groups; when the second log generation rate exceeds a preset rate threshold, the multiple container groups in the object to be collected are divided into a first container group set and a second container group set, wherein the total rate at which the multiple container groups in the first container group set generate application logs is a third log generation rate, and the third log generation rate does not exceed the preset rate threshold; the application logs generated by the multiple container groups in the first container group set are collected using a management mode, wherein the management mode is used A first collection container is run on each computing node of the object to be collected, and the first collection container is used to collect application logs generated by all container groups in the same computing node; a sidecar mode is used to collect application logs generated by multiple container groups in a second container group set, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect application logs generated by the container group. Therefore, according to the log generation rate of the application logs generated by the object to be collected, the management mode and the sidecar mode can be flexibly combined, thereby achieving the technical effect of ensuring the efficiency and accuracy of log collection without wasting collection resources, thereby solving the technical problem of low log collection efficiency.

[0044] In the above step S102, the preset time interval may be a preconfigured polling monitoring time. When the preset time interval is reached, a polling monitoring is performed to update the first log generation rate of the application log generated by each container group in the collection object, thereby ensuring that the collection mode selected based on the first log generation rate can effectively collect logs, thereby avoiding the waste of collection resources caused by excessive use of the sidecar mode, and avoiding the sidecar mode reaching a collection bottleneck without being discovered.

[0045] In the above step S102, the log collection scheme is implemented by adding a collection container to the object to be collected. The collection container can be a containerized application with a log collection function, that is, a collection program can be run in the collection container, and the collection program performs log collection.

[0046] Optionally, the collection container includes: a first collection container and a second collection container.

[0047] In the above step S102, the object to be collected may be a server, and the server deploys applications using containerization technology.

[0048] In the above step S102, the container group Pod is the smallest deployable unit. Each container group Pod includes one or more application containers. When the application in the container group Pod is running, an application log of the application will be generated. The rate at which one or more applications in each container group generate application logs is the log generation rate of the container group.

[0049] In the above step S102, the first log generation rate is the rate at which each container group generates the application log.

[0050] In the above step S102, the second log generation rate is the total rate at which all container groups in the object to be collected generate the application logs; that is, the second log generation rate is the sum of multiple first log generation rates.

[0051] In the above step S104, the preset rate threshold can be determined according to the bottleneck of collecting application logs. When the second log generation rate of the object to be collected exceeds the preset rate threshold, it means that the bottleneck of collecting application logs has been reached. Continuing to collect in the original way will result in inaccurate collection of application logs due to the destruction time limit of the application logs. Therefore, when the second log generation rate exceeds the preset rate threshold, multiple container groups in the object to be collected can be split to obtain a first container group set whose total rate of generating application logs is lower than the preset rate threshold, and a second container group set including multiple container groups in the object to be collected except the first container group set.

[0052] In the above step S104, the third log generation rate may be the total rate of the application logs of all container groups in the first container group set; that is, the third log generation rate is the sum of the first log generation rates of multiple container groups in the first container group set.

[0053] In the above step S106, the management mode DaemonSet may be a default collection mode for collecting application logs generated in the collection object.

[0054] Optionally, the preset rate threshold may be determined according to a collection bottleneck of the management mode.

[0055] In the above step S106, since the collection bottleneck of the management mode is the preset rate threshold, the management mode can be used to effectively collect application logs for multiple container groups whose log generation rates are lower than the preset rate threshold.

[0056] As an optional example, when the second log generation rate does not exceed a preset rate threshold, the management mode may be adopted to collect application logs generated by all container groups in the collection object.

[0057] As another optional example, the third log generation rate of the first container group set does not exceed the preset rate threshold, so the management mode can be used to collect application logs generated by all container groups in the first container group set.

[0058] In the above step S108, the sidecar mode SideCar is adopted to collect application logs for multiple container groups in the second container group set that exceed the preset rate threshold.

[0059] As an optional embodiment, polling and monitoring a first log generation rate of application logs generated by each container group in the object to be collected at a preset time interval to obtain a second log generation rate of application logs generated by the object to be collected includes: detecting a controller type of a controller corresponding to each container group in the object to be collected, wherein the controller type includes at least a first preset type and a second preset type, the container group corresponding to the controller of the first preset type is a first type container group, and the container group corresponding to the controller of the second preset type is a second type container group; collecting application logs generated by multiple first type container groups in a management mode; polling and monitoring a rate at which multiple second type container groups generate application logs at a preset time interval to obtain a first log generation rate, and counting the sum of the first log generation rates to obtain a second log generation rate.

[0060] It should be noted that the container group POD is the basic execution unit of the application. A container group POD can contain one or more containers, which share storage, network and running configurations. The controller of the container group POD is responsible for ensuring the lifecycle management of the container group POD and ensuring that at any given time, there are the correct number of copies of the container group POD running in the cluster. If a container group POD fails or is deleted, the controller of the container group POD will automatically replace it to maintain the continuous availability and elasticity of the service.

[0061] In the above-mentioned embodiment of the present application, each container group has a corresponding controller. The type of the container group can be distinguished according to the type of the controller. Different container groups can use different collection modes for collection. Therefore, when assigning collection modes to container groups, the type of controller corresponding to each container group can be identified first, and then the corresponding collection mode can be selected for different container groups for collection, thereby realizing reasonable allocation of collection modes.

[0062] Optionally, the controller types include: Deployment, suitable for stateless service deployment; StatefulSet, suitable for stateful service deployment; DaemonSet, ensuring that there is only one pod on each node; Job, one-time execution of tasks; CronJob, periodic execution of tasks.

[0063] As an optional example, the first preset type includes: Job and CronJob. For Job and CronJob, two types of containers with high frequency of addition and deletion and short life cycle, the collection container is deployed in SideCar mode to prevent the situation where the container is not discovered or the log collection has not started, and the container application has been destroyed and log data cannot be collected due to the delay of container discovery and data collection (mainly from the polling interval of the docker_file plug-in and the frequency limit of the C++ core part loading the latest container configuration).

[0064] As an optional example, the second preset type includes: Deployment, StatefulSet and DaemonSet. For the controller of the second preset type, the management mode DaemonSet deployment is adopted by default.

[0065] Optionally, as the number of Pods deployed on the server nodes increases, the amount of logs that need to be collected will also increase significantly. When the throughput bottleneck is exceeded, the log collection speed will be significantly slowed down. Suppose the controllers that use the management mode DaemonSet to collect logs on the current node are C1, C2, ...C i , the corresponding number of copies are R1, R2, ...R j , there are i*j pods in total, and their corresponding log generation rates are S1, S2...S i*j , then the log generation rate of the controller of the current server node that uses the management mode DaemonSet to collect logs is S = S1 + S2 + ... + S i*j .

[0066] As an optional embodiment, after collecting application logs generated by multiple container groups in the first container group set in a management mode, the method also includes: monitoring whether there is a newly added container application in the object to be collected, wherein the newly added container application is used to generate multiple replica container groups, and the replica container group is a replica of the container group existing in the object to be collected; when there is a newly added container application in the object to be collected, determining a fourth log generation rate of the application logs generated by the object to be collected, wherein the fourth log generation rate is the sum of the third log generation rate and a fifth log generation rate of the application logs generated by the multiple replica container groups; when the fourth log generation rate exceeds a preset rate threshold, collecting the application logs generated by the replica container group in a sidecar mode; when the fourth log generation rate does not exceed the preset rate threshold, collecting the application logs generated by the replica container group in a management mode.

[0067] In the above embodiment of the present application, the management mode DaemonSet should be used to collect application logs. Therefore, when there is a newly added container application in the object to be collected, the rate of application logs generated by the newly added replica container group can be used as the fifth log generation rate, and then the fifth log production rate is summed with the third log production rate originally collected by the management mode DaemonSet. It can be obtained that, assuming that the management mode DaemonSet is used to collect application logs for the newly added replica container group, the total rate of application logs generated by all container groups collected by the management mode DaemonSet, that is, the fourth log generation rate, can be obtained. Further, when the fourth log generation rate exceeds the preset rate threshold, it means that the management mode DaemonSet cannot collect application logs for the newly added container well. Therefore, when the fourth log generation rate exceeds the preset rate threshold, the sidecar mode SideCar is needed to collect the application logs generated by the replica container group; and when the fourth log generation rate does not exceed the preset rate threshold, the management mode DaemonSet can be used to collect the application logs generated by the replica container group, thereby realizing a reasonable selection of the collection mode of the application logs, thereby improving the collection efficiency of the application logs.

[0068] As an optional embodiment, after collecting application logs generated by multiple container groups in the first container group set in a management mode, the method also includes: monitoring a first rate change of a third log generation rate, wherein the first rate change is used to indicate that the third log generation rate exceeds a first duration of a preset rate threshold; when the first duration exceeds the first duration threshold, selecting multiple container groups from the first container group set to obtain a first container group subset, wherein the difference between the third log generation rate and a sixth log generation rate of application logs generated by multiple container groups in the first container group subset does not exceed the preset rate threshold; and collecting application logs generated by multiple container groups in the first container group subset in a sidecar mode.

[0069] In the above-mentioned embodiment of the present application, when an application in an existing container group is running, the rate at which the application logs generated by the already running application may change. Therefore, by monitoring the third log generation rate of the first container group set that uses the management mode for collection, it is possible to avoid using the management mode for application log collection, resulting in the log collection rate exceeding the collection bottleneck. Moreover, by monitoring the first duration for which the third log generation rate exceeds the preset rate threshold, it is possible to avoid the instantaneous rate at which the first container group set generates application logs exceeding the preset rate threshold, thereby causing frequent changes in the collection mode. Furthermore, when the third log generation rate of the first container group set exceeds the preset rate threshold and the first duration for which the third log generation rate exceeds the preset rate threshold exceeds the first duration threshold, some container groups are selected from the first container group set to switch from the management mode to the sidecar mode for application log collection. This allows flexible scheduling of the collection mode of the application logs, thereby improving the collection efficiency of the application logs.

[0070] It should be noted that, when some container groups are selected from the first container group set as the first container group subset, it is necessary to ensure that the total rate at which multiple container groups other than the first container group subset in the first container group set generate application logs does not exceed a preset rate threshold, so as to ensure that the management mode can effectively collect application logs for the first container group set excluding the first container group subset.

[0071] As an optional embodiment, when the first duration exceeds the first duration threshold, multiple container groups are selected from the first container group set to obtain a first container group subset, including: arranging the seventh log generation rate of the application log generated by each container group in the first container group set in order from large to small to obtain a first rate queue; when the first duration exceeds the first duration threshold, intercepting multiple container groups from the tail of the first rate queue to obtain a first container group subset, wherein the sum of the seventh log generation rate of the application log generated by each intercepted container group is the sixth log generation rate.

[0072] In the above embodiment of the present application, when the first container group subset is selected from the first container group set, multiple container groups in the first container group set can be arranged in descending order according to the rate of generating application logs to obtain a first rate queue, and then multiple container groups are intercepted from the tail of the first rate queue as the first container group subset for switching to the sidecar mode for collection, which can ensure that the management mode is used preferentially to collect application logs for the container group, avoiding excessive occupation of collection resources for collecting application logs.

[0073] As an optional embodiment, after collecting application logs generated by multiple container groups in the first container group set in management mode, the method also includes: monitoring a second rate change of a third log generation rate, and whether there is a situation in which the application logs are collected in sidecar mode in the object to be collected, wherein the second rate change is used to indicate that the third log generation rate does not exceed a second duration of a preset rate threshold; when the second duration exceeds the second duration threshold, and there is a situation in which the application logs are collected in sidecar mode in the object to be collected, multiple container groups are selected from the second container group set to obtain a second container group subset, wherein the sum of the third log generation rate and the eighth log generation rate of the application logs generated by the multiple container groups in the second container group subset does not exceed the preset rate threshold; and collecting the application logs generated by the multiple container groups in the second container group subset in management mode.

[0074] In the above embodiment of the present application, when an application in an existing container group is running, the rate at which the application logs are generated by the already running application may change. Therefore, by monitoring the rate at which the third logs are generated by the first container group set that is collected using the management mode, it is possible to avoid using the management mode to collect application logs, which results in a waste of collection resources in the management mode. Moreover, by monitoring the second duration for which the third log generation rate does not exceed the preset rate threshold, it is possible to avoid the instantaneous rate at which the first container group set generates application logs not exceeding the preset rate threshold, which results in frequent changes in the collection mode. Furthermore, when the rate at which the third logs are generated by the first container group set does not exceed the preset rate threshold and the second duration for which the third logs do not exceed the preset rate threshold exceeds the second duration threshold, some container groups are selected from the second container group set to switch from the sidecar mode to the management mode for collecting application logs. This allows for flexible scheduling of the collection mode of application logs, thereby improving the collection efficiency of application logs.

[0075] Optionally, the second duration threshold may be the same as the first duration threshold.

[0076] It should be noted that, when some container groups are selected from the second container group set as the second container group subset, it is necessary to ensure that the total rate at which the first container group set and the plurality of container groups in the second container group subset generate application logs does not exceed a preset rate threshold, so as to ensure that the management mode can effectively collect application logs for the first container group set excluding the second container group subset.

[0077] As an optional embodiment, when the second duration exceeds the second duration threshold and there are application logs collected in sidecar mode among the objects to be collected, multiple container groups are selected from the second container group set to obtain a second container group subset, including: arranging the ninth log generation rate of the application logs generated by each container group in the second container group set in order from large to small to obtain a second rate queue; when the second duration exceeds the second duration threshold and there are application logs collected in sidecar mode among the objects to be collected, multiple container groups are intercepted from the head of the second rate queue to obtain a second container group subset, wherein the sum of the ninth log generation rate of the application logs generated by each intercepted container group is the eighth log generation rate.

[0078] In the above embodiment of the present application, when selecting the second container group subset from the second container group set, multiple container groups in the second container group set can be arranged in descending order according to the rate of generating application logs to obtain a second rate queue, and then multiple container groups are intercepted from the head of the second rate queue as the second container group subset for switching to the management mode for collection, which can ensure that the management mode is used preferentially to collect application logs for the container group, thereby avoiding excessive occupation of collection resources for collecting application logs.

[0079] The present invention also provides an optional embodiment, which provides a container log collection method that can automatically adapt to the collection throughput. The method comprehensively considers the application container type to adopt different collection methods, and also considers the current throughput bottleneck of the collection container. According to the log generation rate during the log collection process, the collection method of the adapted log is automatically adjusted to achieve the collection of container logs while minimizing the intrusion.

[0080] Figure 2 is a schematic diagram of a container log collection method capable of automatically adapting to the log generation rate according to an embodiment of the present invention, such as Figure 2 As shown, the container log collection method process may include: if it is necessary to collect the system log of the server node (that is, to collect the host log), the collection container is deployed in the form of DaemonSet; if it is necessary to collect the application log generated by the container running in the server node (that is, to collect the container log), it is necessary to judge the type of the controller that manages the container in the server node.

[0081] It should be noted that common controllers in k8s clusters include the following types: Deployment, suitable for stateless service deployment; StatefulSet, suitable for stateful service deployment; DaemonSet, ensuring that there is only one Pod on each node; Job, one-time execution of tasks; CronJob, periodic execution of tasks.

[0082] As an optional example, for Job and CronJob, two types of controllers with high addition and deletion frequencies and short life cycles, the collection container is deployed in SideCar mode to prevent the situation where the container is not discovered or log collection has not started, and the container application is destroyed and log data cannot be collected due to the time-consuming container discovery and data collection delays (mainly from the polling interval of the docker_file plug-in and the frequency limit of loading the latest container configuration of the C++ core part).

[0083] As another optional example, for the controllers of Deployment, StatefulSet, and DaemonSet, the DaemonSet mode is used by default. As the number of Pods deployed on the server nodes increases, the amount of logs that need to be collected will also increase significantly. When the throughput bottleneck is exceeded, the log collection speed will be significantly slowed down.

[0084] Optionally, assume that the DaemonSet mode is used on the current server node, and the controllers for collecting logs are C1, C2, ...C i , the corresponding number of Pod replicas are R1, R2, ...R j , there are a total of i*j Pods, and the log generation rates corresponding to each Pod are S1, S2...S i*j , the current server node uses the DaemonSet mode to collect logs, and the log generation rate of its controller is S = S1 + S2 + ... + S i*j .

[0085] It should be noted that, when the collection container runs on the server node in DaemonsSet mode, the throughput bottleneck of the collection container is the maximum throughput of the collection container.

[0086] Optionally, the collection containers mainly use: FileBeat (i.e., log data collector) and Loggie (i.e., log data collector), etc. Among them, the maximum throughput of FileBeat is about 80M / s, and the maximum throughput of Loggie is about 200M / s, which varies depending on the node server configuration.

[0087] Optionally, for a single application, its log generation rate is gradually increased from the initial value of 10M / s to 300M / s in steps of 10M / s, and the collected logs are sent to Kafka (i.e., a distributed stream computing platform). When the throughput bottleneck is not reached, the number of messages accumulated in Kafka increases in direct proportion to the log generation rate. When the message accumulation in Kafka begins to reach the bottleneck and does not grow linearly, it is considered that the throughput bottleneck has been reached, and the log generation rate at this time is called the bottleneck rate B.

[0088] Optionally, when adding a new container application T, the number of its Pod replicas is N, and the log generation rate of each Pod of the application is monitored as S T1 ,S T2 ,...S TN . When M = S + S T1 + S T2 +...+ S TN ≤ B, the DaemonSet mode is used to collect the logs of the application; when M > B, the SideCar method is used to collect the logs.

[0089] Optionally, during the operation of the application, a timer is used to poll and monitor the log generation rate of each controller. The log production rates of the N Pod replicas currently collected in the DaemonSet mode are sorted in descending order L(S) = [S1, S2... S N , and their corresponding controllers are [C1, C2... C N ; the log generation rates of the M Pod replicas currently collected in the SideCar mode are sorted in descending order L(P) = [P1, P2... P M , and their corresponding controllers are [Q1, Q2... Q M .

[0090] Optionally, when S > B and lasts for a period of time, it is considered that a switch in the collection method is required. Select the first i values of L(S) such that S - S1 - S2 -... - S i ≤ B, and switch the collection method for C1, C2... C i , and use the SideCar mode to collect the container logs of the corresponding controllers. The system automatically modifies the controller's orchestration file and configures a container for it.

[0091] Optionally, when S < B and lasts for a period of time, and there are currently controllers using the SideCar mode for log collection, then the collection method of some controllers needs to be switched. Select the last j values of L(P) such that S + P M , P M- 1... P M-j+1 ≤ B, and switch the collection method for Q M , Q M-1 ... Q M-j+1 , and use the DaemonSet mode to collect the container logs of the corresponding controllers. The system automatically modifies the controller's orchestration file and adds the path of the controller log file to be monitored by the DaemonSet at the same time.

[0092] According to an embodiment of the present invention, a log collection device embodiment is also provided. It should be noted that the log collection device can be used to execute the log collection method in the embodiment of the present invention, and the log collection method in the embodiment of the present invention can be executed in the log collection device.

[0093] Figure 3 is a schematic diagram of a log collection device according to an embodiment of the present invention. Figure 3 As shown, the device may include: a monitoring module 32, which is used to poll and monitor the first log generation rate of application logs generated by each container group in the object to be collected at a preset time interval, and obtain a second log generation rate of application logs generated by the object to be collected, wherein the object to be collected includes multiple container groups; a division module 34, which is used to divide the multiple container groups in the object to be collected into a first container group set and a second container group set when the second log generation rate exceeds a preset rate threshold, wherein the total rate of application logs generated by the multiple container groups in the first container group set is a third log generation rate, and the third log generation rate does not exceed the preset rate threshold; a first collection module 36, which is used to collect application logs generated by multiple container groups in the first container group set in a management mode, wherein the management mode is used to run a first collection container on each computing node of the object to be collected, and the first collection container is used to collect application logs generated by all container groups in the same computing node; a second collection module 38, which is used to collect application logs generated by multiple container groups in the second container group set in a sidecar mode, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect application logs generated by the container group.

[0094] It should be noted that the monitoring module 32 in this embodiment can be used to execute step S102 in the embodiment of the present application, the division module 34 in this embodiment can be used to execute step S104 in the embodiment of the present application, the first acquisition module 36 in this embodiment can be used to execute step S106 in the embodiment of the present application, and the second acquisition module 38 in this embodiment can be used to execute step S108 in the embodiment of the present application. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0095] In an embodiment of the present invention, a first log generation rate of application logs generated by each container group in an object to be collected is polled and monitored at a preset time interval to obtain a second log generation rate of application logs generated by the object to be collected, wherein the object to be collected includes multiple container groups; when the second log generation rate exceeds a preset rate threshold, the multiple container groups in the object to be collected are divided into a first container group set and a second container group set, wherein the total rate at which the multiple container groups in the first container group set generate application logs is a third log generation rate, and the third log generation rate does not exceed the preset rate threshold; the application logs generated by the multiple container groups in the first container group set are collected using a management mode, wherein the management mode is used A first collection container is run on each computing node of the object to be collected, and the first collection container is used to collect application logs generated by all container groups in the same computing node; a sidecar mode is used to collect application logs generated by multiple container groups in a second container group set, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect application logs generated by the container group. Therefore, according to the log generation rate of the application logs generated by the object to be collected, the management mode and the sidecar mode can be flexibly combined, thereby achieving the technical effect of ensuring the efficiency and accuracy of log collection without wasting collection resources, thereby solving the technical problem of low log collection efficiency.

[0096] As an optional embodiment, the monitoring module includes: a detection unit, which is used to detect the controller type of the controller corresponding to each container group in the object to be collected, wherein the controller type includes at least: a first preset type and a second preset type, the container group corresponding to the controller of the first preset type is a first type container group, and the container group corresponding to the controller of the second preset type is a second type container group; a first collection unit, which is used to collect application logs generated by multiple first type container groups in a management mode; a monitoring unit, which is used to poll and monitor the rate at which multiple second type container groups generate application logs at a preset time interval to obtain a first log generation rate, and to count the sum of the first log generation rates to obtain a second log generation rate.

[0097] As an optional embodiment, the device also includes: a first monitoring submodule, which is used to monitor whether there is a new container application in the object to be collected after collecting application logs generated by multiple container groups in the first container group set in a management mode, wherein the new container application is used to generate multiple replica container groups, and the replica container group is a replica of the container group existing in the object to be collected; a determination submodule, which is used to determine a fourth log generation rate of the application logs generated by the object to be collected when there is a new container application in the object to be collected, wherein the fourth log generation rate is the sum of the third log generation rate and the fifth log generation rate of the application logs generated by the multiple replica container groups; a first collection submodule, which is used to collect the application logs generated by the replica container group in a sidecar mode when the fourth log generation rate exceeds a preset rate threshold; and a second collection submodule, which is used to collect the application logs generated by the replica container group in a management mode when the fourth log generation rate does not exceed the preset rate threshold.

[0098] As an optional embodiment, the device also includes: a second monitoring submodule, which is used to monitor a first rate change of a third log generation rate after collecting application logs generated by multiple container groups in the first container group set in a management mode, wherein the first rate change is used to indicate that the third log generation rate exceeds a first duration of a preset rate threshold; a first selection submodule, which is used to select multiple container groups from the first container group set to obtain a first container group subset when the first duration exceeds the first duration threshold, wherein the difference between the third log generation rate and a sixth log generation rate of application logs generated by multiple container groups in the first container group subset does not exceed the preset rate threshold; and a third collection submodule, which is used to collect application logs generated by multiple container groups in the first container group subset in a sidecar mode.

[0099] As an optional embodiment, the first selection submodule includes: a first arrangement subunit, used to arrange the seventh log generation rate of the application log generated by each container group in the first container group set in order from large to small, to obtain a first rate queue; a first interception subunit, used to intercept multiple container groups from the tail of the first rate queue when the first duration exceeds the first duration threshold, to obtain a first container group subset, wherein the sum of the seventh log generation rate of the application log generated by each intercepted container group is the sixth log generation rate.

[0100] As an optional embodiment, the device also includes: a third monitoring submodule, which is used to monitor the second rate change of the third log generation rate after collecting application logs generated by multiple container groups in the first container group set in management mode, and whether there is a situation in which the application logs are collected in the sidecar mode in the object to be collected, wherein the second rate change is used to indicate that the third log generation rate does not exceed the second duration of the preset rate threshold; a second selection submodule, which is used to select multiple container groups from the second container group set to obtain a second container group subset when the second duration exceeds the second duration threshold and there is a situation in which the application logs are collected in the sidecar mode in the object to be collected, wherein the sum of the third log generation rate and the eighth log generation rate of the application logs generated by the multiple container groups in the second container group subset does not exceed the preset rate threshold; a fourth collection submodule, which is used to collect application logs generated by multiple container groups in the second container group subset in management mode.

[0101] As an optional embodiment, the second selection submodule includes: a second arrangement subunit, used to arrange the ninth log generation rate of the application log generated by each container group in the second container group set in order from large to small, to obtain a second rate queue; a second interception subunit, used to intercept multiple container groups from the head of the second rate queue when the second duration exceeds the second duration threshold and there are application logs collected in the sidecar mode in the objects to be collected, to obtain a second container group subset, wherein the sum of the ninth log generation rate of the application log generated by each intercepted container group is the eighth log generation rate.

[0102] An embodiment of the present invention may provide an electronic device, which may be a computer terminal, and the computer terminal may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0103] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0104] In this embodiment, the computer terminal can execute the program code of the following steps in the log collection method: at a preset time interval, poll and monitor the first log generation rate of application logs generated by each container group in the object to be collected, and obtain the second log generation rate of application logs generated by the object to be collected, wherein the object to be collected includes multiple container groups; when the second log generation rate exceeds the preset rate threshold, divide the multiple container groups in the object to be collected into a first container group set and a second container group set, wherein the total rate of application logs generated by the multiple container groups in the first container group set is a third log generation rate, and the third log generation rate does not exceed the preset rate threshold; adopt the management mode to collect the application logs generated by the multiple container groups in the first container group set, wherein the management mode is used to run a first collection container on each computing node of the object to be collected, and the first collection container is used to collect the application logs generated by all container groups in the same computing node; adopt the sidecar mode to collect the application logs generated by the multiple container groups in the second container group set, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect the application logs generated by the container group.

[0105] Figure 4 is a structural block diagram of a computer terminal according to an embodiment of the present invention. Figure 4 As shown, the computer terminal 40 may include: one or more (only one is shown in the figure) processors 42 , and a memory 44 .

[0106] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the log collection method and device in the embodiment of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the above-mentioned log collection method is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the terminal 40 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0107] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: at a preset time interval, poll and monitor the first log generation rate of application logs generated by each container group in the object to be collected to obtain a second log generation rate of application logs generated by the object to be collected, wherein the object to be collected includes multiple container groups; when the second log generation rate exceeds a preset rate threshold, divide the multiple container groups in the object to be collected into a first container group set and a second container group set, wherein the total rate of application logs generated by the multiple container groups in the first container group set is a third log generation rate, and the third log generation rate does not exceed the preset rate threshold; adopt a management mode to collect application logs generated by multiple container groups in the first container group set, wherein the management mode is used to run a first collection container on each computing node of the object to be collected, and the first collection container is used to collect application logs generated by all container groups in the same computing node; adopt a sidecar mode to collect application logs generated by multiple container groups in the second container group set, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect application logs generated by the container group.

[0108] Optionally, the processor may also execute program code of the following steps: detecting a controller type of a controller corresponding to each container group in the object to be collected, wherein the controller type includes at least a first preset type and a second preset type, the container group corresponding to the controller of the first preset type is a first type container group, and the container group corresponding to the controller of the second preset type is a second type container group; collecting application logs generated by multiple first type container groups in a management mode; and polling and monitoring the rate at which multiple second type container groups generate application logs at a preset time interval to obtain a first log generation rate, and counting the sum of the first log generation rates to obtain a second log generation rate.

[0109] Optionally, the processor may also execute program code of the following steps: monitoring whether there is a newly added container application in the object to be collected, wherein the newly added container application is used to generate multiple replica container groups, and the replica container group is a replica of the container group existing in the object to be collected; when there is a newly added container application in the object to be collected, determining a fourth log generation rate of the application log generated by the object to be collected, wherein the fourth log generation rate is the sum of the third log generation rate and the fifth log generation rate of the application log generated by the multiple replica container groups; when the fourth log generation rate exceeds a preset rate threshold, using the sidecar mode to collect the application logs generated by the replica container group; when the fourth log generation rate does not exceed the preset rate threshold, using the management mode to collect the application logs generated by the replica container group.

[0110] Optionally, the processor may further execute program code of the following steps: monitoring a first rate change of a third log generation rate, wherein the first rate change is used to indicate that the third log generation rate exceeds a first duration of a preset rate threshold; when the first duration exceeds the first duration threshold, selecting multiple container groups from the first container group set to obtain a first container group subset, wherein a difference between the third log generation rate and a sixth log generation rate of application logs generated by multiple container groups in the first container group subset does not exceed the preset rate threshold; and using a sidecar mode to collect application logs generated by multiple container groups in the first container group subset.

[0111] Optionally, the processor may further execute program codes of the following steps: arranging the seventh log generation rate of the application log generated by each container group in the first container group set in descending order to obtain a first rate queue; and when the first duration exceeds a first duration threshold, intercepting multiple container groups from the tail of the first rate queue to obtain a first container group subset, wherein the sum of the seventh log generation rate of the application log generated by each intercepted container group is the sixth log generation rate.

[0112] Optionally, the processor may also execute program code of the following steps: monitoring a second rate change of a third log generation rate, and whether there is a situation in which application logs are collected in sidecar mode in the objects to be collected, wherein the second rate change is used to indicate that the third log generation rate does not exceed a second duration of a preset rate threshold; when the second duration exceeds the second duration threshold, and there is a situation in which application logs are collected in sidecar mode in the objects to be collected, selecting multiple container groups from the second container group set to obtain a second container group subset, wherein the sum of the third log generation rate and an eighth log generation rate of application logs generated by multiple container groups in the second container group subset does not exceed the preset rate threshold; and collecting application logs generated by multiple container groups in the second container group subset in management mode.

[0113] Optionally, the processor may further execute program code of the following steps: arranging the ninth log generation rate of the application log generated by each container group in the second container group set in descending order to obtain a second rate queue; when the second duration exceeds a second duration threshold and there are application logs collected in the sidecar mode among the objects to be collected, intercepting multiple container groups from the head of the second rate queue to obtain a second container group subset, wherein the sum of the ninth log generation rate of the application log generated by each intercepted container group is the eighth log generation rate.

[0114] According to an embodiment of the present invention, a log collection scheme is provided. In the embodiment of the present invention, at a preset time interval, the first log generation rate of each container group in the object to be collected is polled and monitored to obtain the second log generation rate of the application log generated by the object to be collected, wherein the object to be collected includes multiple container groups; when the second log generation rate exceeds the preset rate threshold, the multiple container groups in the object to be collected are divided into a first container group set and a second container group set, wherein the total rate of the application logs generated by the multiple container groups in the first container group set is a third log generation rate, and the third log generation rate does not exceed the preset rate threshold; the application logs generated by the multiple container groups in the first container group set are collected using a management mode, wherein the management mode is used A first collection container is run on each computing node of the object to be collected, and the first collection container is used to collect application logs generated by all container groups in the same computing node; a sidecar mode is used to collect application logs generated by multiple container groups in a second container group set, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect application logs generated by the container group. Therefore, according to the log generation rate of the application logs generated by the object to be collected, the management mode and the sidecar mode can be flexibly combined, thereby achieving the technical effect of ensuring the efficiency and accuracy of log collection without wasting collection resources, thereby solving the technical problem of low log collection efficiency.

[0115] It can be understood by those skilled in the art that Figure 4 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 4 The structure of the electronic device is not limited. For example, the computer terminal 40 may also include Figure 4 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 4 Different configurations shown.

[0116] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a computer program. The computer program can be stored in a non-volatile medium. The non-volatile storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0117] The embodiment of the present invention further provides a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the log collection method provided in the above embodiment.

[0118] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0119] Optionally, in this embodiment, the non-volatile storage medium is configured to store program codes for executing the following steps: at a preset time interval, polling and monitoring a first log generation rate of application logs generated by each container group in the object to be collected, to obtain a second log generation rate of application logs generated by the object to be collected, wherein the object to be collected includes multiple container groups; when the second log generation rate exceeds a preset rate threshold, dividing the multiple container groups in the object to be collected into a first container group set and a second container group set, wherein the total rate at which the multiple container groups in the first container group set generate application logs is a third log generation rate, and the third log generation rate does not exceed the preset rate threshold; using a management mode to collect application logs generated by multiple container groups in the first container group set, wherein the management mode is used to run a first collection container on each computing node of the object to be collected, and the first collection container is used to collect application logs generated by all container groups in the same computing node; using a sidecar mode to collect application logs generated by multiple container groups in the second container group set, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect application logs generated by the container group.

[0120] Optionally, in this embodiment, the non-volatile storage medium is configured to store program codes for executing the following steps: detecting a controller type of a controller corresponding to each container group in the object to be collected, wherein the controller type includes at least a first preset type and a second preset type, the container group corresponding to the controller of the first preset type is a first type container group, and the container group corresponding to the controller of the second preset type is a second type container group; collecting application logs generated by multiple first type container groups in a management mode; and polling and monitoring the rate at which multiple second type container groups generate application logs at a preset time interval to obtain a first log generation rate, and counting the sum of the first log generation rates to obtain a second log generation rate.

[0121] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: monitoring whether there is a new container application in the object to be collected, wherein the new container application is used to generate multiple replica container groups, and the replica container group is a replica of the container group existing in the object to be collected; when there is a new container application in the object to be collected, determining a fourth log generation rate of application logs generated by the object to be collected, wherein the fourth log generation rate is the sum of the third log generation rate and a fifth log generation rate of application logs generated by multiple replica container groups; when the fourth log generation rate exceeds a preset rate threshold, using a sidecar mode to collect application logs generated by the replica container group; when the fourth log generation rate does not exceed the preset rate threshold, using a management mode to collect application logs generated by the replica container group.

[0122] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: monitoring a first rate change of a third log generation rate, wherein the first rate change is used to indicate that the third log generation rate exceeds a first duration of a preset rate threshold; when the first duration exceeds the first duration threshold, selecting multiple container groups from the first container group set to obtain a first container group subset, wherein a difference between the third log generation rate and a sixth log generation rate of application logs generated by multiple container groups in the first container group subset does not exceed the preset rate threshold; and using a sidecar mode to collect application logs generated by multiple container groups in the first container group subset.

[0123] Optionally, in this embodiment, the non-volatile storage medium is configured to store a program code for executing the following steps: arranging the seventh log generation rate of the application log generated by each container group in the first container group set in descending order to obtain a first rate queue; when the first duration exceeds a first duration threshold, intercepting multiple container groups from the tail of the first rate queue to obtain a first container group subset, wherein the sum of the intercepted seventh log generation rate of the application log generated by each container group is the sixth log generation rate.

[0124] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for executing the following steps: monitoring a second rate change of a third log generation rate, and whether there is a situation in which application logs are collected in sidecar mode in the object to be collected, wherein the second rate change is used to indicate that the third log generation rate does not exceed a second duration of a preset rate threshold; when the second duration exceeds the second duration threshold, and there is a situation in which application logs are collected in sidecar mode in the object to be collected, selecting multiple container groups from the second container group set to obtain a second container group subset, wherein the sum of the third log generation rate and an eighth log generation rate of application logs generated by multiple container groups in the second container group subset does not exceed the preset rate threshold; and collecting application logs generated by multiple container groups in the second container group subset in management mode.

[0125] Optionally, in this embodiment, the non-volatile storage medium is configured to store a program code for executing the following steps: arranging the ninth log generation rate of the application log generated by each container group in the second container group set in order from large to small to obtain a second rate queue; when the second duration exceeds a second duration threshold and there are application logs collected in sidecar mode in the objects to be collected, intercepting multiple container groups from the head of the second rate queue to obtain a second container group subset, wherein the sum of the ninth log generation rate of the application log generated by each intercepted container group is the eighth log generation rate.

[0126] The embodiment of the present invention further provides a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the steps of the log collection method provided in the above embodiment are implemented.

[0127] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0128] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0131] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a non-volatile storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned non-volatile storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0133] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A log collection method, characterized in that: include: According to a preset time interval, poll and monitor a first log generation rate of application logs generated by each container group in the object to be collected, and obtain a second log generation rate of the application logs generated by the object to be collected, wherein the object to be collected includes a plurality of the container groups; In the case where the second log generation rate exceeds a preset rate threshold, the plurality of container groups in the to-be-collected object are divided into a first container group set and a second container group set, wherein a total rate at which the plurality of container groups in the first container group set generate the application logs is a third log generation rate, and the third log generation rate does not exceed the preset rate threshold; The application logs generated by the plurality of container groups in the first container group set are collected in a management mode, wherein the management mode is used to run a first collection container on each computing node of the object to be collected, and the first collection container is used to collect the application logs generated by all container groups in the same computing node; The application logs generated by multiple container groups in the second container group set are collected using a sidecar mode, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect the application logs generated by the container group.

2. The method according to claim 1, characterized in that According to a preset time interval, polling and monitoring a first log generation rate of application logs generated by each container group in the object to be collected, and obtaining a second log generation rate of the application logs generated by the object to be collected includes: Detecting the controller type of the controller corresponding to each container group in the object to be collected, wherein the controller type includes at least: a first preset type and a second preset type, the container group corresponding to the controller of the first preset type is a first type of container group, and the container group corresponding to the controller of the second preset type is a second type of container group; The management mode is used to collect the application logs generated by the plurality of the first-type container groups; According to a preset time interval, the rate at which the multiple second-type container groups generate the application logs is polled and monitored to obtain the first log generation rate, and the sum of the first log generation rates is counted to obtain the second log generation rate.

3. The method according to claim 1, characterized in that After collecting the application logs generated by the plurality of container groups in the first container group set in a management mode, the method further includes: Monitoring whether there is a newly added container application in the object to be collected, wherein the newly added container application is used to generate a plurality of replica container groups, and the replica container groups are replicas of the container group existing in the object to be collected; In the case where the newly added container application exists in the object to be collected, determining a fourth log generation rate of the application log generated by the object to be collected, wherein the fourth log generation rate is the sum of the third log generation rate and a fifth log generation rate of the application log generated by the plurality of replica container groups; When the fourth log generation rate exceeds the preset rate threshold, using the sidecar mode to collect the application logs generated by the replica container group; When the fourth log generation rate does not exceed the preset rate threshold, the management mode is adopted to collect the application logs generated by the replica container group.

4. The method according to claim 1, characterized in that: After collecting the application logs generated by the plurality of container groups in the first container group set in a management mode, the method further includes: Monitoring a first rate change of the third log generation rate, wherein the first rate change is used to indicate a first duration during which the third log generation rate exceeds the preset rate threshold; When the first duration exceeds a first duration threshold, selecting a plurality of the container groups from the first container group set to obtain a first container group subset, wherein a difference between the third log generation rate and a sixth log generation rate of the application logs generated by the plurality of the container groups in the first container group subset does not exceed the preset rate threshold; The sidecar mode is used to collect the application logs generated by the plurality of container groups in the first container group subset.

5. The method according to claim 4, characterized in that When the first duration exceeds a first duration threshold, a plurality of the container groups are selected from the first container group set to obtain a first container group subset, including: Arrange the seventh log generation rate of the application log generated by each container group in the first container group set in descending order to obtain a first rate queue; When the first duration exceeds a first duration threshold, a plurality of the container groups are intercepted from the tail of the first rate queue to obtain the first container group subset, wherein a sum of the seventh log generation rate of the application log generated by each of the intercepted container groups is the sixth log generation rate.

6. The method according to claim 1, characterized in that After collecting the application logs generated by the plurality of container groups in the first container group set in a management mode, the method further includes: Monitor a second rate change of the third log generation rate, and whether there is a situation in which the application log is collected in the sidecar mode in the object to be collected, wherein the second rate change is used to indicate a second duration during which the third log generation rate does not exceed the preset rate threshold; When the second duration exceeds a second duration threshold and the object to be collected includes the application log collected by the sidecar mode, multiple container groups are selected from the second container group set to obtain a second container group subset, wherein the sum of the third log generation rate and an eighth log generation rate of the application logs generated by the multiple container groups in the second container group subset does not exceed the preset rate threshold; The management mode is adopted to collect the application logs generated by the plurality of the container groups in the second container group subset.

7. The method according to claim 6, characterized in that When the second duration exceeds a second duration threshold and the object to be collected includes an application log collected in the sidecar mode, a plurality of the container groups are selected from the second container group set to obtain a second container group subset, including: Arrange the ninth log generation rate of the application log generated by each container group in the second container group set in descending order to obtain a second rate queue; When the second duration exceeds a second duration threshold and the object to be collected includes an application log collected in the sidecar mode, a plurality of the container groups are intercepted from the head of the second rate queue to obtain a subset of the second container groups, wherein a sum of the ninth log generation rate of the application log generated by each of the intercepted container groups is the eighth log generation rate.

8. A log collection device, characterized in that: include: A monitoring module, configured to poll and monitor, at a preset time interval, a first log generation rate of application logs generated by each container group in an object to be collected, to obtain a second log generation rate of the application logs generated by the object to be collected, wherein the object to be collected includes a plurality of the container groups; a division module, configured to divide the plurality of container groups in the to-be-collected object into a first container group set and a second container group set when the second log generation rate exceeds a preset rate threshold, wherein the total rate at which the plurality of container groups in the first container group set generate the application logs is a third log generation rate, and the third log generation rate does not exceed the preset rate threshold; A first collection module, configured to collect the application logs generated by the plurality of container groups in the first container group set in a management mode, wherein the management mode is configured to run a first collection container on each computing node of the object to be collected, and the first collection container is configured to collect the application logs generated by all container groups in the same computing node; The second collection module is used to collect the application logs generated by multiple container groups in the second container group set by adopting a sidecar mode, wherein the sidecar mode is used to run a second collection container in each container group of the object to be collected, and the second collection container is used to collect the application logs generated by the container group.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to execute the log collection method according to any one of claims 1 to 7 through the computer program.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the log collection method described in any one of claims 1 to 7 are implemented.

Images