Edge cloud system log speed-limiting collection method

Abstract

The invention provides a speed-limiting collection method for logs of an edge cloud system, and belongs to the technical field of flow control. According to the method, log speed-limiting collection of an edge cloud system is achieved through a log integration system EFK, and the log integration system EFK comprises a storage system ES, a log collection module Fluentd and a visual platform Kibana. According to the method, log collection modules Fluentd are deployed on edge cloud nodes, each edge cloud node is provided with one log collection module Fluentd, each log collection module Fluentd comprises a collection speed control plug-in, log data collected by the log collection modules Fluentd are integrated and summarized to a storage system ES, and visualization is achieved through a visualization platform Kibana. According to the method, the log collection process is optimized, the flow is collected, log collection is controlled according to the collected flow, and the log collection speed is controlled.

Description

Technical Field

[0001] The present invention belongs to the technical field of flow control, and in particular relates to a method for collecting logs at a limited speed in an edge cloud system. Background Art

[0002] With the development of edge cloud technology, cloud technology has become more and more mature in the civilian field; in the containerized observable platform, the container log integration system has been verified and affirmed in most cloud-edge system projects. However, due to changes in scenarios and business logic, there are some defects in the log integration system; for example, in the process of collecting logs of edge cloud devices, as the amount of logs increases, the traffic in special network environments is occupied by the log integration system, while the available bandwidth traffic of other core services is reduced, thus affecting the stability of the edge cloud. In the existing technology, the industry's EFK / ELK is a real-time / quasi-real-time log integration solution. When the service logs in the container are produced too frequently, the bandwidth of the edge device is greatly occupied; in this case, it is necessary to design a log integration method that does not occupy traffic to implement a log integration system. Summary of the invention

[0003] In view of the above technical problems, the present invention proposes a log collection scheme with limited speed for edge cloud system.

[0004] A first aspect of the present invention provides a method for collecting edge cloud system logs at a limited speed.

[0005] The method uses the log integration system EFK to realize the log speed limit collection of the edge cloud system. The log integration system EFK includes a storage system ES, a log collection module Fluentd, and a visualization platform Kibana.

[0006] The log collection module Fluentd is deployed on the edge cloud node. Each edge cloud node deploys a log collection module Fluentd, and each log collection module Fluentd contains a plug-in to control the collection speed. The log data collected by each log collection module Fluentd is integrated and summarized into the storage system ES and visualized by the visualization platform Kibana.

[0007] The method specifically includes: step S1, deploying a traffic collection and calculation module on each edge cloud node to determine the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node; step S2, using the control collection speed plug-in contained in the log collection module Fluentd deployed on each edge cloud node, based on the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node, to control the speed of log collection by the log collection module Fluentd.

[0008] According to the method of the first aspect of the present invention, in step S1, the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node are determined by using the traffic collection and calculation module; specifically, the following steps are included: using a periodic collection method, obtaining the system network card traffic, including input traffic and output traffic, through the system file / prpc / net and the system command ifstat to calculate the total network traffic of the edge cloud node; using a periodic collection method, obtaining the network traffic associated with the container Pod on the edge cloud node through the instructions iftop and iptraf.

[0009] According to the method of the first aspect of the present invention, the traffic collection and calculation module is a DaemonSet controller.

[0010] According to the method of the first aspect of the present invention, in step S2: request the node traffic collection service to obtain the traffic situation of the current edge cloud node, and receive the total network traffic of the current edge cloud node and the network traffic associated with the container Pod from the traffic collection calculation module; obtain the configuration information of the log collection plug-in used to collect logs in the log collection module Fluentd, and the configuration information includes the upper limit of the traffic occupied during the log collection process, the lower limit of the traffic, and the interval time of log collection; based on the total network traffic of the current edge cloud node and the network traffic associated with the container Pod, combined with the configuration information, use the network traffic control algorithm to control the speed of log collection by the log collection module Fluentd.

[0011] According to the method of the first aspect of the present invention, in step S2, the speed of collecting logs by the log collection module Fluentd is controlled; specifically, the following steps are performed: when the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the first target range, log collection is performed with the upper limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted; when the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the second target range, log collection is performed with the lower limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted; when the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the third target range, log collection is performed with the traffic value between the lower limit and the upper limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted.

[0012] A second aspect of the present invention provides a device for collecting logs at a limited speed in an edge cloud system.

[0013] The device uses the log integration system EFK to realize the speed-limited collection of logs of the edge cloud system. The log integration system EFK includes a storage system ES, a log collection module Fluentd, and a visualization platform Kibana.

[0014] The log collection module Fluentd is deployed on the edge cloud node. Each edge cloud node deploys a log collection module Fluentd, and each log collection module Fluentd contains a plug-in to control the collection speed. The log data collected by each log collection module Fluentd is integrated and summarized into the storage system ES and visualized by the visualization platform Kibana.

[0015] The device specifically includes: a traffic collection and calculation module; wherein a traffic collection and calculation module is deployed on each edge cloud node, and the traffic collection and calculation module is used to determine the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node; a log collection module Fluentd; wherein the log collection module Fluentd deployed on each edge cloud node includes a control collection speed plug-in and a log collection plug-in, and the log collection module Fluentd controls the speed of collecting logs based on the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node.

[0016] According to the device of the second aspect of the present invention, the traffic collection and calculation module is configured to: use the traffic collection and calculation module to determine the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node; wherein: adopt a periodic collection method to obtain the system network card traffic, including input traffic and output traffic, through the system file / prpc / net and the system command ifstat to calculate the total network traffic of the edge cloud node; adopt a periodic collection method to obtain the network traffic associated with the container Pod on the edge cloud node through the instructions iftop and iptraf.

[0017] According to the device of the second aspect of the present invention, the traffic collection and calculation module is a DaemonSet controller.

[0018] According to the device of the second aspect of the present invention, the log collection module Fluentd is configured to: request the node traffic collection service to obtain the traffic situation of the current edge cloud node, and receive the total network traffic of the current edge cloud node and the network traffic associated with the container Pod from the traffic collection calculation module; obtain the configuration information of the log collection plug-in, the configuration information including the upper limit of the traffic occupied during the log collection process, the lower limit of the traffic, and the interval time of the log collection; based on the total network traffic of the current edge cloud node and the network traffic associated with the container Pod, combined with the configuration information, use the network traffic control algorithm to control the speed at which the log collection module Fluentd collects logs.

[0019] According to the device of the second aspect of the present invention, the log collection module Fluentd is configured to: control the speed at which the log collection module Fluentd collects logs; wherein: when the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the first target range, log collection is performed with the upper limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted; when the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the second target range, log collection is performed with the lower limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted; when the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the third target range, log collection is performed with the traffic value between the lower limit and the upper limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted.

[0020] In summary, the present invention optimizes and modifies the log collection module in the log integration system EFK, collects traffic, and collects logs according to traffic control, thereby achieving control over the log collection speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 A schematic diagram of speed-limited collection of edge cloud system logs according to an embodiment of the present invention.

[0023] Figure 2 Schematic diagram of a flow collection and calculation module according to an embodiment of the present invention.

[0024] Figure 3 Schematic diagram of the log collection module Fluentd according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0026] The first aspect of the present invention provides a method for collecting edge cloud system logs at a limited speed. Figure 1 As shown, the method uses the log integration system EFK to realize the speed-limited log collection of the edge cloud system, and the log integration system EFK includes the storage system ES, the log collection module Fluentd, and the visualization platform Kibana. The log collection module Fluentd is deployed on the edge cloud node, each edge cloud node deploys a log collection module Fluentd, and each log collection module Fluentd contains a control collection speed plug-in. The log data collected by each log collection module Fluentd is integrated and summarized into the storage system ES, and visualized by the visualization platform Kibana.

[0027] The method specifically includes: step S1, deploying a traffic collection and calculation module on each edge cloud node to determine the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node; step S2, using the control collection speed plug-in contained in the log collection module Fluentd deployed on each edge cloud node, based on the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node, to control the speed of log collection by the log collection module Fluentd.

[0028] According to the method of the first aspect of the present invention, in step S1, the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node are determined by using the traffic collection and calculation module; specifically, the following steps are included: using a periodic collection method, obtaining the system network card traffic, including input traffic and output traffic, through the system file / prpc / net and the system command ifstat to calculate the total network traffic of the edge cloud node; using a periodic collection method, obtaining the network traffic associated with the container Pod on the edge cloud node through the instructions iftop and iptraf.

[0029] According to the method of the first aspect of the present invention, the traffic collection and calculation module is a DaemonSet controller.

[0030] like Figure 2 As shown in the figure, the edge node traffic collection and calculation mainly includes: obtaining the system's network card traffic through system files / prpc / net, system commands ifstat, etc.; including input, output, etc. Obtain network traffic statistics associated with the pod through iftop, iptraf, etc. The above processes all use the periodic collection method to organize the process traffic and total network card traffic into memory for use by the traffic control algorithm. The logic of this part uses the DaemonSet controller method when running on edge cloud devices, so that each device can have a traffic collection and calculation service.

[0031] According to the method of the first aspect of the present invention, in step S2: request the node traffic collection service to obtain the traffic situation of the current edge cloud node, and receive the total network traffic of the current edge cloud node and the network traffic associated with the container Pod from the traffic collection calculation module; obtain the configuration information of the log collection plug-in used to collect logs in the log collection module Fluentd, and the configuration information includes the upper limit of the traffic occupied during the log collection process, the lower limit of the traffic, and the interval time of log collection; based on the total network traffic of the current edge cloud node and the network traffic associated with the container Pod, combined with the configuration information, use the network traffic control algorithm to control the speed of log collection by the log collection module Fluentd.

[0032] According to the method of the first aspect of the present invention, in step S2, the speed of collecting logs by the log collection module Fluentd is controlled; specifically, the following steps are performed: when the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the first target range, log collection is performed with the upper limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted; when the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the second target range, log collection is performed with the lower limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted; when the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the third target range, log collection is performed with the traffic value between the lower limit and the upper limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted.

[0033] like Figure 3 As shown in the figure, the algorithm controls the log collection speed mainly including: This part runs on Fluentd as a plug-in (Fluentd is a log collection module of the log integration system EFK). Request the node traffic collection service to obtain the network traffic of the node network card and the network traffic of the pod. Obtain the configuration information of the log collection plug-in, including: the upper and lower limits of the maximum traffic occupied by log collection, the interval time, etc. Execute the network traffic control algorithm according to the above situation to control the log collection speed of Fluentd.

[0034] A second aspect of the present invention provides a device for collecting logs at a limited speed in an edge cloud system.

[0035] The device uses the log integration system EFK to realize the speed-limited collection of logs of the edge cloud system. The log integration system EFK includes a storage system ES, a log collection module Fluentd, and a visualization platform Kibana.

[0036] The log collection module Fluentd is deployed on the edge cloud node. Each edge cloud node deploys a log collection module Fluentd, and each log collection module Fluentd contains a plug-in to control the collection speed. The log data collected by each log collection module Fluentd is integrated and summarized into the storage system ES and visualized by the visualization platform Kibana.

[0037] The device specifically includes: a traffic collection and calculation module; wherein a traffic collection and calculation module is deployed on each edge cloud node, and the traffic collection and calculation module is used to determine the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node; a log collection module Fluentd; wherein the log collection module Fluentd deployed on each edge cloud node includes a control collection speed plug-in and a log collection plug-in, and the log collection module Fluentd controls the speed of collecting logs based on the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node.

[0038] According to the device of the second aspect of the present invention, the traffic collection and calculation module is configured to: use the traffic collection and calculation module to determine the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node; wherein: adopt a periodic collection method to obtain the system network card traffic, including input traffic and output traffic, through the system file / prpc / net and the system command ifstat to calculate the total network traffic of the edge cloud node; adopt a periodic collection method to obtain the network traffic associated with the container Pod on the edge cloud node through the instructions iftop and iptraf.

[0039] According to the device of the second aspect of the present invention, the traffic collection and calculation module is a DaemonSet controller.

[0040] According to the device of the second aspect of the present invention, the log collection module Fluentd is configured to: request the node traffic collection service to obtain the traffic situation of the current edge cloud node, and receive the total network traffic of the current edge cloud node and the network traffic associated with the container Pod from the traffic collection calculation module; obtain the configuration information of the log collection plug-in, the configuration information including the upper limit of the traffic occupied during the log collection process, the lower limit of the traffic, and the interval time of the log collection; based on the total network traffic of the current edge cloud node and the network traffic associated with the container Pod, combined with the configuration information, use the network traffic control algorithm to control the speed at which the log collection module Fluentd collects logs.

[0041] According to the device of the second aspect of the present invention, the log collection module Fluentd is configured to: control the speed at which the log collection module Fluentd collects logs; wherein: when the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the first target range, log collection is performed with the upper limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted; when the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the second target range, log collection is performed with the lower limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted; when the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the third target range, log collection is performed with the traffic value between the lower limit and the upper limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted.

[0042] In summary, the present invention optimizes and modifies the log collection module in the log integration system EFK, collects traffic, and collects logs according to traffic control, thereby achieving control over the log collection speed.

[0043] Please note that the technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several variations and improvements can be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the patent in this application shall be based on the attached claims.

Claims

1. A method for collecting edge cloud system logs at a limited speed, characterized in that: The method uses the log integration system EFK to realize the speed-limited collection of logs of the edge cloud system. The log integration system EFK includes a storage system ES, a log collection module Fluentd, and a visualization platform Kibana; The log collection module Fluentd is deployed on the edge cloud node. Each edge cloud node deploys a log collection module Fluentd, and each log collection module Fluentd contains a plug-in to control the collection speed. The log data collected by each log collection module Fluentd is integrated and summarized into the storage system ES, and visualized by the visualization platform Kibana; The method specifically comprises: Step S1: deploy a traffic collection and calculation module on each edge cloud node to determine the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node; Step S2, using the control collection speed plug-in contained in the log collection module Fluentd deployed on each edge cloud node, based on the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node, control the speed at which the log collection module Fluentd collects logs.

2. According to claim 1, a method for collecting edge cloud system logs at a limited speed is characterized in that: In step S1, the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node are determined by using the traffic collection and calculation module; specifically, the following steps are included: The system network card traffic, including input and output traffic, is obtained through the system file / prpc / net and the system command ifstat in a periodic collection mode to calculate the total network traffic of the edge cloud node. The network traffic associated with the container Pod on the edge cloud node is obtained by periodic collection through the iftop and iptraf commands.

3. A method for collecting edge cloud system logs at a limited speed according to claim 2, characterized in that: The traffic collection and calculation module is the DaemonSet controller.

4. A method for collecting edge cloud system logs at a limited speed according to claim 3, characterized in that: In step S2: Request the node traffic collection service to obtain the traffic status of the current edge cloud node, and receive the total network traffic of the current edge cloud node and the network traffic associated with the container Pod from the traffic collection calculation module; Obtain configuration information of the log collection plug-in used to collect logs in the log collection module Fluentd, the configuration information including the upper and lower limits of the flow occupied during the log collection process, and the interval time of log collection; Based on the total network traffic of the current edge cloud node and the network traffic associated with the container Pod, combined with the configuration information, a network traffic control algorithm is used to control the speed at which the log collection module Fluentd collects logs.

5. A method for collecting edge cloud system logs at a limited speed according to claim 4, characterized in that: In step S2, the speed of collecting logs by the log collection module Fluentd is controlled; specifically, the following steps are performed: When the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the first target range, log collection is performed based on the upper limit of the traffic occupied during log collection, and the interval of log collection is adaptively adjusted; When the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the second target range, log collection is performed based on the lower limit of the traffic occupied during log collection, and the interval of log collection is adaptively adjusted; When the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the third target range, log collection is performed based on the traffic value between the lower limit and the upper limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted.

6. A device for collecting logs at a limited speed on an edge cloud system, characterized in that: The device uses the log integration system EFK to realize the speed-limited collection of logs of the edge cloud system. The log integration system EFK includes a storage system ES, a log collection module Fluentd, and a visualization platform Kibana; The log collection module Fluentd is deployed on the edge cloud node. Each edge cloud node deploys a log collection module Fluentd, and each log collection module Fluentd contains a plug-in to control the collection speed. The log data collected by each log collection module Fluentd is integrated and summarized into the storage system ES, and visualized by the visualization platform Kibana; The device specifically comprises: Traffic collection and calculation module; wherein a traffic collection and calculation module is deployed on each edge cloud node, and the traffic collection and calculation module is used to determine the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node; The log collection module Fluentd is deployed on each edge cloud node and includes a collection speed control plug-in and a log collection plug-in. The log collection module Fluentd controls the speed of collecting logs based on the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node.

7. The device for collecting logs at a limited speed in an edge cloud system according to claim 6, characterized in that: The traffic collection and calculation module is configured to: determine the total network traffic of the edge cloud node and the network traffic of the container Pod on the edge cloud node by using the traffic collection and calculation module; wherein: The system network card traffic, including input and output traffic, is obtained through the system file / prpc / net and the system command ifstat in a periodic collection mode to calculate the total network traffic of the edge cloud node. The network traffic associated with the container Pod on the edge cloud node is obtained by periodic collection through the iftop and iptraf commands.

8. The device for collecting logs at a limited speed in an edge cloud system according to claim 7, characterized in that: The traffic collection and calculation module is the DaemonSet controller.

9. The device for collecting logs at a limited speed in an edge cloud system according to claim 8, characterized in that: The log collection module Fluentd is configured as follows: Request the node traffic collection service to obtain the traffic status of the current edge cloud node, and receive the total network traffic of the current edge cloud node and the network traffic associated with the container Pod from the traffic collection calculation module; Obtain the configuration information of the log collection plug-in, which includes the upper and lower limits of the traffic volume occupied during the log collection process, and the interval time of log collection; Based on the total network traffic of the current edge cloud node and the network traffic associated with the container Pod, combined with the configuration information, a network traffic control algorithm is used to control the speed at which the log collection module Fluentd collects logs.

10. The device for collecting logs at a limited speed in an edge cloud system according to claim 9, characterized in that: The log collection module Fluentd is configured to: control the speed at which the log collection module Fluentd collects logs; wherein: When the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the first target range, log collection is performed based on the upper limit of the traffic occupied during log collection, and the interval of log collection is adaptively adjusted; When the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the second target range, log collection is performed based on the lower limit of the traffic occupied during log collection, and the interval of log collection is adaptively adjusted; When the total network traffic of the current edge cloud node and the network traffic associated with the container Pod fall into the third target range, log collection is performed based on the traffic value between the lower limit and the upper limit of the traffic occupied during the log collection process, and the interval time of log collection is adaptively adjusted.

Images