Log processing method, server, system, readable storage medium and program product

Abstract

The invention discloses a log processing method, a server, a system, a readable storage medium and a program product in the technical field of information, a node server collects log items based on collection frequency to obtain log information, and records log traffic information; sending the log information to a log server, and obtaining a log bandwidth of the log server for processing the log information; determining a flow bandwidth upper limit in combination with the network bandwidth and the log bandwidth; traffic prediction is carried out based on the log traffic information, and the total log traffic of the next time period is obtained; and when the log total flow is greater than the flow bandwidth upper limit, adjusting the collection frequency of the log items, so that the log total flow is not greater than the flow bandwidth upper limit. The method has the technical effects that the log acquisition frequency can be adaptively adjusted according to the current transmission and the processing capacity of the log server, and the stability and the monitoring performance of the system can be ensured.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a log processing method, a server, a system, a readable storage medium, and a program product. Background Art

[0002] System log collection and analysis involves collecting, storing, processing, and analyzing log data from various systems and applications. Stable and reliable log transmission is a key component to ensure system stability and monitorability, which involves efficiently and reliably transmitting log data from the generation location to the log processing and storage system.

[0003] During the log collection process, if the log volume increases over a period of time, sending the collected logs to the log server may increase the server memory pressure. In addition, due to bandwidth limitations, if the log volume sent exceeds the network bandwidth, data loss may occur.

[0004] In summary, how to effectively solve problems such as flow control of log processing is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The purpose of this application is to provide a log processing method, server, system, readable storage medium and program product. Based on the log bandwidth and network bandwidth of the log server processing log information, the upper limit of the traffic bandwidth can be determined, and the total log traffic in the next time period can be obtained by performing traffic prediction based on a neural network model. Combining the total log traffic and the upper limit of the traffic bandwidth, the collection frequency of log items is adjusted so that the total log traffic does not exceed the upper limit of the traffic bandwidth, thereby avoiding excessive log occupation of server memory and ensuring that log transmission loss does not result in log loss.

[0006] In order to solve the above technical problems, this application provides the following technical solutions:

[0007] A log processing method, applied to a node server, comprising:

[0008] Collect log items based on the collection frequency to obtain log information, and record log flow information;

[0009] Sending the log information to a log server and obtaining a log bandwidth for the log server to process the log information;

[0010] Determine the upper limit of traffic bandwidth by combining the network bandwidth and the log bandwidth;

[0011] Performing flow prediction based on the log flow information to obtain the total log flow for the next time period;

[0012] When the total log flow is greater than the flow bandwidth upper limit, the collection frequency of log items is adjusted so that the total log flow is not greater than the flow bandwidth upper limit.

[0013] Preferably, traffic prediction is performed based on the log traffic information to obtain the total log traffic in the next time period, including:

[0014] Inputting the log traffic information into a neural network model;

[0015] The neural network model is used to perform flow prediction on the log flow information to obtain the total log flow in the next time period.

[0016] Preferably, when the total log flow is greater than the flow bandwidth upper limit, adjusting the collection frequency of log items so that the total log flow is not greater than the flow bandwidth upper limit includes:

[0017] Determine whether the total log traffic is greater than the traffic bandwidth upper limit;

[0018] If yes, the collection frequency of the log items is lowered, and after the adjustment, the neural network model is used to perform traffic prediction based on the new collection frequency to obtain the target total log traffic for the next time period;

[0019] If not, it is determined that there is no need to reduce the acquisition frequency.

[0020] Preferably, when the total log flow is not greater than the flow bandwidth upper limit, the method further includes:

[0021] Determine whether the target log total flow is less than the flow bandwidth lower limit;

[0022] If yes, the collection frequency of the log items is increased, and after the adjustment, the neural network model is used to perform traffic prediction based on the new collection frequency to obtain the target total log traffic for the next time period;

[0023] If not, it is determined that there is no need to increase the acquisition frequency.

[0024] Preferably, the flow bandwidth upper limit is determined by combining the network bandwidth and the log bandwidth, including:

[0025] The minimum value of the network bandwidth and the log bandwidth is determined as the traffic bandwidth upper limit.

[0026] Preferably, sending the log information to a log server includes:

[0027] Cache the log information in a log waiting to be sent queue;

[0028] The log information is sent to the log server by utilizing the log waiting to be sent queue.

[0029] Preferably, using the log to-be-sent queue to send the log information to the log server includes:

[0030] Obtaining the bandwidth limit from the log server;

[0031] Reading logs whose log volume matches the bandwidth limit from the log to be sent queue;

[0032] The read log is sent to the log server.

[0033] Preferably, the initialization stage includes:

[0034] Receiving log configuration information sent by the log server;

[0035] The log configuration information is used to determine log items and the initial collection frequency of the log items.

[0036] Preferably, it also includes:

[0037] Receiving log configuration adjustment information sent by the log server;

[0038] The log item is re-determined according to the log configuration adjustment information.

[0039] A log processing method, applied to a log server, comprising:

[0040] Send log configuration information to the node server;

[0041] Receiving log information sent by the node server;

[0042] Processing the received log information and recording the log bandwidth for processing the log information;

[0043] The log bandwidth is sent to a node server so that the node server adjusts the frequency of collecting log items based on the log bandwidth.

[0044] A node server, comprising:

[0045] Memory for storing computer programs;

[0046] A processor is used to implement the steps of the above-mentioned log processing method when executing the computer program.

[0047] A log server, comprising:

[0048] Memory for storing computer programs;

[0049] A processor is used to implement the steps of the above-mentioned log processing method when executing the computer program.

[0050] A log processing system, comprising:

[0051] One log server as described above, multiple node servers as described above, and a gateway device for connecting the log server and the node servers.

[0052] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log processing method are implemented.

[0053] A computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned log processing method are implemented.

[0054] A log processing device applied to a node server, comprising:

[0055] A log collection unit, used to collect log items based on a collection frequency to obtain log information, and record log flow information;

[0056] A log bandwidth acquisition unit, used for sending the log information to a log server and obtaining a log bandwidth of the log server for processing the log information;

[0057] An upper limit determination unit, used to determine the upper limit of the traffic bandwidth by combining the network bandwidth and the log bandwidth;

[0058] A traffic prediction unit, configured to perform traffic prediction based on the log traffic information to obtain the total log traffic in the next time period;

[0059] The flow regulating unit is used to adjust the collection frequency of log items when the total log flow is greater than the flow bandwidth upper limit, so that the total log flow is not greater than the flow bandwidth upper limit.

[0060] A log processing device applied to a log server, comprising:

[0061] A configuration unit, used to send log configuration information to the node server;

[0062] A log receiving unit, used to receive log information sent by the node server;

[0063] A log processing unit, used to process the received log information and record the log bandwidth for processing the log information;

[0064] The log bandwidth feedback unit is used to send the log bandwidth to the node server so that the node server adjusts the collection frequency of log items based on the log bandwidth.

[0065] The node server applies the method provided by the embodiment of the present application, including: collecting log items based on the collection frequency to obtain log information, and recording log traffic information; sending the log information to the log server, and obtaining the log bandwidth of the log server for processing the log information; determining the traffic bandwidth upper limit in combination with the network bandwidth and the log bandwidth; performing traffic prediction based on the log traffic information to obtain the total log traffic in the next time period; when the total log traffic is greater than the traffic bandwidth upper limit, adjusting the collection frequency of the log items so that the total log traffic is not greater than the traffic bandwidth upper limit.

[0066] In the present application, the node server collects log items according to the collection frequency to obtain log information. In order to facilitate log flow control, log flow information will be recorded while collecting logs. The collected log information is sent to the log server. The log server processes the received log information. The node server can obtain the log bandwidth for processing log information from the log server. Combining the network bandwidth and the log bandwidth, the upper limit of the traffic bandwidth can be determined. Based on the log traffic information, the traffic forecast can be obtained to obtain the total log flow in the next time period. When the total log flow is greater than the upper limit of the traffic bandwidth, the collection frequency of the log items is adjusted so that the total log flow is not greater than the upper limit of the traffic bandwidth. In this way, the node server collects logs for the log items based on the adjusted collection frequency in the next time period, and its total log flow will not exceed the upper limit of the traffic bandwidth. In this way, it is possible to avoid occupying the memory data of the node server, and it is also possible to avoid losing logs during the transmission of the collected log information to the log server.

[0067] The technical effect of the present application is as follows: the frequency of log collection can be adaptively adjusted according to the current transmission and processing capabilities of the log server, thereby ensuring system stability and monitorability.

[0068] Accordingly, the embodiments of the present application also provide a node server, a log server, a log processing system, a readable storage medium and a computer program product corresponding to the above-mentioned log processing method, which have the above-mentioned technical effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0070] Figure 1 This is a flow chart of an implementation of a log processing method in an embodiment of the present application;

[0071] Figure 2 This is a flowchart of another log processing method in the embodiment of the present application.

[0072] Figure 3 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application;

[0073] Figure 4 This is a schematic diagram of a specific structure of an electronic device in an embodiment of the present application;

[0074] Figure 5 A schematic diagram of a log processing system in an embodiment of the present application;

[0075] Figure 6 This is a schematic diagram of specific functional modules of a log processing system in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0077] Please refer to Figure 1 , Figure 1 This is a flow chart of a log processing method in an embodiment of the present application. The method is applied to a node server and includes the following steps:

[0078] S101. Collect log items based on a collection frequency to obtain log information, and record log flow information.

[0079] The log item is the name of the monitoring indicator. For a node server, it may correspond to multiple monitoring items, and different monitoring items correspond to the same or different collection frequencies according to different monitoring requirements.

[0080] When collecting logs, log items are collected based on the collection frequency to obtain log information. In order to facilitate subsequent log flow control, log flow information can also be recorded during the log collection process.

[0081] Log flow information may specifically include information such as the collected log items, collection frequency, log volume, and collection time.

[0082] In a specific implementation of the present application, in the initialization phase, it includes:

[0083] Step 1: Receive log configuration information sent by the log server;

[0084] Step 2: Use the log configuration information to determine the log items and the initial collection frequency of the log items.

[0085] For ease of description, the above steps are combined for explanation below.

[0086] In this embodiment, the log server may send log configuration information to the node server. The log configuration information may include information such as which log items to be collected and the initial collection frequency. After the node server obtains the log configuration information, it may determine the log items to be collected and the initial collection frequency of the log items.

[0087] In a specific implementation of the present application, the following steps may also be performed:

[0088] Step 1: Receive log configuration adjustment information sent by the log server;

[0089] Step 2: Adjust the information according to the log configuration and redefine the log items.

[0090] For ease of description, the above steps are combined for explanation below.

[0091] That is to say, the log server can adjust the log configuration, and after the adjustment, it can send the log configuration adjustment information to the node server. After the node server receives the log configuration adjustment information, it can restart the log item. For example, according to the needs, it is necessary to add a new log item, or according to the needs, it is necessary to delete some log items, which can be adjusted by sending the log configuration adjustment information to the node server.

[0092] S102: Send the log information to a log server, and obtain the log bandwidth of the log server for processing the log information.

[0093] After the node server collects the log information, it needs to send the log information to the log server. In the actual transmission process of the log information, the node server can send the log information to the gateway, and then the gateway forwards the log information to send the log information to the log server.

[0094] After receiving the log information, the log server can process the log information.

[0095] In a specific implementation of the present application, sending log information to a log server includes:

[0096] Cache the log information in the log queue to be sent;

[0097] Use the log waiting queue to send log information to the log server.

[0098] For ease of description, the above two steps are combined for explanation below.

[0099] In this embodiment, the log information can be buffered in a log waiting to be sent queue, and the log information is sent to the log server through the log waiting to be sent queue. In this way, transmission control can be achieved by controlling the log waiting to be sent queue.

[0100] In a specific implementation of the present application, using a log to-be-sent queue to send log information to a log server includes:

[0101] Get bandwidth limits from the log server;

[0102] Read the logs whose log volume matches the bandwidth limit from the log waiting to be sent queue;

[0103] Send the read logs to the log server.

[0104] For ease of description, the above three steps are combined for explanation below.

[0105] The node server can obtain the bandwidth limit from the log server, and then read the logs whose log volume matches the bandwidth limit from the log waiting queue when sending logs, and send the read logs to the log server.

[0106] The log bandwidth may be obtained by receiving feedback from a log server, or may be obtained by actively pulling the log bandwidth from the log server.

[0107] Log bandwidth refers to the bandwidth of the log server processing log information.

[0108] In this embodiment, in order to effectively perform log traffic control, the log bandwidth of the log server for processing log information can be obtained in real time.

[0109] S103. Determine the upper limit of the traffic bandwidth based on the network bandwidth and the log bandwidth.

[0110] Network bandwidth is related to log transmission, and log bandwidth is related to the processing pressure of the log server.

[0111] In order to ensure that log collection and analysis are carried out stably and reliably, it is necessary not only to ensure that log data is not lost during log transmission, but also to avoid excessive pressure on the log server. Therefore, the traffic bandwidth upper limit can be determined based on the network bandwidth and log bandwidth. The traffic bandwidth upper limit can be used to constrain the flow control of log collection.

[0112] In a specific implementation of the present application, the upper limit of the traffic bandwidth is determined by combining the network bandwidth and the log bandwidth, including:

[0113] The minimum value of the network bandwidth and the log bandwidth is determined as the upper limit of the traffic bandwidth.

[0114] Specifically, if the log bandwidth w on the log server 1 , the system's network bandwidth w 2 , then the upper limit of traffic bandwidth: w = min{w 1 , w 2}, that is, select the minimum value from w1 and w2 as the upper limit of the traffic bandwidth.

[0115] S104: Perform traffic prediction based on the log traffic information to obtain the total log traffic for the next time period.

[0116] The log acquisition situation of the next time period can refer to the current log collection traffic situation. Therefore, the total log traffic of the next time period can be inferred based on the log traffic information. Specifically, in actual applications, the average log acquisition bandwidth can be calculated based on the log traffic information, combined with the length of the time period for each collection statistic, and the total log traffic of the next time period can be obtained by multiplying the two. Of course, in actual applications, considering that there will be differences in log acquisition situations, it is also possible to use a model with predictive capabilities to make accurate predictions, so as to obtain a more accurate total log traffic of the next time period. There is no restriction on the model with predictive capabilities itself, it is only required to be able to predict the total log traffic of the next time period based on the log traffic information.

[0117] In a specific implementation of the present application, log flow information is input into a neural network model; wherein the log flow information includes each log item, and the collection frequency, log time and log flow of the log item;

[0118] The neural network model is used to predict the log traffic information and obtain the total log traffic in the next time period.

[0119] For ease of description, the above steps are combined for explanation below.

[0120] In this embodiment, the training samples related to the log traffic information can be used in advance to train the neural network model for traffic prediction, so that the neural network model can perform traffic prediction for the input log traffic information and obtain the total log traffic for the next time period.

[0121] After the training of the neural network model is completed, when the log information is collected and the log flow information is recorded, the log flow information can be input into the neural network model for flow prediction, so as to obtain the total log flow in the next time period.

[0122] In a specific implementation of the present application, the log flow includes each log item, and the collection frequency, log time and log flow of the log item;

[0123] The log traffic information is input into the neural network model for traffic prediction to obtain the total log traffic for the next time period, including:

[0124] Each log item, as well as the collection frequency, log time and log flow of the log item, is input into the neural network model for flow prediction to obtain the total log flow in the next time period.

[0125] That is to say, the log flow information input to the neural network model includes each log item, its collection frequency, log time and log flow. The neural network model predicts the flow based on this information and can obtain the total log flow in the next time period.

[0126] S105 . When the total log flow is greater than the upper limit of the flow bandwidth, adjust the collection frequency of log items so that the total log flow is not greater than the upper limit of the flow bandwidth.

[0127] After predicting the total log traffic, compare the total log traffic with the traffic bandwidth upper limit. If it is found that the total log traffic is not greater than the traffic bandwidth upper limit, it indicates that the currently collected logs will not cause problems during transmission and processing, and no adjustment is required at this time; if it is found that the total log traffic is greater than the traffic bandwidth upper limit, it indicates that the currently collected logs may cause problems during transmission or log server processing, such as log transmission loss or excessive processing pressure on the metadata server.

[0128] At this time, the collection frequency of the log items may be adjusted until the total log traffic predicted based on the adjusted collection frequency is no greater than the traffic bandwidth upper limit.

[0129] In a specific implementation of the present application, when the total log flow is greater than the flow bandwidth upper limit, adjusting the collection frequency of log items so that the total log flow is not greater than the flow bandwidth upper limit includes:

[0130] Step 1: Determine whether the total log traffic is greater than the traffic bandwidth limit;

[0131] Step 2: If yes, the collection frequency of the log items is lowered, and after the adjustment, the neural network model is used to predict the flow based on the new collection frequency to obtain the target total log flow for the next time period;

[0132] Step 3: If no, it is determined that there is no need to lower the collection frequency.

[0133] For ease of description, the above three steps are combined for explanation below.

[0134] In this embodiment, after predicting the total log flow, it is determined whether the total log flow is greater than the flow bandwidth upper limit. If it is greater than the log bandwidth upper limit, the collection frequency of the log items needs to be lowered to reduce the target total log flow. In order to ensure accurate adjustment, after lowering the collection frequency of the log items, the neural network model can be used to perform flow prediction based on the new collection frequency to obtain the total log flow for the next time period.

[0135] After obtaining the new total log flow, you can return to step 1 again to make a judgment until the judgment result is that the total log flow is not greater than the flow bandwidth upper limit.

[0136] In a specific implementation of the present application, when the total log traffic is not greater than the traffic bandwidth upper limit, the following is also included:

[0137] Step 1: Determine whether the total target log traffic is less than the traffic bandwidth lower limit;

[0138] Step 2: If yes, the collection frequency of the log items is increased, and after the adjustment, the neural network model is used to predict the traffic based on the new collection frequency to obtain the target total log traffic for the next time period;

[0139] Step 3: If not, it is determined that there is no need to increase the collection frequency.

[0140] For ease of description, the above three steps are combined for explanation below.

[0141] In this embodiment, in order to ensure that the log collection frequency is not too low, when the total log flow is not greater than the upper limit of the flow bandwidth, it is also possible to determine whether the target total log flow is less than the lower limit of the flow bandwidth. The lower limit of the flow bandwidth can be set and adjusted according to the log analysis requirements, and will not be described in detail here.

[0142] If the target log total flow is less than the lower limit of the flow bandwidth, the collection frequency of the log items can be increased. Accordingly, in order to ensure the reliability of the increase, it is necessary to use the neural network model to predict the flow based on the new collection frequency, re-obtain the target log total flow for the next time period, and then return to make a judgment until it is determined that there is no need to increase the collection frequency.

[0143] The node server applies the method provided by the embodiment of the present application, including: collecting log items based on the collection frequency to obtain log information, and recording log traffic information; sending the log information to the log server, and obtaining the log bandwidth of the log server for processing the log information; determining the traffic bandwidth upper limit in combination with the network bandwidth and the log bandwidth; performing traffic prediction based on the log traffic information to obtain the total log traffic in the next time period; when the total log traffic is greater than the traffic bandwidth upper limit, adjusting the collection frequency of the log items so that the total log traffic is not greater than the traffic bandwidth upper limit.

[0144] In the present application, the node server collects log items according to the collection frequency to obtain log information. In order to facilitate log flow control, log flow information will be recorded while collecting logs. The collected log information is sent to the log server. The log server processes the received log information. The node server can obtain the log bandwidth for processing log information from the log server. Combining the network bandwidth and the log bandwidth, the upper limit of the traffic bandwidth can be determined. Based on the log traffic information, the traffic forecast can be obtained to obtain the total log flow in the next time period. When the total log flow is greater than the upper limit of the traffic bandwidth, the collection frequency of the log items is adjusted so that the total log flow is not greater than the upper limit of the traffic bandwidth. In this way, the node server collects logs for the log items based on the adjusted collection frequency in the next time period, and its total log flow will not exceed the upper limit of the traffic bandwidth. In this way, it is possible to avoid occupying the memory data of the node server, and it is also possible to avoid losing logs during the transmission of the collected log information to the log server.

[0145] The technical effect of the present application is as follows: the frequency of log collection can be adaptively adjusted according to the current transmission and processing capabilities of the log server, thereby ensuring system stability and monitorability.

[0146] Please refer to Figure 2 The present application provides a log processing method applicable to a log server, the method comprising:

[0147] Step S201: Send log configuration information to the node server.

[0148] The log server may send log configuration information to the node server. The log configuration information may specifically include log items and the initial collection frequency of the log items.

[0149] Step S202: Receive log information sent by the node server.

[0150] After the node server is initialized based on the log configuration information, log collection can be performed, and the collected log information can be sent to the node server.

[0151] After receiving the log information, the node server may execute the operation of step S203.

[0152] Step S203: Process the received log information and record the log bandwidth of the processed log information.

[0153] For how the log server specifically processes log information, please refer to the log analysis related solutions, which will not be described here one by one. In the process of processing log information, the log server can record the log bandwidth for processing log information. The log bandwidth refers to the throughput of the log server for processing log information.

[0154] Step S204: Send the log bandwidth to the node server so that the node server adjusts the collection frequency of log items based on the log bandwidth.

[0155] The log server sends the log bandwidth to the node server.

[0156] After receiving the log bandwidth, the node server can execute the following Figure 1 The related collection control operations of the log processing method shown.

[0157] In a specific implementation of the present application, when there are multiple node servers, sending the log bandwidth to the node server includes:

[0158] According to the log collection weight of each node server, the log bandwidth is divided into multiple log sub-bandwidths;

[0159] The log sub-bandwidth is sent to each node server so that each node server adjusts the collection frequency of log items based on the received log sub-bandwidth.

[0160] The log collection weight may be predetermined according to the business importance of the node server.

[0161] For example, when the log server needs to manage logs for three node servers, the importance of the services running on these three node servers is different, among which node server 1 is the most important and node server 3 is the least important. Therefore, the log collection weights of node server 1, node server 2 and node server 3 can be 3:2:1; accordingly, the log bandwidth of the log bandwidth server is divided and distributed based on 3:2:1. After node server 1 is adjusted based on the allocated log sub-bandwidth, its log item collection frequency is the highest, and after node server 3 is adjusted based on the allocated log sub-bandwidth, its log item collection frequency is the lowest. This allows the log server to focus on analyzing and processing the logs of more important node servers, thereby realizing the regulation and allocation of log monitoring resources.

[0162] By applying the method provided in the embodiment of the present application, log configuration information can be sent to the node server, so that the node server is initialized based on the log configuration information and collects log information. In the process of processing the log information sent by the node server, the log bandwidth is fed back to the node server, so that the node server can adapt and adjust the collection frequency of log items based on the log bandwidth, which can avoid the loss of logs during log transmission and the overload operation of the log server, and can effectively ensure the reliability of log collection and analysis processing.

[0163] Corresponding to the above method embodiment, the embodiment of the present application further provides a node server and a log server. The node server and the log server described below can correspond to the log processing method described above.

[0164] The internal structure of the node server and log server can refer to Figure 3 and Figure 4 Electronic equipment shown.

[0165] See also Figure 3 As shown, the electronic device includes:

[0166] A memory 332, for storing computer programs;

[0167] The processor 322 is used to implement the steps of the log processing method of the above method embodiment when executing a computer program.

[0168] For details, please refer to Figure 4 , Figure 4A specific structural diagram of an electronic device provided for this embodiment, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the electronic device 301.

[0169] The electronic device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0170] The steps in the log processing method described above can be implemented by the structure of the electronic device. Specifically, when the electronic device is a node server, the processor is used to implement the following when executing the computer program: Figure 1 When the electronic device is a log server, the processor is used to execute the computer program to implement the following steps. Figure 2 The steps of the log processing method are shown.

[0171] Corresponding to the above method embodiment, the embodiment of the present application further provides a log processing system. The log processing system described below and the log processing method described above can be referred to in correspondence with each other.

[0172] Please refer to Figure 5 , the log processing system includes:

[0173] One log server 10 as described above, a plurality of node servers 20 as described above, and a gateway device 30 for connecting the log server and the node servers.

[0174] In this log processing system, the node server can execute the following Figure 1 The log processing method shown in the figure, the log server can execute the following Figure 2 The log processing method shown.

[0175] To help those skilled in the art better understand how to implement the log processing method in a log processing system, a detailed description is given below with reference to a specific application scenario as an example.

[0176] Please refer to Figure 6, the log processing method is implemented in the log processing system (only node server 1 to node server N and the log server are drawn in the figure), and its overall architecture is mainly divided into four modules: log collection configuration module, log collection control module, log transmission control module and log receiving control module.

[0177] Among them, the log collection configuration module is located in the log server, which mainly controls the log items that need to be collected to the node server, and the node server collects logs according to the configured log collection items. The log collection configuration module mainly has the following functions:

[0178] (1) Dynamically configure log collection items. You can preset the log items that need to be collected, or manually add the log items that need to be collected later.

[0179] (2) After the log collection items are configured, the log items that need to be collected can be automatically sent to each node server, and the node server will collect the logs;

[0180] (3) It can automatically send the currently processed log bandwidth according to its own log processing capability and transmit it to the node server, which controls the log sending bandwidth to be limited to the processing capability of the log server.

[0181] The log collection control module is located in the node server and is mainly responsible for receiving the log collection item configuration, collecting logs according to the configuration, and intelligently adjusting the frequency of log collection. Its main responsibilities include:

[0182] (1) Perform incremental log collection based on the preset collection frequency for each log collection item, and save the total amount of logs collected each time;

[0183] (2) Use a neural network model to train log collection. Based on the trained model, predict the amount of logs to be collected in the next time period. Adjust the collection frequency based on the current network bandwidth and the log bandwidth that the log server can process, so that the collected logs can meet the current bandwidth capacity.

[0184] That is to say, after receiving the log collection item configuration, the log collection control module can intelligently adjust the log collection frequency so as to meet the log collection bandwidth limit. The specific adjustment process includes:

[0185] The log collection configuration module will issue log collection items. If the current average log bandwidth w for each node server log processing is 1 , the system's network bandwidth w 2 , then the overall traffic bandwidth upper limit: w = min{w 1 , w 2}; The acquisition control module sets the initial acquisition frequency δ according to the acquisition items (the total number is N). i (1≤i≤N), log collection is performed for each log item, and the collected log volume can be saved in time series. When predicting the target total log flow in the next time period, a neural network model is introduced, and T is used to represent the collected log volume, i is the log item, and t is the time. The log volume collected for each log item is as follows:

[0186] T i =f(δ i ,t); train the module according to the collected log items, collection frequency, log time, and log volume, and then predict the log collection volume in the next time period, and get: T′ i =f(δ i ,t′);

[0187] The predicted target log total flow is: Sum the predicted log collection volume. If it exceeds the traffic bandwidth limit w, reduce the frequency δ one by one. i , so that M≤w. It can be seen that by adjusting δ i , and finally the predicted log collection traffic can be within the entire traffic bandwidth limit.

[0188] The log transmission control module is located in the node server and includes a log waiting to be sent queue. The logs collected by the log collection control module are sent to the queue. In addition, the module takes out the log content from the queue according to the limited bandwidth capacity and sends it to the log receiving control module of the log server.

[0189] By adding a log waiting to be sent queue in the log transmission control module, the queue is controlled by the program internally and acts as a log cache, which can effectively alleviate the fluctuation in log collection. Specifically, the log collection module stores the collected logs in the log waiting to be sent queue according to the optimal collection frequency.

[0190] The log transmission control module takes out the log quantity within the bandwidth limit from the queue to be sent according to the bandwidth limit sent by the log server; and sends the taken out logs to the log receiving control module of the log server;

[0191] The log receiving control module is located in the log server and is mainly used to receive the log information sent by the log transmission control module and save and process the received log content.

[0192] In the log processing system, the frequency of log collection is dynamically adjusted and controlled through artificial intelligence, thereby effectively controlling the flow of log collection, making full use of network bandwidth, and reducing the load of the log system. Through intelligent flow control and increasing log queues, the log sending bandwidth can be limited, thereby reducing the failure and loss of log sending.

[0193] Corresponding to the above method embodiment, the embodiment of the present application further provides a readable storage medium. The readable storage medium described below and the log processing method described above can refer to each other.

[0194] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log processing method of the above method embodiment are implemented.

[0195] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0196] Corresponding to the above method embodiment, the embodiment of the present application further provides a computer program product. The computer program product described below and the log processing method described above can refer to each other.

[0197] A computer program product includes a computer program / instruction. When the computer program / instruction is executed by a processor, the steps of the above-mentioned log processing method are implemented.

[0198] Corresponding to the above method embodiment, the embodiment of the present application further provides a log processing device. The log processing device described below and the log processing method described above can refer to each other.

[0199] A log processing device applied to a node server, comprising:

[0200] A log collection unit, used to collect log items based on a collection frequency to obtain log information, and record log flow information;

[0201] The log bandwidth acquisition unit is used to send the log information to the log server and obtain the log bandwidth of the log server for processing the log information;

[0202] An upper limit determination unit, used to determine the upper limit of the traffic bandwidth by combining the network bandwidth and the log bandwidth;

[0203] A traffic prediction unit, used to perform traffic prediction based on log traffic information to obtain the total log traffic in the next time period;

[0204] The flow regulating unit is used to adjust the collection frequency of log items when the total log flow is greater than the flow bandwidth upper limit, so that the total log flow is not greater than the flow bandwidth upper limit.

[0205] In a specific implementation of the present application, the traffic prediction unit is specifically used to

[0206] Inputting log flow information into the neural network model; wherein the log flow information includes each log item, and the collection frequency, log time and log flow of the log item;

[0207] The neural network model is used to predict the log traffic information and obtain the total log traffic in the next time period.

[0208] In a specific implementation of the present application, the traffic regulating unit is specifically used to determine whether the total log traffic is greater than the traffic bandwidth upper limit;

[0209] If yes, the collection frequency of the log items is lowered, and after the adjustment, the neural network model is used to predict the traffic based on the new collection frequency to obtain the target total log traffic for the next time period;

[0210] If not, it is determined that there is no need to reduce the acquisition frequency.

[0211] In a specific implementation of the present application, the flow regulating unit is further used to determine whether the target log total flow is less than the flow bandwidth lower limit when the log total flow is not greater than the flow bandwidth upper limit;

[0212] If yes, the collection frequency of the log items is increased, and after the adjustment, the neural network model is used to predict the traffic based on the new collection frequency to obtain the target total log traffic for the next time period;

[0213] If not, it is determined that there is no need to increase the acquisition frequency.

[0214] In a specific implementation manner of the present application, the upper limit determination unit is specifically configured to determine the minimum value of the network bandwidth and the log bandwidth as the flow bandwidth upper limit.

[0215] In a specific implementation of the present application, the log sending unit is specifically used to cache the log information in a log to-be-sent queue;

[0216] Use the log waiting queue to send log information to the log server.

[0217] In a specific implementation of the present application, the log sending unit is specifically used to obtain the bandwidth limit from the log server;

[0218] Read the logs whose log volume matches the bandwidth limit from the log waiting to be sent queue;

[0219] Send the read logs to the log server.

[0220] In a specific implementation of the present application, the initialization unit is used to receive log configuration information sent by the log server during the initialization phase;

[0221] Use the log configuration information to determine the log items and the initial collection frequency of the log items.

[0222] In a specific implementation of the present application, it also includes:

[0223] A log item adjustment unit, used for receiving log configuration adjustment information sent by a log server;

[0224] Re-determine the log items according to the log configuration adjustment information.

[0225] A log processing device applied to a log server, comprising:

[0226] A configuration unit, used to send log configuration information to the node server;

[0227] A log receiving unit, used to receive log information sent by the node server;

[0228] A log processing unit, used to process the received log information and record the log bandwidth of the processed log information;

[0229] The log bandwidth feedback unit is used to send the log bandwidth to the node server so that the node server can adjust the collection frequency of log items based on the log bandwidth.

[0230] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0231] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0232] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0233] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms include, include or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0234] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A log processing method, characterized in that: Applicable to node servers, including: Collect log items based on the collection frequency to obtain log information, and record log flow information; Sending the log information to a log server and obtaining a log bandwidth for the log server to process the log information; Determine the upper limit of traffic bandwidth by combining the network bandwidth and the log bandwidth; Performing flow prediction based on the log flow information to obtain the total log flow for the next time period; When the total log flow is greater than the flow bandwidth upper limit, the collection frequency of log items is adjusted so that the total log flow is not greater than the flow bandwidth upper limit.

2. The method according to claim 1, characterized in that Based on the log traffic information, traffic prediction is performed to obtain the total log traffic in the next time period, including: Inputting the log traffic information into a neural network model; The neural network model is used to perform flow prediction on the log flow information to obtain the total log flow in the next time period.

3. The method according to claim 2, characterized in that When the total log flow is greater than the flow bandwidth upper limit, adjusting the collection frequency of log items so that the total log flow is not greater than the flow bandwidth upper limit includes: Determine whether the total log traffic is greater than the traffic bandwidth upper limit; If yes, the collection frequency of the log items is lowered, and after the adjustment, the neural network model is used to perform traffic prediction based on the new collection frequency to obtain the target total log traffic for the next time period; If not, it is determined that there is no need to reduce the acquisition frequency.

4. The method according to claim 2, characterized in that: In the case where the total log traffic is not greater than the traffic bandwidth upper limit, the method further includes: Determine whether the target log total flow is less than the flow bandwidth lower limit; If yes, the collection frequency of the log items is increased, and after the adjustment, the neural network model is used to perform traffic prediction based on the new collection frequency to obtain the target total log traffic for the next time period; If not, it is determined that there is no need to increase the acquisition frequency.

5. The method according to claim 1, characterized in that Combine the network bandwidth and the log bandwidth to determine the upper limit of the traffic bandwidth, including: The minimum value of the network bandwidth and the log bandwidth is determined as the traffic bandwidth upper limit.

6. The method according to claim 1, characterized in that Sending the log information to a log server includes: Cache the log information in a log waiting to be sent queue; The log information is sent to the log server by utilizing the log waiting to be sent queue.

7. The method according to claim 6, characterized in that Using the log to-be-sent queue to send the log information to the log server includes: Obtaining the bandwidth limit from the log server; Reading logs whose log volume matches the bandwidth limit from the log to be sent queue; The read log is sent to the log server.

8. The method according to claim 1, characterized in that During the initialization phase, it includes: Receiving log configuration information sent by the log server; The log configuration information is used to determine log items and the initial collection frequency of the log items.

9. The method according to claim 1, characterized in that: Also includes: Receiving log configuration adjustment information sent by the log server; The log item is re-determined according to the log configuration adjustment information.

10. A log processing method, characterized in that: Applicable to log servers, including: Send log configuration information to the node server; Receiving log information sent by the node server; Processing the received log information and recording the log bandwidth for processing the log information; The log bandwidth is sent to a node server so that the node server adjusts the frequency of collecting log items based on the log bandwidth.

11. A node server, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the log processing method according to any one of claims 1 to 9 when executing the computer program.

12. A log server, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of the log processing method as claimed in claim 10 when executing the computer program.

13. A log processing system, characterized in that: include: 1 log server as claimed in claim 12, multiple node servers as claimed in claim 11 and a gateway device for connecting the log server and the node servers.

14. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the log processing method according to any one of claims 1 to 10 are implemented.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the log processing method described in any one of claims 1 to 10 are implemented.

Images