Log processing method, server, system, readable storage medium and program product

Abstract

In the field of information technology, the present application discloses a log processing method, server, system, readable storage medium, and program product. The node server collects log items based on the collection frequency to obtain log information and records log traffic information; sends the log information to the log server and obtains the log bandwidth used by the log server to process the log information; determines the traffic bandwidth upper limit based on the network bandwidth and the log bandwidth; performs traffic prediction based on the log traffic information to obtain the total log traffic for the next time period; and, when the total log traffic exceeds the traffic bandwidth upper limit, adjusts the collection frequency of the log items so that the total log traffic does not exceed the traffic bandwidth upper limit. The technical effect of the present application is that the log collection frequency can be adaptively adjusted according to the current transmission and the processing capacity of the log server, thereby ensuring system stability and monitorability.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a log processing method, a server, a system, a readable storage medium, and a program product. Background Art

[0002] System log collection and analysis involves collecting, storing, processing, and analyzing log data from various systems and applications. Stable and reliable log transmission is a key component to ensuring system stability and monitorability. It involves efficiently and reliably transmitting log data from its generation point to the log processing and storage systems.

[0003] During the log collection process, if the log volume increases over a period of time, sending the collected logs to the log server may increase the server memory pressure. In addition, due to bandwidth limitations, if the log volume sent exceeds the network bandwidth, data loss may occur.

[0004] In summary, how to effectively solve problems such as flow control in log processing is a technical problem that those skilled in the art urgently need to solve. Summary of the Invention

[0005] The purpose of this application is to provide a log processing method, server, system, readable storage medium, and program product. Based on the log bandwidth and network bandwidth of the log server's processed log information, the traffic bandwidth upper limit can be determined. Traffic prediction based on a neural network model can be used to obtain the total log traffic for the next time period. By combining the total log traffic and the traffic bandwidth upper limit, the frequency of log item collection can be adjusted to ensure that the total log traffic does not exceed the traffic bandwidth upper limit, thereby preventing excessive log usage on the server memory and ensuring that log transmission losses do not cause log loss.

[0006] To solve the above technical problems, this application provides the following technical solutions:

[0007] A log processing method, applied to a node server, comprising:

[0008] Collect log items based on the collection frequency to obtain log information and record log traffic information;

[0009] Sending the log information to a log server and obtaining a log bandwidth of the log server for processing the log information;

[0010] Determine the upper limit of traffic bandwidth based on the network bandwidth and the log bandwidth;

[0011] Performing traffic prediction based on the log traffic information to obtain the total log traffic for the next time period;

[0012] When the total log flow is greater than the flow bandwidth upper limit, the frequency of collecting log items is adjusted so that the total log flow is not greater than the flow bandwidth upper limit.

[0013] Preferably, traffic prediction is performed based on the log traffic information to obtain the total log traffic for the next time period, including:

[0014] Inputting the log traffic information into a neural network model;

[0015] The neural network model is used to perform traffic prediction on the log traffic information to obtain the total log traffic in the next time period.

[0016] Preferably, when the total log flow is greater than the flow bandwidth upper limit, adjusting the collection frequency of log items so that the total log flow is not greater than the flow bandwidth upper limit includes:

[0017] Determine whether the total log traffic is greater than the traffic bandwidth upper limit;

[0018] If yes, the collection frequency of the log items is lowered, and after the adjustment, the neural network model is used to perform traffic prediction based on the new collection frequency to obtain the target total log traffic for the next time period;

[0019] If not, it is determined that there is no need to lower the acquisition frequency.

[0020] Preferably, when the total log traffic is not greater than the traffic bandwidth upper limit, the method further includes:

[0021] Determine whether the total target log flow is less than the flow bandwidth lower limit;

[0022] If yes, the collection frequency of the log items is increased, and after the adjustment, the neural network model is used to perform traffic prediction based on the new collection frequency to obtain the target total log traffic for the next time period;

[0023] If not, it is determined that there is no need to increase the acquisition frequency.

[0024] Preferably, the flow bandwidth upper limit is determined by combining the network bandwidth and the log bandwidth, including:

[0025] The minimum value of the network bandwidth and the log bandwidth is determined as the traffic bandwidth upper limit.

[0026] Preferably, sending the log information to a log server includes:

[0027] Cache the log information in a log waiting to be sent queue;

[0028] The log information is sent to the log server using the log waiting queue.

[0029] Preferably, using the log to-be-sent queue to send the log information to the log server includes:

[0030] Obtaining the bandwidth limit from the log server;

[0031] Reading logs whose log volume matches the bandwidth limit from the log to be sent queue;

[0032] The read log is sent to the log server.

[0033] Preferably, the initialization phase includes:

[0034] Receiving log configuration information sent by the log server;

[0035] The log configuration information is used to determine log items and an initial collection frequency of the log items.

[0036] Preferably, it also includes:

[0037] Receiving log configuration adjustment information sent by the log server;

[0038] The log items are re-determined according to the log configuration adjustment information.

[0039] A log processing method, applied to a log server, comprising:

[0040] Send log configuration information to the node server;

[0041] Receiving log information sent by the node server;

[0042] Processing the received log information and recording the log bandwidth for processing the log information;

[0043] The log bandwidth is sent to a node server so that the node server adjusts a frequency of collecting log items based on the log bandwidth.

[0044] A node server, comprising:

[0045] Memory for storing computer programs;

[0046] A processor is used to implement the steps of the log processing method as described above when executing the computer program.

[0047] A log server, comprising:

[0048] Memory for storing computer programs;

[0049] A processor is used to implement the steps of the log processing method as described above when executing the computer program.

[0050] A log processing system, comprising:

[0051] One log server as described above, multiple node servers as described above, and a gateway device for connecting the log server and the node servers.

[0052] A readable storage medium stores a computer program, which implements the steps of the log processing method when executed by a processor.

[0053] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned log processing method.

[0054] A log processing device applied to a node server, comprising:

[0055] A log collection unit, configured to collect log items based on a collection frequency to obtain log information and record log flow information;

[0056] a log bandwidth obtaining unit, configured to send the log information to a log server and obtain a log bandwidth for the log server to process the log information;

[0057] an upper limit determination unit, configured to determine a flow bandwidth upper limit by combining the network bandwidth and the log bandwidth;

[0058] A traffic prediction unit, configured to perform traffic prediction based on the log traffic information to obtain the total log traffic for the next time period;

[0059] The flow regulating unit is used to adjust the collection frequency of log items when the total log flow is greater than the flow bandwidth upper limit so that the total log flow is not greater than the flow bandwidth upper limit.

[0060] A log processing device applied to a log server, comprising:

[0061] Configuration unit, used to send log configuration information to the node server;

[0062] A log receiving unit, configured to receive log information sent by the node server;

[0063] A log processing unit, configured to process the received log information and record the log bandwidth for processing the log information;

[0064] The log bandwidth feedback unit is configured to send the log bandwidth to the node server so that the node server adjusts the frequency of collecting log items based on the log bandwidth.

[0065] The node server applies the method provided by the embodiment of the present application, including: collecting log items based on the collection frequency to obtain log information, and recording log traffic information; sending the log information to the log server, and obtaining the log bandwidth of the log server for processing the log information; determining the traffic bandwidth upper limit based on the network bandwidth and the log bandwidth; performing traffic prediction based on the log traffic information to obtain the total log traffic for the next time period; and when the total log traffic is greater than the traffic bandwidth upper limit, adjusting the collection frequency of the log items so that the total log traffic is not greater than the traffic bandwidth upper limit.

[0066] In the present application, the node server collects log items according to the collection frequency to obtain log information. In order to facilitate log flow control, log flow information will be recorded while collecting logs. The collected log information is sent to the log server. The log server processes the received log information. The node server can obtain the log bandwidth for processing log information from the log server. Combining the network bandwidth and the log bandwidth, the upper limit of the traffic bandwidth can be determined. Based on the log flow information, traffic prediction is performed to obtain the total log flow for the next time period. When the total log flow is greater than the upper limit of the traffic bandwidth, the collection frequency of the log items is adjusted so that the total log flow is not greater than the upper limit of the traffic bandwidth. In this way, the node server collects log items based on the adjusted collection frequency in the next time period, and its total log flow will not exceed the upper limit of the traffic bandwidth. In this way, the memory data of the node server can be avoided from being occupied, and the collected log information can be prevented from being lost during transmission to the log server.

[0067] The technical effect of this application is that the log collection frequency can be adaptively adjusted according to the current transmission and processing capabilities of the log server, which can ensure system stability and monitorability.

[0068] Accordingly, the embodiments of the present application also provide a node server, a log server, a log processing system, a readable storage medium and a computer program product corresponding to the above-mentioned log processing method, which have the above-mentioned technical effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 This is a flow chart of an implementation of a log processing method in an embodiment of the present application;

[0071] Figure 2 This is a flowchart of another log processing method in the embodiment of this application.

[0072] Figure 3 This is a schematic structural diagram of an electronic device according to an embodiment of the present application;

[0073] Figure 4 This is a schematic diagram of the specific structure of an electronic device in an embodiment of the present application;

[0074] Figure 5 This is a schematic diagram of a log processing system according to an embodiment of the present application;

[0075] Figure 6 This is a schematic diagram of the specific functional modules of a log processing system in an embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific embodiments. Obviously, the embodiments described are only a part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of the present application.

[0077] Please refer to Figure 1 , Figure 1 This is a flow chart of a log processing method in an embodiment of the present application, which is applied to a node server and includes the following steps:

[0078] S101: Collect log items based on a collection frequency to obtain log information, and record log traffic information.

[0079] The log item is the name of the monitoring indicator. For a node server, it can correspond to multiple monitoring items. Different monitoring items correspond to the same or different collection frequencies based on different monitoring requirements.

[0080] When collecting logs, log items are collected based on the collection frequency to obtain log information. In order to facilitate subsequent log flow control, log flow information can also be recorded during the log collection process.

[0081] Log flow information may specifically include information such as the collected log items, collection frequency, log volume, and collection time.

[0082] In a specific implementation of the present application, during the initialization phase, the following steps are included:

[0083] Step 1: Receive log configuration information sent by the log server;

[0084] Step 2: Use the log configuration information to determine the log items and the initial collection frequency of the log items.

[0085] For ease of description, the above steps are combined and explained below.

[0086] In this embodiment, the log server can send log configuration information to the node server. The log configuration information may include information such as which log items to collect and the initial collection frequency. After receiving the log configuration information, the node server can determine the log items to be collected and the initial collection frequency of the log items.

[0087] In a specific embodiment of the present application, the following steps may also be performed:

[0088] Step 1: Receive log configuration adjustment information sent by the log server;

[0089] Step 2: Adjust the information according to the log configuration and redefine the log items.

[0090] For ease of description, the above steps are combined and explained below.

[0091] In other words, the log server can adjust the log configuration and send the log configuration adjustment information to the node server. After receiving the log configuration adjustment information, the node server can restart the log entry. For example, if you need to add a new log entry or delete certain log entries, the log server can send the log configuration adjustment information to the node server to make the adjustment.

[0092] S102: Send the log information to the log server, and obtain the log bandwidth of the log server for processing the log information.

[0093] After the node server collects the log information, it needs to send the log information to the log server. In the actual transmission process of the log information, the node server can send the log information to the gateway, and then the gateway forwards the log information to send the log information to the log server.

[0094] After receiving the log information, the log server can process the log information.

[0095] In a specific implementation of the present application, sending log information to a log server includes:

[0096] Cache log information in the log sending queue;

[0097] Use the log waiting queue to send log information to the log server.

[0098] For ease of description, the above two steps are combined for explanation below.

[0099] In this embodiment, the log information can be buffered in a log waiting to be sent queue, and the log information is sent to the log server through the log waiting to be sent queue. In this way, transmission control can be achieved by controlling the log waiting to be sent queue.

[0100] In a specific embodiment of the present application, using a log to-be-sent queue to send log information to a log server includes:

[0101] Get bandwidth limits from the log server;

[0102] Read logs whose log volume matches the bandwidth limit from the log sending queue;

[0103] Send the read logs to the log server.

[0104] For ease of description, the above three steps are combined and explained below.

[0105] The node server can obtain the bandwidth limit from the log server, and then read the logs whose log volume matches the bandwidth limit from the log waiting queue when sending logs, and send the read logs to the log server.

[0106] The log bandwidth may be obtained by receiving feedback from a log server, or may be obtained by actively pulling data from the log server.

[0107] Log bandwidth refers to the bandwidth of the log server processing log information.

[0108] In this embodiment, in order to effectively perform log flow control, the log bandwidth of the log server for processing log information can be obtained in real time.

[0109] S103: Determine the upper limit of the traffic bandwidth based on the network bandwidth and the log bandwidth.

[0110] Network bandwidth is related to log transmission, and log bandwidth is related to the processing pressure of the log server.

[0111] To ensure stable and reliable log collection and analysis, it's crucial not only to ensure log data is not lost during transmission but also to avoid placing excessive pressure on the log server. Therefore, a traffic bandwidth cap can be determined based on network bandwidth and log bandwidth. This traffic bandwidth cap can be used to constrain log collection traffic flow.

[0112] In a specific embodiment of the present application, the flow bandwidth upper limit is determined by combining the network bandwidth and the log bandwidth, including:

[0113] The minimum value of the network bandwidth and the log bandwidth is determined as the upper limit of the traffic bandwidth.

[0114] Specifically, if the log bandwidth on the log server is w1 and the system network bandwidth is w2, then the traffic bandwidth upper limit is: w=min{w1, w2}, that is, the minimum value is selected from w1 and w2 as the traffic bandwidth upper limit.

[0115] S104: Perform traffic prediction based on the log traffic information to obtain the total log traffic for the next time period.

[0116] The log acquisition situation at the next time period can refer to the current log collection traffic situation. Therefore, the total log traffic for the next time period can be inferred based on the log traffic information. Specifically, in actual applications, the average log acquisition bandwidth can be calculated based on the log traffic information, and the two can be multiplied by the length of the time period for each collection statistic to obtain the total log traffic for the next time period. Of course, in actual applications, considering that there will be differences in log acquisition situations, it is also possible to use a model with predictive capabilities to make accurate predictions, thereby obtaining a more accurate total log traffic for the next time period. There are no restrictions on the model with predictive capabilities itself, and it is only required to be able to predict the total log traffic for the next time period based on the log traffic information.

[0117] In a specific embodiment of the present application, log flow information is input into a neural network model; wherein the log flow information includes each log item, and the collection frequency, log time and log flow of the log item;

[0118] The neural network model is used to predict the log traffic information and obtain the total log traffic in the next time period.

[0119] For ease of description, the above steps are combined and explained below.

[0120] In this embodiment, the neural network model can be trained in advance with training samples related to log traffic information to perform traffic prediction, so that the neural network model can perform traffic prediction based on the input log traffic information and obtain the total log traffic for the next time period.

[0121] After completing the training of the neural network model, when the log information is collected and the log flow information is recorded, the log flow information can be input into the neural network model for flow prediction, thereby obtaining the total log flow for the next time period.

[0122] In a specific embodiment of the present application, the log flow includes each log item, and the collection frequency, log time and log flow of the log item;

[0123] The log traffic information is input into the neural network model for traffic prediction to obtain the total log traffic for the next time period, including:

[0124] Each log item, as well as its collection frequency, log time, and log traffic, is input into the neural network model for traffic prediction to obtain the total log traffic for the next time period.

[0125] In other words, the log traffic information fed into the neural network model includes each log entry, its collection frequency, log time, and log traffic volume. Based on this information, the neural network model makes traffic predictions, determining the total log traffic volume for the next time period.

[0126] S105 : When the total log flow is greater than the upper limit of the flow bandwidth, adjust the frequency of collecting log items so that the total log flow is not greater than the upper limit of the flow bandwidth.

[0127] After predicting the total log traffic, compare the total log traffic with the traffic bandwidth upper limit. If the total log traffic is not greater than the traffic bandwidth upper limit, it indicates that the currently collected logs will not cause problems during transmission and processing, and no adjustment is required. If the total log traffic is greater than the traffic bandwidth upper limit, it indicates that the currently collected logs may cause problems during transmission or log server processing, such as log transmission loss or excessive processing pressure on the metadata server.

[0128] At this time, the frequency of collecting log items can be adjusted until the total log traffic predicted based on the adjusted frequency is no greater than the traffic bandwidth upper limit.

[0129] In a specific embodiment of the present application, when the total log flow exceeds the upper limit of the flow bandwidth, the frequency of collecting log items is adjusted so that the total log flow does not exceed the upper limit of the flow bandwidth, including:

[0130] Step 1: Determine whether the total log traffic is greater than the traffic bandwidth limit;

[0131] Step 2: If yes, then the frequency of collecting log items is lowered. After the adjustment, the neural network model is used to predict the traffic based on the new frequency of collection to obtain the target total log traffic for the next time period.

[0132] Step 3: If no, it is determined that there is no need to lower the collection frequency.

[0133] For ease of description, the above three steps are combined and explained below.

[0134] In this embodiment, after predicting the total log traffic, we determine whether it exceeds the traffic bandwidth limit. If so, we adjust the frequency of log item collection downwards to reduce the target total log traffic. To ensure accurate adjustments, after reducing the frequency of log item collection, we use a neural network model to perform traffic prediction based on the new frequency and re-determine the total log traffic for the next time period.

[0135] After obtaining the new total log traffic, you can return to step 1 again and make a judgment until the judgment result is that the total log traffic is not greater than the traffic bandwidth upper limit.

[0136] In a specific implementation of the present application, when the total log traffic is not greater than the traffic bandwidth upper limit, the following is further included:

[0137] Step 1: Determine whether the total target log traffic is less than the traffic bandwidth lower limit;

[0138] Step 2: If yes, then the frequency of collecting log items is adjusted upwards. After the adjustment, the neural network model is used to predict the traffic based on the new frequency of collection to obtain the target total log traffic for the next time period.

[0139] Step 3: If no, it is determined that there is no need to increase the collection frequency.

[0140] For ease of description, the above three steps are combined and explained below.

[0141] In this embodiment, to ensure that the log collection frequency is not too low, if the total log flow rate is not greater than the upper flow bandwidth limit, it is also possible to determine whether the target total log flow rate is less than the lower flow bandwidth limit. This lower flow bandwidth limit can be set and adjusted according to the log analysis requirements, and will not be detailed here.

[0142] If the target total log traffic volume is determined to be less than the traffic bandwidth lower limit, the log item collection frequency can be increased. To ensure the reliability of the increase, a neural network model is used to predict traffic volume based on the new collection frequency. This model then re-derives the target total log traffic volume for the next time period and returns to the original prediction until it is determined that the collection frequency does not need to be increased.

[0143] The node server applies the method provided by the embodiment of the present application, including: collecting log items based on the collection frequency to obtain log information, and recording log traffic information; sending the log information to the log server, and obtaining the log bandwidth of the log server for processing the log information; determining the traffic bandwidth upper limit based on the network bandwidth and the log bandwidth; performing traffic prediction based on the log traffic information to obtain the total log traffic for the next time period; and when the total log traffic is greater than the traffic bandwidth upper limit, adjusting the collection frequency of the log items so that the total log traffic is not greater than the traffic bandwidth upper limit.

[0144] In the present application, the node server collects log items according to the collection frequency to obtain log information. In order to facilitate log flow control, log flow information will be recorded while collecting logs. The collected log information is sent to the log server. The log server processes the received log information. The node server can obtain the log bandwidth for processing log information from the log server. Combining the network bandwidth and the log bandwidth, the upper limit of the traffic bandwidth can be determined. Based on the log flow information, traffic prediction is performed to obtain the total log flow for the next time period. When the total log flow is greater than the upper limit of the traffic bandwidth, the collection frequency of the log items is adjusted so that the total log flow is not greater than the upper limit of the traffic bandwidth. In this way, the node server collects log items based on the adjusted collection frequency in the next time period, and its total log flow will not exceed the upper limit of the traffic bandwidth. In this way, the memory data of the node server can be avoided from being occupied, and the collected log information can be prevented from being lost during transmission to the log server.

[0145] The technical effect of this application is that the log collection frequency can be adaptively adjusted according to the current transmission and processing capabilities of the log server, which can ensure system stability and monitorability.

[0146] Please refer to Figure 2 The present application provides a log processing method applicable to a log server, the method comprising:

[0147] Step S201: Send log configuration information to the node server.

[0148] The log server can send log configuration information to the node server. The log configuration information can specifically include log items and the initial collection frequency of the log items.

[0149] Step S202: Receive log information sent by the node server.

[0150] After the node server is initialized based on the log configuration information, log collection can be performed, and the collected log information will be sent to the node server.

[0151] After receiving the log information, the node server may execute the operation of step S203.

[0152] Step S203: Process the received log information and record the log bandwidth of the processed log information.

[0153] For details on how the log server processes log information, please refer to the log analysis solution and will not be discussed in detail here. During log processing, the log server can record the log bandwidth used to process log information. Log bandwidth refers to the log server's log processing throughput.

[0154] Step S204: Send the log bandwidth to the node server so that the node server adjusts the frequency of collecting log items based on the log bandwidth.

[0155] The log server sends the log bandwidth to the node server.

[0156] After receiving the log bandwidth, the node server can execute the following Figure 1 Related collection control operations of the log processing method shown.

[0157] In a specific implementation of the present application, when there are multiple node servers, sending the log bandwidth to the node server includes:

[0158] According to the log collection weight of each node server, the log bandwidth is divided into multiple log sub-bandwidths;

[0159] The log sub-bandwidth is sent to each node server so that each node server adjusts the frequency of collecting log items based on the received log sub-bandwidth.

[0160] The log collection weight can be predetermined based on the business importance of the node server.

[0161] For example, when the log server needs to manage logs for three node servers, the importance of the businesses running on these three node servers is different, among which node server 1 is the most important and node server 3 is the least important. Therefore, the log collection weights of node server 1, node server 2 and node server 3 can be 3:2:1; accordingly, the log bandwidth of the log bandwidth server is divided and distributed based on 3:2:1. This allows node server 1 to have the highest frequency of log item collection after adjustment based on the allocated log sub-bandwidth, and node server 3 to have the lowest frequency of log item collection after adjustment based on the allocated log sub-bandwidth. This allows the log server to focus on analyzing and processing the logs of more important node servers, thereby realizing the regulation and allocation of log monitoring resources.

[0162] By applying the method provided in the embodiments of the present application, log configuration information can be sent to a node server, thereby enabling the node server to initialize and collect log information based on the log configuration information. During the process of processing the log information sent by the node server, the log bandwidth is fed back to the node server, allowing the node server to adaptively adjust the frequency of log item collection based on the log bandwidth. This can avoid log loss during log transmission and avoid overloading the log server, effectively ensuring the reliability of log collection, analysis and processing.

[0163] Corresponding to the above method embodiment, the embodiment of the present application further provides a node server and a log server. The node server and the log server described below can correspond to the log processing method described above.

[0164] The internal structure of the node server and log server can refer to Figure 3 and Figure 4 Electronic devices shown.

[0165] See also Figure 3 As shown, the electronic device includes:

[0166] Memory 332, for storing computer programs;

[0167] The processor 322 is configured to implement the steps of the log processing method of the above method embodiment when executing a computer program.

[0168] For details, please refer to Figure 4 , Figure 4 This is a schematic diagram of the specific structure of an electronic device provided in this embodiment. The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the electronic device 301.

[0169] The electronic device 301 may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input / output interfaces 358 , and / or one or more operating systems 341 .

[0170] The steps in the log processing method described above can be implemented by the structure of the electronic device. Specifically, when the electronic device is a node server, the processor is used to implement the following when executing the computer program: Figure 1 When the electronic device is a log server, the processor is used to execute the computer program to implement the following steps. Figure 2 The steps of the log processing method are shown.

[0171] Corresponding to the above method embodiment, the embodiment of the present application further provides a log processing system. The log processing system described below and the log processing method described above can be referred to in correspondence with each other.

[0172] Please refer to Figure 5 , the log processing system includes:

[0173] One log server 10 as described above, multiple node servers 20 as described above, and a gateway device 30 for connecting the log server and the node servers.

[0174] In this log processing system, the node server can execute the following Figure 1 The log processing method shown in the following example can be executed by the log server. Figure 2 Log processing method shown.

[0175] To help those skilled in the art better understand how to implement the log processing method in a log processing system, a detailed description is given below with reference to a specific application scenario as an example.

[0176] Please refer to Figure 6 ,The log processing method is implemented in the log processing system (only node server 1 to node server N and the log server are drawn in the figure), and its overall architecture is mainly divided into four modules: log collection configuration module, log collection control module, log transmission control module and log receiving control module.

[0177] Among them, the log collection configuration module, which is located in the log server, mainly controls the log items that need to be collected to the node server, and the node server collects logs according to the configured log collection items. The log collection configuration module mainly has the following functions:

[0178] (1) Ability to dynamically configure log collection items, pre-set log items to be collected, and manually add log items to be collected later;

[0179] (2) After the log collection items are configured, the log items that need to be collected can be automatically sent to each node server, and the node server will collect the logs;

[0180] (3) It can automatically send the log bandwidth that can be processed according to its own log processing capabilities and pass it to the node server. The node server controls the log sending bandwidth to be limited to the processing capacity of the log server.

[0181] The log collection control module, located on the node server, is responsible for receiving log collection item configurations, collecting logs based on the configurations, and intelligently adjusting the frequency of log collection. Its main responsibilities include:

[0182] (1) Perform incremental log collection based on the preset collection frequency for each log collection item, and save the total amount of logs collected each time;

[0183] (2) Use a neural network model to train log collection. Based on the trained model, predict the amount of logs to be collected in the next time period. Adjust the collection frequency based on the current network bandwidth and the log bandwidth that the log server can process, so that the collected logs can meet the current bandwidth capacity.

[0184] In other words, after receiving the log collection item configuration, the log collection control module can intelligently adjust the log collection frequency to meet the log collection bandwidth limit. The specific adjustment process includes:

[0185] The log collection configuration module will issue log collection items. If the current average log bandwidth of each node server log processing is w1 and the system network bandwidth is w2, then the overall traffic bandwidth upper limit is: w = min{w1, w2}; the collection control module sets the initial collection frequency δ according to the collection items (the total number is N). i (1≤i≤N), logs are collected for each log item, and the collected log volume can be saved in time series. When predicting the target total log flow for the next time period, a neural network model is introduced, with T representing the collected log volume, i representing the log item, and t representing the time. The log volume collected for each log item is as follows:

[0186] T i =f(δ i ,t); train the module according to the collected log items, collection frequency, log time, and log volume, and then predict the log collection volume in the next time period, and get: T′ i =f(δ i ,t′);

[0187] The predicted target total log traffic is: Sum the predicted log collection volume. If it exceeds the traffic bandwidth limit w, reduce the frequency δ one by one. i , so that M≤w. It can be seen that by adjusting δ i, which can ultimately satisfy the predicted log collection traffic within the entire traffic bandwidth limit.

[0188] The log transmission control module is located in the node server and includes a log waiting queue. The logs collected by the log collection control module are sent to this queue. In addition, this module takes out the log content from the queue according to the limited bandwidth capacity and sends it to the log receiving control module of the log server.

[0189] By adding a log queue to the log transmission control module, which is controlled internally by the program and acts as a log buffer, fluctuations in log collection can be effectively mitigated. Specifically, the log collection module stores collected logs in the log queue based on the optimal collection frequency.

[0190] The log transmission control module takes out the logs with bandwidth limit from the queue to be sent according to the bandwidth limit sent by the log server; and sends the taken out logs to the log receiving control module of the log server;

[0191] The log receiving control module is located in the log server and is mainly used to receive the log information sent by the log transmission control module and save and process the received log content.

[0192] In the log processing system, artificial intelligence is used to dynamically adjust and control the frequency of log collection, effectively controlling the flow of log collection, fully utilizing network bandwidth, and reducing the load on the log system. Intelligent flow control and the addition of log queues can limit the bandwidth for log transmission, thereby reducing log transmission failures and losses.

[0193] Corresponding to the above method embodiment, an embodiment of the present application further provides a readable storage medium. The readable storage medium described below and the log processing method described above can refer to each other.

[0194] A readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the log processing method of the above method embodiment.

[0195] The readable storage medium may specifically be any readable storage medium that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0196] Corresponding to the above method embodiment, an embodiment of the present application further provides a computer program product. The computer program product described below and the log processing method described above can refer to each other.

[0197] A computer program product includes a computer program / instruction. When the computer program / instruction is executed by a processor, the steps of the log processing method are implemented.

[0198] Corresponding to the above method embodiment, an embodiment of the present application further provides a log processing device. The log processing device described below and the log processing method described above can refer to each other.

[0199] A log processing device applied to a node server, comprising:

[0200] A log collection unit, configured to collect log items based on a collection frequency to obtain log information and record log flow information;

[0201] The log bandwidth acquisition unit is used to send log information to the log server and obtain the log bandwidth of the log server for processing the log information;

[0202] An upper limit determination unit, used to determine the upper limit of traffic bandwidth by combining network bandwidth and log bandwidth;

[0203] A traffic prediction unit is used to perform traffic prediction based on log traffic information to obtain the total log traffic for the next time period;

[0204] The flow regulating unit is used to adjust the collection frequency of log items when the total log flow exceeds the flow bandwidth upper limit so that the total log flow does not exceed the flow bandwidth upper limit.

[0205] In a specific embodiment of the present application, the traffic prediction unit is specifically used to

[0206] Inputting log flow information into the neural network model; wherein the log flow information includes each log item, the collection frequency, log time and log flow of the log item;

[0207] The neural network model is used to predict the log traffic information and obtain the total log traffic in the next time period.

[0208] In a specific embodiment of the present application, the traffic regulating unit is specifically configured to determine whether the total log traffic is greater than the traffic bandwidth upper limit;

[0209] If so, the frequency of log item collection is lowered. After the adjustment, the neural network model is used to predict traffic based on the new frequency to obtain the target total log traffic for the next time period.

[0210] If not, it is determined that there is no need to lower the acquisition frequency.

[0211] In a specific embodiment of the present application, the flow regulating unit is further configured to determine whether the target total log flow is less than the flow bandwidth lower limit when the total log flow is not greater than the flow bandwidth upper limit;

[0212] If yes, the frequency of log item collection is increased. After the adjustment, the neural network model is used to predict traffic based on the new frequency to obtain the target total log traffic for the next time period.

[0213] If not, it is determined that there is no need to increase the acquisition frequency.

[0214] In a specific implementation of the present application, the upper limit determination unit is specifically configured to determine the minimum value of the network bandwidth and the log bandwidth as the traffic bandwidth upper limit.

[0215] In a specific embodiment of the present application, the log sending unit is specifically configured to cache the log information in a log to-be-sent queue;

[0216] Use the log waiting queue to send log information to the log server.

[0217] In a specific implementation of the present application, the log sending unit is specifically configured to obtain the bandwidth limit from the log server;

[0218] Read logs whose log volume matches the bandwidth limit from the log sending queue;

[0219] Send the read logs to the log server.

[0220] In a specific embodiment of the present application, the initialization unit is configured to receive log configuration information sent by the log server during the initialization phase;

[0221] Use the log configuration information to determine the log items and the initial collection frequency of the log items.

[0222] In a specific embodiment of the present application, it also includes:

[0223] A log item adjustment unit, configured to receive log configuration adjustment information sent by a log server;

[0224] Adjust the log configuration information and redefine the log items.

[0225] A log processing device applied to a log server, comprising:

[0226] Configuration unit, used to send log configuration information to the node server;

[0227] A log receiving unit, used to receive log information sent by the node server;

[0228] A log processing unit, configured to process the received log information and record the log bandwidth of the processed log information;

[0229] The log bandwidth feedback unit is used to send the log bandwidth to the node server so that the node server can adjust the frequency of collecting log items based on the log bandwidth.

[0230] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0231] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0232] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0233] Finally, it should be noted that, in this document, relationships such as first and second, etc., are used solely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms include, comprise, or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0234] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A log processing method, characterized in that: Applicable to node servers, including: Collect log items based on the collection frequency to obtain log information and record log traffic information; Sending the log information to a log server and obtaining a log bandwidth of the log server for processing the log information; Determine the upper limit of traffic bandwidth based on the network bandwidth and the log bandwidth; Performing traffic prediction based on the log traffic information to obtain the total log traffic for the next time period; When the total log flow is greater than the flow bandwidth upper limit, the frequency of collecting log items is adjusted so that the total log flow is not greater than the flow bandwidth upper limit.

2. The method according to claim 1, characterized in that Based on the log traffic information, traffic prediction is performed to obtain the total log traffic for the next time period, including: Inputting the log traffic information into a neural network model; The neural network model is used to perform traffic prediction on the log traffic information to obtain the total log traffic in the next time period.

3. The method according to claim 2, characterized in that When the total log flow rate is greater than the flow bandwidth upper limit, adjusting the frequency of collecting log items so that the total log flow rate does not exceed the flow bandwidth upper limit includes: Determine whether the total log traffic is greater than the traffic bandwidth upper limit; If yes, the collection frequency of the log items is lowered, and after the adjustment, the neural network model is used to perform traffic prediction based on the new collection frequency to obtain the total log traffic for the next time period; If not, it is determined that there is no need to lower the acquisition frequency.

4. The method according to claim 2, characterized in that If the total log traffic is not greater than the traffic bandwidth upper limit, the method further includes: Determine whether the total log traffic is less than the traffic bandwidth lower limit; If yes, the collection frequency of the log items is increased, and after the adjustment, the neural network model is used to perform traffic prediction based on the new collection frequency to obtain the total log traffic for the next time period; If not, it is determined that there is no need to increase the acquisition frequency.

5. The method according to claim 1, wherein Combine the network bandwidth and the log bandwidth to determine the upper limit of the traffic bandwidth, including: The minimum value of the network bandwidth and the log bandwidth is determined as the traffic bandwidth upper limit.

6. The method according to claim 1, characterized in that Sending the log information to the log server includes: Cache the log information in a log waiting to be sent queue; The log information is sent to the log server using the log waiting queue.

7. The method according to claim 6, characterized in that Utilizing the log to-be-sent queue to send the log information to the log server includes: Obtaining bandwidth limits from the log server; Reading logs whose log volume matches the bandwidth limit from the log to be sent queue; The read log is sent to the log server.

8. The method according to claim 1, characterized in that During the initialization phase, it includes: Receiving log configuration information sent by the log server; The log configuration information is used to determine log items and an initial collection frequency of the log items.

9. The method according to claim 1, characterized in that Also includes: Receiving log configuration adjustment information sent by the log server; The log items are re-determined according to the log configuration adjustment information.

10. A log processing method, characterized in that: Applicable to log servers, including: Send log configuration information to the node server; Receiving log information sent by the node server; Processing the received log information and recording the log bandwidth for processing the log information; Sending the log bandwidth to a node server so that the node server adjusts the frequency of collecting log items based on the log bandwidth comprises the step of the node server executing the log processing method according to any one of claims 1 to 9.

11. A node server, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the log processing method according to any one of claims 1 to 9 when executing the computer program.

12. A log server, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the log processing method according to claim 10 when executing the computer program.

13. A log processing system, characterized in that: include: A log server as claimed in claim 12, a plurality of node servers as claimed in claim 11, and a gateway device for connecting the log server and the node servers.

14. A readable storage medium, characterized in that The readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the log processing method according to any one of claims 1 to 10.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the log processing method according to any one of claims 1 to 10 are implemented.

Images