Log data processing method and device, terminal equipment and readable storage medium
By collecting and judging the number of logs and traffic quotas in the log data processing system, combined with repetition rate analysis, the problem of crashing in traditional systems when receiving a large number of duplicate logs is solved, reducing the risk of crashes and improving system stability.
Patent Information
- Application Number
- CN202411084441.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional log collection and analysis systems are prone to crash when receiving a large number of duplicate meaningless log files from the target business system, and there is a risk of crash.
By collecting log information of the target business system, obtaining traffic quota, determining the relationship between the number of logs and traffic quota, sending log information to the storage module when the number of logs is less than or equal to the traffic quota, and performing repetition rate when the repetition rate is lower than the preset threshold value Parses, avoid sending duplicate logs, and reduce the risk of crashes.
Reduces the number of log files sent to the storage module, avoids serial crashes caused by a large number of logs, and reduces the risks in log data processing.
Smart Images

Figure CN118939615A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of log data processing, and in particular to a log data processing method, device, terminal equipment and readable storage medium. Background Art
[0002] In today's information society, log collection and analysis systems have become an indispensable part of enterprise information. The log collection and analysis systems currently on the market, such as ELK, Flume, Logstash, Filebeat, Kibana, etc., can efficiently collect and analyze scattered information system log files. However, traditional log collection and analysis systems have no protection mechanism. When receiving a large number of repeated and meaningless log files caused by downtime of the target business system, they are prone to chain crashes and have a certain risk of crashes. Summary of the invention
[0003] The present invention provides a log data processing method, apparatus, terminal device and readable storage medium to solve the technical problem of easy chain crash when receiving a large number of repeated and meaningless log files of a target business system, thereby reducing the risk in the log data processing process.
[0004] In order to solve the above technical problems, an embodiment of the present invention provides a log data processing method, including:
[0005] Collect log information of the target business system; obtain the flow quota of the target business system;
[0006] According to the log information, the number of logs within a first preset time is obtained; and the relationship between the number of logs and the flow quota of the target business system is determined;
[0007] When the number of logs is less than the flow quota, sending the log information to a storage module;
[0008] When the number of logs is greater than or equal to the traffic quota, the log information is analyzed for repetition rate, and when the repetition rate of the log information is less than a preset first repetition threshold, the log information is sent to a storage module.
[0009] As a preferred solution, after performing repetition rate analysis on the log information, the method further includes:
[0010] When the repetition rate of the log information is greater than or equal to the first repetition threshold, the repeated logs are discarded, and the abnormal information of the target business system is sent to the user.
[0011] As a preferred solution, the log data processing method further includes:
[0012] When the number of logs is greater than or equal to the traffic quota and lasts for a second preset time, remove the information in the log information that can change with the system status and extract the business log; perform a repetition rate check on the business log, and when the repetition rate is greater than or equal to a preset second repetition threshold, send the abnormal information of the target business system to the user.
[0013] As a preferred solution, before obtaining the traffic quota of the target business system, the method further includes:
[0014] Access the target business system and allocate an initial flow value to the target business system; after running for a third preset time, adjust the initial flow value according to the average flow during the third preset time to obtain the flow quota.
[0015] As a preferred solution, before sending the log information to the storage module, the method further includes:
[0016] According to the log format of each target business system, the logs that cannot be analyzed in the log information are discarded; the format of the erroneous logs in the log information is corrected; the logs in the log information that exceed the preset length are intercepted to obtain the cleaned log information.
[0017] As a preferred solution, before sending the log information to the storage module, the method further includes:
[0018] The timestamp format in each log entry in the log information is converted into the same timestamp format; the levels of the log information of different target business systems are converted into the same level, and the log information after the conversion process is obtained.
[0019] As a preferred solution, the log data processing method further includes:
[0020] According to the demand information of the target business system, the log information in the storage module is displayed in categories through a preset display interface.
[0021] Correspondingly, an embodiment of the present invention further provides a log data processing device, comprising a collection module, a judgment module and a sending module; wherein,
[0022] The collection module is used to collect log information of the target business system; obtain the flow quota of the target business system;
[0023] The judgment module is used to obtain the number of logs within a first preset time according to the log information; and judge the relationship between the number of logs and the flow quota of the target business system;
[0024] The sending module is used to send the log information to the storage module when the number of logs is less than the flow quota;
[0025] When the number of logs is greater than or equal to the traffic quota, the log information is analyzed for repetition rate, and when the repetition rate of the log information is less than a preset first repetition threshold, the log information is sent to a storage module.
[0026] As a preferred solution, the log data processing device also includes a first early warning module, which is used to discard duplicate logs and send abnormal information of the target business system to the user after the repetition rate of the log information is parsed: when the repetition rate of the log information is greater than or equal to the first repetition threshold.
[0027] As a preferred solution, the log data processing device also includes a second early warning module, which is used to remove the information that can change with the system status in the log information and extract the business log when the number of logs is greater than or equal to the traffic quota and lasts for a second preset time; perform a repetition rate check on the business log, and when the repetition rate is greater than or equal to a preset second repetition threshold, send the abnormal information of the target business system to the user.
[0028] As a preferred solution, the log data processing device also includes a quota configuration module, which is used to: access the target business system and allocate an initial traffic value to the target business system before obtaining the traffic quota of the target business system; after running for a third preset time, adjust the initial traffic value according to the average traffic of the third preset time to obtain the traffic quota.
[0029] As a preferred solution, the log data processing device also includes a cleaning module, which is used to, before sending the log information to the storage module: discard the logs that cannot be analyzed in the log information according to the log format of each target business system; correct the format of erroneous logs in the log information; and intercept the logs that exceed a preset length in the log information to obtain the cleaned log information.
[0030] As a preferred solution, the log data processing device also includes a conversion module, which is used to convert the timestamp format in each log entry in the log information into the same timestamp format before sending the log information to the storage module; convert the level of log information of different target business systems to the same level, and obtain the log information after conversion.
[0031] As a preferred solution, the log data processing device further includes a display module, which is used to display the log information in the storage module by category through a preset display interface according to the demand information of the target business system.
[0032] Correspondingly, an embodiment of the present invention further provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the log data processing method when executing the computer program.
[0033] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the log data processing method.
[0034] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0035] The present invention provides a log data processing method, device, terminal equipment and readable storage medium, wherein the log data processing method comprises: collecting log information of a target business system; obtaining the flow quota of the target business system; obtaining the number of logs within a first preset time according to the log information; judging the size relationship between the number of logs and the flow quota of the target business system; sending the log information to a storage module when the number of logs is less than the flow quota; performing repetition rate analysis on the log information when the number of logs is greater than or equal to the flow quota, and sending the log information to the storage module when the repetition rate of the log information is less than a preset first repetition threshold. The present invention sends the log information to the storage module when the number of logs is less than the flow quota, which can reduce the number of log files sent to the storage module and avoid sending a large number of logs to the storage module, causing a chain crash; in addition, when the number of logs is greater than or equal to the flow quota, performing repetition rate analysis on the log information, and sending the log information to the storage module when the repetition rate of the log information is less than a preset first repetition threshold, thus avoiding sending duplicate and meaningless log files to the storage module and reducing the risk of crash. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 : A flow chart of an embodiment of the log data processing method provided by the present invention.
[0037] Figure 2 : A structural diagram of an embodiment of the log data processing device provided by the present invention. DETAILED DESCRIPTION
[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0039] Embodiment one:
[0040] Please refer to Figure 1 , Figure 1 A log data processing method provided by an embodiment of the present invention includes steps S1 to S3; wherein:
[0041] Step S1, collecting log information of the target business system; obtaining the flow quota of the target business system.
[0042] In this step, the target business system can be a front-line power production system. According to the needs of the power plant, the specific business log information of the front-line power production system is customized and analyzed, and key logs are extracted from the huge log data and integrated into separate categories to help the power plant monitor the operation of the production system, such as including but not limited to the quick screening of work tickets and operation tickets.
[0043] The log data processing method of the present application can be applied to a log collection system, which can be applied to a computer device, including but not limited to a smart phone, a laptop, a tablet computer, a desktop computer, and a physical server and a cloud server connected to a display unit. The log collection system can be built using an e lk component.
[0044] The log collection system can access the target business system to obtain the traffic quota. Before obtaining the traffic quota of the target business system, an initial traffic value is allocated to the target business system; after running for a third preset time (for example, one week), the initial traffic value is adjusted according to the average traffic of the third preset time, and the traffic quota is configured. Among them, the initial value can be set relatively large to ensure that the system can receive and forward the logs in the first week after access.
[0045] In addition, since the log collection system may be connected to multiple business systems, when configuring the traffic quota, the situation of each business system can be considered and the corresponding traffic quota can be allocated to the target business system to further improve the rationality of the traffic quota setting.
[0046] Furthermore, the log information stored in the database table can be collected by installing the logstash-input-jdbc client on the database / application server. Logstash can use this plug-in to implement the function of collecting logs at a fixed time.
[0047] For text logs, you can deploy the FileBeat component in the target business system to monitor the log files of the target business system, with a configuration limit of 1 CPU. FileBeat reads log files and pushes messages to the Kafka component. The logs collected by FileBeat are sent to the message queue, and then sent to the corresponding Logstash through the message queue, where filtering operations are performed. This implementation method can decouple log collection from the business system, thereby reducing the impact of the business system on the log collection system.
[0048] Step S2: obtaining the number of logs within a first preset time according to the log information; and determining the relationship between the number of logs and the flow quota of the target business system.
[0049] In this step, the first preset time may be one minute. The size relationship is obtained by obtaining the number of logs within one minute and comparing it with the traffic quota.
[0050] In this step, the traffic quota can be used as a limit to ensure the amount of system log forwarding, to prevent the log collection system from crashing due to errors and generating a large number of logs, exhausting the resources of the log collection system, and causing other connected business system logs to be unable to be collected normally.
[0051] In addition, as an example, the traffic quota can also be used as an indicator to determine whether the log collection system is operating normally. Generally speaking, before the target business system undergoes a relatively large business expansion, the number of logs per minute will tend to a stable value (obtained by calculating the average log volume and peak value per minute within a week after the business system is connected to this system), and the traffic quota can be 20% more than this value. If the number of logs per minute exceeds the traffic quota, the repetition rate needs to be analyzed and the user needs to be reminded when necessary to avoid the risk of system crashes.
[0052] If the traffic quota is appropriately lowered and the log forwarding throughput of the business system is reduced, the load on the system can be reduced, while reducing the hardware cost required for the system.
[0053] Furthermore, before sending the log information to the storage module in step S3, the method further includes:
[0054] According to the log format of each target business system, the logs that cannot be analyzed in the log information are discarded; the erroneous logs in the log information are formatted and discarded if the correction fails; the logs in the log information that exceed the preset length are intercepted and the maximum amount allowed is retained to obtain the cleaned log information.
[0055] Furthermore, before sending the log information to the storage module in step S3, the method further includes:
[0056] The timestamp format in each log entry in the log information is converted to the same timestamp format; the levels of log information of different target business systems are converted to the same level to obtain the converted log information. In addition, the user ID of the target business system can be extracted and converted into the user ID of the log collection system.
[0057] Step S3, when the number of logs is less than the traffic quota, the log information is sent to the storage module; when the number of logs is greater than or equal to the traffic quota, the log information is analyzed for repetition rate, and when the repetition rate of the log information is less than a preset first repetition threshold, the log information is sent to the storage module.
[0058] Furthermore, when the repetition rate of the log information is greater than or equal to the first repetition threshold, the repeated logs are discarded, and the abnormal information of the target business system is sent to the user.
[0059] When the number of logs is greater than or equal to the traffic quota and lasts for a second preset time, remove the information in the log information that can change with the system status (such as log time, process number, etc.) and extract the business log; perform a repetition rate check on the business log, and when the repetition rate is greater than or equal to a preset second repetition threshold, that is, the repetition rate of the queued logs reaches the preset threshold, it is determined that the system is abnormal, and the abnormal information of the target business system is sent to the user.
[0060] Furthermore, the filebeat component is used to parse the log files on the server according to the preset configuration file, and the log text is read and parsed. Before step S3, some identification information of the business system is added to each log message. After receiving the log, the log is classified and stored according to the identification information for use by the data background.
[0061] The parsing content specifically includes: using regular expressions to parse different types of error logs into the same one (for example, in a system developed using Java, the error stack will be divided into many lines, which need to be identified and merged into one error message), and then extracting the content that needs attention in the log according to the needs of different business systems.
[0062] The above cleaning, conversion and parsing (or analysis) can be achieved using logstash. Logstash uses a multi-pipe line design, and accordingly, each target business system log is received using a pipe line.
[0063] As a further preferred solution, the log data processing method further includes:
[0064] A set of preset display interfaces is provided according to the business needs of the target business system, and the log information (such as user operation logs and system logs) in the storage module is displayed in categories through the preset display interface according to the demand information of the target business system. The interface can uniformly display time, log level and user ID to meet user needs.
[0065] Accordingly, refer to Figure 2 The embodiment of the present invention further provides a log data processing device, including a collection module 101, a judgment module 102 and a sending module 103; wherein,
[0066] The collection module 101 is used to collect log information of the target business system; obtain the flow quota of the target business system;
[0067] The determination module 102 is used to obtain the number of logs within a first preset time according to the log information; and determine the relationship between the number of logs and the flow quota of the target business system;
[0068] The sending module 103 is used to send the log information to the storage module when the number of logs is less than the traffic quota;
[0069] When the number of logs is greater than or equal to the traffic quota, the log information is analyzed for repetition rate, and when the repetition rate of the log information is less than a preset first repetition threshold, the log information is sent to a storage module.
[0070] As a preferred solution, the log data processing device also includes a first early warning module, which is used to discard duplicate logs and send abnormal information of the target business system to the user after the repetition rate of the log information is parsed: when the repetition rate of the log information is greater than or equal to the first repetition threshold.
[0071] As a preferred solution, the log data processing device also includes a second early warning module, which is used to remove the information that can change with the system status in the log information and extract the business log when the number of logs is greater than or equal to the traffic quota and lasts for a second preset time; perform a repetition rate check on the business log, and when the repetition rate is greater than or equal to a preset second repetition threshold, send the abnormal information of the target business system to the user.
[0072] As a preferred solution, the log data processing device also includes a quota configuration module, which is used to: access the target business system and allocate an initial traffic value to the target business system before obtaining the traffic quota of the target business system; after running for a third preset time, adjust the initial traffic value according to the average traffic of the third preset time to obtain the traffic quota.
[0073] As a preferred solution, the log data processing device also includes a cleaning module, which is used to, before sending the log information to the storage module: discard the logs that cannot be analyzed in the log information according to the log format of each target business system; correct the format of erroneous logs in the log information; and intercept the logs that exceed a preset length in the log information to obtain the cleaned log information.
[0074] As a preferred solution, the log data processing device also includes a conversion module, which is used to convert the timestamp format in each log entry in the log information into the same timestamp format before sending the log information to the storage module; convert the level of log information of different target business systems to the same level, and obtain the log information after conversion.
[0075] As a preferred solution, the log data processing device further includes a display module, which is used to display the log information in the storage module by category through a preset display interface according to the demand information of the target business system.
[0076] Correspondingly, an embodiment of the present invention further provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the log data processing method when executing the computer program.
[0077] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal, and various interfaces and lines are used to connect various parts of the entire terminal.
[0078] The memory can be used to store the computer program, and the processor realizes various functions of the terminal by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0079] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the log data processing method.
[0080] Wherein, if the module integrated in the log data processing device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.
[0081] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0082] The present invention provides a log data processing method, device, terminal equipment and readable storage medium, wherein the log data processing method comprises: collecting log information of a target business system; obtaining the flow quota of the target business system; obtaining the number of logs within a first preset time according to the log information; judging the size relationship between the number of logs and the flow quota of the target business system; sending the log information to a storage module when the number of logs is less than the flow quota; performing repetition rate analysis on the log information when the number of logs is greater than or equal to the flow quota, and sending the log information to the storage module when the repetition rate of the log information is less than a preset first repetition threshold. The present invention sends the log information to the storage module when the number of logs is less than the flow quota, which can reduce the number of log files sent to the storage module and avoid sending a large number of logs to the storage module, causing a chain crash; in addition, when the number of logs is greater than or equal to the flow quota, performing repetition rate analysis on the log information, and sending the log information to the storage module when the repetition rate of the log information is less than a preset first repetition threshold, thus avoiding sending duplicate and meaningless log files to the storage module and reducing the risk of crash.
[0083] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.
Claims
1. A log data processing method, characterized in that: include: Collect log information of the target business system; Obtaining the traffic quota of the target business system; According to the log information, the number of logs within a first preset time is obtained; and the relationship between the number of logs and the flow quota of the target business system is determined; When the number of logs is less than the flow quota, sending the log information to a storage module; When the number of logs is greater than or equal to the traffic quota, the log information is analyzed for repetition rate, and when the repetition rate of the log information is less than a preset first repetition threshold, the log information is sent to a storage module.
2. A log data processing method according to claim 1, characterized in that: After the log information is analyzed for repetition rate, the method further includes: When the repetition rate of the log information is greater than or equal to the first repetition threshold, the repeated logs are discarded, and the abnormal information of the target business system is sent to the user.
3. A log data processing method as claimed in claim 1, characterized in that: The log data processing method further includes: When the number of logs is greater than or equal to the traffic quota and lasts for a second preset time, remove the information in the log information that can change with the system status and extract the business log; perform a repetition rate check on the business log, and when the repetition rate is greater than or equal to a preset second repetition threshold, send the abnormal information of the target business system to the user.
4. A log data processing method as claimed in claim 1, characterized in that: Before obtaining the traffic quota of the target business system, the method further includes: Access the target business system and allocate an initial flow value to the target business system; after running for a third preset time, adjust the initial flow value according to the average flow during the third preset time to obtain the flow quota.
5. A log data processing method as claimed in claim 1, characterized in that: Before sending the log information to the storage module, the method further includes: According to the log format of each target business system, the logs that cannot be analyzed in the log information are discarded; the format of the erroneous logs in the log information is corrected; the logs in the log information that exceed the preset length are intercepted to obtain the cleaned log information.
6. A log data processing method as claimed in claim 1, characterized in that: Before sending the log information to the storage module, the method further includes: The timestamp format in each log entry in the log information is converted into the same timestamp format; the levels of the log information of different target business systems are converted into the same level, and the log information after the conversion process is obtained.
7. A log data processing method according to claim 1, characterized in that: The log data processing method further includes: According to the demand information of the target business system, the log information in the storage module is displayed in categories through a preset display interface.
8. A log data processing device, characterized in that: It includes a collection module, a judgment module and a sending module; wherein, The collection module is used to collect log information of the target business system; obtain the flow quota of the target business system; The judgment module is used to obtain the number of logs within a first preset time according to the log information; and judge the relationship between the number of logs and the flow quota of the target business system; The sending module is used to send the log information to the storage module when the number of logs is less than the flow quota; When the number of logs is greater than or equal to the traffic quota, the log information is analyzed for repetition rate, and when the repetition rate of the log information is less than a preset first repetition threshold, the log information is sent to a storage module.
9. A terminal device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the log data processing method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the log data processing method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Method for recording log data
CN102594581A
Method and device for controlling system logs
CN103838659A
Log duplication eliminating processing method and device
CN106844143A
Method and device for determining log flow limiting value, storage medium and electronic device
CN117591346A
