A log processing method, apparatus, device, and computer storage medium

Abstract

Embodiments of the present application disclose a log processing method, device and equipment and a computer storage medium. The method comprises: receiving log data to be processed; classifying the log data to be processed according to log types to obtain at least one group of log data, and storing the at least one group of log data in at least one storage directory file of a preset cache warehouse; wherein the preset cache warehouse comprises at least one storage directory file, and different log types correspond to different storage directory files; reading log data from the at least one storage directory file of the preset cache warehouse, performing analysis on the log data, and writing analysis results into a target database. In this way, by classifying the log data to be processed according to log types and storing the log data in a plurality of storage directory files of a preset cache warehouse, unified management and standardized storage of log data can be realized, and the speed of log processing can be improved.

Description

Technical Field

[0001] This application relates to the field of computer data processing technology, and in particular to a log processing method, apparatus, device, and computer storage medium. Background Technology

[0002] In today's information age, the rapid development of computer technology has placed higher demands on information security. Firewall devices from various manufacturers are widely deployed across different sectors, inevitably generating massive amounts of firewall logs. These logs are crucial for network security auditing, and how to receive and store these massive amounts of firewall logs is a primary issue that needs to be considered and resolved.

[0003] Currently, log formats vary widely across different manufacturers and firewall devices. SYSLOG, as the primary log format, is widely used in various network and security devices and is supported by most operating systems, making it an important logging standard. However, even within SYSLOG format, log types and specific content formats differ. When receiving and storing log data, issues such as excessively large log volumes and inconsistent log types arise, making unified management and standardized storage of logs difficult, thus slowing down log processing. Summary of the Invention

[0004] This application provides a log processing method, apparatus, device, and computer storage medium, which can realize unified management and standardized storage of log data, and also improve the speed of log processing.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a log processing method, the method comprising:

[0007] Receive log data to be processed;

[0008] Log data to be processed is classified according to log type to obtain at least one set of log data, and the at least one set of log data is stored in at least one storage directory file in a preset cache repository; wherein, the preset cache repository includes at least one storage directory file, and different log types correspond to different storage directory files;

[0009] Log data is read from at least one storage directory file in the preset cache repository, parsed, and the parsing results are written to the target database.

[0010] Secondly, embodiments of this application provide a log processing apparatus, which includes a receiving unit, a storage unit, and a parsing unit, wherein...

[0011] The receiving unit is configured to receive log data to be processed.

[0012] The storage unit is configured to classify the log data to be processed according to the log type to obtain at least one set of log data, and store the at least one set of log data in at least one storage directory file of the preset cache repository; wherein, the preset cache repository includes at least one storage directory file, and different log types correspond to different storage directory files;

[0013] The parsing unit is configured to read log data from at least one storage directory file in a preset cache repository, parse it, and write the parsing results to the target database.

[0014] Thirdly, embodiments of this application provide a log processing device, which includes a memory and a processor, wherein...

[0015] Memory is used to store computer programs that can run on a processor;

[0016] A processor is used to execute log processing methods, such as those described in the first aspect, while running computer programs.

[0017] Fourthly, embodiments of this application provide a computer storage medium storing a computer program that, when executed by at least one processor, implements the log processing method as described in the first aspect.

[0018] The log processing method, apparatus, device, and computer storage medium provided in this application embodiment receive log data to be processed; classify the log data to be processed according to log type to obtain at least one set of log data; and store the at least one set of log data in at least one storage directory file in a preset cache repository; wherein the preset cache repository includes at least one storage directory file, and different log types correspond to different storage directory files; read log data from at least one storage directory file in the preset cache repository, parse it, and write the parsing results to a target database. In this way, after classifying the log data to be processed according to log type and storing it in several storage directory files of the preset cache repository, not only is unified management of the log data to be processed according to its type achieved, but also standardized storage of log data in the storage directory files is achieved; furthermore, it allows parallel parsing of log data according to the storage directory files during log data parsing, thereby improving the log processing speed. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a log processing method provided in an embodiment of this application;

[0020] Figure 2 A flowchart illustrating another log processing method provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the application framework for a log processing method provided in an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of the composition structure of a log processing device provided in an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of the composition structure of another log processing device provided in an embodiment of this application;

[0024] Figure 6 A schematic diagram of the specific hardware structure of a log processing device provided in this application embodiment;

[0025] Figure 7 This is a schematic diagram of the composition structure of another log processing device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts related to the relevant application are shown in the accompanying drawings.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0028] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0029] It should also be noted that the terms "first, second, and third" used in the embodiments of this application are only used to distinguish similar objects and do not represent a specific order of objects. It is understood that "first, second, and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0030] In related technologies, how to receive and store massive amounts of firewall logs is the first problem that needs to be considered and solved; and the log formats of different firewall devices from different manufacturers vary greatly, so how to uniformly parse and store various firewall log data is also an important problem to be solved. SYSLOG, as the main log type, is widely used in various network and security devices and is supported by most operating systems. It has now become an important log standard. As long as the firewall logs meet the SYSLOG format standard, they can be uniformly received and processed. Taking SYSLOG logs as an example, the log format is as follows (for details, please refer to RFC3164, which defines the SYSLOG format standard and various rules for transmission in the network):

[0031] <pri>HEADER MSG < / pri>

[0032] here, <pri>Priority is a combination of facility and severity. Specifically, PRI = Facility × 8 + Severity value. Severity is further divided into eight different log levels: 0, Emergency; 1, Alert; 2, Critical; 3, Error; 4, Warning; 5, Notice; 6, Informational; 7, Debug.

[0033] Additionally, the HEADER can include a timestamp (TIMESTAMP) and a hostname (HOSTNAME). If the hostname is empty or unrecognized, the Internet Protocol (IP) address will be displayed. The MSG can contain additional information about the process that generated the message, as well as the text portion of the message. The MSG portion can be further divided into two parts: TAG and Content. The Content portion is the log content.

[0034] Currently, the industry's optimization of receiving and storing massive firewall logs mainly focuses on software, with little attention paid to overall software and hardware optimization. Moreover, the related technologies still have many shortcomings in receiving and storing log data. For example: (1) In related technologies, a single thread (or process) is generally used to listen to port 514 of the User Datagram Protocol (UDP) to receive logs. That is, when receiving logs, data transmission is completed through a single network card. When the peak value of log reception is too large, the single network card can easily become a bottleneck for log reception. (2) In related technologies, when receiving SYSLOG logs, the log data generally needs to be normalized before storage. This is not conducive to the dynamic addition of firewall log content format and cannot achieve dynamic addition and plug-and-play functionality. (3) In related technologies, when receiving logs, the log receiving module generally writes the log data to a local file first and then parses it into the database. However, after writing the logs to the local file, there is a lack of unified management of the local log files, such as how to classify log files, control the size of log files, control the number of log files, and handle log accumulation.

[0035] Based on this, this application provides a log processing method. The basic idea of ​​this method is: receiving log data to be processed; classifying the log data to be processed according to log type to obtain at least one set of log data, and storing the at least one set of log data in at least one storage directory file of a preset cache repository; wherein, the preset cache repository includes at least one storage directory file, and different log types correspond to different storage directory files; reading log data from at least one storage directory file of the preset cache repository, parsing it, and writing the parsing result into a target database. In this way, by classifying the log data to be processed according to log type and storing it in several storage directory files of the preset cache repository, not only is unified management of the log data to be processed according to its type achieved, but also standardized storage of log data in storage directory files is achieved; at the same time, it also enables parallel parsing of log data according to storage directory files during log data parsing, thereby improving the speed of log processing.

[0036] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0037] In one embodiment of this application, see [link to embodiment]. Figure 1 This illustrates a flowchart of a log processing method provided in an embodiment of this application. Figure 1 As shown, the method may include:

[0038] S101, Receive log data to be processed.

[0039] It should be noted that the log processing method provided in this application embodiment can be applied to a log processing device or a log processing equipment integrated with such a device. Here, the log processing device can be various network devices, security devices, firewall devices, etc., such as computers, smartphones, tablets, laptops, handheld computers, personal digital assistants (PDAs), navigation devices, servers, etc. This application embodiment does not specifically limit the specific devices in this regard.

[0040] It should also be noted that while SYSLOG logs, as the primary log type, can be uniformly received and processed as long as the firewall log format conforms to the SYSLOG standard, even if the log format meets the SYSLOG standard, the specific types and content formats of various log data still differ, leading to some issues in actual reception, storage, and parsing. Therefore, this application provides a log processing method, still using SYSLOG logs as an example, to improve the process of receiving, storing, and parsing log data.

[0041] During the reception of log data to be processed, when the peak log reception value is too large, in order to solve the problem that a single network interface card (NIC) can easily become a bottleneck for log reception, this application embodiment can use multiple physical NICs listening on different UDP ports to achieve multi-process concurrent reception of log data to be processed, thereby improving the efficiency of log reception. Therefore, in some embodiments, receiving log data to be processed may include:

[0042] Listen on at least two UDP ports using at least two physical network interface cards;

[0043] Log data obtained from at least two UDP ports is identified as log data to be processed; each physical network card listens on one UDP port, and the process of obtaining log data from at least two UDP ports is executed concurrently.

[0044] It should be noted that, in this embodiment of the application, at least two physical network interface cards (NICs) listen to at least two UDP ports, and the log data obtained from these at least two UDP ports is determined as the log data to be processed. The log data obtained from the UDP ports may be SYSLOG logs sent by the firewall. Furthermore, each physical NIC can listen to a corresponding UDP port, and the process of obtaining log data by listening to multiple UDP ports through multiple physical NICs can be executed concurrently.

[0045] In this way, by using multiple physical network cards to listen to multiple UDP ports, multi-process concurrent reception of log data to be processed is achieved. Even if the peak value of the received log data is large, the log data can be received smoothly, avoiding the formation of a log reception bottleneck.

[0046] S102. Classify the log data to be processed according to the log type to obtain at least one set of log data, and store the at least one set of log data in at least one storage directory file of the preset cache repository.

[0047] In this embodiment of the application, the preset cache repository may include at least one storage directory file, and different log types correspond to different storage directory files.

[0048] It should be noted that after receiving the log data to be processed, the log data can be categorized according to its log type, thus obtaining at least one set of log data. These sets of log data are then stored in at least one storage directory file within a pre-defined cache repository. Here, each set of log data belongs to a specific log type; that is, sets of log data of different log types are stored in the same storage directory file.

[0049] In other words, in this embodiment, the storage directory file corresponds to the log type of the log data. In the preset cache repository, one storage directory file stores log data of one log type. For example, assuming the log type of the log data to be processed is a Session log, then the log data is stored in the storage directory file corresponding to the Session log in the preset cache repository, thereby achieving standardized storage of log data.

[0050] It should also be noted that in actual business scenarios, the log type of the log data to be processed may not correspond to any of the existing storage directory files. That is, a new log type may appear for the existing storage directory files. In this case, a new storage directory file can be created to store the log data of the new log type. Therefore, in some embodiments, the method may further include:

[0051] When the log type of the log data to be processed is a new log type, a storage directory file corresponding to the new log type is created in the preset cache repository.

[0052] In this way, the log data to be processed is stored in the corresponding storage directory file of the preset cache repository according to its log type. In addition, a corresponding storage directory file is created for new log types, thereby realizing unified management of log data and dynamically updating the storage directory file of the preset cache repository.

[0053] It is understandable that, since the embodiments of this application can concurrently execute the process of receiving log data to be processed by listening to multiple UDP ports through multiple physical network cards, a large amount of log data may be received at the same time. Storing and processing such a large amount of log data in a preset cache warehouse may overload the Central Processing Unit (CPU). Therefore, to avoid overloading the CPU by processing a large number of tasks simultaneously, or to avoid the CPU's current resources being unable to support processing so much data at the same time, the log data to be processed can be stored in at least one preset queue first. Then, when the CPU is relatively idle, the log data to be processed can be transferred to the preset cache warehouse. Next, the log data to be processed can be classified according to log type, and the resulting at least one set of log data can be stored in at least one storage directory file in the preset cache warehouse.

[0054] Therefore, in some embodiments, after receiving the log data to be processed, the method may further include:

[0055] Store the log data to be processed into at least one preset queue;

[0056] Log data is read from at least one preset queue and sent to a preset cache repository by at least one sending thread; wherein each preset queue corresponds to one sending thread, and at least one sending thread executes concurrently;

[0057] Accordingly, classifying the log data to be processed according to log type to obtain at least one set of log data, and storing the at least one set of log data in at least one storage directory file in a preset cache repository, may include:

[0058] After receiving the log data to be processed in the preset cache warehouse, the log data to be processed is classified according to the log type to obtain at least one set of log data.

[0059] At least one set of log data is stored in at least one storage directory file.

[0060] It should be noted that after storing the log data to be processed into at least one preset queue, when reading log data from the preset queue and sending it to the preset cache warehouse, the sending thread can read the log data from the preset queue and send it to the preset cache warehouse. Then, after receiving the log data to be processed, the preset cache warehouse divides it into at least one group of log data according to the log type and stores the at least one group of log data in at least one storage directory file.

[0061] Here, both the number of preset queues and the number of sending threads are at least one, and the number of at least one sending thread and the number of at least one preset queue can be the same. When reading log data from preset queues and sending it to preset cache warehouses, at least one sending thread can execute concurrently. For example, if there are two preset queues, the number of sending threads is also two; if there are three preset queues, the number of sending threads is also three. In this way, by concurrently executing multiple sending threads to retrieve log data to be processed from multiple preset queues and send it to preset cache warehouses, the efficiency of log transmission can be improved.

[0062] It should also be noted that the number of preset queues can be dynamically determined based on CPU load status and / or set according to a custom value. Specifically, the number of preset queues that the current CPU resources can support can be determined based on the current CPU load status. Here, the number of preset queues that the current CPU resources can support is referred to as the first quantity value. The custom value is a preset value pre-written into the code, which can be a desired reference value that log administrators can customize based on actual needs, code status, and other information.

[0063] In one example, the number of preset queues is dynamically determined based on the CPU load status. This can be based on CPU resource utilization or the number of CPU cores. Specifically, if CPU resource utilization is low or the number of CPU cores is high (i.e., current resources are sufficient), then a larger number of preset queues can be set; conversely, if CPU resource utilization is high or the number of CPU cores is low (i.e., current resources are insufficient), then a smaller number of preset queues can be set. The terms "more" and "fewer" are relative; the specific number of preset queues is determined based on the resources the CPU can provide.

[0064] In another example, the number of preset queues can be determined based on a custom value. This can be achieved by directly writing the custom value into the code and setting that custom value as the number of preset queues.

[0065] In addition, in this embodiment of the application, there is a situation where the current CPU resources are insufficient to support a preset queue of a custom value, or although the current CPU resources can support a large number of preset queues (greater than the custom value), it is not actually necessary to use that many preset queues. Therefore, in another example, the number of preset queues can be dynamically determined based on a combination of CPU load status and the custom value. The specific implementation method is as follows:

[0066] Compare the first quantity value with the user-defined value;

[0067] If the first quantity value is less than the custom value, then the first quantity value will be determined as the number of preset queues;

[0068] If the first quantity value is greater than or equal to the custom value, then the custom value will be set as the number of the preset queue.

[0069] For example, suppose the preset custom value in the code is 5. If CPU resources are sufficient to support 8-10 preset queues, but the preset number of preset queues is 5, then only 5 preset queues will be allocated to cache log data to be processed. If CPU resources are insufficient and can only support a maximum of 3 preset queues, instead of 5, then although the custom value is 5, due to insufficient CPU resources, 3 preset queues will be allocated based on the actual CPU resource situation. In other words, in this method, the custom value can be seen as a guideline for the number of preset queues. The number of preset queues is dynamically determined by combining the CPU load status and the custom value, so that data transmission efficiency is met while CPU resources are sufficient.

[0070] It should also be noted that, since a large amount of log data to be processed needs to be stored in the preset queues, a certain allocation method needs to be set for how to store the log data into multiple preset queues.

[0071] In some embodiments, storing the log data to be processed into at least one preset queue may include:

[0072] The log data to be processed is stored into at least one preset queue according to a preset allocation method;

[0073] The preset allocation method includes at least one of the following: random allocation, round-robin allocation, and consistent hashing algorithm allocation.

[0074] It should be noted that when storing a large amount of log data to be processed into multiple preset queues, there can be various allocation methods. Preferably, the embodiments of this application can use one or more of the following allocation methods to store the log data to be processed into several preset queues: random allocation, round-robin allocation, or consistent hashing algorithm allocation. Of course, other allocation methods can also be used, and the embodiments of this application do not specifically limit them.

[0075] Random allocation can involve randomly distributing log data to be processed into at least one preset queue, or combining this with a priority algorithm. Round-robin allocation can be performed according to the request time order; that is, log data to be processed can be distributed and stored in at least one preset queue in chronological order. If one preset queue is full, the log data to be processed can be distributed to another preset queue. Consistent hashing, by mapping the entire hash value space as a virtual ring organized clockwise, can solve the classic load balancing problem. In this embodiment, consistent hashing can evenly distribute log data to be processed into at least one preset queue.

[0076] S103. Read log data from at least one storage directory file in the preset cache repository, parse it, and write the parsing results to the target database.

[0077] It should be noted that, in this embodiment of the application, the log data to be processed also needs to be parsed and stored in a database. That is, the log data is read from the storage directory file, parsed, and after obtaining the parsing result, written to the target database. Here, the target database can be a database, a file system, or any storage location specified by the user or operating system. It can be customized according to user needs, and this embodiment of the application does not impose any specific limitations on it.

[0078] In some embodiments, the step of reading and parsing log data from at least one storage directory file in a preset cache repository may include: reading target log data to be parsed from at least one storage directory file and parsing the target log data through at least one parsing thread; wherein each parsing thread corresponds to one storage directory file, and at least one parsing thread executes concurrently;

[0079] When a field in the target log data is a public field, the public field is parsed using a regular expression to obtain the parsing result of the public field;

[0080] When a field in the target log data is a private field, the private field is parsed according to the preset configuration rules to obtain the parsing result of the private field;

[0081] The parsing results of the target log data are obtained based on the parsing results of the public fields and the parsing results of the private fields.

[0082] It should be noted that even log data that meets the SYSLOG format standard varies in its specific type and content format. For custom log content and other data in the log data, the relevant technologies cannot fully and accurately parse them.

[0083] Here, when parsing log data, a parsing thread can be allocated to each storage directory file. In this way, each parsing thread parses the log data in one storage directory file. The log data to be parsed is called the target log data. Multiple parsing threads can concurrently execute the process of parsing the log data in the storage directory files and writing the parsing results to the target database, effectively improving the efficiency of log data parsing.

[0084] Specifically, each parsing thread reads and parses the target log data line by line from the storage directory file. The fields parsed in the target log data can be divided into public fields and private fields. Fields that conform to the SYSLOG format standard are public fields, which can be matched using regular expressions to obtain the parsing results. Fields that do not conform to the SYSLOG format standard are private fields, which need to be parsed using preset configuration rules.

[0085] In other words, the log data can include public and private fields. Different parsing methods are used for each public and private field to obtain their respective parsing results. Then, the parsing results of the public and private fields are combined to obtain the parsing result of the target log data. In this embodiment, the parsing results of the public and private fields can be combined according to the order of the fields in the log data to obtain the parsing result of the log data. Alternatively, the parsing results of the public and private fields can be combined or processed according to actual needs to obtain the parsing result of the log data.

[0086] For example, taking the aforementioned SYSLOG format standard as an example, TIMESTAMP and HOSTNAME in HEADER are public fields of the SYSLOG format standard. The timestamp and hostname of the target log data can be directly matched using regular expressions.

[0087] For private fields in the log data that do not conform to the SYSLOG format standard, they need to be parsed according to preset configuration rules. For example, when parsing private fields using preset configuration rules, this can be done through custom configuration items in the log data. The code can identify the Identity Document (ID) of the custom configuration item. This ID can be an eight-digit hexadecimal number, followed by template information for parsing the private fields in the log content. In other words, the custom configuration item provides template information for parsing the log content, and the log content can be parsed based on this template information. For example, custom configuration items can include log type, log level, device manufacturer information, system module, and other configuration items.

[0088] In addition, when new log formats exist in the log data to be processed, in order to address the problem that related technologies are not conducive to the dynamic addition of firewall log formats, in some embodiments, the method may further include:

[0089] During the process of receiving log data to be processed, if the received log data is in a new log format, a new configuration item is set and added to the preset configuration rules; the new configuration item is used to achieve correct parsing of log data in the new log format.

[0090] It should be noted that if a new log format appears during the process of receiving log data to be processed, the existing preset configuration rules will no longer be able to parse the new log format. In this case, a new configuration item can be set and added to the preset configuration rules to ensure correct parsing of the new log format.

[0091] The new log format typically includes private fields. Since these private fields often do not conform to the SYSLOG standard, existing configuration items in the default configuration rules cannot parse them correctly. New configuration items need to be added to parse the new log format. For example, a new configuration item could be template information corresponding to the new log format. Adding this to the default configuration rules updates the rules, ensuring correct parsing of the new format not only in the current instance but also in subsequent log processing.

[0092] In other words, the embodiments of this application can achieve correct parsing of log data through configuration items. The preset configuration rules include a variety of configuration items. When a new log format appears, the preset configuration rules can be updated using the new configuration items. Moreover, since the log data is normalized by adding new configuration items, it is also beneficial to dynamically add firewall log formats.

[0093] For example, the new configuration item can be the LogID configuration item. The LogID configuration item can include the following parts: (1) Log type, such as Session log, Thread log, IM log, URL log, etc.; (2) Log level, 8 levels; (3) Device manufacturer information; (4) System module.

[0094] In this way, when a new log format appears, the log data can be correctly parsed by adding new configuration items to the preset configuration rules, realizing dynamic addition and plug-and-play functionality.

[0095] Furthermore, in this embodiment, the health status of the preset cache repository can also be monitored to ensure that the preset cache repository does not encounter problems during the processing of log data.

[0096] In some embodiments, the method may further include:

[0097] Determine if there is log backlog in the preset cache repository;

[0098] If there is log accumulation in the preset cache repository, the preset cache repository will be processed according to the preset cleanup strategy.

[0099] The preset cleanup strategy includes at least one of the following: deleting log data with the lowest log level, deleting the earliest received log data, discarding the latest received log data, and deleting log data of the preset log type.

[0100] It should be noted that storing log data in the preset cache repository and reading and parsing log data from the preset cache repository are dynamic processes. During this process, log data may accumulate in the preset cache repository due to reasons such as failure to read and parse log data in a timely manner. In this case, it is necessary to process the accumulated log data.

[0101] Specifically, a daemon process can be established for the preset cache repository. This daemon process monitors and manages the health status of the preset cache repository. When it is determined that there is log accumulation in the preset cache repository, the daemon process will process the log accumulation according to a preset cleanup strategy. Here, the preset cleanup strategy may include one or more of the following: deleting log data with the lowest log level, deleting the earliest received log data, discarding the most recently received log data, and deleting log data of a preset log type. Of course, the daemon process may also process log accumulation in the preset cache repository in other ways, and is not limited to those listed here. This application embodiment does not specifically limit this.

[0102] In some embodiments, determining whether there is log backlog in the preset cache repository may include:

[0103] Determine the first speed of storing log data to be processed in a preset cache repository and the second speed of reading log data from the preset cache repository, parsing it, and writing it to the target database;

[0104] If the second speed is less than the first speed, it is determined that there is a backlog of logs in the preset cache repository;

[0105] If the second speed is greater than or equal to the first speed, then it is determined that there is no log backlog in the preset cache repository.

[0106] It should be noted that the first speed represents the log data generation speed when storing log data in the preset cache repository, while the second speed represents the data insertion speed when reading log data from the preset cache repository and parsing it into the database.

[0107] If the second speed is less than the first speed, that is, for the preset cache repository, the speed of reading and parsing log data from it is less than the speed of storing log data in it, which is equivalent to the total amount of log data in the preset cache repository increasing, then it may cause log accumulation in the preset cache repository.

[0108] If the second speed is greater than or equal to the first speed, that is, for the preset cache repository, the speed of reading log data from it is greater than or equal to the speed of storing log data in it, which is equivalent to the total amount of log data in the preset cache repository decreasing or remaining unchanged, then the preset cache repository will not generate log accumulation.

[0109] It should also be noted that in this embodiment, there may be a situation where, although the second speed is less than the first speed, the space utilization rate of the preset cache repository is still very low. For example, if only 10% of the space in the preset cache repository is occupied, then although the total amount of log data in the preset cache repository is increasing, it may not necessarily lead to log accumulation because the space in the preset cache repository is still sufficient. Therefore, a space utilization rate threshold needs to be set in this case. In some embodiments, the method may further include: determining that there is log accumulation in the preset cache repository when the second speed is less than the first speed and the space utilization rate of the preset cache repository is greater than the space utilization rate threshold. For example, the space utilization rate threshold can be a value such as 70%, 80%, or 90%, and this embodiment does not specifically limit it.

[0110] Furthermore, in some embodiments, the method may further include:

[0111] When an anomaly occurs in the preset cache repository, corresponding anomaly alarm information is generated; among which, the anomaly alarm information includes: insufficient space alarm information, and / or, log backlog alarm information.

[0112] It should be noted that the aforementioned daemon process can also monitor the log backlog and space usage status of the preset cache repository to determine if there are any anomalies. If an anomaly is found, an anomaly alarm message will be generated. Here, the anomaly alarm message can include one or more of the following: insufficient space alarm message, log backlog alarm message, etc.

[0113] Here, anomalies in the preset cache repository may include: the space utilization rate of the preset cache repository has exceeded the space utilization threshold, or log accumulation has occurred in the preset cache repository.

[0114] For example, if the space utilization rate is greater than the space utilization rate threshold, a space shortage alarm message is generated and a space shortage alarm is triggered; if the log backlog may occur because the second speed is lower than the first speed, a log backlog alarm message is generated and a log backlog alarm is triggered; if both space shortage and log backlog occur, both space shortage alarm message and log backlog alarm message are generated and an alarm is triggered.

[0115] In this way, by monitoring the health status of the preset cache repository, log backlog can be handled in a timely manner, and timely alerts can be issued when there are anomalies in the preset cache repository.

[0116] This embodiment provides a log processing method. The method involves receiving log data to be processed; classifying the log data according to log type to obtain at least one set of log data; storing the at least one set of log data in at least one storage directory file in a preset cache repository; wherein the preset cache repository includes at least one storage directory file, and different log types correspond to different storage directory files; parsing the log data in the at least one storage directory file; and writing the parsing results to a target database. In this way, after classifying the log data to be processed according to log type and storing it in several storage directory files of the preset cache repository, not only is unified management of the log data to be processed according to its type achieved, but also standardized storage of log data in the storage directory files is realized. Simultaneously, during log data parsing, log data can be parsed in parallel according to the storage directory files, thereby improving the log processing speed. Furthermore, when receiving log data, multiple processes receive the log data to be processed by listening on multiple ports using multiple network cards, improving the log data reception efficiency. When transmitting log data from several preset queues to the preset cache repository, each preset queue corresponds to a sending thread, and several sending threads are used to transmit the log data. The system uses concurrent threads to read log data from a preset queue and send it to a preset cache repository, improving log data transmission efficiency. When parsing log data, each storage directory file corresponds to a parsing thread. Multiple parsing threads concurrently execute log data parsing, and preset configuration rules can accurately parse the log content. Even if a new log format appears, only a new configuration item needs to be added to the configuration file to accurately parse the log content, improving parsing efficiency and enabling dynamic addition of log formats. Furthermore, the system monitors the health status of the preset cache repository, effectively handling log backlogs and issuing alerts when anomalies occur, preventing log data loss or errors due to anomalies.

[0117] In another embodiment of this application, see Figure 2 This illustrates a flowchart of another log processing method provided in an embodiment of this application. Figure 2 As shown, the method may include:

[0118] S201. Run multiple log receiving processes concurrently and cache the received SYSLOG logs into at least one preset queue.

[0119] It should be noted that this application embodiment still uses SYSLOG logs as an example, meaning that the log data involved in this application embodiment can all refer to SYSLOG logs. Here, multiple log receiving processes run concurrently, listening on different UDP ports through multiple physical network cards to receive SYSLOG logs sent by the firewall, and caching the received SYSLOG logs into at least one preset queue (also called a cache queue). Preferably, the number of preset queues can be multiple.

[0120] Specifically, when multiple physical network interface cards (NICs) listen on different UDP ports, the data packets contain Internet Protocol (IP) information. This IP information allows the system to identify the source device and corresponding port of the received data. Thus, a single physical NIC can listen on a specific port to receive the corresponding device's SYSLOG logs. As long as the port is not occupied by other nodes, it can be listened to via the physical NIC, such as UDP ports 514, 515, and others, enabling concurrent reception of SYSLOG logs by multiple processes.

[0121] After receiving the SYSLOG logs, the SYSLOG logs need to be stored in at least one preset queue. The number of preset queues can be dynamically determined based on the number of CPU cores or set by a custom value.

[0122] Specifically, for each process listening on a UDP port to obtain SYSLOG logs, the number of its threads can be controlled, and the number of threads is consistent with the number of preset queues. In this embodiment, the custom value is a value written into the code, equivalent to an expected reference value. It can be a specific value set by the log administrator according to actual needs or device configuration. The custom value has a certain guiding role in the actual number of preset queues allocated during the SYSLOG log reception and storage process. During the reception and storage of SYSLOG logs, the current CPU load status can be obtained in real time and dynamically determined based on the custom value. Specifically, if the current CPU load is too high or the number of CPU cores is too small, i.e., the current CPU resources are insufficient to support a large number of preset queues (the number of preset queues that the CPU can currently support is less than the custom value), then the number of preset queues is determined based on the current CPU status; if the current CPU load is low or the number of CPU cores is large, i.e., the current CPU resources are sufficient and the number of preset queues that can be supported is greater than or equal to the custom value, then the number of preset queues is determined to be the custom value.

[0123] It should also be noted that when storing SYSLOG logs in preset queues, log data can be allocated to different preset queues using random allocation, round-robin allocation, or consistent hashing algorithm allocation methods. This application embodiment does not specifically limit this.

[0124] S202. Run at least one sending thread concurrently to transfer SYSLOG logs from at least one preset queue to a preset cache repository.

[0125] It should be noted that, in this embodiment, the preset queue is equivalent to a temporary storage area for the SYSLOG log. After storing the SYSLOG log in the preset queue, if the CPU has too many tasks to execute, the SYSLOG log will only be retrieved from the preset queue when the CPU is idle. In this way, the preset queue ensures that the SYSLOG log is not lost, and the SYSLOG log can be retrieved from the preset queue when the CPU is idle, thus ensuring the CPU's working efficiency.

[0126] When retrieving SYSLOG logs from a preset queue and sending them to a preset cache repository (also known as a log file cache repository), each preset queue corresponds to a sending thread (also known as a log sending thread). Each sending thread retrieves SYSLOG logs from its corresponding preset queue and sends them to the preset cache repository, and multiple sending threads can run concurrently.

[0127] S203. In the preset cache repository, SYSLOG logs are classified according to log type and stored in at least one storage directory file.

[0128] When the default cache repository receives SYSLOG logs, it categorizes them according to log type and stores them in different storage directories. Each log type has its own corresponding storage directory file (also called a storage directory). When a new log type is introduced, a new storage directory file is created to store the SYSLOG logs for that new log type.

[0129] Here, the log type can be the log type configured in the preset configuration rules. For example, the log type can be Session log, Thread log, IM log or URL log, etc. For each log type, there is a corresponding storage directory file.

[0130] S204. Run at least one log parsing and database insertion thread concurrently, and write the parsing results to the target database.

[0131] It should be noted that the log parsing and database insertion thread (i.e., the parsing thread in the aforementioned embodiment) can monitor whether there are log files in the preset cache repository. When there are log files in the preset cache repository, the log files in the preset cache repository will be parsed and written to the target database, which is "log parsing and database insertion". Here, "database" refers to the target database, which can be a file system or a database, etc. In practice, it is necessary to combine the user's needs and write the parsing results to the corresponding target database. This application embodiment does not make specific limitations in this regard.

[0132] When parsing logs and writing the parsing results to the target database, a parsing thread (also known as a log parsing and database insertion thread) is allocated for each log type, i.e. each storage directory file. At the same time, multiple parsing threads can concurrently parse and insert SYSLOG logs from multiple storage directory files into the database.

[0133] Specifically, each parsing thread reads and processes SYSLOG logs one by one from the corresponding storage directory file. For parsing public fields, regular expressions can be used for matching; for parsing private fields, the private fields are parsed according to configuration items in the preset configuration rules. In this embodiment, the configuration item can be the LogID configuration item, which can provide a log content template for parsing private fields. Here, public fields can be fields within the scope of the SYSLOG format standard, while private fields are fields outside the scope of the SYSLOG format standard that need to be parsed using the LogID configuration item.

[0134] For example, the following is an example of a SYSLOG log with a LogID configuration item:

[0135] <pri>TIMESTAMP HOSTNAME LOGID:CONTENT < / pri>

[0136] Wherein, PRI represents priority; TIMESTAMP represents timestamp; HOSTNAME represents hostname; LogID is a custom configuration item in the preset configuration rules of the specific scheme. In this embodiment, the LogID configuration item is used to parse the log content; and Content represents the log content.

[0137] In this embodiment of the application, different firewall logs are distinguished by configuration items such as the LogID configuration item in the preset configuration rules. Different firewall log content formats correspond to different LogID configuration items, and their IDs are 8-digit hexadecimal numbers.

[0138] For example, the LogID configuration item may include the following parts: (1) Log type, such as Session log, Thread log, IM log, URL log, etc.; (2) Log level, 8 levels; (3) Device manufacturer information; (4) System module.

[0139] The following is an example of the LogID configuration item (where module is not shown):

[0140]

[0141] In this way, the LogID configuration item can accurately parse the log content. If a new log format (also known as the log content format, i.e., the specific content format of the log) appears in the SYSLOG log, you only need to add a LogID configuration item to the preset configuration rules to update the log format, and the content of the SYSLOG log can be parsed correctly.

[0142] After parsing the log file, the parsed data can be written to the target database, completing the reception, storage, parsing, and database entry of the SYSLOG log.

[0143] S204. Establish a daemon process to monitor the health status of the default cache repository.

[0144] It should be noted that, in this embodiment of the application, a daemon process (also known as a cache repository daemon process) is also established for the preset cache repository to monitor the health status of the preset cache repository and to perform corresponding processing or alarms when the health status of the preset cache repository is abnormal.

[0145] Specifically, when the speed of parsing SYSLOG logs and writing the parsing results to the target database is slower than the speed at which the preset cache repository receives SYSLOG logs, log accumulation may occur in the preset cache repository. In this case, log accumulation processing is required. For example, log accumulation processing can include strategies such as deleting SYSLOG logs with the lowest log level, deleting the earliest received SYSLOG logs, discarding the most recently received SYSLOG logs, or deleting SYSLOG logs of a certain log type.

[0146] Additionally, when the default cache repository malfunctions, corresponding alerts should be generated. These could include: insufficient space alerts (such as disk space shortage alerts) or log backlog alerts. It's important to note that while insufficient space may be caused by log backlog, insufficient space alerts and log backlog alerts are not the same.

[0147] For example, if disk space usage reaches 80% or more, it is considered insufficient disk space, and a disk space shortage alert will be issued. However, if, although disk space is insufficient, no new SYSLOG logs are being sent to the preset cache repository, the log files originally piled up in the preset cache repository will gradually decrease as SYSLOG logs are read and parsed. Alternatively, if the speed of parsing and writing SYSLOG logs to the target database is greater than the speed at which the preset cache repository receives SYSLOG logs, then no log backlog will occur. However, if the speed of parsing and writing SYSLOG logs to the target database is less than the speed at which the preset cache repository receives SYSLOG logs, and this cleanup continues unabated, then log backlog has occurred, and a corresponding log backlog alert needs to be issued.

[0148] Based on the log processing methods described above, see [link to documentation]. Figure 3 This illustrates a schematic diagram of the application framework for a log processing method provided in an embodiment of this application. For example... Figure 3 As shown, the application framework may include the following parts: log receiving section 301; preset queue processing section 302; cache repository processing section 303; log parsing and storage section 304; and cache repository guardian section 305. The functions of each part are as follows:

[0149] The log receiving section 301 is used to implement the log receiving process, which may include multiple log receiving processes. Specifically, it may involve receiving log data sent by the firewall through multiple physical network cards listening on different UDP ports and storing the received log data into multiple different preset queues. The number of preset queues can be dynamically determined based on the number of CPU cores or set as a custom value. Allocation strategies such as random allocation, round-robin allocation, or consistent hashing can be used to distribute log data to different preset queues.

[0150] The preset queue processing section 302 is used to send log data from the preset queue to the preset cache warehouse. It can include multiple sending threads, that is, each preset queue corresponds to one sending thread. The sending thread retrieves log data from the preset queue and sends it to the preset cache warehouse. Multiple sending threads execute concurrently.

[0151] The cache warehouse processing section 303 is used to receive log data sent by the preset queue and store it in different storage directory files according to the log type. That is, the preset cache warehouse is responsible for receiving log data sent by the preset queue, classifying the log data according to the log type, and storing them in different storage directory files. Each log type corresponds to one storage directory file.

[0152] The log parsing and writing section (304) is used to parse log data and write it to the target database. It can include a log parsing and writing process, which may include at least one parsing thread. Specifically, the log parsing and writing process monitors a preset cache repository for log files. If a log file is detected in the preset cache repository, the log is parsed and written to the database. A parsing thread can be assigned to each log type, and multiple storage directory files can be parsed and written concurrently. Specifically, it can read log data line by line from the storage directory files in the preset cache repository and process it. For parsing public fields in the log data, regular expressions can be used for matching; for parsing private fields in the log data, parsing is required based on the log content template provided by LogID.

[0153] The cache repository guardian component 305 is used to monitor the health status of the preset cache repository. It can establish a cache repository guardian process to monitor the cache repository's health status and perform necessary log backlog handling and alarm processing. Specifically, when the speed of parsing and writing log data to the target database is slower than the speed of storing log data in the preset cache repository, resulting in log backlog, log backlog handling is required. Several different strategies are available, such as deleting log data with the lowest log level, deleting the earliest received log data, discarding the latest received log data, and deleting log data of a certain log type. When the cache repository experiences an anomaly, corresponding alarms need to be generated, such as insufficient disk space alarms and log backlog alarms.

[0154] In other words, this application provides a log processing method that uses multiple physical network cards to listen to different UDP ports and run multiple log receiving processes concurrently to solve the performance bottleneck of a single network card when the log receiving peak is too large, thereby accelerating the log receiving rate. At the same time, when there is a new log format, only a LogID configuration item needs to be added to the configuration file to set the new log content format, and the log data can be correctly parsed, achieving dynamic addition and plug-and-play functionality. The received log data is first stored in a preset cache repository, which has a dedicated daemon process responsible for the health management of the preset cache repository and provides functions such as log backlog processing, disk threshold analysis, and log alarms.

[0155] Furthermore, the application framework of the log processing method provided in this application includes a log receiving part, a log transmission part, a log parsing and storage part, and a cache repository guardian part. These parts cooperate with each other, using multiple physical network cards to listen on different UDP ports, running multiple log receiving processes concurrently, and storing the received log data into a preset cache repository through a preset queue. The log parsing and storage process listens to whether there are log files in the preset cache repository; if so, it parses and stores the logs. The cache repository guardian process is responsible for monitoring the health status of the preset cache repository and performing necessary log backlog processing, alarm processing, etc.

[0156] Compared with related technologies, the log processing method provided in this application has at least the following advantages:

[0157] (1) Multiple network cards concurrently receive log data.

[0158] In related technologies, a single thread (or process) typically listens on UDP port 514 to receive logs. Data transmission during log reception is usually completed through a single network interface card (NIC). When the peak log reception volume is too high, this single NIC can easily become a bottleneck. In this embodiment, multiple physical NICs can be added, each listening on a different UDP port, and multiple log receiving processes can be run concurrently to address the performance bottleneck of a single NIC when the peak log reception volume is too high, thereby accelerating the log reception rate.

[0159] (2) Use the LogID configuration item to dynamically add new firewall log content format.

[0160] In related technologies, when receiving SYSLOG logs, it is generally necessary to normalize the log data before storing it, which is not conducive to the dynamic addition of firewall log formats. In the embodiments of this application, when there is a new firewall log format, it is only necessary to add a LogID configuration item to the configuration file to set the new log content format, and the firewall log data can be correctly parsed, achieving dynamic addition and plug-and-play functionality.

[0161] (3) Unified management of cached files.

[0162] In related technologies, log receiving modules typically write log data to local files first before analysis. However, writing logs to local files lacks unified management of these files, such as log file classification, log file size control, log file quantity control, and log backlog handling. In this embodiment, the received log data is first stored in a preset cache repository. This preset cache repository has a dedicated daemon process responsible for its health management, providing functions such as log backlog handling, disk threshold analysis, and log alerts.

[0163] This embodiment provides a log processing method. It can be seen that the log processing method provided in this application embodiment is applicable to an application framework including a log receiving section, a preset queue processing section, a cache repository processing section, a log parsing and storage section, and a cache repository guardian section. Multiple physical network cards listen on different UDP ports, and multiple log receiving processes run concurrently. The received log data is first stored in a preset cache repository. The cache repository classifies the log data according to log type and stores it in different storage directory files, with each log type corresponding to a storage directory file. Multi-threaded parsing of log data according to log type is also possible. A LogID configuration item is used to dynamically add firewall log content formats. The cache repository guardian process monitors the health status of the preset cache repository and performs necessary log backlog processing and alarm handling. This enables unified management and standardized storage of log data and improves the efficiency of log receiving, storage, parsing, and storage.

[0164] In another embodiment of this application, see [link to application]. Figure 4 This illustrates a schematic diagram of the composition of a log processing device 40 provided in an embodiment of this application. Figure 4 As shown, the log processing device includes a receiving unit 401, a storage unit 402, and a parsing unit 403, wherein...

[0165] Receiving unit 401 is configured to receive log data to be processed;

[0166] Storage unit 402 is configured to classify log data to be processed according to log type to obtain at least one set of log data, and store the at least one set of log data in at least one storage directory file of a preset cache repository; wherein, the preset cache repository includes at least one storage directory file, and different log types correspond to different storage directory files;

[0167] The parsing unit 403 is configured to read log data from at least one storage directory file in a preset cache repository, parse it, and write the parsing result to the target database.

[0168] In some embodiments, the receiving unit 401 is specifically configured to listen to at least two User Data Packet Protocol (UDP) ports through at least two physical network cards; and to determine the log data obtained from the at least two UDP ports as log data to be processed; wherein each physical network card listens to one UDP port, and the process of obtaining log data from the at least two UDP ports is executed concurrently.

[0169] In some embodiments, the storage unit 402 is specifically configured to store log data to be processed in at least one preset queue; and to read log data from at least one preset queue and send it to a preset cache warehouse according to at least one sending thread; wherein each preset queue corresponds to one sending thread, and at least one sending thread is executed concurrently; and after the preset cache warehouse receives the log data to be processed, it classifies the log data to be processed according to the log type to obtain at least one set of log data; and stores the at least one set of log data in at least one storage directory file.

[0170] In some embodiments, the storage unit 402 is specifically configured to store the log data to be processed into at least one preset queue according to a preset allocation method; wherein the preset allocation method includes at least one of the following: random allocation method, round-robin allocation method, and consistent hashing algorithm allocation method.

[0171] In some embodiments, the parsing unit 403 is specifically configured to read target log data to be parsed from at least one storage directory file through at least one parsing thread; wherein each parsing thread corresponds to one storage directory file, and at least one parsing thread executes concurrently; and when a field in the target log data is a public field, the public field is parsed using a regular expression to obtain the parsing result of the public field; and when a field in the target log data is a private field, the private field is parsed using a preset configuration rule to obtain the parsing result of the private field; and the parsing result of the target log data is obtained based on the parsing results of the public field and the parsing results of the private field.

[0172] In some embodiments, see Figure 5 This illustrates a schematic diagram of the composition of another log processing apparatus provided in an embodiment of this application. For example... Figure 5 As shown, the log processing device 40 may further include a setting unit 404, configured to set a new configuration item and add the new configuration item to the preset configuration rules if the received log data is a new log format during the process of receiving log data to be processed; wherein, the new configuration item is used to achieve correct parsing of log data in the new log format.

[0173] In some embodiments, such as Figure 5 As shown, the log processing device 40 may further include a monitoring unit 405, configured to determine whether there is log accumulation in the preset cache repository; if there is log accumulation in the preset cache repository, then the preset cache repository is processed for log accumulation according to a preset cleanup strategy; wherein, the preset cleanup strategy includes at least one of the following: deleting log data with the lowest log level, deleting the earliest received log data, discarding the latest received log data, and deleting log data of a preset log type.

[0174] In some embodiments, the monitoring unit 405 is specifically configured to determine a first speed at which log data to be processed is stored in a preset cache warehouse and a second speed at which log data to be processed is read from the preset cache warehouse and parsed and written to a target database; and if the second speed is less than the first speed, it is determined that there is log accumulation in the preset cache warehouse; and if the second speed is greater than or equal to the first speed, it is determined that there is no log accumulation in the preset cache warehouse.

[0175] In some embodiments, the monitoring unit 405 is further configured to generate an anomaly alarm message when an anomaly occurs in the preset cache repository; wherein the anomaly alarm message includes: insufficient space alarm message, and / or, log accumulation alarm message.

[0176] Understandably, in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The integrated unit can be implemented in hardware or as a software functional module.

[0177] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] Therefore, this embodiment provides a computer storage medium storing a computer program that, when executed by at least one processor, implements the steps of any of the methods described in the foregoing embodiments.

[0179] Based on the above-described composition of a log processing device 40 and computer storage medium, see [link to documentation]. Figure 6 This illustrates a schematic diagram of the specific hardware structure of a log processing device 50 provided in an embodiment of this application. For example... Figure 6 As shown, the log processing device 50 may include a communication interface 501, a memory 502, and a processor 503; the various components are coupled together via a bus system 504. It is understood that the bus system 504 is used to implement communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 All buses are labeled as bus system 504. Among them, communication interface 501 is used for receiving and sending signals during information exchange with other external network elements;

[0180] Memory 502 is used to store computer programs that can run on processor 503;

[0181] Processor 503 is used to perform the following when running computer programs:

[0182] Receive log data to be processed;

[0183] Log data to be processed is classified according to log type to obtain at least one set of log data, and the at least one set of log data is stored in at least one storage directory file in a preset cache repository; wherein, the preset cache repository includes at least one storage directory file, and different log types correspond to different storage directory files;

[0184] Log data is read from at least one storage directory file in the preset cache repository, parsed, and the parsing results are written to the target database.

[0185] It is understood that the memory 502 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 502 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0186] The processor 503 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 503 or by instructions in software form. The processor 503 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 502, and the processor 503 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the above method.

[0187] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0188] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0189] Alternatively, as another embodiment, the processor 503 is further configured to perform the steps of the method described in any of the foregoing embodiments when running the computer program.

[0190] In another embodiment of this application, based on the schematic diagram of the above-described log processing apparatus 40, see [link to diagram]. Figure 7 This illustrates a schematic diagram of the composition of another log processing device 50 provided in an embodiment of this application. For example... Figure 7 As shown, the log processing device 50 includes at least the log processing apparatus 40 described in any of the foregoing embodiments.

[0191] For the log processing device 50, when receiving log data, it can achieve multi-process reception of log data by listening to multiple ports through multiple network cards, thereby improving the efficiency of log data reception. When transmitting log data from multiple preset queues to preset cache warehouses, each preset queue corresponds to a sending thread. Multiple sending threads concurrently read log data from preset queues and send it to preset cache warehouses, thereby improving the efficiency of log data transmission. When parsing log data, each storage directory file corresponds to a parsing thread. Multiple parsing threads concurrently execute the parsing of log data. At the same time, the log content can be accurately parsed through preset configuration rules. Even if a new log format appears, only a new configuration item needs to be added to the configuration file to accurately parse the log content, thereby improving the efficiency of log data parsing and realizing dynamic updates of log formats. In addition, the health status of the preset cache warehouse is monitored, log backlog is effectively handled in a timely manner, and corresponding alarms are issued when anomalies occur, thereby avoiding log data loss or errors due to anomalies.

[0192] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

[0193] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0194] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0195] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0196] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0197] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0198] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / pri>

Claims

1. A log processing method, characterized in that, The method includes: Receive log data to be processed; during the process of receiving the log data to be processed, if the received log data is in a new log format, set a new configuration item and add the new configuration item to the preset configuration rules; wherein, the new configuration item is used to achieve correct parsing of the log data in the new log format; The log data to be processed is classified according to log type to obtain at least one set of log data, and the at least one set of log data is stored in at least one storage directory file in a preset cache repository; wherein, the preset cache repository includes at least one storage directory file, and different log types correspond to different storage directory files; Log data is read from at least one storage directory file of the preset cache repository, parsed, and the parsing results are written to the target database; The step of reading and parsing log data from at least one storage directory file of the preset cache repository includes: The target log data to be parsed is read from the at least one storage directory file by at least one parsing thread; wherein each parsing thread corresponds to one storage directory file, and the at least one parsing thread executes concurrently; When a field in the target log data is a public field, the public field is parsed using a regular expression to obtain the parsing result of the public field; When a field in the target log data is a private field, the private field is parsed according to the preset configuration rules to obtain the parsing result of the private field; The parsing result of the target log data is obtained based on the parsing results of the public fields and the parsing results of the private fields.

2. The method according to claim 1, characterized in that, The receipt of log data to be processed includes: Listen on at least two User Datagram Protocol (UDP) ports using at least two physical network interface cards (NICs). The log data obtained from the at least two UDP ports is determined as the log data to be processed; wherein each physical network card listens to one UDP port, and the process of obtaining log data from the at least two UDP ports is executed concurrently.

3. The method according to claim 1, characterized in that, After receiving the log data to be processed, the method further includes: The log data to be processed is stored in at least one preset queue; Log data is read from at least one preset queue and sent to the preset cache repository by at least one sending thread; wherein each preset queue corresponds to one sending thread, and the at least one sending thread executes concurrently; Accordingly, classifying the log data to be processed according to log type to obtain at least one set of log data, and storing the at least one set of log data in at least one storage directory file of a preset cache repository, includes: After receiving the log data to be processed in the preset cache warehouse, the log data to be processed is classified according to the log type to obtain the at least one set of log data; The at least one set of log data is stored in the at least one storage directory file.

4. The method according to claim 3, characterized in that, The step of storing the log data to be processed into at least one preset queue includes: The log data to be processed is stored in at least one preset queue according to a preset allocation method; The preset allocation method includes at least one of the following: random allocation, round-robin allocation, and consistent hashing algorithm allocation.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Determine whether there is log backlog in the preset cache repository; If the preset cache repository has log accumulation, then the preset cache repository will be processed according to the preset cleanup strategy. The preset cleanup strategy includes at least one of the following: deleting log data with the lowest log level, deleting the earliest received log data, discarding the latest received log data, and deleting log data of a preset log type.

6. The method according to claim 5, characterized in that, The step of determining whether there is log backlog in the preset cache repository includes: Determine a first speed for storing the log data to be processed in the preset cache repository, and a second speed for reading log data from the preset cache repository and parsing and writing it to the target database; If the second speed is less than the first speed, then it is determined that there is a log backlog in the preset cache repository; If the second speed is greater than or equal to the first speed, then it is determined that there is no log backlog in the preset cache repository.

7. The method according to claim 5, characterized in that, The method further includes: When the preset cache repository is abnormal, corresponding abnormal alarm information is generated; wherein, the abnormal alarm information includes: insufficient space alarm information, and / or, log backlog alarm information.

8. A log processing device, characterized in that, The log processing device includes a receiving unit, a storage unit, and a parsing unit, wherein, The receiving unit is configured to receive log data to be processed; during the process of receiving the log data to be processed, if the received log data is in a new log format, a new configuration item is set and the new configuration item is added to the preset configuration rules; wherein, the new configuration item is used to achieve correct parsing of the log data in the new log format; The storage unit is configured to classify the log data to be processed according to the log type to obtain at least one set of log data, and store the at least one set of log data in at least one storage directory file of a preset cache repository; wherein, the preset cache repository includes at least one storage directory file, and different log types correspond to different storage directory files; The parsing unit is configured to read log data from at least one storage directory file of the preset cache repository, parse it, and write the parsing result to the target database; The step of reading and parsing log data from at least one storage directory file of the preset cache repository includes: The target log data to be parsed is read from the at least one storage directory file by at least one parsing thread; wherein each parsing thread corresponds to one storage directory file, and the at least one parsing thread executes concurrently; When a field in the target log data is a public field, the public field is parsed using a regular expression to obtain the parsing result of the public field; When a field in the target log data is a private field, the private field is parsed according to the preset configuration rules to obtain the parsing result of the private field; The parsing result of the target log data is obtained based on the parsing results of the public fields and the parsing results of the private fields.

9. A log processing device, characterized in that, The log processing device includes a memory and a processor, wherein... The memory is used to store computer programs that can run on the processor; The processor is configured to execute the log processing method as described in any one of claims 1 to 7 when running the computer program.

10. A computer storage medium, characterized in that, The computer storage medium stores a computer program that, when executed by at least one processor, implements the log processing method as described in any one of claims 1 to 7.

Images