Log data processing method, system and device and nonvolatile storage medium
By identifying and parsing the type of unstructured logs, generating structured data and storing it in the target table or distributed file system, the problem of low efficiency in unstructured log text processing is solved, achieving efficient log data integration, analysis, and storage, and simplifying Flume cluster management.
Patent Information
- Application Number
- CN202410940494.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-08
AI Technical Summary
In existing technologies, unstructured log text cannot be processed efficiently, making it impossible to perform integrated analysis. Furthermore, Flume does not support direct writing to OLAP databases, resulting in a high probability of data writing failure and low efficiency.
By receiving unstructured log data, the system uses a predefined set of regular expressions to determine the log type, parses the data according to the type, generates structured data and stores it in the target data table, or sends it to a distributed file system using a multiplexer. A custom Iceberg Sink can be used to omit the Kafka middleware.
It enables efficient processing and integrated analysis of unstructured logs, improves data storage efficiency, reduces network transmission frequency, saves disk resources, and simplifies Flume cluster management.
Smart Images

Figure CN118916409A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of log data processing, and in particular, to a log data processing method, system, device and non-volatile storage medium. Background Art
[0002] Currently, in the communication business, network alarms and full traffic monitoring are an indispensable part of maintaining data security and system security. However, the log texts currently transmitted over the network are mostly unstructured data, which is not suitable for being represented by a two-dimensional database table. Even if it is collected and stored in the database through Flume, it is far from meeting the requirements for integrated analysis. In addition, in business scenarios where log data is analyzed, it is suitable to be stored in OLAP databases such as Iceberg that support more complex online analytical processing of large-scale data. However, Flume currently does not support direct writing to OLAP databases and requires the use of Kafka middleware, which increases the probability of failure of the data writing process and reduces efficiency.
[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0004] The embodiments of the present application provide a log data processing method, system, device and non-volatile storage medium to at least solve the technical problem that log text cannot be integrated and analyzed due to the inability to efficiently process unstructured log text in related technologies.
[0005] According to one aspect of an embodiment of the present application, a log data processing method is provided, including: after receiving unstructured log data, determining keywords in the unstructured log data based on a preset regular expression set, thereby determining the log type of the unstructured log data; after determining the log type, processing the unstructured log data according to a parsing method corresponding to the log type to obtain structured data, and storing the structured data in a target data table; when the log type cannot be determined, using a multiplexer to send the unstructured log to a distributed file system.
[0006] Optionally, the log type includes JSON log, and the unstructured log data is processed according to the parsing method corresponding to the log type, and the steps of obtaining structured log data include: when the log type is JSON log, determining the JSON object based on the JSON string in the unstructured log; determining a first preset field list, wherein the first preset field list includes a field name column, a field description column, and a dependent field column; and reassembling the JSON object according to the preset field list to obtain structured JSON format data.
[0007] Optionally, the log type includes a string log, and the unstructured log data is processed according to the parsing method corresponding to the log type, and the step of obtaining structured log data includes: when the log type is a string log, determining a second preset field list, wherein the second preset field list includes a field name column, a field description column, and a subscript column; and segmenting the strings in the unstructured log according to the second preset field list to obtain structured data.
[0008] Optionally, the step of processing unstructured log data according to the parsing method corresponding to the log type includes: reading log parsing rules from the target storage space, wherein the log parsing rules include parsing methods corresponding to various log types, and the log parsing rules are updated according to a preset time period.
[0009] Optionally, storing the structured data in the target data table includes: writing the structured data into a data buffer; and storing the structured data in the data buffer into the target data table through a stream processing framework.
[0010] Optionally, the log data processing method further includes: obtaining unstructured log data distributed by the proxy node, wherein the proxy node is used to obtain the unstructured log data generated by each manufacturer and distribute the unstructured log data to at least one processing node connected to the proxy node.
[0011] Optionally, the log data processing method also includes: changing the configuration information of the processing node or the proxy node in response to the operation instructions of the target device, wherein the target device is also used to display a visualization interface, the visualization interface includes the connection method between the proxy node and the processing node, and the configuration information of the proxy node and the processing node.
[0012] According to another aspect of an embodiment of the present application, a log data processing system is also provided, including a proxy node and a processing node, wherein the proxy node is connected to at least one processing node, and is used to obtain unstructured log data generated by various manufacturers, and distribute the unstructured log data to at least one processing node connected to the proxy node; the processing node is used to determine the keywords in the unstructured log data according to a preset regular expression set after receiving the unstructured log data, so as to determine the log type of the unstructured log data; after determining the log type, the unstructured log data is processed according to the parsing method corresponding to the log type to obtain structured data, and the structured data is stored in a target data table; when the log type cannot be determined, a multiplexer is used to send the unstructured log to a distributed file system.
[0013] Optionally, the log data processing system further includes a target device, and the target device is used to display a visualization interface, wherein the visualization interface includes a connection mode between the proxy node and the processing node, and configuration information of the proxy node and the processing node.
[0014] According to another aspect of an embodiment of the present application, a log data processing device is also provided, including: a first processing module, which is used to determine the keywords in the unstructured log data according to a preset regular expression set after receiving the unstructured log data, so as to determine the log type of the unstructured log data; a second processing module, which is used to process the unstructured log data according to the parsing method corresponding to the log type after determining the log type, obtain structured data, and store the structured data in a target data table; a third processing module, which is used to use a multiplexer to send the unstructured log to a distributed file system when the log type cannot be determined.
[0015] According to another aspect of the embodiment of the present application, a non-volatile storage medium is provided, wherein a program is stored in the non-volatile storage medium, wherein when the program is executed, a device where the non-volatile storage medium is located is controlled to execute a log data processing method.
[0016] According to another aspect of an embodiment of the present application, there is further provided an electronic device, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the log data processing method is executed when the program is run.
[0017] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a computer program, and the computer program implements the log data processing method when executed by a processor.
[0018] In an embodiment of the present application, after receiving unstructured log data, keywords in the unstructured log data are determined according to a preset regular expression set, thereby determining the log type of the unstructured log data; after determining the log type, the unstructured log data is processed according to the parsing method corresponding to the log type to obtain structured data, and the structured data is stored in a target data table; when the log type cannot be determined, a multiplexer is used to send the unstructured log to a distributed file system, and the purpose of efficiently processing various types of unstructured logs is achieved by converting different types of unstructured log data into structured logs according to corresponding parsing methods, thereby achieving the technical effect of storing the data in the log file in a data table for subsequent integrated analysis, thereby solving the technical problem of being unable to integrate and analyze the log text due to the inability to efficiently process the unstructured log text in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0020] Figure 1 This is a log collection process diagram provided according to an embodiment of the present application;
[0021] Figure 2 This is a schematic diagram of a log collection and storage process provided according to an embodiment of the present application;
[0022] Figure 3 It is a structural schematic diagram of a log data processing system provided according to an embodiment of the present application;
[0023] Figure 4 It is a Flume topology diagram provided according to an embodiment of the present application;
[0024] Figure 5 It is a schematic diagram of Flume node configuration provided according to an embodiment of the present application;
[0025] Figure 6 It is a flow chart of a log data processing method provided according to an embodiment of the present application;
[0026] Figure 7 It is a flow chart of a log data parsing and distribution process provided according to an embodiment of the present application;
[0027] Figure 8 This is another log collection and storage process diagram provided according to an embodiment of the present application;
[0028] Fig. 9 It is a schematic diagram of the configuration process of a custom IcebergSink provided according to an embodiment of the present application;
[0029] Fig.10 It is a flowchart of a log data processing process provided according to an embodiment of the present application;
[0030] Fig.11 is a structural diagram of a log data processing device provided according to an embodiment of the present application;
[0031] Fig.12 It is a structural diagram of a computer terminal (mobile terminal) provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0034] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:
[0035] Syslog: System log or system record is a standard for transmitting log file messages on the Internet Protocol (TCP / IP) network.
[0036] Apache Flume: A distributed, highly reliable, and highly available tool for collecting, aggregating, and transferring large amounts of log data from different sources to a central data warehouse.
[0037] Flume Event: The smallest unit of data flow transmission defined by Flume.
[0038] Flume Agent: A Flume instance, essentially a JVM process that controls the flow of Event data from external log producers to the destination.
[0039] Flume Source: The source and method of data. It is mainly responsible for receiving data units.
[0040] Flume Channel: Logically, it is between Source and Sink and is a data buffer pool.
[0041] Flume Sink: the method and destination of data output.
[0042] Apache Iceberg: A new table format for storing and analyzing large, slow-moving tabular data.
[0043] Currently, in telecommunications services, network alarms and full traffic monitoring are an indispensable part of maintaining data security and system security. Among them, the analysis of monitoring device log Syslog has become an important means of monitoring system health. In the log collection stage, Apache Flume is a distributed, highly reliable, and highly available tool for collecting, aggregating, and transferring large amounts of log data from different sources to a central data warehouse. Common log collection processes include: Figure 1 shown.
[0044] Figure 1 The test scenario shown in the figure is the most basic business model, namely the web-collection-forwarding-data warehouse model. The Flume Source component receives data sent from the network through the TCP or UDP protocol, writes the data into the Channel data buffer, and the Sink component reads the buffer data and sends it to the downstream storage system.
[0045] In Flume components, Event is the smallest unit of data stream transmission defined by Flume. Agent is an instance of Flume, which is essentially a JVM process that controls the transmission of Event data stream from external log producers to the destination. A complete Agent contains three necessary components: Source, Channel, and Sink.
[0046] In the above components, Source refers to the source and method of data. It is mainly responsible for receiving data units. The outside sends Event to Flume Source in the format recognized by Flume Source. Source receives data and sinks the data into one or more Channels.
[0047] Channel is logically between Source and Sink, and it is a data buffer pool.
[0048] Sink defines the data output method and destination. It consumes the data in the Channel and sends it to an external repository (such as HDFS, Kafka, etc.). After the data is sent successfully, it is deleted from the Channel.
[0049] Specifically, in the log collection scenario, syslog is transmitted over the network through TCP or UDP protocol, so the syslogudp source and syslogtcp source components provided by Flume are usually used for log collection. In order to meet the storage of large amounts of data and low latency and high throughput, the Kafka Channel component is used as a data buffer. And according to different scenarios, Avro Sink, Kafka Sink, HDFS Sink and other components are used to sink data to different downstreams.
[0050] However, the log data processing methods in related technologies have the following three main disadvantages:
[0051] First, the log texts transmitted over the network are mostly unstructured data, which is not suitable for being represented by a two-dimensional database table. Even if collected and stored in the database through Flume, it is far from meeting the requirements for integrated analysis. Therefore, it is necessary to configure corresponding strategies based on the characteristics of the log data to parse the data into structured data with a fixed format and limited length for easy storage; in addition, the log styles of different security equipment manufacturers are also different. How to adapt the received log data to its parsing strategy is also an urgent problem to be solved.
[0052] Second, for business scenarios of log data analysis, it is suitable to store it in OLAP databases such as Iceberg that support complex online analytical processing of large-scale data, but Flume currently does not support direct writing to OLAP databases. Figure 2 As shown, the related technology uses Kafka middleware to solve this problem. Kafka is a high-throughput, low-latency, disk-based message queue. Common big data processing frameworks such as Flink and Spark have implemented Kafka integration. In this way, in the log collection and storage process, Flume is responsible for sending the collected and cleaned log data to Kafka, and the stream processing engine Flink reads data from Kafka and writes the data to the final destination of the data, Iceberg. The two are decoupled using Kafka middleware. However, the Kafka middleware itself may fail, and the addition of the Kafka middleware will make the data transmission process cumbersome.
[0053] Third, since the throughput of a single Flume Agent is limited, if the amount of log data needs to be evaluated in a production environment, a Flume cluster needs to be built to improve the throughput of the entire Flume log collection system.
[0054] All Flume configurations are based on a text file in the Java Properties file format under its directory; in some scenarios where custom interceptors are needed, the Jar package file needs to be uploaded to its directory file and the node needs to be restarted for it to take effect. As the number of Flume nodes in the production environment increases, Flume itself lacks cluster management devices, so the management of configuration files and interceptor resources will become increasingly cumbersome.
[0055] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.
[0056] According to an embodiment of the present application, a method embodiment of a log data processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0057] The method embodiments provided in the embodiments of the present application can be used in Figure 3 The log data processing system shown is executed. Figure 3 As can be seen in the figure, the system includes an agent node 30 and a processing node 32, wherein the agent node 30 is connected to at least one processing node 32, and is used to obtain the unstructured log data generated by each manufacturer, and distribute the unstructured log data to at least one processing node 32 connected to the agent node 30; the processing node 32 is used to determine the keywords in the unstructured log data according to the preset regular expression set after receiving the unstructured log data, so as to determine the log type of the unstructured log data; after determining the log type, the unstructured log data is processed according to the parsing method corresponding to the log type to obtain structured data, and the structured data is stored in the target data table; when the log type cannot be determined, a multiplexer is used to send the unstructured log to the distributed file system. It should be noted that the above-mentioned agent node 30 and processing node 32 are both Flume nodes equipped with Flume components. Among them, the agent node 30 is mainly responsible for the reception and distribution of syslog data in the network, which can be represented by P. The processing node 32 is responsible for receiving the data distributed by the P node, and cleaning and parsing the data, and finally dropping it into the library or HDFS, which can be represented by C.
[0058] As an optional implementation, the log data processing system also includes a target device, which is used to display a visualization interface, wherein the visualization interface includes a connection method between the proxy node and the processing node, and configuration information of the proxy node and the processing node.
[0059] In some embodiments of the present application, the operation topology diagram of the Flume node displayed in the above visualization interface is as follows: Figure 4 As shown. Figure 4 As can be seen in FIG. 1 , one proxy node 30 can be connected to multiple processing nodes 32, and different processing nodes 32 can send files to different objects, such as the target data table Iceberg, the distributed file system HDFS (Hadoop Distributed File System) or the Kafka middleware.
[0060] It can be seen that through the above visualization interface, when there are many Flume nodes in the production environment, the operation and maintenance personnel can clearly grasp the flow of data through the topology diagram. In addition, when the operation and maintenance personnel need to configure a single Flume node, they do not need to log in to the host where the node is located. They can directly perform interface operations, modify the configuration file and upload the jar package to the remote host, and start and stop the Flume node by remotely executing shell commands. The configuration interface interaction diagram for configuring the Flume node is shown below. Figure 5 shown.
[0061] In addition, in the above visualization interface, you can continue to integrate Filebeat, ES, Kibana and other middleware based on the topology diagram to collect, summarize and display the operation logs under the working directory of each node, so that operation and maintenance personnel can have a deeper understanding of the current health status of each node in the cluster.
[0062] In the above operating environment, the present application embodiment provides a log data processing method, which is applicable to the above processing node. Figure 6 As shown, the method comprises the following steps:
[0063] Step S602, after receiving the unstructured log data, determine the keywords in the unstructured log data according to a preset regular expression set, thereby determining the log type of the unstructured log data;
[0064] In the technical solution provided in step S602, the unstructured log data distributed by the proxy node can be obtained through the proxy node, wherein the proxy node is used to obtain the unstructured log data generated by each manufacturer and distribute the unstructured log data to at least one processing node connected to the proxy node.
[0065] Specifically, different equipment manufacturers can send data in a fixed format to the proxy node through one or more fixed IPs, and the proxy node forwards the unstructured log data to each processing node for further processing.
[0066] As an optional implementation, the configuration information of the processing node or the proxy node can also be changed in response to the operation instructions of the target device, wherein the target device is also used to display a visualization interface, which includes the connection method between the proxy node and the processing node, and the configuration information of the proxy node and the processing node.
[0067] In some embodiments of the present application, a custom Flume interceptor can be added to the processing node. The connector has a pre-judgment mechanism. Specifically, since different types of logs have different keywords and the same type of logs have the same keywords, a series of pre-judgment regular expressions can be pre-defined for this feature. That is, it is determined whether the log data carries a pre-defined keyword. In addition, Figure 7 As shown in the figure, only the logs that pass the log prediction are the logs required by the business, and can enter the next parsing task and be stored in Iceberg. Logs that fail the prediction are directly written to the HDFS system for retrospective review or directly discarded. This realizes the structuring of log data.
[0068] Step S604, after determining the log type, processing the unstructured log data according to the parsing method corresponding to the log type to obtain structured data, and storing the structured data in the target data table;
[0069] In the technical solution provided in step S604, the step of processing unstructured log data according to the parsing method corresponding to the log type includes: reading log parsing rules from the target storage space, wherein the log parsing rules include parsing methods corresponding to various log types, and the log parsing rules are updated according to a preset time period.
[0070] As an optional implementation, after parsing the log and converting it into a structured log, you can also add the table name to be written to the parsed structured data to tell the downstream Sink which table to write the data into. In addition, in order to ensure the real-time nature of the log parsing rules, you can use Caffeine Cache (a JAVA library that provides efficient local memory-based caching implementation) to cache all parsing rules in memory and configure a timed update strategy.
[0071] In some embodiments of the present application, the log type includes JSON log, and the unstructured log data is processed according to the parsing method corresponding to the log type, and the steps of obtaining structured log data include: when the log type is JSON log, determining the JSON object based on the JSON string in the unstructured log; determining a first preset field list, wherein the first preset field list includes a field name column, a field description column, and a dependent field column; and reassembling the JSON object according to the preset field list to obtain structured JSON format data.
[0072] Specifically, the SON log consists of a syslog header and a JSON string. The JSON string is the main data information of the log, such as syslog xxxxxx{"alarm_type":"xx attack","sip":"127.0.0.1","sport":"44444"}. In the Flume interceptor, the JSON string of the log can be obtained by interception, and the JSON object can be obtained by the JSON.parseObject(String) method. Then, the dependent field variables in the field list pre-configured by the parsing rule are referred to (as shown in the following table), and the getJSONObject(key) method is called in a loop to reassemble the JSON formatted data as follows:
[0073]
[0074]
[0075]
[0076] As an optional implementation, the log type includes a string log, and the unstructured log data is processed according to the parsing method corresponding to the log type. The step of obtaining structured log data includes: when the log type is a string log, determining a second preset field list, wherein the second preset field list includes a field name column, a field description column, and a subscript column; and segmenting the strings in the unstructured log according to the second preset field list to obtain structured data.
[0077] Specifically, the characteristic of string logs is that the core data information is divided by fixed and uncommon strings. For example, in the string log xx attack|||! 127.0.0.1|||! 44444, an array is obtained by dividing the string "|||!". In this way, you only need to refer to the corresponding subscripts of the pre-configured fields (as shown in the table below) and loop to obtain the value of the field subscript in the array. In this way, you can also get the same JSON formatted data after JSON parsing.
[0078] Field Name Field Description Dependent subscript attackType Attack Types 0 attackIp Attack source IP 1 attackPort Attack source port 2
[0079] In some embodiments of the present application, storing the structured data in the target data table includes: writing the structured data into a data buffer; and storing the structured data in the data buffer into the target data table through a stream processing framework.
[0080] Specifically, since Flume does not provide a Sink (data output method and purpose) component that can be directly written to Iceberg, the traditional method is to use Kafka middleware for decoupling. Flume sends the data in the Channel (data buffer) to the Kafka middleware, and then the Flink or Spark engine reads the Kafka data and writes it to Iceberg. In the implementation of this application, since the unstructured log data has been processed in advance, Figure 8 As shown, the method provided by this application customizes Iceberg Sink and integrates it into Flume, and Flink directly writes Channel into Iceberg, omitting the Kafka middleware and decoupling link. And because the decoupling link is omitted, the number of network transmissions of log data is greatly reduced, the log parsing-storage process is shortened, the log collection efficiency is improved, and the server disk resources are greatly saved.
[0081] As an optional implementation, the configuration process of customizing Iceberg is as follows: Fig. 9 As shown. Alternatively, you can also use the following process to configure custom Iceberg:
[0082] The first step is to customize the iceberg-sink component and distribute the Jar package of the code implementation to the class loading path of each Flume processing node;
[0083] Step 2: Configure the Flume Sink type to iceberg sink.
[0084] The third step is that the IcebergSink class implements the Configure(Context context) method of the Configurable interface to read relevant configuration parameters such as the Iceberg address;
[0085] In the fourth step, the IcebergSink class inherits the process() method of the AbstractSink class, reads the data units in the Channel, and writes them to Iceberg through the Flink real-time stream.
[0086] Step S606: When the log type cannot be determined, a multiplexer is used to send the unstructured log to the distributed file system.
[0087] In some embodiments of the present application, the complete process of processing log data is as follows: Fig.10 As shown. Fig.10It can be seen that in the embodiment of the present application, after receiving the unstructured log data, the keywords in the unstructured log data are determined according to the preset regular expression set, so as to determine the log type of the unstructured log data; after determining the log type, the unstructured log data is processed according to the parsing method corresponding to the log type to obtain structured data, and the structured data is stored in the target data table; when the log type cannot be determined, a multiplexer is used to send the unstructured log to the distributed file system, and the purpose of efficiently processing various types of unstructured logs is achieved by converting different types of unstructured log data into structured logs according to the corresponding parsing methods, thereby achieving the technical effect of storing the data in the log file in the data table for subsequent integrated analysis, thereby solving the technical problem of being unable to integrate and analyze the log text due to the inability to efficiently process the unstructured log text in the related technology.
[0088] In summary, a method for parsing unstructured logs to obtain structured data is provided in the embodiment of the present application, which can parse various common format logs and flexibly configure the parsing method to complete the log parsing required by the business. TB-level throughput, second-level low-latency data integration, formatted warehousing capabilities are achieved, and the data accuracy is also higher than the relevant technology. In addition, the Flume SInk output terminal is expanded by customizing the IcebergSink in this application, the Kafka Sink sinking component is omitted, the data transmission link is shortened, and it is more suitable for data integration in the field of big data. On the original basis, about 50% of the disk space is saved and the log drop delay is reduced by about 500ms. In addition, the present application also avoids frequent logins to the host through unified Flume cluster management, and node-related information can be modified remotely. The operations of modifying the configuration file, uploading jar, and restarting the node for a single Flume node are reduced from 5 minutes to less than 1 minute. For the configuration of multiple nodes, the time saved is doubled.
[0089] The present application embodiment provides a log data processing device, Fig.11 is a schematic diagram of the structure of the device. Fig.11 It can be seen that the device includes: a first processing module 110, which is used to determine the keywords in the unstructured log data according to a preset regular expression set after receiving the unstructured log data, so as to determine the log type of the unstructured log data; a second processing module 112, which is used to process the unstructured log data according to the parsing method corresponding to the log type after determining the log type, obtain structured data, and store the structured data in a target data table; a third processing module 114, which is used to use a multiplexer to send the unstructured log to the distributed file system when the log type cannot be determined.
[0090] In some embodiments of the present application, the first processing module 110 is also used to: obtain unstructured log data distributed by the proxy node, wherein the proxy node is used to obtain the unstructured log data generated by each manufacturer and distribute the unstructured log data to at least one processing node connected to the proxy node.
[0091] In some embodiments of the present application, the log data processing device is also used to: respond to the operation instructions of the target device to change the configuration information of the processing node or the proxy node, wherein the target device is also used to display a visual interface, which includes the connection method between the proxy node and the processing node, and the configuration information of the proxy node and the processing node.
[0092] In some embodiments of the present application, the step of the second processing module 112 processing unstructured log data according to the parsing method corresponding to the log type includes: reading log parsing rules from the target storage space, wherein the log parsing rules include parsing methods corresponding to each type of log type, and the log parsing rules are updated according to a preset time period.
[0093] In some embodiments of the present application, the log type includes a JSON log, and the second processing module 112 processes the unstructured log data according to the parsing method corresponding to the log type, and the steps of obtaining structured log data include: when the log type is a JSON log, determining a JSON object based on a JSON string in the unstructured log; determining a first preset field list, wherein the first preset field list includes a field name column, a field description column, and a dependent field column; and reassembling the JSON object according to the preset field list to obtain structured JSON format data.
[0094] In some embodiments of the present application, the log type includes a string log, and the second processing module 112 processes the unstructured log data according to the parsing method corresponding to the log type, and the step of obtaining structured log data includes: when the log type is a string log, determining a second preset field list, wherein the second preset field list includes a field name column, a field description column, and a subscript column; and segmenting the strings in the unstructured log according to the second preset field list to obtain structured data.
[0095] In some embodiments of the present application, the second processing module 112 storing the structured data into the target data table includes: writing the structured data into a data buffer; and storing the structured data in the data buffer into the target data table through a stream processing framework.
[0096] It should be noted that the various modules in the above-mentioned log data processing device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.
[0097] The method embodiments provided in the embodiments of the present application may be executed in a mobile terminal, a computer terminal or similar electronic devices. Fig.12 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a log data processing method. Fig.12 As shown, the computer terminal 120 (or mobile device 120) may include one or more (shown as 1202a, 1202b, ..., 1202n in the figure) processors 1202 (the processor 1202 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1204 for storing data, and a transmission module 1206 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Fig.12 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.12 More or fewer components as shown, or with Fig.12 Different configurations are shown.
[0098] It should be noted that the above-mentioned one or more processors 1202 and / or other log data processing circuits may generally be referred to as "log data processing circuits" in this article. The log data processing circuit may be embodied in whole or in part as software, hardware, firmware or any other combination. In addition, the log data processing circuit may be a single independent processing module, or may be fully or partially integrated into any of the other components in the computer terminal 120 (or mobile device). As involved in the embodiments of the present application, the log data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0099] The memory 1204 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the log data processing method in the embodiment of the present application. The processor 1202 executes various functional applications and log data processing by running the software programs and modules stored in the memory 1204, that is, the above-mentioned log data processing method is realized. The memory 1204 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1204 may further include a memory remotely arranged relative to the processor 1202, and these remote memories may be connected to the computer terminal 120 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.
[0100] The transmission device 1206 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 120. In one example, the transmission device 1206 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 1206 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0101] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 120 (or mobile device).
[0102] According to an embodiment of the present application, a non-volatile storage medium is also provided. A program is stored in the non-volatile storage medium, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the following data processing method: after receiving unstructured log data, the keywords in the unstructured log data are determined according to a preset regular expression set, thereby determining the log type of the unstructured log data; after determining the log type, the unstructured log data is processed according to the parsing method corresponding to the log type to obtain structured data, and the structured data is stored in a target data table; when the log type cannot be determined, a multiplexer is used to send the unstructured log to a distributed file system.
[0103] According to an embodiment of the present application, a computer program product is also provided, including a computer program. When the computer program is executed by a processor, the following log data processing method is implemented: after receiving unstructured log data, keywords in the unstructured log data are determined according to a preset regular expression set, thereby determining the log type of the unstructured log data; after determining the log type, the unstructured log data is processed according to the parsing method corresponding to the log type to obtain structured data, and the structured data is stored in a target data table; when the log type cannot be determined, a multiplexer is used to send the unstructured log to a distributed file system.
[0104] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0105] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.
[0109] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A log data processing method, characterized in that: include: After receiving the unstructured log data, determining keywords in the unstructured log data according to a preset regular expression set, thereby determining the log type of the unstructured log data; After determining the log type, processing the unstructured log data according to the parsing method corresponding to the log type to obtain structured data, and storing the structured data in a target data table; When the log type cannot be determined, a multiplexer is used to send the unstructured log to a distributed file system.
2. The log data processing method according to claim 1, characterized in that: The log type includes a JSON log, and the steps of processing the unstructured log data according to the parsing method corresponding to the log type to obtain the structured log data include: In the case where the log type is the JSON log, determining a JSON object according to a JSON string in the unstructured log; Determine a first preset field list, wherein the first preset field list includes a field name column, a field description column, and a dependent field column; The JSON object is reassembled according to the preset field list to obtain structured JSON format data.
3. The log data processing method according to claim 1, characterized in that: The log type includes a string log, and the steps of processing the unstructured log data according to the parsing method corresponding to the log type to obtain the structured log data include: In the case where the log type is the string log, determining a second preset field list, wherein the second preset field list includes a field name column, a field description column, and a subscript column; The strings in the unstructured log are segmented according to the second preset field list to obtain the structured data.
4. The log data processing method according to claim 1, characterized in that: The step of processing the unstructured log data according to the parsing method corresponding to the log type includes: The log parsing rules are read from the target storage space, wherein the log parsing rules include parsing methods corresponding to each type of the log, and the log parsing rules are updated according to a preset time period.
5. The log data processing method according to claim 1, characterized in that: Storing the structured data in the target data table includes: Writing the structured data into a data buffer; The structured data in the data buffer is stored in the target data table through a stream processing framework.
6. The log data processing method according to claim 1, characterized in that: The log data processing method further includes: The unstructured log data distributed by the proxy node is obtained, wherein the proxy node is used to obtain the unstructured log data generated by each manufacturer and distribute the unstructured log data to at least one processing node connected to the proxy node.
7. The log data processing method according to claim 6, characterized in that: The log data processing method further includes: The configuration information of the processing node or the proxy node is changed in response to an operation instruction of the target device, wherein the target device is also used to display a visualization interface, and the visualization interface includes a connection method between the proxy node and the processing node, and the configuration information of the proxy node and the processing node.
8. A log data processing system, characterized in that: Including agent nodes and processing nodes, among which, The proxy node is connected to at least one of the processing nodes, and is used to obtain unstructured log data generated by various manufacturers, and distribute the unstructured log data to at least one of the processing nodes connected to the proxy node; The processing node is used to determine the keywords in the unstructured log data according to a preset regular expression set after receiving the unstructured log data, so as to determine the log type of the unstructured log data; after determining the log type, process the unstructured log data according to the parsing method corresponding to the log type to obtain structured data, and store the structured data in a target data table; if the log type cannot be determined, use a multiplexer to send the unstructured log to a distributed file system.
9. The log data processing system according to claim 8, characterized in that: The log data processing system also includes a target device, which is used to display a visualization interface, wherein the visualization interface includes a connection method between the proxy node and the processing node, and the configuration information of the proxy node and the processing node.
10. A log data processing device, characterized in that: include: A first processing module is used to determine keywords in the unstructured log data according to a preset regular expression set after receiving the unstructured log data, so as to determine the log type of the unstructured log data; A second processing module is used to process the unstructured log data according to the parsing method corresponding to the log type after determining the log type, obtain structured data, and store the structured data in a target data table; The third processing module is used to use a multiplexer to send the unstructured log to a distributed file system when the log type cannot be determined.
11. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the log data processing method according to any one of claims 1 to 7.
12. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the log data processing method according to any one of claims 1 to 7 when running.
13. A computer program product, characterized in that The invention comprises a computer program, which implements the log data processing method according to any one of claims 1 to 7 when being executed by a processor.
Citation Information
Patent Citations
Log data analysis method and device, computer equipment and storage medium
CN109582551A
Logistics data real-time processing method and device, equipment and storage medium
CN113220756A
Application log classification method and device, equipment and medium
CN116821067A
