Log data processing method and computing device

Abstract

The embodiment of the invention provides a log data processing method and computing equipment, and the method comprises the steps: obtaining at least one piece of to-be-processed log data; based on the original field characteristics of the to-be-processed log data, matching a first log template of the to-be-processed log data; wherein the original field characteristics indicate a field value of each original field in the to-be-processed log data; the first log template comprises a processing rule of to-be-processed log data, and the processing rule is used for converting field values of a plurality of original fields in the to-be-processed log data into a plurality of preset standard fields; performing mapping conversion on an original field of the to-be-processed log data according to the first log template to obtain target log data with a standard field; the first log template further comprises a pre-configured standard log template and a mapping relation between an original field and a standard field of the to-be-processed log data; and storing the target log data in a database.

Description

Technical Field

[0001] The present application relates to the technical field of log data processing, and in particular to a log data processing method and computing device. Background Art

[0002] With the development of information technology, log data has become an important means for enterprises to monitor the security and operation of network systems. Multi-source heterogeneous logs refer to a collection of log data that comes from different sources (multiple sources) and has different formats, structures, or storage methods (heterogeneous).

[0003] In the field of network security, enterprises need to collect security-related information from multiple log sources and formats for unified analysis and processing. However, because different vendors and types of security devices (such as firewalls, intrusion detection systems, and antivirus software) generate logs in different formats, including but not limited to text files, XML (eXtensible markup language), and JSON (JavaScript object notation), log fields in different log formats vary significantly. For example, fields with the same meaning may have different names in different logs (such as "timestamp" vs. "event_time"), resulting in a high degree of diversity and complexity in log data.

[0004] When storing these heterogeneous logs from multiple sources, if standardized storage is not implemented, automatic correlation cannot be achieved during log queries and association. Consequently, security analysts must perform tedious field queries. This inconsistency increases the complexity of data association and threat analysis and requires urgent resolution. Summary of the Invention

[0005] The embodiments of the present application provide a log data processing method and computing device that can efficiently process multi-source heterogeneous logs, achieve standardized storage of log data, improve the accuracy of log data, and facilitate log query and retrieval.

[0006] To this end, the following technical solutions are provided in the embodiments of the present application:

[0007] In a first aspect, an embodiment of the present application provides a log data processing method, comprising: obtaining at least one log data to be processed; matching a first log template of the log data to be processed based on the original field characteristics of the log data to be processed; wherein the original field characteristics indicate the field values of each original field in the log data to be processed; the first log template includes processing rules for the log data to be processed, and the processing rules are used to convert the field values of multiple original fields in the log data to be processed into multiple preset standard fields; mapping and converting the original fields of the log data to be processed according to the first log template to obtain target log data with standard fields; wherein the first log template also includes a pre-configured standard log template and a mapping relationship between the original fields of the log data to be processed and the standard fields, and the standard log template includes a set of field values of the preset standard fields of the log data to be processed; storing the target log data in a database.

[0008] One of the biggest challenges of multi-source heterogeneous logs is field inconsistency. The names and formats of log fields from different sources vary greatly, which makes query and analysis difficult. The log data processing method provided in the embodiment of the present application aims to convert log data from different sources into standard log data in a unified format through standardized processing, thereby improving the accuracy of log data and query efficiency. Specifically, the embodiment of the present application collects log data from different network security devices (such as firewalls, intrusion detection systems, antivirus software, etc.). According to the original field characteristics of the log data to be processed, the system will match it according to a pre-defined "first log template". The first log template contains processing rules for the log data to be processed, which are used to convert the field values of multiple original fields in the log data to be processed into multiple preset standard fields. For example, in different security log data, the original field of "source IP" may be "source_ip" or "src_ip". In the first log template, the above-mentioned "source IP" can be uniformly modified to the standard field "source_ip", and the Chinese name of the field is unique and corresponds one to one with the English name of the field. Based on the first log template, the original fields in the log data to be processed are mapped and converted. Through this conversion, all log fields are mapped to unified, standard fields, ensuring the consistency of log data. After the mapping conversion, the generated target log data with standard fields is stored in the database. This step ensures standardized log data storage, facilitating subsequent querying and analysis.

[0009] In this way, after standardized log data storage, the data becomes unified and standardized, which makes log query and analysis more efficient and facilitates subsequent data management and maintenance. Security analysts can query directly through standard fields, no longer needing to manually process different field names and formats, avoiding tedious query steps and improving query accuracy. Through automated field mapping and data conversion, manual intervention can also be reduced, the error rate can be lowered, and the reliability of overall data processing can be improved. At the same time, converting all log data into a unified standard format can greatly reduce data inconsistency issues and improve data integration and consistency. Moreover, it facilitates the automated correlation of log data, allowing security analysts to more quickly identify potential security threats or abnormal behavior.

[0010] In summary, the core advantage of this technical solution lies in its standardized processing of multi-source, heterogeneous log data, which resolves the issue of log field inconsistency and makes data storage, querying, and association more efficient and accurate. Standardized log data not only improves overall system performance but also simplifies log data management and maintenance, reducing the risk of manual errors. This has important practical implications for enterprise network security monitoring and threat analysis.

[0011] In an achievable implementation manner, after obtaining at least one log data to be processed, the method further includes: performing log format conversion on the log data to be processed to obtain log data to be processed in a unified target log format.

[0012] In this implementation, log data from different sources uses different storage formats and structures (such as XML, JSON, and text), making it difficult to effectively analyze and correlate these logs directly within the system. Log format conversion eliminates these format differences, ensuring that all log data conforms to a unified target format. This allows enterprises to easily integrate log data from different devices or systems, enhancing system compatibility.

[0013] In an achievable implementation, the target log format is a JSON log format.

[0014] In this embodiment, the JSON format not only has the characteristics of clear structure, easy parsing and expansion, but also has wide compatibility and is suitable for the integration and processing of cross-platform logs. It improves the query efficiency and analysis capabilities of log data, supports dynamic expansion, is convenient for integration with various analysis tools and platforms, and has good automated processing capabilities. By converting log data into a unified JSON format, the technical solution provided by the embodiment of the present application provides significant advantages in data processing, storage, analysis and transmission. Overall, the use of the JSON format can significantly improve the efficiency and flexibility of log processing, enhance data security, and provide strong support for enterprises in log monitoring, troubleshooting and security analysis.

[0015] In one achievable implementation, the log format of the log data to be processed is converted using a regular expression.

[0016] In this implementation, regular expressions not only efficiently extract and convert log content, but also offer strong adaptability and scalability, enabling them to handle complex and diverse log formats. Using regular expressions to convert log formats is an efficient and flexible technical solution suitable for processing log data of various formats and types, and is particularly well-suited for scenarios requiring rapid processing of large volumes of log data.

[0017] As an achievable implementation, the matching of the first log template of the log data to be processed based on the original field characteristics of the log data to be processed includes: parsing the log data to be processed to obtain field values of multiple original fields of the log data to be processed, matching the field values of the original fields of the log data to be processed with the field values of the original fields in each first log template, and determining that the field values of the original fields of the log data to be processed are the same as the field values of the original fields in the first log template; and determining the first log template of the log data to be processed when the field values of the original fields of the log data to be processed are the same as the field values of the original fields in the standard log template.

[0018] In this embodiment, the technical solution of matching the first log template based on the original field characteristics can efficiently and accurately determine the first log template of the log to be processed by comparing the log data to be processed with the original fields in the predefined first log template one by one, so as to facilitate the standard field mapping conversion of the log to be processed. The advantages of this solution are high-precision matching, multi-format adaptation, automated processing, data consistency assurance, etc., and it is suitable for processing multi-source, multi-format, and dynamically changing log data. In this way, the efficiency and quality of log data processing can be significantly improved, manual intervention can be reduced, and a solid foundation can be laid for subsequent log analysis, storage, and query.

[0019] As an achievable implementation, the mapping and conversion of the original fields of the log data to be processed is performed based on the first log template, including: extracting the field values of the original fields of the log data to be processed to form a field value set of the original fields; according to the processing rules preset in the first log template, and based on the field values of the standard fields in the standard log template and the mapping relationship between the original fields of the log data to be processed and the standard fields, converting the field values of the original fields of the log data to be processed into the field values of the standard fields in the standard log template.

[0020] In this embodiment, first, the field values of the original fields are extracted from the log data to be processed to form a field value set of the original fields. The field value set of the original field contains all the key information items in the log record, which may come from different log sources or formats. The extracted field value set of the original field is mapped to the field value of the standard field of the preset standard log template. The standard log template is a pre-defined normalized structure that contains a field value set of the standard field. Each standard field represents a specific log information, and different log sources may use different field names or formats. According to the field value set of the standard field preset in the standard log template, the field values in the original field set are converted into field values of the standard field according to a one-to-one corresponding rule. Even if the original field names or formats are inconsistent, through mapping conversion, all log data will be uniformly converted into a predefined standard format, so that the log data from various sources can be compatible and unified, which is convenient for subsequent processing and analysis.

[0021] This technical solution effectively normalizes log data to be processed into a unified, standardized format through mapping and conversion of standard log templates. This solution not only improves data processing efficiency and accuracy, but also ensures cross-platform and cross-system compatibility, greatly facilitating the automated processing, analysis, and monitoring of large-scale log data.

[0022] In a feasible implementation, the standard field set includes any one or more standard fields of quintuple information, network protocol, host type, vulnerability type, and threat type log information.

[0023] In this embodiment, during log processing, various fields of information are first extracted from the log data to be processed. For quintuple information, network protocol, host type, vulnerability type, and threat type log information, the system extracts the corresponding fields based on the definition of the standard field set and matches them with the content in the log data. For example, if the log to be processed contains fields such as source IP, destination IP, and destination port, these fields are matched as quintuple information. Once the relevant field values are extracted, the system matches the log data with the fields in the standard field set to standardize the logs. Standardized log data helps unify log formats and field types, ensuring consistent processing of various log data types during subsequent analysis. By extracting and matching the standard field set, the system can perform security analysis based on the field content. For example, using quintuple information, the system can analyze the direction, source, and destination of traffic and identify abnormal traffic. Using the threat and vulnerability fields, the system can identify potential attack activities and respond promptly. Another key application of this technical solution is log classification and archiving. By using a standard set of fields, the system can categorize logs by type (e.g., attack type, host type, protocol type, etc.), facilitating subsequent query, storage, and analysis. Structured log data also helps improve efficiency in large-scale log data storage and management.

[0024] This technical solution, based on a standard set of fields (such as quintuple information, network protocol, host type, vulnerability type, and threat type log information), significantly improves the efficiency, accuracy, and security of log analysis by standardizing, classifying, and structuring log data. Its advantages include enhanced security analysis capabilities, reduced false positives, improved query efficiency, and the ability to flexibly respond to diverse systems and emerging needs. It is particularly suitable for scenarios such as large-scale log processing, network security monitoring, cross-system log integration, and compliance auditing.

[0025] As an achievable implementation, the obtaining of at least one log data to be processed includes: collecting at least one log data to be processed from the source system based on a distributed data stream processing cluster, and obtaining at least one log data to be processed from the distributed data stream processing cluster based on any data processing and distribution node in the data processing and distribution cluster.

[0026] In this implementation, a log collection technology solution based on a distributed data stream processing cluster provides strong support for the real-time collection, storage, and processing of log data through its advantages of high throughput, low latency, distributed architecture, and high reliability. This solution enables efficient transmission and processing of log data, decouples producers and consumers, and improves the scalability and reliability of the system, making it suitable for a variety of application scenarios such as large-scale log management, real-time security monitoring, data analysis, and compliance auditing. In addition, by setting the number of Apache distributed data stream processing, Apache data processing, and distribution cluster nodes based on the log volume, scalable deployment maximizes the efficiency of hardware resource utilization, reduces the waste of idle resources, enables on-demand expansion, and avoids the cost waste caused by over-configuration.

[0027] In one achievable implementation, the source end system includes any one of a firewall, a Web application protection system, a terminal detection and response system, a network detection and response system, an intrusion prevention system, and an antivirus software.

[0028] In this implementation, security devices or software such as firewalls, web application firewalls (WAFs), endpoint detection and response (EDRs), network detection and response (NDRs), intrusion prevention systems (IPSs), and antivirus software serve as source systems. Log data is transmitted to the backend for real-time analysis and processing via distributed data stream processing or other log collection and transmission platforms. By recording and generating log data, these security devices and software provide critical security information for timely detection and response to potential network security threats.

[0029] In an achievable implementation, mapping and converting the original fields of the log data to be processed according to the standard log template to obtain target log data with standard fields includes:

[0030] Creating a data stream based on the data processing and distribution cluster, wherein the data stream is used to process the to-be-processed log data obtained from the distributed data stream processing cluster based on a first log template;

[0031] Data processing and distribution

[0032] The data processing and distribution cluster parses the field values of the original fields of the log data to be processed based on the mapping relationship between the original fields of the log data to be processed and the predefined standard fields in the first log template, and maps the field values of the original fields of the log data to be processed to the field values of the predefined standard fields, thereby obtaining target log data with standard fields;

[0033] Store the target log data into the object storage table according to the storage object of the data flow of the predefined standard fields.

[0034] In this embodiment, the distributed data stream processing cluster acts as a distributed stream platform, capable of collecting log data from different data sources (such as applications, services, etc.). The data processing and distribution cluster obtains the log data stream to be processed by configuring a suitable distributed data stream processing connector (such as the Consume distributed data stream processing processor) or by subscribing to a topic in the distributed data stream processing. By using the data processing and distribution cluster and the distributed data stream processing cluster to obtain the log data to be processed, the first log template is used for field mapping, and the log data is standardized and stored in the object storage system. Its main advantages include data standardization, flexible scalability, real-time processing capabilities, efficient storage and access, and easy operation and maintenance management. Such a processing flow can greatly simplify the unified processing of cross-system and cross-platform log data, and improve the efficiency of log data analysis, monitoring and query. At the same time, one or more data processing and distribution nodes in the data processing and distribution cluster ensure that the service can still operate normally in the event of a single point failure through redundant design and load balancing, thereby improving system stability, reducing downtime, and having high availability.

[0035] In the second aspect, the embodiment of the present application also provides a log data processing device, including an acquisition module and a processing module. The acquisition module is used to acquire at least one log data to be processed; the processing module is used to match the first log template of the log data to be processed based on the original field characteristics of the log data to be processed; the original field characteristics indicate the field values of each original field in the log data to be processed; the first log template includes processing rules for the log data to be processed, and the processing rules are used to convert the field values of multiple original fields in the log data to be processed into multiple preset standard fields; the original fields of the log data to be processed are mapped and converted according to the first log template to obtain target log data with standard fields; the first log template also includes a pre-configured standard log template and a mapping relationship between the original fields of the log data to be processed and the standard fields; the target log data is stored in a database.

[0036] In a third aspect, an embodiment of the present application further provides a computing device comprising at least one memory for storing programs; and at least one processor for executing the programs stored in the memory; wherein the memory is coupled to the processor, and when the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0037] In a fourth aspect, an embodiment of the present application further provides a server cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method described in any possible implementation of the first aspect.

[0038] In a fourth aspect, an embodiment of the present application further provides a computing device cluster, comprising a management node and multiple computing nodes, wherein the management node is used to execute the method described in the first aspect.

[0039] In the fifth aspect, an embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions in the computer-readable storage medium are executed by a computing device, the computing device executes the method involved in the first aspect and its possible implementation methods.

[0040] In a sixth aspect, an embodiment of the present application further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computing device, the computing device executes the method described in the first aspect and its possible implementation methods.

[0041] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0042] In summary, this application has at least one of the following advantages:

[0043] 1. Automatic field mapping: Automatically identify and map log fields in different formats through the data processing and distribution API (application programming interface), converting them into a unified standard format. The automatic field mapping function reduces the need for manual intervention, improves work efficiency, and reduces the risk of human error.

[0044] 2. High-performance processing engine: Using distributed data stream processing, data processing and distribution and other efficient data processing engines to achieve real-time large-scale log data processing. The high-performance processing engine ensures that the system can process large amounts of log data in real time to meet the needs of high-load scenarios;

[0045] 3. Flexible configuration mechanism: Provides flexible configuration options, allowing users to customize parsing rules and mapping relationships. The flexible configuration mechanism enables the system to be adjusted according to specific needs, improving adaptability and scalability;

[0046] 4. Multi-source heterogeneous log parsing: Supports heterogeneous log data generated by various security devices. The multi-source heterogeneous log parsing capability enables the system to be compatible with more types of security devices, increasing the system's applicability.

[0047] 5. Improve data consistency: Convert log data from different sources and formats into a unified standard format, improving data consistency and availability;

[0048] 6. Simplified Management and Analysis: Standardizing log data simplifies subsequent data management and analysis, improving work efficiency. You can easily perform correlation queries by searching standardized fields. For example, by configuring the standard fields Source IP and Destination IP English names as sip and dip, you can search to see if a specific IP matches the sip and dip values in all standard fields, allowing for correlation analysis of related behaviors. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 FIG. 1 is a schematic diagram of the architecture of a log data processing system provided by an embodiment of the present application;

[0050] Figure 2 : shows a schematic diagram of the structure of a server cluster provided by an embodiment of the present application;

[0051] Figure 3 A flow chart of a log data processing method provided in an embodiment of the present application is shown;

[0052] Figure 4 A flow chart showing another log data processing method provided in an embodiment of the present application is shown;

[0053] Figure 5 A schematic diagram of the structure of a log data processing device provided in an embodiment of the present application is shown;

[0054] Figure 6 A schematic diagram of the structure of a computing device provided in an embodiment of the present application is shown;

[0055] Figure 7 A schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application;

[0056] Figure 8 This is a schematic diagram of the architecture of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0058] In the description of the embodiments of the present application, words such as "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of the present application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0059] In the description of the embodiments of this application, the term "and / or" is simply a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "plurality" means two or more. For example, "multiple systems" refers to two or more systems, and "multiple terminals" refers to two or more terminals.

[0060] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly identifying the technical features being referred to. Thus, features specified as "first" or "second" may explicitly or implicitly include one or more of such features. The terms "include," "comprising," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0061] In the description of the embodiments of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

[0062] In the description of the embodiments of the present application, the terms "first\second\third, etc." or module A, module B, module C, etc. are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that the specific order or sequence can be interchanged where permitted so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0063] In the description of the embodiments of the present application, the numbers representing the steps, such as S101, S102, etc., do not necessarily mean that the steps must be executed in this manner. If permitted, the order of the previous and next steps can be interchanged, or they can be executed simultaneously.

[0064] Related terms involved in the embodiments of this application:

[0065] Log: refers to the recorded information generated by systems, applications, devices, etc., which is used to record operating status, events, error messages, or operational behaviors. It is usually used for monitoring, troubleshooting, analysis, auditing, etc.

[0066] Multi-source heterogeneous logs refer to log data from different sources with different formats or structures. They typically contain log information from different systems, applications, devices, or platforms. These logs may use different formats (such as text, JSON, XML, etc.) or contain different types of data (such as event records, error information, performance data, etc.).

[0067] Log normalization: refers to the unification of log formats generated by different sources, different devices, or different manufacturers so that they follow a unified format and structure, making them more suitable for subsequent analysis and processing.

[0068] Cluster: refers to a group of servers (or computing nodes) that work together through a network and form a unified system. Although these servers are independent physical machines, they collaborate through cluster management software or protocols to provide a combined service or resource to the outside world. Usually, to users and applications, these servers look like a single system. Therefore, the cluster can act as a single system to provide services, process tasks, or store data. The purpose of the cluster is to improve the reliability, availability, and performance of the system by increasing redundancy and scalability. Clusters are widely used in computing and storage tasks that require high performance and high reliability. Through the cluster architecture, the system can maintain stable and efficient operation in the face of failures, increased load, or a surge in data volume.

[0069] Data Processing and Distribution: Data processing software is primarily used for processing and distributing data streams. It supports highly configurable directed graphs for data routing, transformation, and system transit logic. Data Processing and Distribution provides a web-based graphical interface that allows users to program flow-based data collection through drag-and-drop, connection, and configuration.

[0070] Data flow: In data processing and distribution, the data flow is divided into multiple processing units, each of which performs certain operations on the data, such as extraction, transformation, and merging. These processing units are connected together in a certain order to form a complete data processing flow.

[0071] Distributed data stream processing: It is an open-source distributed streaming data platform, also known as a distributed message queue, used for high-throughput, low-latency data publishing and subscription.

[0072] Elasticsearch is an open-source, distributed, real-time search and analytics engine designed to provide fast, scalable, and high-performance search solutions. It supports a variety of data formats, including text, numbers, and geolocation, and provides a flexible query language to meet diverse search needs. Elasticsearch is primarily used for large-scale text search, log analysis, and real-time data analysis. Elasticsearch has a database where data can be stored and queried.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0074] The embodiment of the present application provides a log data processing method, which can uniformly convert the log format of multi-source heterogeneous log data into JSON format when collecting log data, and then convert the original fields in the heterogeneous logs into standard fields according to the mapping relationship to obtain standard log data, and distribute the standard log data to achieve normalization of multi-source heterogeneous logs. That is, the embodiment of the present application improves the consistency of log data: log data from different sources and different formats are converted into a unified standard format, which improves the consistency and availability of data. After the multi-source heterogeneous logs are standardized, the log analysis is simplified through association rules, and logs from multiple sources can be associated with each other for query, which improves the accuracy of log data, facilitates log query retrieval, and effectively reduces the complexity of log data association processing and threat analysis.

[0075] Next, the architecture of a log data processing system provided in an embodiment of the present application is introduced.

[0076] Figure 1 The diagram in FIG. 1 is a schematic diagram of the architecture of a log data processing system provided by an embodiment of the present application. Figure 1 As shown, the log data processing system includes at least one terminal device 10, a storage device 11 and a server cluster 12. The terminal device 10, the storage device 11 and the server cluster 12 can be directly or indirectly connected through a wired network or a wireless network for data transmission, which is not limited in this embodiment of the present application.

[0077] Exemplarily, the server cluster 12 may include multiple computing devices (or computing nodes), for example, the multiple computing devices may include computing device 12-1, computing device 12-2, computing device 12-3, etc. The number of server clusters 12 may be more or less, and the embodiments of the present application do not limit this. It should be noted that the computing device 12 may be a server, a computer, a smart phone, or other devices. The multiple computing devices 12 may be a single device or a combination of multiple devices. The multiple computing devices 12 may be deployed with an application program that runs a log data processing system to perform the functions of the log data processing system, such as data acquisition, cleaning, data storage and management, data conversion and integration, data analysis and mining, real-time data processing, security and privacy protection, etc. For example, the server cluster in the embodiments of the present application may include an Apache data processing and distribution cluster deployed with a log data processing system and an Apache distributed data stream processing cluster for obtaining the unprocessed log data generated by the log production source, and the data processing and distribution cluster may create a data stream for centrally processing the unprocessed log data under the distributed data stream processing cluster topic.

[0078] Optionally, the computing device may be provided with a storage unit, which may be a cache, dynamic random access memory (DRAM), static random access memory (SRAM), or other memory for temporarily storing data, and is used to store data that needs to be temporarily stored during the operation of the log data processing system for access or operation by the processor or other hardware. In an embodiment of the present application, the storage unit may store data corresponding to the computing tasks to be executed by the data processing system, data corresponding to the computing tasks that have not been executed in the interrupted computing tasks, intermediate data during the processing process, processed data, and other cached data.

[0079] A computing device may be provided with a communication interface to enable data transmission with other computing devices, storage devices, and other devices. The communication interface may be a wired transmission interface, such as a Compute Express Link (CXL) interface, a Peripheral Component Interconnect Express (PCIe) interface, or a Universal Serial Bus (USB) interface. The communication interface may also be a wireless transmission interface, such as a Bluetooth (BT) module, a Wireless Fidelity (WI-FI) module, or a wireless communication module.

[0080] In one embodiment, the distributed data stream processing cluster in the server cluster 12 can be used to collect various log data to be processed. It should be noted that distributed data stream processing is a distributed stream processing platform that is widely used to process real-time data streams. Distributed data stream processing adopts a publish / subscribe model, allowing log producers to publish log data to distributed data stream processing topics, and then consumers can subscribe to these topics and consume data. Distributed data stream processing has the characteristics of high throughput, low latency, and strong fault tolerance, and can efficiently process massive amounts of log data. Using a distributed data stream processing cluster as middleware can achieve reliable transmission and decoupling of log data, improve the scalability and stability of the system, and is suitable for a variety of application scenarios such as large-scale log management, real-time security monitoring, data analysis, and compliance auditing.

[0081] In this embodiment, the distributed data stream processing cluster is communicatively connected to a source-end system that serves as a log production source. In one embodiment, the log information generated by the source-end system can be sent to the distributed data stream processing cluster through a configured distributed data stream processing client. For example, the Producer API of the distributed data stream processing client can be used to send different types of log data to the distributed data stream processing cluster. Exemplarily, the logs of the source-end system can be various types of log data such as system logs, application logs, security logs, business logs, or operation and maintenance logs. Exemplarily, the source-end system can be implemented as different types of devices and system programs that can generate a variety of log data. For example, the source-end system can be a server, a network device, a security device, and an application.

[0082] In one embodiment, the server of the source system can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud business libraries, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and big data and artificial intelligence platforms.

[0083] Optionally, the network device may be a router, a switch, a sensor, or a load balancer.

[0084] Optionally, the security device can be a hardware device such as a hardware firewall, a secure boot device, or a hardware token. In some implementations, the security device can also be a virtualized security system, such as a Web Application Protection System (WAF): Detects and protects against attacks on Web applications (such as SQL injection, XSS, etc.), generating detailed logs about Web requests, including access URLs, HTTP methods, attack signatures, etc. Endpoint Detection and Response (EDR): Monitors suspicious activities on terminal devices, such as malicious programs and unauthorized access, and generates detailed terminal behavior logs. Network Detection and Response (NDR): Monitors network traffic and detects abnormal traffic or potential security threats, generating logs about network communications and data flows. Intrusion Detection System (IDS) / Intrusion Prevention System (IPS): Generates logs about malicious activities, abnormal traffic, and attack signature matching. Security Information and Event Management (SIEM) System: Centrally collects security event and alarm logs from various devices (such as firewalls, IDS / IPS, terminals, etc.). Antivirus software: Logs virus detection, malware scan results, cleanup operations, and system health. Authentication systems: Log user logins, authentication failures, permission changes, and more. Data loss prevention (DLP) systems: Log potential data leaks, file transfers, and interceptions.

[0085] Optionally, when the source system is used as an application, it can be an application running on a user-side terminal device, or a web browser provided by the data processing system. For example, the client can be a desktop application, mobile application, web application, or web-based application, etc., without limitation here. The user-side terminal device can be, but is not limited to, a smartphone, tablet computer, laptop computer, or desktop computer.

[0086] The distributed data stream processing cluster acts as middleware, receiving log data from source systems and storing it in multiple topics. Each topic represents a log type or log source, which can be flexibly defined based on requirements. Log data is stored in the distributed data stream processing cluster as messages (in the form of message queues). The distributed architecture of the distributed data stream processing cluster enables it to store large amounts of log data, thus supporting large-scale data inflows. Partitioning achieves load balancing, improving the system's fault tolerance and high availability. Even if some nodes fail, the distributed data stream processing's replication mechanism ensures data is not lost, ensuring high availability and reliability of log data. Furthermore, distributed data stream processing supports high concurrency and elastic scalability, enabling distributed data stream processing to support high-concurrency log data collection and dynamically add or remove consumer nodes based on demand to flexibly respond to fluctuations in log traffic. For example, when log data volume surges, the system can expand collection capacity by adding consumer nodes. Distributed data stream processing decouples the source systems that produce logs from the data processing and distribution cluster. The source systems and the data processing and distribution cluster do not interact directly, but rather use the distributed data stream processing as an intermediary for data transmission. This makes the system architecture more flexible, able to cope with the expansion, update, and change of the source system and data processing and distribution cluster. The source system of log data and the data processing and distribution cluster can operate independently, reducing the coupling between systems.

[0087] The data processing and distribution cluster can obtain the log data to be processed by configuring a suitable distributed data stream processing connector (such as the Consume distributed data stream processing processor) or by subscribing to the topic in the distributed data stream processing. The data processing and distribution cluster will further process the log as needed, such as parsing, analysis, mapping conversion, etc. Exemplarily, the data processing and distribution cluster reads the mapping relationship between the original fields of the log data to be processed and the pre-configured standard fields from the field mapping table of the standard log template. It should be noted that the standard log template defines pre-agreed standard fields (for example: source IP address (Source IP), target IP address (Destination IP), source port number (Source Port), destination port number (Destination Port), transport protocol (Protocol, such as TCP, UDP), etc.). At the same time, the field mapping table records the mapping relationship between the original fields (for example: src_ip, dst_ip, proto, etc.) and standard fields (for example: source_ip, destination_ip, protocol) in each log data to be processed. The data processing and distribution cluster reads these predefined mapping rules from the field mapping table to obtain the corresponding relationship between the original fields and the standard fields. According to the mapping relationship, the data processing and distribution cluster uses stream processing tools (such as ExtractText, EvaluateJsonPath and other data processing and distribution processors) to parse the original fields of the log data to be processed. These original fields will be converted and mapped according to the requirements of the standard fields to obtain target log data that conforms to the standardized format. For example, if the original log contains the src_ip field, and the standard template requires the field name to be source_ip, the original field will be mapped to the standard field through the data flow of the data processing and distribution cluster. The mapped target log data (i.e., data containing standard fields) will be stored in the object storage storage device 11 (for example: Elasticsearch, Amazon S3, etc.) database table according to the predefined standard field format. It can be understood that the data still maintains the standard format when stored in the database table, which is convenient for subsequent analysis and query.

[0088] Optionally, the storage device 11 is an independent storage device. Exemplarily, the storage device 11 can be a hard disk drive (HDD), a solid state drive (SSD), a NAND flash memory, a disk, etc. Exemplarily, the storage device 11 can be used to store overflow cache data when multiple computing devices run the log data processing system, as well as other data, such as programs for running the log data processing system and data processed by the log data processing system. Exemplarily, the storage device 11 can be used to store standard business data that has been successfully processed by the data processing system; the data lake is used to store unprocessed data that the data processing system fails to identify, and is archived for centralized analysis of the data stored in the storage device. It should be noted that, Figure 1 The connection between the memory 11 and the computing device 12 is not limited to establishing a communication connection between the memory 11 and the computing device 12, but also means that the memory 11 establishes a communication connection with the data processing system deployed by the computing device 12, so that the memory 11 can establish a communication connection with any computing device 12.

[0089] The storage device 11 may be provided with a communication interface to enable data transmission with multiple computing devices 12 and other devices. The communication interface may be a wired transmission interface, such as a CXL interface, a PCIe interface, a USB interface, etc. The communication interface may also be a wireless transmission interface, such as a BT module, a Wi-Fi module, a wireless communication module, etc.

[0090] In one embodiment, storage device 11 may also be a hardware storage device deployed on an instance (Elastic node) of an Elasticsearch cluster. Elasticsearch is an open-source distributed search and analysis engine used for real-time search, analysis, and storage of large amounts of data. The Bulk API is a batch operation interface provided by Elasticsearch that can be used to efficiently index, update, or delete multiple documents in batches. Elasticsearch, as an open-source distributed search engine, is used for real-time search and analysis of large amounts of data. In this embodiment, it is used to store and retrieve standardized log data.

[0091] As an example, in the data processing and distribution cluster, three data streams generate three standard log data, namely standard log 1, standard log 2 and standard log 3. The three standardized log data are distributed to multiple Elastic nodes. On each Elastic node, the Bulk API is used to store the standardized log data distributed to the node in batches in Elasticsearch. For example, standard log 1, standard log 2 and standard log 3 are stored in the corresponding Elasticsearch node 1, Elasticsearch node 2 and Elasticsearch node 3 respectively. Elasticsearch will index the standardized log data into the appropriate index according to the configured index and mapping rules for subsequent search and analysis operations. In this way, the system realizes distributed storage. Since the data is stored in Elasticsearch, users can query and analyze related logs in real time, support complex logical condition queries, and improve the real-time and accuracy of related analysis.

[0092] The terminal device 10 can be an entity on the user side for receiving or transmitting signals. The terminal device 10 can be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal equipment, industrial control terminal equipment, UE unit, UE station, mobile station, remote station, remote terminal equipment, mobile device, wireless communication equipment, UE agent or UE device, etc. The terminal device can be fixed or mobile. It should be noted that the terminal device can support at least one wireless communication technology, such as long time evolution (LTE), NR, sixth generation mobile communication system (6th-generation, 6G) or next generation wireless communication technology, etc.

[0093] For example, the terminal device can be a mobile phone, a tablet computer (pad), a desktop computer, a laptop computer, an all-in-one computer, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a cellular phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, a wearable device, a terminal device in a future mobile communication network, or a terminal device in a future evolved public land mobile network (PLMN), etc. The embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device. A corresponding application is arranged on the terminal device 10 , and the user can use the application to obtain the standard log data stored in the storage device 11 after being processed by the data processing and distribution cluster, so as to retrieve and analyze the above log data.

[0094] In addition, the user can also configure a preset first standard log template on the terminal device 10. The first log template includes processing rules for the log data to be processed, and the processing rules are used to convert the field values of multiple original fields in the log data to be processed into multiple preset standard fields. It can be understood that the first log template also includes a pre-configured standard log template and a mapping relationship between the original fields of the log data to be processed and the standard fields. The standard log template includes a set of field values of the preset standard fields of the log data to be processed; wherein the first log template is pre-configured based on the second log template, and the second log template includes a set of field values of multiple original fields when the log data to be processed is used as a sample log. The first log template can be sent to the Apache data processing and distribution cluster via the data processing and distribution API, so that the data processing and distribution cluster creates a data stream, uses the defined standard log template for conversion, and writes the standardized log to the database of the storage device 11 (such as Elasticsearch).

[0095] Exemplarily, the above-mentioned wired network or wireless network can use standard communication technologies and / or protocols, including but not limited to local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired network, private network or any combination of virtual private networks.

[0096] Thus, the log data processing system provided by the embodiments of the present application enables automated processing of multi-source heterogeneous log data, collecting and standardizing logs generated by different security devices. Specifically, this system extracts key information fields and converts them into unified, standard fields, enabling standardized log storage. Standardized logs use unique, standard fields across different log metadata, improving data accuracy and making them easier to manage. It also allows for faster and simpler searching and analysis, enabling automated correlation during log retrieval.

[0097] It can be understood that the data processing system described in the embodiment of the present application is for the purpose of more clearly illustrating the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. Those skilled in the art will know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0098] To facilitate understanding of the technical solution of the present application, the data processing principle of the embodiment of the present application is introduced in detail below.

[0099] Figure 2 The diagram in FIG. 1 is a schematic diagram of a server cluster structure provided by an embodiment of the present application. Figure 2As shown, in some possible implementations, on the server cluster side of the log data processing system, the server cluster is a data processing and distribution cluster 20. The data processing and distribution cluster 20 includes a plurality of data processing and distribution nodes, each of which is deployed with a data processing and distribution instance, which work together to process the data stream (FlowFile). The log data processing method in this embodiment can be executed by any one or more data processing and distribution nodes in the plurality of data processing and distribution nodes. The one or more data processing and distribution nodes in the data processing and distribution cluster 20 ensure that the service can still operate normally in the event of a single point failure through redundant design and load balancing, thereby improving system stability and reducing downtime. It is worth mentioning that the plurality of data processing and distribution nodes include a management node 21 and a plurality of computing nodes. For example, the plurality of computing nodes include computing node A, computing node B and computing node C. The number of computing nodes is not limited in this application. Among them, the plurality of computing nodes coordinate work through the management node 21, share status and data, and ensure high availability and load balancing of the entire data processing system. In the data processing and distribution cluster 20, the data flow (FlowFile) is shared by all nodes, and the management node 21 in the data processing and distribution cluster will decide which nodes process which tasks. The management node 21 in the data processing and distribution cluster 20 will automatically assign tasks to each computing node AC in the data processing and distribution cluster 20 to improve processing efficiency and scalability. It should be noted that in the data processing and distribution cluster 20, any computing node can play the role of management node 21. In other words, the management node 21 is elected on any computing node in the data processing and distribution cluster 20. The election process is automatically handled by the data processing and distribution cluster 20 itself. Based on factors such as the health status and load of the computing node, the data processing and distribution cluster 20 will select a node as the management node 21.

[0100] For example, when the data processing and distribution cluster 20 starts up, all compute nodes attempt to compete for the role of management node 21. At any given moment in the data processing and distribution cluster 20, only one compute node will be elected as management node 21 (typically through an election algorithm such as Zookeeper or Raft). If this compute node becomes unavailable for some reason (such as a failure or reboot), the other compute nodes will automatically initiate an election and elect a new management node 21. In other words, the management node 21 is actually assumed by any compute node in the data processing and distribution cluster 20. This compute node serves not only as the "control center" of the data processing and distribution cluster 20 but also as part of the data processing and distribution cluster 20, executing data flow tasks. Although only one management node 21 is responsible for coordination, all compute nodes participate in actual data processing, task execution, and other tasks. In addition to performing data processing tasks, the management node 21 is also responsible for coordinating and managing the data processing and distribution cluster 20 status, task scheduling, resource allocation, data flow control, and other tasks.

[0101] In one embodiment, the data processing and distribution cluster 20 is pre-configured with a first log template, which includes processing rules for the log data to be processed, and the processing rules are used to convert the field values of multiple original fields in the log data to be processed into multiple preset standard fields. It is understandable that the first log template also includes a pre-configured standard log template and a mapping relationship between the original fields of the log data to be processed and the standard fields, and the standard log template includes a set of field values of the preset standard fields of the log data to be processed; wherein the first log template is pre-configured based on a second log template, and the second log template includes a set of field values of multiple original fields when the log data to be processed is used as a sample log.

[0102] Among them, the standard log template in the first log template supports user customization. When the user customizes the standard log template, he needs to provide a set of field values of the original fields of the original sample log data (referred to as the second log template above), and then configure the mapping relationship between the field values of the original fields of the original sample log data and the field values of the standard fields to be converted. After that, the logs received with the same original fields as the sample log data can be parsed and mapped through the field values of the standard fields in this standard log template. At the same time, the user also needs to mark which manufacturer and model of equipment the standard log template is for. The first log template supports manual configuration by the user to provide flexible configuration options, allowing users to customize parsing rules and mapping relationships. The flexible configuration mechanism enables the system to be adjusted according to specific needs, improving adaptability and scalability.

[0103] The management node 21 distributes the multiple log data to be processed obtained from the distributed data stream processing cluster to different computing nodes for processing, and creates a data stream for processing the multiple log data to be processed. It should be noted that in Apache data processing and distribution, the data stream is defined by a series of processors, and the log data to be processed in the data stream is processed based on the processing rules of the log data to be processed. For example, the processing rules can be used to convert the field values of multiple original fields in the log data to multiple preset standard fields. Specifically, the processing rules can include a description file of the data stream. For example, when the data stream adopts data processing and distribution processing, the description file can be a data processing and distribution file. The description file can include the processor type and the configuration content of the processor in multiple computing nodes. Each processor performs a task in the data stream, and the data stream gradually completes the collection, processing and transmission of data from source to target.

[0104] For example, in a data processing and distribution cluster, you can use the Consume distributed data stream processor to retrieve log data to be processed from the distributed data stream processing cluster. Configure the Consume distributed data stream processor to communicate with the distributed data stream processing cluster and ensure that the connection parameters of the distributed data stream processing cluster (such as the distributed data stream processing broker address) are correctly configured.

[0105] In the Consume distributed data stream processing processor, configure the following properties:

[0106] Distributed Data Stream Processing Brokers: Set the broker address of the distributed data stream processing cluster (for example, distributed data stream processing 1:9092, distributed data stream processing 2:9092).

[0107] Topic: Specifies the topic of the log messages to be processed in distributed data stream processing.

[0108] Group ID: Set the ID of the distributed data stream processing consumer group.

[0109] Auto Offset Reset: Set the offset reset policy (such as earliest or latest). If the consumer group has not consumed messages from this topic, select earliest.

[0110] In the Consume distributed data stream processing processor, set the Message Selector to specify the messages to be consumed, and set the Batch Size and Max Wait Time to control the number of messages pulled each time.

[0111] With this configuration, data processing and distribution obtains log data to be processed from the distributed data stream processing topic.

[0112] Messages in distributed data stream processing are usually unstructured text data (such as raw log syslogkey / value, text files, XML or CSV format), so they need to be converted into structured JSON format. In this embodiment, the ConvertRecord or ReplaceText processor of data processing and distribution is used to perform this operation. Specifically, in the ConvertRecord processor, set the Record Reader and Record Writer. Among them, Record Reader: Select the appropriate reader (for example, CSVReader, if your distributed data stream processing message is in CSV format). RecordWriter: Select JsonRecordSetWriter as the writer to convert the data into JSON format.

[0113] With these configurations, data processing and distribution will convert the messages consumed from the distributed data stream processing into JSON format.

[0114] Convert fields based on the standard log template in the first log template:

[0115] By configuring the field values of specific standard fields (such as timestamp, ip_address, error_code, source_ip, destination_ip, destination_port, protocol, etc.) in the preset standard log template, we can map the field values of the original fields of the original log data to the field values of the standard fields through the expression language and processors of data processing and distribution.

[0116] Exemplarily, use the ExtractText or EvaluateJsonPath processor to extract the original fields from the JSON data. Use the EvaluateJsonPath processor to extract the fields in the distributed data stream processing message and store them as attributes. For example, extract the src_ip field, the dst_ip original field, and so on. Then use the ReplaceText or UpdateAttribute processor to process the log fields according to the rules of the standard log template. For example, the standard log template needs to map the original field "src_ip" to the standard field "source_ip". You can define a regular expression in the ReplaceText processor for replacement. Use the AttributesToJson processor to convert the updated attributes back to JSON format to ensure that the fields are generated in accordance with the standard format.

[0117] Once the data is converted to a standard format JSON, it can be written to the target storage object (such as Elasticsearch, database, etc.). Data processing and distribution support interacting with various databases through different database processors (such as PutElasticsearchHttp or PutSQL).

[0118] In one embodiment, the PutElasticsearchHttp processor is used to write the standardized JSON data to the Elasticsearch index. It should be noted that in order to write data to Elasticsearch, it is necessary to configure the connection to the target database. For example, a database connection pool (such as DBCPConnectionPool) is configured to store the connection information of the target database (such as IP address, port, account, password, etc.). Configure the connection information of Elasticsearch in the PutElasticsearchHttp processor: IP address, port, cluster name, user name and password. For example, configure the IP address through the PutElasticsearchHttp processor: configure the connected Elasticsearch address (such as http: / / localhost:9200).

[0119] In Elasticsearch, you can configure indexes (equivalent to tables in relational databases). You can set different indexes (table names) for each log type. For example, in the PutElasticsearchHttp processor, you can dynamically set the index name. Typically, the index name will contain the date or other related information. For example, you can use the expression language for data processing and distribution to dynamically generate index names in the log processing flow. For example, the index name logstash-YYYY.MM.dd is generated by date. This format can dynamically generate date-related index names.

[0120] In one embodiment, a SQL database (e.g., MySQL, PostgreSQL) is used to store standardized log data. The target table name can be configured in the PutSQL processor. The table name can also be dynamically passed through the data processing and distribution expression language. For example, the database connection pool (e.g., DBCPConnectionPool) and the target table name can be configured to map standardized fields to database table columns.

[0121] For example, suppose we need to normalize log data and store the normalized log data in Elasticsearch. We can create the following data flow in the data processing and distribution cluster:

[0122] Get log data:

[0123] Use the Consume distributed data stream processing processor to obtain log data to be processed from the distributed data stream processing cluster.

[0124] Log parsing and mapping transformation:

[0125] Use the regular expression defined in the ReplaceText processor to replace the log data in different log formats and convert them into JSON format.

[0126] Use EvaluateJsonPath to extract the required original fields, such as source IP (source_ip), destination IP (destination_ip), source port (source_port), destination port (destination_port), protocol (protocol), etc.

[0127] Standardized logs:

[0128] After parsing the original log fields, compare them with the standard log templates in the template library through string matching or regular expressions. If a match is successful, the AttributesToJson processor can be used to convert the original log fields into standard fields while maintaining the JSON format for compatibility with Elasticsearch indexes.

[0129] Configure the Elasticsearch connection:

[0130] When configuring an Elasticsearch connection, you typically need to specify information such as the IP address, port, username, and password to connect and authenticate access. In data processing and distribution clusters, you can use the PutElasticsearchHttp handler to configure the IP address, port, and authentication information for connecting to the Elasticsearch cluster. This is because, for Elasticsearch, the username and password can be configured directly in the PutElasticsearchHttp handler or set in the connection pool configuration.

[0131] Dynamically set the index name, such as logstash-YYYY.MM.dd. In the PutElasticsearchHttp processor, you can set the dynamic index name through the data processing and distribution expression language, for example: logstash-${now():format('yyyy.MM.dd')}, so that the log will be dynamically indexed according to the date.

[0132] Write normalized logs to Elasticsearch:

[0133] Use PutElasticsearchHttp to write the processed log data to Elasticsearch.

[0134] Thus, in this implementation, the data stream FlowFile mechanism and processor for data processing and distribution are used to automatically process log files, and the storage table name (index name) is dynamically configured as needed. In this way, when the log stream comes in, the data is standardized according to the predefined template and written to the configured database table (Elasticsearch index).

[0135] By leveraging an Apache data processing and distribution cluster, this embodiment efficiently creates a complete data flow for acquiring, converting, and storing log data. The entire process includes acquiring logs from the data source, mapping and standardizing the log format using templates, configuring a connection to a database (such as Elasticsearch), and finally writing the processed log data to a configured storage table. The entire data flow can flexibly manage storage device configurations, including IP addresses, account numbers, and passwords, ensuring automated and efficient data flow.

[0136] The above is an introduction to the log data processing system and process provided by the embodiments of the present application. Based on the above content, the technical solution of the present application is described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0137] Figure 3 FIG. 1 shows a flow chart of a log data processing method provided by an embodiment of the present application. Figure 3 As shown, it can be understood that the method can be executed by any device, equipment, platform, or computing device cluster with computing and processing capabilities. Exemplarily, the method can be executed by a data processing device, wherein the device can be implemented by software and / or hardware, and can be, but not limited to, configured in a computing device cluster comprising at least one computing device, typically, it can be configured in a server. For ease of description, this embodiment will be described with any computing device in the computing device cluster as the execution subject. Figure 3 As shown, the data processing method may include the following steps:

[0138] S101: Obtain at least one log data to be processed.

[0139] In this embodiment, the log data to be processed is log data collected from multiple source end systems, which include various applications and devices. For example, the log data to be processed can be log data generated by security devices. Since security devices involve firewalls, WAFs, IPSs, IDSs, EDRs, NDRs, antivirus software, etc., and may even involve dozens of security vendors, different security vendors often have their own log formats. Even the log formats of different types of security devices from the same security vendor are not the same. As a result, log data from different security devices will be stored in different formats and structures due to different designs and business requirements. Therefore, it is necessary to standardize the log data of different structures. After collecting the log data, it is necessary to extract the original fields from these data and convert the original fields into standard fields for subsequent log analysis and processing. For example, it is convenient for network security operation and maintenance personnel to conduct related queries, screening, analysis and judgment without having to understand different log formats, and then respond to and deal with network attacks, thereby improving the efficiency of network attack analysis.

[0140] In one embodiment, log data from multiple source systems can be obtained through the Apache data processing and distribution cluster. Exemplarily, log data is generated by different types of devices. Among them, log data from different log production sources can first be collected by a log agent deployed on the source system, and then sent by the log agent to the data processing and distribution cluster listening entry. The listening entry is opened by various processors in the data processing and distribution cluster, for example, through listeners (Listener) and get processors (Get Processor) to collect logs and messages from different data sources. These processors allow data processing and distribution to obtain data from different input channels and serve as the starting point for subsequent data processing and forwarding. Specific examples: File system class: such as ListenFTP processor and GetFTP processor, suitable for collecting logs from file transfer protocols. Message queue class: such as Consume distributed data stream processing processor and ConsumeAMQP processor, used to consume logs from message queues in real time. Database class: such as QueryDatabaseTable processor, suitable for extracting log records from a database. Internet of Things device class: such as ConsumeMQTT processor, obtains log data from Internet of Things devices. Interface and protocol classes: such as the ListenHCP processor, ListenUDP processor, ListenSyslog processor, and GetSplunk processor, which are used to receive and obtain log data published through protocols such as HTTP, UDP, Syslog, and Splunk.

[0141] In one embodiment, the log data of the source system can be collected through the Apache distributed data stream processing cluster and then sent to the Apache data processing and distribution cluster for subsequent processing. For example, you can open the data processing and distribution Web UI (Website user interface) and drag the Consume distributed data stream processing processor to the data stream canvas. Configure the processor to communicate with the distributed data stream processing cluster and ensure that the connection parameters of the distributed data stream processing cluster (such as the distributed data stream processing broker address) are correctly configured. The specific configuration method can be found in the description above and will not be repeated here.

[0142] Both distributed data stream processing and data processing and distribution clusters support high concurrency and elastic scalability. The elastically scalable architecture of distributed data stream processing + data processing and distribution clusters allows for scalable deployment based on log volume, offering the advantages of high concurrency, high throughput, and high availability. The number of nodes in the Apache distributed data stream processing and Apache data processing and distribution clusters can be adjusted based on the volume of log data to be processed. This scalable deployment maximizes hardware resource utilization, reduces idle resources, enables on-demand expansion, and avoids the costly over-provisioning. For example, distributed data stream processing is a highly scalable distributed system that supports horizontal scaling, increasing cluster throughput by adding more nodes. As data volumes increase, distributed data stream processing can easily scale to accommodate increased traffic. Data processing and distribution also offer strong scalability, increasing data processing capacity by adding more nodes. Data processing and distribution supports horizontal scaling through clustering to accommodate large-scale data stream processing needs. In this implementation, Apache distributed data stream processing is used to collect log data and send it to Apache data processing and distribution for subsequent processing. This solution combines the high throughput and scalability of distributed data stream processing with the flexible data stream management capabilities of data processing and distribution, offering multiple advantages such as high reliability, real-time performance, scalability, and decoupling. This solution is suitable for large-scale log data processing, real-time monitoring, and complex data stream processing tasks, providing powerful support for scenarios such as log analysis, security monitoring, and business analysis.

[0143] S103: Match the first log template of the log data to be processed based on the original field characteristics of the log data to be processed.

[0144] The original field characteristics indicate the field values of each original field in the log data to be processed. The first log template includes processing rules for the log data to be processed, and the processing rules are used to convert the field values of multiple original fields in the log data to be processed into multiple preset standard fields. The first log template also includes a pre-configured standard log template and a mapping relationship between the original fields of the log data to be processed and the standard fields, and the standard log template includes a set of field values for the preset standard fields of the log data to be processed. The first log template is pre-configured based on the second log template, and the second log template includes a set of field values for multiple original fields when the log data to be processed is used as a sample log. It is understandable that the second log template is derived from log data generated by security devices. Security devices include firewalls, WAFs, IPSs, IDSs, EDRs, NDRs, antivirus software, etc. It is understandable that even the same security device may involve dozens of security vendors, and different security vendors often have their own log formats. Even different types of security devices from the same security vendor have different log formats. Therefore, it is necessary to collect log data from different security devices to build a second log template, and to aggregate the original field sets of sample log data of different log formats of the same security device into the same second log template. Then, according to different design and business requirements, the first log template is configured based on the second log template. For example, assuming that the field values of the original fields provided by different manufacturers of the same security device, such as WAF, are original field value 1, original field value 2, and original field value 3, respectively, the original field value 1, original field value 2, and original field value 3 are aggregated into a second log template, and then the first log template is configured. The configured first log template is used to indicate the processing rules of the log data to be processed, and includes a pre-configured standard log template and a mapping relationship between the original fields of the log data to be processed and the standard fields. Among them, the standard log template includes a set of field values of the preset standard fields of the log data to be processed, such as the set of field values of the preset standard fields is preset field value 1, preset field value 2, and preset field value 1. Based on the mapping relationship between the original fields and the standard fields, the original field value 1 is mapped to the preset field value 1, the original field value 2 is mapped to the preset field value 2, and the original field value 3 is mapped to the preset field value 3, thereby achieving data normalization.

[0145] Specifically, in this embodiment, the original field characteristics indicate the field values of each original field in the log data to be processed, and the field values of each original field indicate the specific attributes and structural information of the original field of the log data to be processed. Specifically, the field value of the original field may include information such as the name, data type, format, value range, etc. of the field, which reflect the content and data representation of each field in the log data. The field value of the original field mainly describes the various characteristics of the fields in the log data to be processed (such as timestamp, log level, IP address, etc.). For each field, the field value of the original field includes: field name: that is, the identifier of the field, such as timestamp, level, message, etc. Field data type: such as string, integer, float, date and time, etc. Field format: such as the format of the timestamp is YYYY-MM-DD HH:MM:SS, the log level may be an enumeration type, such as INFO, ERROR, DEBUG, etc. Field value range or constraints: For example, a field may only have a specific value or range, such as the status field can only take 200 or 500.

[0146] In the cluster architecture of the Apache Distributed Data Stream Processing Cluster and the Apache Data Processing and Distribution Cluster, we mentioned that log data is transmitted from the source system to the data processing and distribution system via distributed data stream processing, and then processed in the data processing and distribution system. The original field characteristics here are reflected in the fact that the log data stored in the distributed data stream processing is typically raw and unprocessed, and may have different formats, different field names, and different data types. Data processing and distribution can flexibly convert and process these original fields based on their characteristics. For example, a log field originally containing a string timestamp can be converted to a datetime type in the standardized target log data.

[0147] For example, suppose we have a raw log data: 2025-03-31 14:25:01, INFO, 192.168.1.1, User login successful. It contains the following fields: timestamp: 2025-03-31 14:25:01 (timestamp, string type); level: INFO (log level, string type); ip_address: 192.168.1.1 (IP address, string type); message: User login successful (log message, string type). These fields are raw fields, and their raw field characteristics include: the timestamp field is of time type, in the format of YYYY-MM-DD HH:MM:SS. The level field is the log level, which belongs to the predefined enumeration value (such as INFO, ERROR, etc.). The ip_address field is a standard IP address string. The message field contains text describing the log.

[0148] When mapping these raw field data to the standard log template in the first log template, we can convert it into a more standardized structure, such as:

[0149] {

[0150] "timestamp":"2025-03-31 14:25:01",

[0151] "log_level":"INFO",

[0152] "source_ip":"192.168.1.1",

[0153] "message":"User login successful"

[0154] }

[0155] In this example, the mapping between the original fields and the standard fields in the first log template is as follows:

[0156] timestamp→timestamp

[0157] level→log_level

[0158] ip_address→source_ip

[0159] message→message.

[0160] Therefore, the field values of the original fields determine how the fields are extracted from the log data and how they are converted into the standardized target field format. This conversion can be automatically completed by the data processing and distribution cluster and finally stored in the database.

[0161] In other words, in this embodiment, the field value of the original field refers to the specific attributes of each field in the log data to be processed (such as field name, data type, format, etc.). In this log data processing method, the characteristics of the original field are first analyzed, and then these fields are converted according to the field values of the standard fields in the standard log template, ultimately forming target log data that meets the standards.

[0162] Specifically, the log data to be processed is parsed to obtain the field values of multiple original fields of multiple log data to be processed, and the field values of the original fields of the log data to be processed are matched one by one with the field values of the original fields of the sample logs in each first log template, and it is determined that the field values of the original fields of the log data to be processed are the same as those of the original fields of the sample logs in the first log template; when the field values of the original fields of the log data to be processed are the same as those of the original fields of the sample logs in the first log template, the first log template of the log data to be processed is determined.

[0163] When comparing the original fields of the log to be processed with the field values of the standard fields in the first log template one by one, the matching criteria here can be the field name, field type, field format, etc. For example, whether the timestamp field in the log to be processed conforms to the same format and type as the timestamp field in the first log template, whether the error code field matches, and so on. During the comparison process, if the field of the log to be processed matches the original field information of the sample log in a certain first log template, it can be determined that the log to be processed belongs to the first log template. The higher the degree of matching of the original field, the more accurate the matching result. At this point, the log to be processed can be classified as the first log template, which facilitates subsequent processing and analysis.

[0164] It's understandable that because the mapping relationship between original fields and standard fields needs to be preset, that is, the field values of the original fields are associated with the field values of the standard fields, network security personnel can customize and edit the mapping of the relevant standard fields according to actual needs. Then, by setting the mapping relationship between the original fields and the standard fields according to actual conditions, the original fields of the log data to be processed can be standardized and converted, making log data standardization more flexible and convenient. Since the standard log template supports user customization, when customizing the standard log template, the user needs to provide an original sample log as the second log template, and then configure the first log template that contains the standard log template. Subsequently, all logs received with the same original fields as the sample log can be mapped and converted using this first log template. It should be noted that in one case, if the same security device comes from different manufacturers, the original fields of the log data it generates will still have significant differences. Therefore, for the same security device from different manufacturers, two standard log templates can be created as needed. Users can also indicate which manufacturer and model of device the different standard log templates are for. In one scenario, if the original fields of the same security device from different manufacturers are not significantly different (for example, the field names differ only in uppercase and lowercase letters), a standard log template can be created. Based on the mapping relationship between the field values of the standard fields in the standard log template and the field values of the original fields, the mapping relationship between the field values of the original fields and the field values of the standard fields of multiple different security devices can optionally be sent to the data processing and distribution cluster via the data processing and distribution API for subsequent use.

[0165] Exemplarily, the log data to be processed is the log data generated by security devices. In the field of network security and log analysis, standard field sets are used to unify and normalize key information in different log data formats. Standardized log fields can make log data from different sources consistent, which facilitates cross-system analysis, search, and association. Exemplarily, the standard field set includes five-tuple information, network protocol, host class, vulnerability class, threat class and other log information. The content contained in the above standard field set is the standard field used to describe and analyze network traffic, process activity and vulnerability information in the field of network security. They belong to different categories and are often used in scenarios such as log analysis, intrusion detection, and attack tracing.

[0166] The following explains in detail the contents of the standard field set and the mapping relationship between original fields and standard fields.

[0167] Regarding the quintuple in the standard field:

[0168] For example, a five-tuple refers to the five key fields involved in network communication that describe a data transmission. It is used to uniquely identify a network connection and is widely used in firewalls, traffic analysis, intrusion detection, etc. It includes the following fields:

[0169] source_ip: Source IP address, indicating the IP address of the communication initiator, that is, the IP address of the host that sends the data packet.

[0170] destination_ip: Destination IP address, indicating the IP address of the communication recipient, that is, the IP address of the host that receives the data packet.

[0171] source_port: Source port number, indicating the port of the communication initiator, that is, the sending port, usually the client's port number, used to identify the source application process.

[0172] destination_port: Destination port number, indicating the port of the communication recipient, that is, the receiving port, usually the port number of the server, used to identify the target application process.

[0173] Protocol: Communication protocol, indicating the protocol type used for data transmission (e.g., TCP, UDP, ICMP, etc.). It should be noted that the original fields of log data from different security devices (e.g., security device A or security device B) may be different, but the Chinese names of their original fields are the same. Therefore, in this embodiment, different English original fields with the same Chinese names on different security devices are mapped to a unified standard field.

[0174] For example, in one embodiment, the mapping relationship between the original fields and the standard fields of a quintuple in a standard log template is shown in Table 1-1.

[0175] Table 1-1 Mapping relationship between original fields and standard fields of quintuple log data to be processed in the standard log template

[0176] Field Chinese name Standard English fields Device A field Device B field Source IP sip s_ip src_ip Destination IP dip d_ip dest_ip Source Port sport s_port dest_port Destination port dport d_port dest_port protocol protocol proto protocol

[0177] As shown in Table 1-1, the standard fields referred to in this embodiment are the standard English fields in Table 1-1. The English original field s_ip in the device A field and the English original field src_ip in the device B field have the unique Chinese name of source IP. In this embodiment, the standard English field of source IP is configured as sip, so both s_ip and src_ip are mapped to sip. Similarly, the field Chinese name of destination IP corresponding to d_ip in the device A field and dest_ip in the device B field are mapped to the pre-configured dip. The source port corresponding to s_port in the device A field and dest_port in the device B field are mapped to the pre-configured port. The destination port corresponding to d_port in the device A field and dest_port in the device B field are mapped to the pre-configured dport. The protocol corresponding to proto in the device A field and protocol in the device B field are mapped to the pre-configured protocol.

[0178] In another possible implementation, in another different standard log template, the standard fields of the quintuple may also be:

[0179] Table 1-2 Mapping relationship between original fields and standard fields of five-tuple log data to be processed in another standard log template

[0180]

[0181]

[0182] As shown in Table 1-2, src_ip, source_ip, and client_ip all represent source IP addresses and are therefore mapped to source_ip. dst_ip, destination_ip, and server_ip all represent destination IP addresses and are therefore mapped to destination_ip. src_port maps to source_port, and dst_port maps to destination_port. proto represents the protocol type and is mapped to protocol.

[0183] Regarding process information in standard fields:

[0184] Process Information: Process information is used to record detailed information about the process, which is used to analyze process activities in the system, and is usually used to detect malicious processes and activities. For example, the standard fields for process information can be: process_id: Process ID (PID), a unique identifier for a process running in the system. process_name: Process name, which represents the file name or execution program name of the process (such as chrome.exe or java). parent_process_id: Parent process ID (PPID), which represents the parent process ID that started the current process, which helps to track the startup chain of the process. These fields help understand the startup of the process, the parent process, and the relationship between processes, and are often used for behavioral analysis and malware detection.

[0185] For example, the mapping relationship between the original fields and the standard fields of the process information log in this embodiment is shown in Table 2:

[0186] Table 2 Mapping relationship between original fields and standard fields of log data to be processed regarding process information in the standard log template

[0187] Original fields Standard fields pid process_id process_id process_id proc_name process_name parent_pid parent_process_id

[0188] As shown in Table 2: In this embodiment, pid or process_id is mapped to process_id, proc_name is mapped to process_name, and parent_pid is mapped to parent_process_id.

[0189] Regarding vulnerability information in standard fields:

[0190] Vulnerability Information: Vulnerability information is used to record details related to security vulnerabilities, to describe vulnerabilities in a system or application, and to help identify potential security risks in the system. Exemplarily, the standard fields for vulnerability information in this embodiment may be vulnerability_id: vulnerability ID, a number or identifier that uniquely identifies a vulnerability, typically conforming to certain standards (such as a CVE number). vulnerability_name: vulnerability name, describing the name or type of vulnerability, such as "SQL Injection" or "Buffer Overflow". severity_level: the severity level of the vulnerability, used to assess the degree of harm caused by the vulnerability, typically divided into several levels (such as high, medium, low, or a score of 1-10) to assess the impact of the vulnerability. These fields provide key information about known vulnerabilities in a system or application, and are typically used for vulnerability management, risk assessment, and security incident response.

[0191] For example, the mapping relationship between the original fields and the standard fields of the vulnerability information log in this embodiment is shown in Table 3:

[0192] Table 3-1 Mapping relationship between original fields and standard fields of log data to be processed regarding vulnerability information in the standard log template

[0193] Original fields Standard fields vuln_id vulnerability_id vulnerability_id vulnerability_id vuln_name vulnerability_name severity severity_level

[0194] As shown in Table 3-1, in this implementation, the original fields vuln_id and vulnerability_id are mapped to the standard field vulnerability_id, vuln_name is mapped to vulnerability_name, and severity is mapped to severity_level.

[0195] In one embodiment, the developer may also set the standard fields of the vulnerability class to the standard English fields in Table 3-2. The specific examples are as follows:

[0196] Table 3-2 Mapping relationship between original fields and standard fields of vulnerability-related log data to be processed in another standard log template

[0197]

[0198]

[0199] Regarding threat class information in standard fields:

[0200] Threat Information: used to record potential threats in a network or system. This type of information may include attack type, source, impact, etc. Exemplarily, in this embodiment, the standard fields of the threat information log may be: threat_id: threat ID, which uniquely identifies a specific security threat. threat_type: threat type, which describes the type of threat (for example: network attack, malware, etc.). attack_pattern: attack pattern, which records the specific method of attack (for example: DDoS attack, SQL injection, etc.). threat_source: threat source, which records the identifier or IP address of the threat initiator. threat_severity: the severity level of the threat.

[0201] For example, the mapping relationship between the original fields and the standard fields of the threat information log in this embodiment is shown in Table 4-1:

[0202] Table 4-1 Mapping relationship between original fields and standard fields of log data to be processed for threat information in the standard log template

[0203] Original fields Standard fields threat_id threat_id attack_type threat_type attack_pattern attack_pattern source_ip threat_source severity threat_severity

[0204] As shown in Table 4-1, in this implementation, the original field attack_type is mapped to the standard field threat_type, source_ip is mapped to threat_source, and severity is mapped to threat_severity.

[0205] In one embodiment, the developer may also set the standard fields of the threat class to be standard English fields as shown in Table 4-2. The specific example is as follows:

[0206] Table 4-2 Mapping relationship between original fields and standard fields of threat-related log data to be processed in another standard log template

[0207] Field Chinese name Standard English fields Device A field Device B field Threat Name threat_name t_name ThreatName Attack threat level threat_level t_level ThreatLevel Attack Type attack_type a_type AttackType Attack Results attack_result a_result AttackResult Threat Information threat_info t_info ThreatInfo

[0208] Regarding the network protocol in the Standard field:

[0209] Network Protocol: The Network Protocol field typically contains information related to the communication, such as the protocol type and port number. This helps analyze the nature and transmission methods of network traffic. For example, the Network Protocol standard field can be configured as follows: protocol: Indicates the transport layer protocol type, such as TCP, UDP, ICMP, etc. ip_version: The IP protocol version, such as IPv4 or IPv6. transport_protocol: The transport protocol, which may be TCP or UDP.

[0210] For example, the mapping relationship between the original fields and the standard fields of the network protocol log in this embodiment is shown in Table 5-1:

[0211] Table 5-1 Mapping relationship between original fields and standard fields of log data to be processed for network protocols in the standard log template

[0212] Original fields Standard fields protocol_type protocol ip_version ip_version transport_proto transport_protocol

[0213] As shown in Table 5-1, in this implementation, protocol_type is mapped to protocol, and transport_proto is mapped to transport_protocol.

[0214] In one embodiment, the developer may also set the standard fields of the network protocol to the standard English fields shown in Table 5-2. The specific examples are as follows:

[0215] Table 5-2 Mapping relationship between original fields and standard fields of log data to be processed for network protocols in another standard log template

[0216] Field Chinese name Standard English fields Device A field Device B field HTTP request domain name http_domain domain Domain HTTP request method http_method method Method HTTP request body http_req_body body HttpBoday HTTP request header http_req_header header HttpHeader HTTP request parameters http_query_string query_sting QueryStr HTTP request cookies http_cookie cookies Cookie HTTP request referer http_referer referer Referer Proxy Address http_xff xff Xff HTTP request URL http_url url Url

[0217] As shown in Table 5-2, it's understood that the network protocol field is used to analyze and record detailed HTTP request information. For example, the HTTP request domain name refers to the server domain name used when initiating the request, such as www.example.com. In this embodiment, the domain field in Device A and the domain field in Device B, corresponding to the HTTP request domain name in Chinese, are mapped to the preconfigured standard English field http_domain.

[0218] HTTP request method: refers to the HTTP method used in the request, such as GET, POST, PUT, DELETE, etc. In this embodiment, the method field in device A and the Method field in device B corresponding to the HTTP request method in Chinese are mapped to the pre-configured standard English field http_method.

[0219] HTTP Request Body: This refers to the main content of the request, typically used for POST and PUT requests, and contains the data being sent. In this embodiment, the "body" field in Device A and the "HttpHeader" field in Device B, corresponding to the Chinese name of the HTTP Request Body, are mapped to the preconfigured standard English field http_req_body.

[0220] HTTP request header: This refers to the header information included in the request, such as User-Agent and Content-Type, which describes the request. In this embodiment, the fields named "header" in Device A and "HttpHeader" in Device B, which correspond to the HTTP request header in Chinese, are mapped to the preconfigured standard English field "http_req_header".

[0221] HTTP request parameters: These are parameters passed via the query string in a URL, typically used for GET requests, in the form of "key=value." In this embodiment, the fields named "query_sting" in Device A and "QueryStr" in Device B, corresponding to HTTP request parameters, are mapped to the preconfigured standard English field http_query_string.

[0222] HTTP request cookie: This refers to the cookie information carried in the request, used to maintain state between the client and the server. In this embodiment, the cookie in the device A field and the cookie in the device B field corresponding to the field named HTTP request cookie in Chinese are mapped to the pre-configured standard English field http_cookie.

[0223] HTTP request referer: refers to the source URL of the current request, usually indicating the page the user was redirected from. In this embodiment, the referer field in device A and the referer field in device B corresponding to the field named HTTP request referer in Chinese are mapped to the pre-configured standard English field http_referer.

[0224] Proxy address: refers to the server address used when initiating a request through a proxy server. In this embodiment, the xff in the device A field and the Xff in the device B field corresponding to the field Chinese name proxy address are mapped to the pre-configured standard English field http_xff.

[0225] HTTP request URL: refers to the complete request URL, including the protocol (such as http: / / or https: / / ), domain name, path, query parameters, etc. In this embodiment, the url field in device A and the Url field in device B, which correspond to the HTTP request URL in Chinese, are mapped to the pre-configured standard English field http_url.

[0226] In this embodiment, the destination IP address (d_ip) in the Device A field and the dest_ip in the Device B field are mapped to the preconfigured dip. The source port (s_port) in the Device A field and the dest_port in the Device B field are mapped to the preconfigured port. The destination port (d_port) in the Device A field and the dest_port in the Device B field are mapped to the preconfigured dport. The protocol (proto) in the Device A field and the protocol in the Device B field are mapped to the preconfigured protocol.

[0227] Regarding host class information in standard fields:

[0228] Host Information: Host information records key information related to the host, typically including the host name and operating system type. host_name: The host name, which represents the name of the computer or device. For example, the standard fields for host information can be configured as follows: os_type: The operating system type, which describes the operating system running on the host (e.g., Linux, Windows, etc.). host_ip: The IP address of the host.

[0229] For example, the mapping relationship between the original fields and the standard fields of the host information log in this embodiment is shown in Table 6-1:

[0230] Table 6-1 Mapping relationship between original fields and standard fields of log data to be processed for host information in the standard log template

[0231] Original fields Standard fields hostname host_name os_type os_type ip_address host_ip

[0232] As shown in Table 6-1, in this implementation, the original field hostname is mapped to the standard field host_name, and ip_address is mapped to host_ip.

[0233] In one embodiment, in another standard field template, the host type log field information may also be:

[0234] Table 6-2 Mapping relationship between original fields and standard fields of host-type log data to be processed in another standard log template

[0235] Field Chinese name Standard English fields Device A field Device B field Corporate ID enterprise_id e_id enterpriseID Host IP host_ip h_id hostID Host intranet IP host_internal_ip inter_ip interID Host external IP host_external_ip external_ip externalIP All IP addresses of the host host_ips ips Ips Host Manager host_charge_name charge_name chargeName Host person in charge's email address host_charge_email charge_email chargeEmail Host computer room location host_location location Location Host asset number host_asset_id assert_id assert_ID Host remarks host_remark remark Remark System version host_os_version os_ver osVer

[0236] In summary, the standard field set in this embodiment may include:

[0237] Quintuple information: including source IP (source_ip), destination IP (destination_ip), source port (source_port), destination port (destination_port), and protocol (protocol). It is mainly used to identify a unique network connection and help analyze the source, destination, and communication protocol of network traffic.

[0238] Process information: includes process ID (process_id), process name (process_name), and parent process ID (parent_process_id), which are used to track and analyze process activities.

[0239] Vulnerability information: This includes vulnerability ID (vulnerability_id), vulnerability name (vulnerability_name), severity level (severity_level), weak password type (vul_weak_passwd), whether the vulnerability allows local privilege escalation (vul_is_local_elevation), whether the vulnerability allows remote execution (vul_is_rce), whether the vulnerability has a proof of concept (vul_has_poc), applications affected (vul_impact_apps), recommended fixes (vul_fix_recommend), severity (vul_severity), whether the vulnerability has an exploit (vul_has_exp), whether it is a kernel vulnerability (vul_is_kernel), and recommended fixes (vul_weak_passwd). These standard log fields help identify potential security risks. These fields help accurately track and fix known vulnerabilities during vulnerability scanning and management, reducing the attack surface.

[0240] Threat information: This includes threat ID (threat_id), threat type (threat_type), attack pattern (attack_pattern), threat source (threat_source), threat severity (threat_severity), threat name (threat_name), attack threat level (threat_level), attack type (attack_type), attack result (attack_result), and threat information (threat_info). This information is primarily used for network security monitoring and defense. This standard field is crucial for detecting and responding to intrusion incidents. For example, identifying DDoS attacks, SQL injection attacks, or malware propagation allows for timely defensive measures.

[0241] Network protocol: including protocol type (protocol), IP protocol version (ip_version) and transport protocol (transport_protocol). The network protocol standard field records the protocol type used in the data transmission process, mainly including common protocols such as TCP, UDP, ICMP, HTTP, DNS, etc. Among them, the included standard field set can be HTTP request domain name (http_domain), HTTP request method (http_method), HTTP request body (http_req_body), HTTP request header (http_req_header), HTTP request parameters (http_query_string), HTTP request cookie (http_cookie), HTTP request referer (http_referer), proxy address (http_xff), HTTP request URL (http_url), to help analyze network communication status.

[0242] Host type: This includes the host name (host_name), operating system type (os_type), and host IP address (host_ip). This standard information helps analyze device characteristics on the network and determine whether a host may be compromised or exhibiting abnormal behavior. Host type log information can include the following standard fields: enterprise ID (enterprise_id), host IP (host_ip), host internal IP (host_internal_ip), host external IP (host_external_ip), host IP addresses (host_ips), host charge person (host_charge_name), host charge person email (host_charge_email), host computer room location (host_location), host asset ID (host_asset_id), host remark (host_remark), and system version (host_os_version).

[0243] This allows for the development of a standardized set of fields based on unified standards, with the Chinese and English names of these fields uniquely matched. By mapping these standard fields, the system can uniformly process logs from various sources, providing a consistent log format for more efficient data analysis, search, and correlation. Together, these standard field sets form a comprehensive security log analysis framework, enabling security analysts to more efficiently identify, analyze, and respond to security incidents.

[0244] In addition, for ordinary system logs, users can also add prefixes to construct standard fields. For example, for log information to be processed that is a JSON data structure used to describe information related to an email, the field name of the original data can be modified and prefixed with mail_ to construct a standard field.

[0245] Original data:

[0246]

[0247] The meaning of each field:

[0248] attachment_num: indicates the number of attachments in the email. Here, "0" means there is no attachment.

[0249] bcc: represents the Blind Carbon Copy (BCC) field of the email. If it is empty, it means there are no BCC recipients.

[0250] cc: represents the Carbon Copy field of the email. Here it is "xxxxx@xfusion.com", which means that the email is copied to xxxxx@xfusion.com.

[0251] device_name: indicates the name of the device that sends the email, here it is "mailgw242", which refers to the identifier of a mail gateway or server.

[0252] direction: Indicates the direction of the email. Here it is "incoming", which means that this email is in the "receiving" direction.

[0253] from_ditai: represents the sender address of the email, here it is "xxxxx@xprecision.net".

[0254] The converted data:

[0255]

[0256] Major changes:

[0257] attachment_num->mail_attachment_num: Added the mail_ prefix to make it clearer that the field is related to an email.

[0258] bcc->mail_bcc: Similarly, the mail_ prefix is added to indicate the private carbon copy information of the email.

[0259] cc->mail_cc: Adds the mail_ prefix to indicate the carbon copy information of the email.

[0260] direction->mail_direction: The mail_ prefix is added to indicate the direction of the mail.

[0261] from_ditai->mail_from_ditai: The mail_ prefix is added to indicate the sender address of the email.

[0262] This conversion code essentially adds the prefix "mail_" to the field names in the original data to make them more specific and clearly indicate their relevance to email. For example, "attachment_num" is changed to "mail_attachment_num" and "direction" is changed to "mail_direction". This avoids data conflicts with other fields (such as device or system configuration) while also improving readability and semantic clarity.

[0263] Similarly, in a scenario where the first log template does not have a mapping relationship between the original fields and the standard fields in the log data to be processed, the original fields in the log data to be processed can be obtained and converted into the corresponding Chinese names of the fields. The Chinese names of the fields are then translated into standard English fields. The standard English fields are the full English spelling of the Chinese names of the fields. The standard English fields are used as the standard fields of the original fields for mapping and conversion.

[0264] S104 : Map and convert the original fields of the log data to be processed according to the first log template to obtain target log data with standard fields.

[0265] In this step, Apache Data Processing and Distribution creates a data stream and uses the defined first log template to map and transform the log data to be processed. After the log standardization data stream is started, it will first initialize and read the standardized log templates from all first log templates in the data processing and distribution cluster. After initialization, multiple threads are started to periodically read log information from the distributed data stream processing cluster queue, select the corresponding first log template, and unify the log data to be processed into target log data with the same standard field.

[0266] Specifically, you can obtain the target log data by following the steps below:

[0267] S1041: Extract the field values of the original fields of the log data to be processed to form a field value set of the original fields.

[0268] For each log record to be processed, a regular expression is used to obtain the field values of the original fields of the log record to be processed, forming a set of original field values. The set of original field values includes key original fields in the log record, such as source IP address, destination IP address, source port number, and destination port number.

[0269] S1042: Map the field value set of the original field to the field value of the standard field in the standard log template.

[0270] Map the extracted field value set of the original field to the preset first log template. The standard log template in the first log template is a predefined, standardized structure that contains a set of field values for standard fields. Each standard field represents a specific type of log information, and different log sources use different standard field names.

[0271] S1043. According to the processing rules preset in the first log template, and based on the field values of the standard fields in the standard log template and the mapping relationship between the original fields of the log data to be processed and the standard fields, the field values of the original fields of the log data to be processed are converted into the field values of the standard fields in the standard log template.

[0272] Based on the pre-set standard field set in the standard log template, the fields in the original field set are converted to standard fields according to a one-to-one correspondence rule. Through mapping conversion, all log data is uniformly converted to pre-defined standard fields, making log data from various sources compatible and unified, facilitating subsequent processing and analysis. Standardized logs use unique standard fields in different log metadata, improving data accuracy, making it easier to manage, and allowing for faster and simpler searching and analysis. Automated correlation can be used during log retrieval.

[0273] It should be noted that the processing logic of steps S1041-S1043 can be implemented using a data processing and distribution cluster. Data processing and distribution allows users to write customized mapping logic using Groovy or Python scripts and integrate it into the data flow through the ExecuteScript processor, further enhancing the flexibility of automatic mapping.

[0274] S105: Store the target log data in a database.

[0275] In this step, after the fields of the log data to be processed are mapped and standardized, the target log data can be exported to other storage systems for storage or further processing. For example, the mapped data can be sent to a database such as HDFS (Hadoop distributed file system), distributed data stream processing, Elasticsearch, or other data storage systems.

[0276] For example, the mapped target log data (i.e., data containing standard fields) is stored in an object storage system (e.g., Amazon S3, OSS, etc.) or a database table using appropriate data processing and distribution storage processors (e.g., PutHDFS, PutDatabaseRecord, or PutObjectStorage) in a predefined standard field format. The data remains in the standard format during storage, facilitating subsequent analysis and querying.

[0277] In one embodiment, after step S101, the log data processing method provided in this embodiment may further include step S102. In other words, step S102 is an optional step, which is represented by a dotted line in the figure. It can be understood that JSON is a lightweight data exchange format that is commonly used for serialization and transmission of data. It uses concise text to represent structured data, is easy to read and write, and can be parsed and generated by a variety of programming languages. Therefore, the log data to be processed in step S103 is in JSON format. If the log format of the log data to be processed is not in JSON format, step S102 is executed.

[0278] S102 : performing log format conversion on the log data to be processed to obtain log data to be processed in a unified target log format.

[0279] Log files come in a variety of formats, including common syslog key / value, JSON, XML, CSV, and text files. Log data in different formats has different structures and parsing methods. When processing logs, it is necessary to accurately determine the log format in order to apply the parsing and mapping conversion methods provided in this embodiment. To achieve this goal, the following steps and technical solutions can be used:

[0280] S1021. Determine the file extension.

[0281] First, you can use the file extension (such as .json, .xml, .txt, .csv) to preliminarily determine the log format. Most file systems and logging systems determine the format type based on the log file extension. The following is the correspondence between common file extensions and format types:

[0282] .json → JSON format, .xml → XML format, .txt → plain text format, .csv → CSV format (comma separated values).

[0283] The correspondence between file suffix and format type is a quick and preliminary way to make a judgment, but it is not always accurate because the file content may not match the extension (for example, the file suffix has been changed, or the file format is mixed).

[0284] S1022: Parsing the log message content.

[0285] If the file suffix cannot fully determine the log format or there is confusion about the file suffix, you can further parse and determine the format based on the file content. Common parsing methods for different types of log content include JSON, XML, and text file parsing. The following is the specific judgment process:

[0286] JSON format determination: Use JsonSlurper (a JSON parsing library in Groovy) to parse the log content. If parsing succeeds, the file is in valid JSON format. JSON data is typically wrapped in {} and consists of key-value pairs, typically in the form {"key":"value"}.

[0287] XML format determination: Use XmlSlurper (Groovy's XML parsing library) to parse the file content. XML data is usually in the form of <tag>< / tag> The log file is in XML format and contains nested tags and attributes. If the file can be successfully parsed into XML, it means that the log file is in XML format.

[0288] Text format determination: If the file content cannot be successfully parsed using a JSON or XML parser, the file is likely in plain text format. In this case, you can simply assume it is a plain text log, or infer whether the data is in a format such as CSV or TSV by using specific delimiters in the content.

[0289] S1023. Log format conversion.

[0290] Once the log format (JSON, XML, etc.) is determined, the next step is to convert the data. In this embodiment, the ConvertRecord or ReplaceText processors used in data processing and distribution are used to perform this operation. Specifically, the regular expression defined in the ReplaceText processor is used to replace the log data in different log formats and convert them to JSON format.

[0291] In this step, after the log data to be processed is converted into JSON format, it is not only convenient for developers and operation and maintenance personnel to view and understand it, but also can be easily parsed and mapped through the data processing and distribution cluster.

[0292] Next, a data processing method provided in a specific embodiment of the present application is described with reference to the accompanying drawings.

[0293] Example 1

[0294] For example, Figure 4 FIG. 1 shows a flow chart of another log data processing method provided by an embodiment of the present application. Figure 4 As shown, the method includes:

[0295] Step 1: Create a standard field set according to unified standards, and ensure that the Chinese and English names of the fields are unique.

[0296] The standard field set includes log information such as quintuple information, network protocol, host type, vulnerability type, and threat type;

[0297] Step 2: Based on the characteristics of the original log fields, a standard log template is developed, and a mapping relationship between the original fields and the standard fields is created. The system sends the defined field mapping relationship to the Apache data processing and distribution cluster through the data processing and distribution API.

[0298] Step 3: Use distributed data stream processing to receive security device logs

[0299] (1) Apache distributed data stream processing cluster is used to accept multi-source heterogeneous logs. The Apache data processing and distribution cluster background reads logs from Kafka.

[0300] (2) Determine the log format. If the log is in XML or Syslog key / value format, convert it to Json. If the log format is Json, do not process it.

[0301] (3) Send Json logs to the Apache data processing and distribution cluster.

[0302] Step 4: Use Apache Data Processing and Distribution to create a data stream, use the defined log template to perform mapping transformations, and write the standardized logs to a database (such as Elasticsearch).

[0303] Standardizing log data simplifies subsequent data management and analysis, improving work efficiency. Using standardized fields makes it easy to perform correlation queries and analyze behaviors related to specific IP addresses. For example, you can easily perform correlation queries by searching standardized fields. For example, if you configure the standard fields "Source IP" and "Destination IP" to "sip" and "dip," you can search for a specific IP address and see if it matches the "sip" and "dip" values in all standard fields, allowing you to analyze related behaviors.

[0304] Specifically, source IP and destination IP have different names in different logs (such as source_ip, destination_ip, etc.), but through standardization, they are unified into standard field names, such as sip (source IP) and dip (destination IP). By querying these standardized fields, you can find relevant logs under specific conditions. For example, you can query whether a specific IP address appears as the source IP (sip) or destination IP (dip) in multiple log entries. Search whether a specific IP address matches all the standard sip and dip values: Suppose you want to find whether a certain IP (such as 192.168.1.1) appears as the source IP or destination IP in the log. By querying the source_ip (sip) or destination_ip (dip) fields, you can quickly find all related behaviors for this IP. Correlate and analyze related behaviors: By finding all logs related to a specific IP, analyze its behavior patterns. For example, whether a certain IP has initiated multiple requests as the source IP or the frequency of being accessed as the destination IP, these behaviors can help analyze security incidents or network traffic anomalies.

[0305] For example, if you want to find out whether 192.168.1.100 appears as the source IP or destination IP, you can query using the standardized fields:

[0306] SELECT * FROM logs

[0307] WHERE sip='192.168.1.100'OR dip='192.168.1.100';

[0308] Then, you can view the interaction between the IP address and other devices to see if there are any suspicious behavior patterns, such as attacks, abnormal access, etc. In this way, by using standardized fields (such as sip and dip) for query and correlation analysis, you can quickly find all log records related to a specific IP address and analyze the behavior of the IP address or network security events.

[0309] It is understandable that the size of the serial number of each step in the above-mentioned embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in some possible implementations, the steps in the above-mentioned embodiments can be selectively executed according to actual conditions, and can be partially executed or fully executed, which is not limited here. In addition, all or part of any features of any of the above-mentioned embodiments can be freely and arbitrarily combined without contradiction; the combined technical solutions are also within the scope of this application.

[0310] like Figure 5 As shown, the embodiment of the present application also provides a log data processing device 500, including: an acquisition module 501 and a processing module 502. The acquisition module 501 is used to acquire at least one log data to be processed; the processing module 502 is used to match the first log template of the log data to be processed based on the original field characteristics of the log data to be processed; the original field characteristics indicate the field values of each original field in the log data to be processed; the first log template includes processing rules for the log data to be processed, and the processing rules are used to convert the field values of multiple original fields in the log data to be processed into multiple preset standard fields; the original fields of the log data to be processed are mapped and converted according to the first log template to obtain target log data with standard fields; the first log template also includes a pre-configured standard log template and a mapping relationship between the original fields of the log data to be processed and the standard fields, and the standard log template includes a set of field values of the preset standard fields of the log data to be processed; the target log data is stored in a database.

[0311] In a possible implementation, the processing module 502 performs log format conversion on the log data to be processed to obtain log data to be processed in a unified target log format.

[0312] In a possible implementation, the processing module 502 converts the log format of the log data to be processed by using a regular expression.

[0313] In one possible implementation, the first log template also includes a preconfigured standard log template and a mapping relationship between the original fields of the log data to be processed and the standard fields, the standard log template including a set of field values of preset standard fields of the log data to be processed; wherein, the first log template is preconfigured based on the second log template, and the second log template includes a set of field values of multiple original fields when the log data to be processed is used as a sample log; the processing module 502 matches the first log template of the log data to be processed based on the original field characteristics of the log data to be processed, and is specifically used to: parse the log data to be processed to obtain field values of multiple original fields of the log data to be processed, match the field values of the original fields of the log data to be processed with the field values of the original fields of the sample logs in each first log template, and determine that the field values of the original fields of the log data to be processed are the same as the field values of the original fields of the sample logs in the standard log template; when the field values of the original fields of the log data to be processed are the same as the field values of the original fields of the sample logs in the standard log template, determine the first log template of the log data to be processed.

[0314] In one possible implementation, the processing module 502 maps and converts the original fields of the log data to be processed based on the first log template, specifically for: extracting the field values of the original fields of the log data to be processed to form a field value set of the original fields; mapping the original field set to the standard fields in the standard log template; according to the processing rules preset in the first log template, and based on the field values of the standard fields in the standard log template and the mapping relationship between the original fields of the log data to be processed and the standard fields, converting the field values of the original fields of the log data to be processed into the field values of the standard fields in the standard log template.

[0315] It should be understood that both the communication module 501 and the processing module 502 can be implemented in software or hardware. For example, the implementation of the communication module 501 will be described below using the communication module 501 as an example. Similarly, the implementation of the processing module 502 can refer to the implementation of the communication module 501.

[0316] As an example of a software functional unit, the communication module 501 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the communication module 501 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0317] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0318] As an example of a hardware functional unit, communication module 501 may include at least one computing device, such as a server. Alternatively, communication module 501 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0319] The multiple computing devices included in the communication module 501 can be distributed in the same region or in different regions. The multiple computing devices included in the communication module 501 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the communication module 501 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0320] The device can convert log data of different formats into a unified standard format for further analysis and processing. The log standardization device can be connected to a variety of security devices, including but not limited to: firewalls, Web application protection systems (WAF), terminal detection and response (EDR), network detection and response (NDR), intrusion prevention systems (IPS) anti-virus software and other network security devices. The log data generated by these devices usually have different formats and structures. The log data processing device provided by the embodiment of the present application can convert these heterogeneous data into a unified standard format. For example: a large operator has WAF equipment from multiple manufacturers. Through log standardization, all WAF logs can be stored in a table after standardized processing, which is convenient for unified management and query.

[0321] Based on the same inventive concept, the principles and beneficial effects of solving the problems provided by the server in the embodiments of the present application can be referred to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0322] like Figure 6 As shown, the embodiment of the present application further provides a computing device 600. For example, the computing device 600 can be a server or a terminal. When the computing device 600 is running, the computing device 600 can execute the method in the above embodiment.

[0323] The computing device includes a bus 601, a processor 602, a memory 603, and a communication interface 604. The processor 602, the memory 603, and the communication interface 604 communicate with each other via the bus 601. The computing device 600 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 600.

[0324] Bus 601 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, the figure shows only one line, but this does not imply a single bus or type of bus. Bus 601 may include a path for transmitting information between various components of computing device 600 (e.g., processor 602, memory 603, and communication interface 604).

[0325] The processor 602 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0326] The memory 603 may include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0327] The memory 603 stores executable program instructions, and the processor 102 executes the executable program instructions to implement the test methods involved in the above embodiments.

[0328] The communication interface 604 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 600 and other devices or a communication network.

[0329] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0330] like Figure 7 As shown, the computing device cluster includes at least one computing device 600A. The memory 603 in one or more computing devices 600A in the computing device cluster may store the same instructions for executing the operation log recording method.

[0331] In some possible implementations, the memory 603 of one or more computing devices 600A in the computing device cluster may also store partial instructions for executing the operation logging method. In other words, the combination of one or more computing devices 600A can jointly execute the instructions for executing the operation logging method.

[0332] It should be noted that the memory 603 in different computing devices 600A in the computing device cluster can store different instructions, each for executing part of the functions of the above-mentioned multiple modules. In other words, the instructions stored in the memory 603 in different computing devices 600A can implement the functions of one or more modules in the above-mentioned multiple modules.

[0333] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 6 A possible implementation is shown. Figure 6 As shown, two computing devices, computing device 600A and computing device 600B, are connected via a network. Specifically, the connection to the network is achieved through a communication interface within each computing device. In this possible implementation, memory 603 within computing device 600A stores instructions for executing the functions of some of the aforementioned modules. Simultaneously, memory 603 within computing device 600B stores instructions for executing the functions of another portion of the aforementioned modules.

[0334] Figure 8 The connection method between the computing device clusters shown may be based on the consideration that the log data processing method provided in this application requires a large amount of data storage, and therefore the functions implemented by another part of the above-mentioned multiple modules may be handed over to the computing device 600B for execution.

[0335] It should be understood that Figure 8 The functionality of computing device 600A shown in FIG. 6 may also be implemented by multiple computing devices 600. Similarly, the functionality of computing device 600B may also be implemented by multiple computing devices 600.

[0336] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similarly referred to as Figure 7 and Figure 8 The connection mode of the computing device cluster is different in that the memory 603 of one or more computing devices 600 in the computing device cluster may store the same instructions for executing the log data processing method.

[0337] In some possible implementations, the memory 603 of one or more computing devices 600 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 600 can jointly execute the instructions for executing the data processing method.

[0338] It should be noted that the memory 603 in different computing devices 600 in the computing device cluster may store different instructions for executing part of the functions of the computing device 600. That is, the instructions stored in the memory 603 in different computing devices 600 may implement the functions of one or more of the multiple modules described above.

[0339] The present application also provides a computer-readable storage medium. The computer-readable storage medium is used to store computer program instructions. When the computer program instructions are executed on a computing device, the computing device executes the data processing method involved in the above embodiment. The computer-readable storage medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0340] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the data processing method.

[0341] Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present application. Those skilled in the art should understand that, although the present application has been described in detail with reference to the aforementioned embodiments, the technical solutions described in the aforementioned embodiments may be modified or some of the technical features thereof may be replaced with equivalents. However, such modifications or replacements do not deviate from the spirit and scope of the technical solutions in the various embodiments of the present application.

Claims

1. A log data processing method, characterized in that: include: Obtain at least one log data to be processed; Based on the original field characteristics of the log data to be processed, matching the first log template of the log data to be processed; wherein the original field characteristics indicate the field values of each original field in the log data to be processed; the first log template includes processing rules for the log data to be processed, and the processing rules are used to convert the field values of multiple original fields in the log data to be processed into multiple preset standard fields; Mapping and converting the original fields of the log data to be processed according to the first log template to obtain target log data with standard fields; wherein the first log template also includes a pre-configured standard log template and a mapping relationship between the original fields of the log data to be processed and the standard fields, and the standard log template includes a set of field values of preset standard fields of the log data to be processed; The target log data is stored in a database.

2. The method according to claim 1, characterized in that After obtaining at least one log data to be processed, the method further includes: Performing log format conversion on the log data to be processed to obtain log data to be processed in a unified target log format, wherein the target log format is a JSON log format.

3. The method according to claim 2, characterized in that The log format of the log data to be processed is converted using a regular expression.

4. The method according to any one of claims 1 to 3, characterized in that The matching the first log template of the log data to be processed based on the original field characteristics of the log data to be processed includes: Parsing the log data to be processed to obtain field values of multiple original fields of the log data to be processed, matching the field values of the original fields of the log data to be processed with the field values of the original fields in each of the first log templates, and determining that the field values of the original fields of the log data to be processed are the same as the field values of the original fields in the standard log template; In a case where the field value of the original field of the log data to be processed is the same as the field value of the original field in the standard log template, a first log template for the log data to be processed is determined.

5. The method according to claim 4, characterized in that The mapping and converting the original fields of the log data to be processed according to the first log template includes: Extracting the field values of the original fields of the log data to be processed to form a field value set of the original fields; According to the processing rules preset in the first log template, and based on the field values of the standard fields in the standard log template and the mapping relationship between the original fields and the standard fields of the log data to be processed, the field values of the original fields of the log data to be processed are converted into the field values of the standard fields in the standard log template.

6. The method according to claim 5, characterized in that The standard field set includes any one or more of the standard fields of quintuple information, network protocol, host type, vulnerability type, and threat type log information.

7. The method according to any one of claims 1 to 6, characterized in that The obtaining of at least one log data to be processed includes: At least one log data to be processed is collected from the source end system according to the distributed data stream processing cluster, and at least one log data to be processed is obtained from the distributed data stream processing cluster through any data processing and distribution node in the data processing and distribution cluster.

8. The method according to claim 7, characterized in that The source end system includes any one of a firewall, a Web application protection system, a terminal detection and response system, a network detection and response system, an intrusion prevention system and an antivirus software.

9. The method according to claim 7, characterized in that The mapping and converting of the original fields of the log data to be processed according to the standard log template to obtain target log data with standard fields includes: Creating a data stream based on the data processing and distribution cluster, wherein the data stream is used to process the to-be-processed log data obtained from the distributed data stream processing cluster based on a first log template; The data processing and distribution cluster parses the field values of the original fields of the log data to be processed based on the mapping relationship between the original fields of the log data to be processed and the predefined standard fields in the first log template, and maps the field values of the original fields of the log data to be processed to the field values of the predefined standard fields to obtain target log data with standard fields.

10. A computing device, characterized in that include: at least one memory for storing a program; at least one processor, configured to execute the program stored in the memory; The memory is coupled to the processor, and when the program stored in the memory is executed, the processor is configured to execute the method according to any one of claims 1 to 9.

Images