A method and device for detecting and backtracking compromised hosts based on mass network logs

Abstract

The application discloses a kind of based on mass network log's lost host detection and backtracking method and device, belong to network security technical field.The method includes: constructing log library based on columnar storage engine;Network log data is obtained and parsed, and parsed network log data is stored to the log library;External threat intelligence data is obtained, and full intelligence library and incremental intelligence library are respectively constructed;The log data to be detected is divided into multiple segmented data according to time, and each segmented data is respectively associated with full intelligence library and incremental intelligence library and is detected, and intelligence hit result is obtained;Based on the intelligence hit result, using the copy mechanism of columnar storage engine, historical data in the log library is associated backtracking, and historical association data is obtained;The historical association data is aggregated and analyzed, and the list of lost host is determined.The application is combined with full and incremental detection mechanism, to reduce the false negative rate.

Description

Technical Field

[0001] This invention belongs to the field of network security technology, and in particular relates to a method and apparatus for detecting and tracing compromised hosts based on massive network logs. Background Technology

[0002] Currently, common network log detection methods mostly employ real-time stream processing architectures or traditional relational databases for intelligence matching; however, these methods have some shortcomings in practical applications:

[0003] (1) When faced with extremely large files or high-throughput log data, streaming systems often have performance bottlenecks and cannot meet the real-time detection requirements.

[0004] (2) Most solutions only perform online detection on logs and do not store the original data, which makes it impossible to trace back the historical records of alarm events and restore the full picture of the attack;

[0005] (3) As the size of the intelligence database increases, the complexity of related queries increases sharply, further exacerbating performance issues, which is particularly prominent in log scenarios. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method and device for detecting and tracing compromised hosts based on massive network logs. Through a detection mechanism that combines full and incremental data, it can not only ensure the timeliness of the intelligence database, but also promptly discover historical behaviors associated with newly emerging threats, thereby reducing the false negative rate.

[0007] This invention is achieved through the following technical solution:

[0008] The first aspect of this invention discloses a method for detecting and tracing compromised hosts based on massive network logs, comprising:

[0009] Build a log library based on a columnar storage engine;

[0010] Acquire and parse network log data, and store the parsed network log data in the log database;

[0011] Acquire external threat intelligence data and build a full intelligence database and an incremental intelligence database respectively;

[0012] The log data to be detected is divided into multiple segments according to time, and each segment is correlated with the full intelligence database and the incremental intelligence database to obtain intelligence hit results.

[0013] Based on the intelligence hit results, the replication mechanism of the columnar storage engine is used to perform correlation and backtracking on the historical data in the log database to obtain historical correlation data, and the historical correlation data is stored in the hit wide table;

[0014] The historical correlation data is aggregated and analyzed to determine the list of compromised hosts.

[0015] Furthermore, a log library based on a columnar storage engine is constructed, including: using Clickhouse as the columnar storage engine, creating storage tables according to time periods, and the storage tables constituting the log library;

[0016] The log database stores:

[0017] The communication log table is used to store communication logs, which includes device identifier, source IP address, destination IP address, source port, destination port, protocol type, number of bytes sent by the client, number of bytes sent by the server, total number of bytes, source IP country code, source IP operator, destination IP country code, destination IP operator, and extended tags;

[0018] The DNS log table is used to store DNS logs, which include event time, entry time, device identifier, source file identifier, query domain name, IP address returned by resolution, record type, response code, and time to live.

[0019] The SSL log table is used to store SSL logs, which include event time, entry time, device identifier, source file identifier, source IP address, destination IP address, server name in the SSL certificate, TLS client fingerprint, TLS server fingerprint, certificate serial number, and certificate issuer.

[0020] Furthermore, the full intelligence database includes a full IP intelligence database and a full domain name intelligence database, and the incremental intelligence database includes an incremental IP intelligence database and an incremental domain name intelligence database.

[0021] The full IP intelligence database and the incremental IP intelligence database constitute the IP intelligence database, and the full domain name intelligence database and the incremental domain name intelligence database constitute the domain name intelligence database.

[0022] Furthermore, the association detection includes:

[0023] Perform a left join query between the IP intelligence database and the communication relationship log table. The join condition is that the source IP address or destination IP address in the communication relationship log table is equal to the intelligence IP address in the IP intelligence database.

[0024] Perform a left join query between the IP intelligence database and the SSL log table, with the join condition being that the source IP address or destination IP address in the SSL log table is equal to the intelligence IP address.

[0025] Perform a left join query between the domain name intelligence database and the DNS log table, with the join condition being that the queried domain name in the DNS log table equals the intelligence domain name in the domain name intelligence database.

[0026] Perform a left join query between the domain name intelligence database and the SSL log table, with the join condition being that the server name in the SSL log table equals the intelligence domain name.

[0027] Furthermore, the step of dividing the log data to be detected into multiple segments by time includes a task scheduling and management step:

[0028] Obtain task configuration information from the configuration center, the task configuration information including timing expressions and task status;

[0029] Determine whether to create a scheduled task based on the task status. If the task status is enabled, create a scheduled task according to the scheduled expression. If the task status is disabled, delete the created scheduled task.

[0030] When a scheduled task is triggered, the scheduling service is invoked through the scheduled task scheduling framework. The scheduling service obtains the corresponding historical task execution records based on the task configuration information.

[0031] The start and end times of the data query for this detection are determined based on the historical task execution records to form a time segment: if the historical task execution records are empty, the start time is set to a preset historical time point, and the end time is set to the start time plus a preset time interval; if the historical task execution records are not empty, the start time is set to the end time of the previous task, and the end time is set to the start time plus the preset time interval.

[0032] After the detection is completed, the historical task execution record is updated and saved to determine the start time of the next segment of data.

[0033] Furthermore, the correlation detection includes a full-scale intelligence detection step: initializing data processing records; starting the detection process at the first preset time point each day, and repeating the following steps until the current time exceeds the second preset time point:

[0034] Determine if the current time is greater than the sum of the last data processing time and the data processing interval. If so, create a new detection task, using the end time of the previous task as the start time of the current task, and the sum of the start time and the data processing interval as the end time of the current task. Perform a correlation query between the full intelligence database and the network log data within the range of the start time to the end time; record the task execution result, and update the last data processing time to the end time of the current task.

[0035] Furthermore, the correlation detection includes an incremental intelligence detection step: initializing data processing records; starting the detection process at a first preset time point each day, and repeating the following steps until the current time exceeds a second preset time point:

[0036] Determine if the current time is greater than the sum of the last data processing time and the data processing interval. If so, create a new backtracking task, using the end time of the previous task as the start time of the current task, and the sum of the start time and the data processing interval as the end time of the current task. Perform a correlation query between the incremental intelligence database and historical network log data within the range of the start time to the end time; record the task execution result, and update the last data processing time to the end time of the current task.

[0037] Furthermore, the historical data in the log database is correlated and traced back, including:

[0038] The IP intelligence hit result is joined with the DNS log table. The joining conditions are that the resolved IP address in the DNS log table is equal to the intelligence IP address in the intelligence hit result, and the device identifier is consistent.

[0039] The domain name intelligence hit result is connected to the communication relationship log table. The connection condition is that the source IP address or destination IP address in the communication relationship log table is equal to the resolved IP address in the domain name intelligence hit result, and the device identifier is consistent.

[0040] The domain name intelligence hit result is linked to the SSL log table. The connection conditions are that the source IP address or destination IP address in the SSL log table is equal to the resolved IP address in the domain name intelligence hit result, and the device identifier is consistent.

[0041] Furthermore, the historical correlation data is aggregated and analyzed to determine the list of compromised hosts, including:

[0042] Based on the communication relationship logs obtained from the IP intelligence hit results, the communication behavior of the hit host is characterized. If any of the following is identified: communication behavior based on abnormal port, communication behavior based on abnormal communication frequency, communication behavior based on packet size, communication behavior based on session duration, or communication behavior based on the number of connection attempts, then the host is included in the list of compromised hosts.

[0043] Based on the SSL logs obtained from the domain name intelligence hit results, the encrypted communication characteristics of the hit host are analyzed. If any of the following is identified: TLS client fingerprint matching, TLS server fingerprint matching, or abnormal certificate information, the host is included in the list of compromised hosts.

[0044] in,

[0045] The communication behavior based on port anomalies refers to the host initiating a connection to an external IP within a unit of time, with the destination port not belonging to the preset standard port set, or using a preset uncommon port for encrypted transmission, and the number of connections exceeding a first preset threshold.

[0046] The communication behavior based on communication frequency anomalies refers to the host continuously engaging in periodic communication with external IPs during preset non-working periods, with traffic below a preset threshold.

[0047] The communication behavior based on data packet size refers to the fact that the size of the uplink data packet for each interaction between the host and the host is less than the preset upper limit in bytes.

[0048] The communication behavior based on session duration refers to the duration of each host connection being less than a preset duration threshold.

[0049] The communication behavior based on the number of connection attempts refers to the host initiating connection attempts to multiple ports or the same ports of other external or internal hosts within a unit of time, and the number of connection attempts exceeds the second preset threshold.

[0050] The TLS client fingerprint matching refers to matching the TLS client fingerprint in the SSL log with a preset malicious fingerprint database.

[0051] The TLS server fingerprint matching refers to matching the TLS server fingerprint in the SSL log with a preset malicious fingerprint database.

[0052] The certificate information anomaly refers to a certificate that is self-signed, a certificate issuer that does not belong to the preset set of trusted certificate authorities, or a certificate whose generic name does not match the name of the accessed server.

[0053] A second aspect of the present invention discloses a compromised host detection and backtracking device based on massive network logs, comprising:

[0054] The log library building module is used to build a log library based on a columnar storage engine;

[0055] The log storage module is used to acquire and parse network log data, and store the parsed network log data in the log library;

[0056] The intelligence database construction module is used to acquire external threat intelligence data and build a full intelligence database and an incremental intelligence database respectively.

[0057] The correlation detection module is used to divide the log data to be detected into multiple segments according to time, and perform correlation detection on each segment with the full intelligence database and the incremental intelligence database to obtain intelligence hit results;

[0058] The backtracking module is used to perform correlation backtracking on historical data in the log database based on the intelligence hit results and using the replication mechanism of the columnar storage engine to obtain historical correlation data, and store the historical correlation data in the hit wide table;

[0059] The aggregation analysis module is used to perform aggregation analysis on the historical correlation data to determine the list of compromised hosts.

[0060] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0061] (1) The present invention builds a log library based on a columnar storage engine, which greatly improves the efficiency of writing and querying log data; at the same time, it creates storage tables and partitions storage according to time periods, optimizes from the data management granularity level, and overcomes the performance bottleneck of traditional relational databases or streaming processing architectures when processing ultra-large log files.

[0062] (2) The present invention divides the network log data to be detected into multiple segments according to time, and each segment is independently associated with the intelligence database. This can effectively avoid the memory overflow problem caused by the full association of massive data. The size of the segment data can be dynamically adjusted through the configuration center, which can adaptively optimize according to the real-time data volume changes, ensuring that the detection task runs stably and efficiently under limited server resources, reducing the occupation of computing resources, and supporting a larger scale of log detection under the same hardware conditions.

[0063] (3) The present invention constructs a full intelligence database and an incremental intelligence database respectively. Through the detection mechanism that combines full and incremental data, it can not only ensure the timeliness of the intelligence database, but also promptly discover historical behaviors associated with newly emerging threats, effectively reducing the false negative rate. At the same time, the left join query method is used to accurately match intelligence and logs, reducing false alarms and improving the reliability of detection results.

[0064] (4) This invention utilizes the replication mechanism of the columnar storage engine to perform backtracking analysis in parallel without affecting the online query service. By associating the IP intelligence hit results with the DNS logs and the domain name intelligence hit results with the communication logs and SSL logs, it is possible to quickly trace the complete historical behavior chain related to malicious intelligence. The backtracked data is integrated and stored in the hit wide table for aggregation analysis, which can realize high-performance multi-dimensional correlation analysis and improve the efficiency of security analysis. Attached Figure Description

[0065] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0066] Figure 1 This is a flowchart of a compromised host detection and backtracking method in this invention;

[0067] Figure 2 This is a flowchart of the full-scale intelligence detection steps in this invention;

[0068] Figure 3This is a flowchart of one step in the incremental intelligence detection process of this invention;

[0069] Figure 4 This is a block diagram of one component of the compromised host detection and backtracking device in this invention. Detailed Implementation

[0070] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0071] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0072] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "inner," and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the accompanying drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0073] In the description of this invention, unless otherwise explicitly specified and limited, the term "connection" or similar designation indicating a connection between components should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0074] like Figures 1 to 4 As shown in the figure, this embodiment discloses a method and apparatus for detecting and tracing compromised hosts based on massive network logs.

[0075] The first aspect of this embodiment discloses a method for detecting and tracing compromised hosts based on massive network logs, such as... Figure 1 As shown, the method includes steps S100 to S600.

[0076] Step S100. Build a log library based on a columnar storage engine.

[0077] In some embodiments of this example, the columnar storage engine is Clickhouse. Clickhouse has excellent data compression and query performance, making it suitable for storing and analyzing massive amounts of log data.

[0078] In some implementations of this embodiment, constructing a log library based on a columnar storage engine includes: using Clickhouse as the columnar storage engine, creating storage tables according to time periods, and the storage tables constituting the log library.

[0079] In these implementations, creating storage tables according to time periods allows for the storage of network log data in different time intervals, thereby improving data management and efficient retrieval, and facilitating subsequent detection and backtracking based on time ranges.

[0080] Step S200. Obtain and parse the network log data, and store the parsed network log data in the log database.

[0081] In some implementations of this embodiment, acquiring and parsing network log data, and storing the parsed network log data in the log database includes: reading network log data from a data source (e.g., an SFTP server), parsing the network log data, extracting information from each field, adding geographical location tags (such as country codes and operator information) to IP addresses, and writing the parsed network log data into the corresponding storage table in Clickhouse.

[0082] In some embodiments of this example, the network log data includes communication logs, DNS logs, and SSL logs.

[0083] The network log data may also include HTTP logs, TCP logs, UDP logs, certificate logs, etc. The type of network log data is determined according to actual needs.

[0084] In some embodiments of this example, the log database stores a connectivity log table, a DNS log table, and an SSL log table. The connectivity log table stores connectivity logs, recording basic information about network communication sessions; the DNS log table stores DNS logs, recording domain name resolution requests and responses; and the SSL log table stores SSL logs, recording SSL / TLS handshake information.

[0085] The communication log table includes device identifier (device_id), source IP address (src_ip), destination IP address (dst_ip), source port (src_port), destination port (dst_port), protocol type (such as TCP / UDP / ICMP), number of bytes sent by the client (client_bytes), number of bytes sent by the server (server_bytes), total number of bytes (total_bytes, which can be calculated from client_bytes + server_bytes), source IP country code (src_country_code), source IP operator (src_isp), destination IP country code (dst_country_code), destination IP operator (dst_isp), and extended tags (tags).

[0086] The DNS log table includes event time (event_time), ingest time (ingest_time), device identifier (device_id), source file identifier (file_id), query domain (query_domain), IP address returned by resolution (answer_ip), record type (answer_type, such as A, AAAA, CNAME), response code (rcode), and time to live (ttl).

[0087] The SSL log table includes event time (event_time), entry time (ingest_time), device identifier (device_id), source file identifier (file_id), source IP address (src_ip), destination IP address (dst_ip), server name (server_name, i.e., SNI) in the SSL certificate, TLS client fingerprint (ja3), TLS server fingerprint (ja3s), certificate serial number (cert_serial), and certificate issuer (cert_issuer).

[0088] The structures of the connectivity log table, DNS log table, and SSL log table can be adjusted according to actual needs, such as adding indexes or partition keys to optimize query performance.

[0089] Step S300. Obtain external threat intelligence data and construct a full intelligence database and an incremental intelligence database respectively.

[0090] This embodiment acquires external threat intelligence data through both full and incremental methods. The external intelligence interface provides full and incremental intelligence acquisition methods, and intelligence files are acquired daily through encrypted and compressed transmission, with full coverage every day to ensure the timeliness and completeness of the intelligence.

[0091] The incremental intelligence database is used to retrospectively examine historical data, which can effectively detect newly emerging threats.

[0092] In some embodiments of this example, the full intelligence database includes a full IP intelligence database and a full domain name intelligence database.

[0093] The full IP intelligence database is used to store intelligence on all known malicious IPs. The full IP intelligence database includes the following fields: IP address (ip), attacking organization (apt_org), tags (tags), threat type (judgements), threat value (threat_value), reliability (reliability), source (source), extended source information (extends_source), remark (remark), intelligence information (info), record time (record_time), and creation time (create_time).

[0094] The full domain intelligence database is used to store intelligence on all known malicious domains. The full domain intelligence database includes the following fields: domain, attacking organization (apt_org), tags, threat type (judgements), threat value (threat_value), reliability, source (source), extended source information (extends_source), remark, intelligence information (info), record time (record_time), and create time (create_time).

[0095] In some embodiments of this example, the incremental intelligence database includes an incremental IP intelligence database and an incremental domain name intelligence database, and the incremental intelligence database is used to store newly added intelligence data every day.

[0096] The incremental IP intelligence database only stores newly added data daily, and its table structure is the same as the full IP intelligence database. For example, the incremental IP intelligence database includes the following fields: IP address (ip), attacking organization (apt_org), tags (tags), threat type (judgements), threat value (threat_value), reliability (reliability), source (source), source extension information (extends_source), remark (remark), intelligence information (info), record time (record_time), and entry time (create_time).

[0097] The incremental domain intelligence database only stores newly added data daily, and its table structure is the same as that of the full domain intelligence database. For example, the incremental domain intelligence database includes the following fields: domain, attacking organization, tags, judgments, threat value, reliability, source, extended source information, remark, intelligence information, record time, and create time.

[0098] The full IP intelligence database and the incremental IP intelligence database constitute the IP intelligence database, and the full domain name intelligence database and the incremental domain name intelligence database constitute the domain name intelligence database.

[0099] Step S400. Divide the log data to be detected into multiple segments according to time, and perform correlation detection on each segment with the full intelligence database and the incremental intelligence database to obtain intelligence hit results.

[0100] In some implementations of this embodiment, the association detection includes: performing a left join query between the IP intelligence database and the communication relationship log table, wherein the join condition is that the source IP address or destination IP address in the communication relationship log table is equal to the intelligence IP address in the IP intelligence database; performing a left join query between the IP intelligence database and the SSL log table, wherein the join condition is that the source IP address or destination IP address in the SSL log table is equal to the intelligence IP address; performing a left join query between the domain name intelligence database and the DNS log table, wherein the join condition is that the queried domain name in the DNS log table is equal to the intelligence domain name in the domain name intelligence database; and performing a left join query between the domain name intelligence database and the SSL log table, wherein the join condition is that the server name in the SSL log table is equal to the intelligence domain name.

[0101] In some implementations of this embodiment, the size of the segmented data is dynamically adjusted by a configuration center, and the execution of the detection task is controlled by a timer scheduler.

[0102] Since performing join queries on massive logs (such as hundreds of millions of logs) can lead to data memory overflow, this embodiment adds a scheduling strategy to effectively control data overload. Specifically, this embodiment uses Nacos and a timer scheduler to achieve effective model task scheduling, thereby enabling efficient processing of massive amounts of data.

[0103] In some embodiments of this example, dividing the log data to be detected into multiple segments by time includes a task scheduling and management step, specifically steps S410 to S450.

[0104] Step S410. Obtain task configuration information from the configuration center, the task configuration information including timing expressions and task status.

[0105] For example, read a predefined task configuration file (such as consumer-preset.yml) from a configuration center (such as Nacos) to obtain task configuration information (TaskConfig).

[0106] The task configuration information includes: task identifier (id), model identifier (modelId), task status (status), task name (name), cron expression (cron), time interval (intervalTime), source data table name (sourceTableName), target result table name (targetTableName), hit type (hitType), and task type (type, where 1 indicates log detection and 2 indicates intelligence addition detection). This configuration information defines the execution rules for each detection task.

[0107] Step S420. Determine whether to create a scheduled task based on the task status. If the task status is enabled, create a scheduled task according to the scheduled expression. If the task status is disabled, delete the created scheduled task.

[0108] Specifically, the creation of a scheduled task is determined based on the status in the task configuration information. If the status is "1" (enabled), a scheduled task is created according to the timing expression; if the status is "0" (disabled), the created scheduled task is deleted. Batch operations such as "start all tasks" are also supported. The output of this step is a set of scheduled tasks triggered according to the timing expression, with each scheduled task corresponding to a detection model.

[0109] Step S430. When a scheduled task is triggered, the scheduling service is invoked through the scheduled task scheduling framework. The scheduling service obtains the corresponding historical task execution records based on the task configuration information.

[0110] Specifically, when a scheduled task reaches its trigger time, a scheduling service (e.g., DispatchServiceImpl) is invoked through a scheduled task scheduling framework (e.g., QuartzJobFactory) as the execution entry point. The scheduling service retrieves the corresponding historical task execution record (TaskExecuteRecord) based on the task configuration information. This historical task execution record is stored in a relational database (e.g., MySQL) and contains information such as the start time, end time, and execution status of the previous task.

[0111] Step S440. Determine the start and end times of the data query for this detection based on the historical task execution records to form a time segment: if the historical task execution records are empty, set the start time to a preset historical time point (e.g., half a year ago) and set the end time to the start time plus a preset time interval; if the historical task execution records are not empty, set the start time to the end time of the last task and set the end time to the start time plus the preset time interval.

[0112] Step S450. After the detection is completed, update and save the historical task execution record to determine the start time of the next segment data.

[0113] In some implementations of this embodiment, after determining the time segments, the segmented data is associated with the intelligence database in the following manner, specifically including steps S460 to S480.

[0114] Step S460. The log data within the start and end time range is used as the current segment data. The time segment range, task configuration information, and corresponding collision logic SQL statements are encapsulated into model task parameters (modetask), in JSON format. For example, the full intelligence collision parameters include the source table name, the full intelligence table name, and the join condition; the incremental intelligence collision parameters include the source table name, the incremental intelligence table name, and the join condition. The encapsulated parameters are stored in a memory queue to buffer and improve processing performance.

[0115] Step S470. An independent thread pool consumes model task parameters from the memory queue, calls the model analysis interface to execute database queries, and completes the correlation detection of full or incremental intelligence. The detection results (i.e., intelligence hit results) are stored in the corresponding hit result table according to the target result table name (targetTableName) in the task configuration information for subsequent backtracking analysis.

[0116] Step S480. After the task execution is completed, update the status (such as completed or failed), task end time, time taken, number of hits, etc. of the task execution record (TaskExecuteRecord) in MySQL, and back up the execution record to ClickHouse for permanent storage, which will be used for subsequent time segment range calculation and auditing of task scheduling.

[0117] In these implementations, the log data to be detected is automatically divided into multiple consecutive time segments by time through the above task scheduling and management steps. Each segment is treated as a segment of data, and a corresponding task execution record is generated for each segment to ensure the orderly and traceable execution of the detection task.

[0118] In some implementations of this embodiment, such as Figure 2 As shown, the correlation detection includes the following steps: initializing data processing records; starting the detection process at the first preset time point each day, and repeating the following steps until the current time exceeds the second preset time point: determining whether the current time is greater than the sum of the last data processing time and the data processing interval time; if so, creating a new detection task, using the end time of the previous task as the start time of the current task, and using the start time plus the data processing interval time as the end time of the current task; performing correlation queries between the full intelligence database and network log data within the range of the start time to the end time; recording the task execution results, and updating the last data processing time to the end time of the current task.

[0119] In these implementations, if the current time is less than or equal to the sum of the last data processing time and the data processing interval time, the process is skipped directly and polling continues until the condition is met, thereby avoiding duplicate data processing and controlling concurrency.

[0120] In some implementations of this embodiment, such as Figure 3 As shown, the correlation detection includes the incremental intelligence detection steps: initializing data processing records; starting the detection process at the first preset time point each day, repeating the following steps until the current time exceeds the second preset time point: determining whether the current time is greater than the sum of the last data processing time and the data processing interval time; if so, creating a new back-lookup task, using the end time of the previous task as the start time of this task, and using the start time plus the data processing interval time as the end time of this task; performing a correlation query between the incremental intelligence database and historical network log data within the range of the start time to the end time; recording the task execution result, and updating the last data processing time to the end time of this task.

[0121] In these implementations, if the current time is less than or equal to the sum of the last data processing time and the data processing interval time, the process is skipped directly and polling continues until the condition is met, thereby avoiding duplicate data processing and controlling concurrency.

[0122] Step S500. Based on the intelligence hit result, the historical data in the log database is correlated and traced back using the replication mechanism of the columnar storage engine to obtain historical correlation data, and the historical correlation data is stored in the hit wide table.

[0123] In some implementations of this embodiment, the historical data in the log database is correlated and traced back, including: connecting the IP intelligence hit result with the DNS log table, wherein the connection condition is that the resolved IP address in the DNS log table is equal to the intelligence IP address in the intelligence hit result, and the device identifier is consistent; connecting the domain name intelligence hit result with the communication relationship log table, wherein the connection condition is that the source IP address or destination IP address in the communication relationship log table is equal to the resolved IP address in the domain name intelligence hit result, and the device identifier is consistent; and connecting the domain name intelligence hit result with the SSL log table, wherein the connection condition is that the source IP address or destination IP address in the SSL log table is equal to the resolved IP address in the domain name intelligence hit result, and the device identifier is consistent.

[0124] Step S600. Perform aggregation analysis on the historical correlation data to determine the list of compromised hosts.

[0125] In some embodiments of this example, the historical associated data is aggregated and analyzed to determine the list of compromised hosts, including steps S610 to S620.

[0126] Step S610. Based on the communication relationship log obtained from the IP intelligence hit results, perform feature analysis on the communication behavior of the hit host. If any of the following is identified: communication behavior based on abnormal port, communication behavior based on abnormal communication frequency, communication behavior based on packet size, communication behavior based on session duration, or communication behavior based on the number of connection attempts, then the host is included in the list of compromised hosts.

[0127] The communication behavior based on port anomalies refers to the host initiating a connection to an external IP within a unit of time, with the destination port not belonging to the preset standard port set (such as initiating communication to the 80 / 443 port of a non-Web server), or using a preset uncommon port for encrypted transmission, and the number of connections exceeding a first preset threshold (e.g., greater than 50 times / minute or greater than 5 times / second).

[0128] The communication behavior based on communication frequency anomalies refers to the host continuously engaging in periodic communication with external IPs during preset non-working periods, with traffic levels below a preset threshold.

[0129] The communication behavior based on packet size refers to the fact that the size of the uplink packet for each interaction between the host is less than the preset byte limit (e.g., 500 bytes). For example, the packet may contain only the TCP header and a small amount of payload, or only the ACK packet.

[0130] The communication behavior based on session duration refers to the duration of each host connection being less than a preset duration threshold, for example, each connection typically lasts 1-3 seconds before being disconnected.

[0131] The communication behavior based on the number of connection attempts refers to a host initiating connection attempts to multiple ports or the same ports of other external or internal hosts within a unit of time, and the number of connections exceeds a second preset threshold (e.g., accessing more than 200 different ports to the same target IP within 1 minute).

[0132] Step S620. Based on the SSL logs obtained from the domain name intelligence hit results, analyze the encrypted communication characteristics of the hit host. If any of the following is identified: TLS client fingerprint matching, TLS server fingerprint matching, or abnormal certificate information, then include the host in the list of compromised hosts.

[0133] The TLS client fingerprint matching refers to matching the TLS client fingerprint in the SSL log with a preset malicious fingerprint database.

[0134] The TLS server fingerprint matching refers to matching the TLS server fingerprint in the SSL log with a preset malicious fingerprint database.

[0135] The abnormal certificate information refers to a certificate that is self-signed, a certificate issuer that does not belong to the preset set of trusted certificate authorities, or a certificate whose common name does not match the name of the accessed server (for example, the domain name is google.com, but the certificate is issued to baidu.com).

[0136] In some implementations of this embodiment, the historical association data is aggregated and analyzed to determine the list of compromised hosts, including: aggregating the data in the hit wide table to generate a final list containing the IP addresses of compromised hosts.

[0137] This embodiment aggregates data from the hit table, for example, grouping by source IP address, and combines intelligence tags and behavioral characteristics (such as access frequency and protocol type) to output a final list of compromised hosts. Specifically, compromised host IPs can be identified through deduplication, statistics, and scoring. This list can be used for security response and notification.

[0138] The second aspect of this embodiment discloses a compromised host detection and backtracking device based on massive network logs, such as... Figure 4 As shown, the device includes a log library construction module, a log storage module, an intelligence library construction module, an association detection module, a backtracking module, and an aggregation analysis module.

[0139] The log library building module is used to build a log library based on a columnar storage engine.

[0140] The log storage module is used to acquire and parse network log data, and store the parsed network log data in the log library.

[0141] The intelligence database construction module is used to acquire external threat intelligence data and construct a full intelligence database and an incremental intelligence database respectively.

[0142] The correlation detection module is used to divide the log data to be detected into multiple segments according to time, and perform correlation detection on each segment with the full intelligence database and the incremental intelligence database to obtain intelligence hit results.

[0143] The backtracking module is used to perform correlation backtracking on historical data in the log database based on the intelligence hit results and using the replication mechanism of the columnar storage engine to obtain historical correlation data, and store the historical correlation data in the hit wide table.

[0144] The aggregation analysis module is used to perform aggregation analysis on the historical correlation data to determine the list of compromised hosts.

[0145] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0146] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting and tracing compromised hosts based on massive network logs, characterized in that, include: Construct a log library based on a columnar storage engine, wherein the log library stores a communication log table, a DNS log table, and an SSL log table; Acquire and parse network log data, and store the parsed network log data in the log database; Acquire external threat intelligence data and build a full intelligence database and an incremental intelligence database respectively; The log data to be detected is divided into multiple segments according to time, and each segment is correlated with the full intelligence database and the incremental intelligence database to obtain intelligence hit results. Based on the intelligence hit results, the replication mechanism of the columnar storage engine is used to perform correlation and backtracking on the historical data in the log database to obtain historical correlation data, and the historical correlation data is stored in the hit wide table; The historical correlation data was aggregated and analyzed to determine the list of compromised hosts; The correlation detection includes a full intelligence detection step: initializing data processing records; starting the detection process at the first preset time point each day, and repeating the following steps until the current time exceeds the second preset time point: Determine if the current time is greater than the sum of the last data processing time and the data processing interval. If so, create a new detection task, use the end time of the previous task as the start time of the current task, and use the start time plus the data processing interval as the end time of the current task. Perform a correlation query between the full intelligence database and the network log data within the range of the start time to the end time; record the task execution result, and update the last data processing time to the end time of the current task. The correlation detection includes an incremental intelligence detection step: initializing data processing records; starting the detection process at the first preset time point each day, and repeating the following steps until the current time exceeds the second preset time point: Determine whether the current time is greater than the sum of the last data processing time and the data processing interval time. If so, create a new backtracking task, take the end time of the previous task as the start time of the current task, and take the start time plus the data processing interval time as the end time of the current task. Then, perform a correlation query between the incremental intelligence database and the historical network log data within the range of the start time to the end time. Record the task execution results and update the last data processing time to the end time of this task; The historical data in the log database is correlated and traced back, including: linking the IP intelligence hit result with the DNS log table, where the connection condition is that the resolved IP address in the DNS log table is equal to the intelligence IP address in the intelligence hit result, and the device identifier is consistent; linking the domain name intelligence hit result with the communication relationship log table, where the connection condition is that the source IP address or destination IP address in the communication relationship log table is equal to the resolved IP address in the domain name intelligence hit result, and the device identifier is consistent; and linking the domain name intelligence hit result with the SSL log table, where the connection condition is that the source IP address or destination IP address in the SSL log table is equal to the resolved IP address in the domain name intelligence hit result, and the device identifier is consistent.

2. The method for detecting and tracing compromised hosts based on massive network logs according to claim 1, characterized in that, Constructing a log library based on a columnar storage engine includes: using Clickhouse as the columnar storage engine, creating storage tables according to time periods, and the storage tables constituting the log library; in, The communication log table is used to store communication logs, which includes device identifier, source IP address, destination IP address, source port, destination port, protocol type, number of bytes sent by the client, number of bytes sent by the server, total number of bytes, source IP country code, source IP operator, destination IP country code, destination IP operator, and extended tags; The DNS log table is used to store DNS logs, which include event time, entry time, device identifier, source file identifier, query domain name, IP address returned by resolution, record type, response code, and time to live. The SSL log table is used to store SSL logs, which include event time, entry time, device identifier, source file identifier, source IP address, destination IP address, server name in the SSL certificate, TLS client fingerprint, TLS server fingerprint, certificate serial number, and certificate issuer.

3. The method for detecting and tracing compromised hosts based on massive network logs according to claim 2, characterized in that, The full intelligence database includes a full IP intelligence database and a full domain name intelligence database, and the incremental intelligence database includes an incremental IP intelligence database and an incremental domain name intelligence database. The full IP intelligence database and the incremental IP intelligence database constitute the IP intelligence database, and the full domain name intelligence database and the incremental domain name intelligence database constitute the domain name intelligence database.

4. The method for detecting and tracing compromised hosts based on massive network logs according to claim 3, characterized in that, The association detection includes: Perform a left join query between the IP intelligence database and the communication relationship log table. The join condition is that the source IP address or destination IP address in the communication relationship log table is equal to the intelligence IP address in the IP intelligence database. Perform a left join query between the IP intelligence database and the SSL log table, with the join condition being that the source IP address or destination IP address in the SSL log table is equal to the intelligence IP address. Perform a left join query between the domain name intelligence database and the DNS log table, with the join condition being that the queried domain name in the DNS log table equals the intelligence domain name in the domain name intelligence database. Perform a left join query between the domain name intelligence database and the SSL log table, with the join condition being that the server name in the SSL log table equals the intelligence domain name.

5. The method for detecting and tracing compromised hosts based on massive network logs according to claim 1, characterized in that, The step of dividing the log data to be detected into multiple segments by time includes a task scheduling and management step: Obtain task configuration information from the configuration center, the task configuration information including timing expressions and task status; Determine whether to create a scheduled task based on the task status. If the task status is enabled, create a scheduled task according to the scheduled expression. If the task status is disabled, delete the created scheduled task. When a scheduled task is triggered, the scheduling service is invoked through the scheduled task scheduling framework. The scheduling service obtains the corresponding historical task execution records based on the task configuration information. The start and end times of the data query for this test are determined based on the historical task execution records to form a time segment: if the historical task execution records are empty, the start time is set to a preset historical time point, and the end time is set to the start time plus a preset time interval. If the historical task execution record is not empty, the start time is set to the end time of the last task, and the end time is set to the start time plus the preset time interval. After the detection is completed, the historical task execution record is updated and saved to determine the start time of the next segment of data.

6. The method for detecting and tracing compromised hosts based on massive network logs according to claim 1, characterized in that, The historical correlation data is aggregated and analyzed to determine the list of compromised hosts, including: Based on the communication relationship logs obtained from the IP intelligence hit results, the communication behavior of the hit host is characterized. If any of the following is identified: communication behavior based on abnormal port, communication behavior based on abnormal communication frequency, communication behavior based on packet size, communication behavior based on session duration, or communication behavior based on the number of connection attempts, then the host is included in the list of compromised hosts. Based on the SSL logs obtained from the domain name intelligence hit results, the encrypted communication characteristics of the hit host are analyzed. If any of the following is identified: TLS client fingerprint matching, TLS server fingerprint matching, or abnormal certificate information, the host is included in the list of compromised hosts. in, The communication behavior based on port anomalies refers to the host initiating a connection to an external IP within a unit of time, with the destination port not belonging to the preset standard port set, or using a preset uncommon port for encrypted transmission, and the number of connections exceeding a first preset threshold. The communication behavior based on communication frequency anomalies refers to the host continuously engaging in periodic communication with external IPs during preset non-working periods, with traffic below a preset threshold. The communication behavior based on data packet size refers to the fact that the size of the uplink data packet for each interaction between the host and the host is less than the preset upper limit in bytes. The communication behavior based on session duration refers to the duration of each host connection being less than a preset duration threshold. The communication behavior based on the number of connection attempts refers to the host initiating connection attempts to multiple ports or the same ports of other external or internal hosts within a unit of time, and the number of connection attempts exceeds the second preset threshold. The TLS client fingerprint matching refers to matching the TLS client fingerprint in the SSL log with a preset malicious fingerprint database. The TLS server fingerprint matching refers to matching the TLS server fingerprint in the SSL log with a preset malicious fingerprint database. The certificate information anomaly refers to a certificate that is self-signed, a certificate issuer that does not belong to the preset set of trusted certificate authorities, or a certificate whose generic name does not match the name of the accessed server.

7. A compromised host detection and backtracking device based on massive network logs, characterized in that, include: The log library construction module is used to build a log library based on a columnar storage engine. The log library stores a communication log table, a DNS log table, and an SSL log table. The log storage module is used to acquire and parse network log data, and store the parsed network log data in the log library; The intelligence database construction module is used to acquire external threat intelligence data and build a full intelligence database and an incremental intelligence database respectively. The correlation detection module is used to divide the log data to be detected into multiple segments according to time, and perform correlation detection on each segment with the full intelligence database and the incremental intelligence database to obtain intelligence hit results; The backtracking module is used to perform correlation backtracking on historical data in the log database based on the intelligence hit results and using the replication mechanism of the columnar storage engine to obtain historical correlation data, and store the historical correlation data in the hit wide table; The aggregation analysis module is used to perform aggregation analysis on the historical related data to determine the list of compromised hosts. The correlation detection includes a full intelligence detection step: initializing data processing records; starting the detection process at the first preset time point each day, and repeating the following steps until the current time exceeds the second preset time point: Determine if the current time is greater than the sum of the last data processing time and the data processing interval. If so, create a new detection task, use the end time of the previous task as the start time of the current task, and use the start time plus the data processing interval as the end time of the current task. Perform a correlation query between the full intelligence database and the network log data within the range of the start time to the end time; record the task execution result, and update the last data processing time to the end time of the current task. The correlation detection includes an incremental intelligence detection step: initializing data processing records; starting the detection process at the first preset time point each day, and repeating the following steps until the current time exceeds the second preset time point: Determine whether the current time is greater than the sum of the last data processing time and the data processing interval time. If so, create a new backtracking task, take the end time of the previous task as the start time of the current task, and take the start time plus the data processing interval time as the end time of the current task. Then, perform a correlation query between the incremental intelligence database and the historical network log data within the range of the start time to the end time. Record the task execution results and update the last data processing time to the end time of this task; The historical data in the log database is correlated and traced back, including: linking the IP intelligence hit result with the DNS log table, where the connection condition is that the resolved IP address in the DNS log table is equal to the intelligence IP address in the intelligence hit result, and the device identifier is consistent; linking the domain name intelligence hit result with the communication relationship log table, where the connection condition is that the source IP address or destination IP address in the communication relationship log table is equal to the resolved IP address in the domain name intelligence hit result, and the device identifier is consistent; and linking the domain name intelligence hit result with the SSL log table, where the connection condition is that the source IP address or destination IP address in the SSL log table is equal to the resolved IP address in the domain name intelligence hit result, and the device identifier is consistent.

Images