A Method and System for Network Threat Monitoring and Tracing Based on Full Traffic Collection
By bypassing traffic to receive traffic through the switch's mirror port and generating session logs and PCAP files with session timestamps, a combined index is established, solving the problem of maintaining the evidence relationship of threat records in full traffic monitoring. This ensures that the attack chain can still be accurately linked even when rules are updated or external handling fails, achieving continuity and reliability in network threat monitoring and tracing.
Patent Information
- Application Number
- CN202611514922.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-28
- Publication Date
- 2026-11-17
AI Technical Summary
Under the conditions of continuous retention of all original traffic packets, separate organization of structured logs and PCAP, noise reduction of threat records, and dynamic updates of detection rules, how can we ensure that a threat record maintains the same evidentiary relationship from structured retrieval to bidirectional packet replay, attack chain association, and handling audit, and avoid the original evidentiary relationship being overwritten due to external handling failure?
Traffic is received by bypassing the mirror port of the switch at the network location, and Berkeley Packet Filter is executed to generate session logs with session timestamps and 5-tuples. The original packets are archived into PCAP files according to time range. A joint index is established between the structured logs and PCAP files, and rule filtering, cluster deduplication, and behavior baseline determination are performed to generate high-risk threat events. Before generating a work order, the request and response packets are restored to form bidirectional evidence to ensure that external processing does not cover the original evidence relationship.
It ensures the continuity of the original evidence relationship even when rules are updated or external handling fails, ensuring that operations and maintenance personnel can return to the corresponding session along the same evidence location key, ensuring the accuracy and reliability of the attack chain association, and avoiding the loss of evidence due to rule updates or handling failures.
Smart Images

Figure CN123122993A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of network security monitoring and network traffic analysis technology, and in particular to a network threat monitoring and tracing method and system that maintains continuous data transmission by combining full traffic bypass collection, structured retrieval, original packet playback, threat record noise reduction, attack chain association, and coordinated handling audit. Background Technology
[0002] In office network egress, data center egress, and branch office networks, security operations personnel need to simultaneously monitor network traffic, application access, threat alerts, and external response results. When using traffic digest methods such as NetFlow, sFlow, or IPFIX, only digests such as source address, destination address, port, protocol, packet count, and byte count are typically retained, making it difficult to obtain the original packets that generated the alerts after an attack occurs. When using full packet capture and long-term storage, the original packets are scattered across time-based files, requiring repeated retrieval of numerous files to locate sessions and restore interaction packets. Retaining only HTTP headers, DNS queries, or certificate information results in the loss of application-layer interaction processes.
[0003] Full-traffic monitoring platforms typically store structured logs and raw packets separately. Structured logs are suitable for filtering by time, address, and port, while PCAP files are suitable for storing raw packets. However, if the granularity of references and the transmission relationship between the two are changed during threat detection, alert aggregation, or high-risk threat event handling, operations personnel will need to manually reassemble the files, sessions, and packet relationships when returning from an alert to the original session. Especially when the same rule is triggered continuously from the same attack source, clustering deduplication will change the number of threat records; after hot loading of the rule base, the current rule state may differ from the rule state when the historical alert was generated, making it difficult to verify the detection context of historical alerts.
[0004] Existing application identification schemes typically rely on port numbers, DPI payload characteristics, or DFI traffic statistics. Port hopping and proprietary protocols can render port identification ineffective, while TLS encryption, TLS 1.3, and QUIC limit payload characteristic matching. Statistical inference based solely on packet length, arrival interval, and connection mode is insufficient to reliably distinguish between application and tunnel traffic. When application objects cannot consistently be traced back to session records, there is a lack of a unified interface between threat logging, behavior auditing, and raw packets.
[0005] The security operations platform also receives numerous alerts from the rule engine, network devices, and host security devices. Internal monitoring, backups, and automated scans generate low-value records, and repeated triggering of the same rule by the same attack source for the same purpose leads to a deluge of alerts. If operations personnel generate work orders directly before reviewing complete session evidence, the handling may be based solely on summaries or one-way messages; when firewalls, honeypots, or notification interfaces time out, the handling status may also become detached from the original evidence. When deployed at multiple locations, inconsistencies in the time base and source identifiers of different collection probes can also make it difficult to verify the chronological order of attacks.
[0006] Therefore, a technical solution based on full traffic collection is needed, which allows the composite index of structured logs to first constrain the retrieval objects of the original PCAP, and after the threat record is filtered, clustered and baseline determined, it still retains the original message reference and the rule identifier when the record was generated. Before generating the handling work order, the request message and response message are replayed and written to the same high-risk threat event, and the status change of external handling does not cover the already formed two-way evidence relationship. Summary of the Invention
[0007] The technical problem to be solved by this invention is: under the conditions of continuous retention of all original traffic packets, separate organization of structured logs and PCAP, noise reduction of threat records, and dynamic updates of detection rules, how to maintain the same evidentiary relationship from structured retrieval to bidirectional packet replay, attack chain association, and handling audit of a threat record, and avoid the original evidentiary relationship being overwritten due to external handling failure.
[0008] In a first aspect, the present invention provides a network threat monitoring and tracing method based on full traffic collection, comprising: receiving traffic in bypass mode at a switch mirror port in at least one network location and executing Berkeley Packet. The filter process generates session logs, URL logs, and a key object sorting statistics log for recording traffic object sorting results, all with session timestamps and 5-tuples. The filtered raw packets are then archived into a PCAP file based on time ranges. The session logs and threat records with rule identifiers generated by the rule engine are written to a time-partitioned wide table in the database. A composite index is created on the session timestamp, source IP, and destination IP fields. Original packet references pointing to the corresponding raw packets within the PCAP file are stored in the structured logs. The session timestamp, 5-tuple, and original packet references form an evidence locating key for locating the PCAP file and its raw packet location information. For threat records, rule filtering, clustering deduplication within a time window based on attack source, attack purpose, and hit rules, and deviation judgment based on historical behavior baselines are performed sequentially. This ensures that clustering deduplication only processes threat records output by rule filtering, and deviation judgment only processes threat records output by clustering deduplication. High-risk threat events carrying evidence locating keys and rule identifiers are output. Before generating a work order, the composite index is used to determine the time. The scope and session records are used to determine the corresponding PCAP file and message location information using the evidence location key. Within the determined scope, request and response messages are reconstructed to form request-direction replay records and response-direction replay records. The request-direction replay records and response-direction replay records are written to the same high-risk threat event along with the rule identifier. The request-direction replay records and response-direction replay records together constitute bidirectional evidence of the high-risk threat event. The high-risk threat event is then associated with session logs, URL logs, HTTP behavior logs, threat intelligence, malicious file information, and sandbox reports to form an attack timeline, target asset topology, and attack chain. After the bidirectional evidence is written to the high-risk threat event, a work order is generated. After the work order is confirmed, the ipset interface, firewall application interface, or honeypot interface is called to perform the action and record the person in charge, the time of action, the action, and the result. The action record is associated with the evidence location key and the rule identifier and written to the audit log. When the action interface times out or fails, the failure status and failure reason are appended without changing the replay record, evidence location key, and rule identifier.
[0009] Furthermore, structured logs include at least a time field, a source IP field, a source port field, a destination IP field, a destination port field, a protocol field, a threat level field, a rule signature field, a handling status field, and an original message reference field. The database wide table establishes a composite index based on the time field, the source IP field, and the destination IP field and partitions it by time. During retrieval, the composite index first determines the time range and session records, and then the original message reference determines the target file and message location information from the PCAP file candidate set.
[0010] Furthermore, when restoring request and response messages separately, a replay record with a direction identifier, session timestamp, 5-tuple, and original message reference is generated for each direction. The two replay records are written together into the same high-risk threat event, and the original message references for each direction are saved in the high-risk threat event, so that subsequent association processing can distinguish between the request direction and the response direction.
[0011] Furthermore, rule filtering excludes known normal business traffic and known scanning noise; cluster deduplication uses the same attack source, the same attack purpose, and the same hit rule as clustering conditions, retains the first occurrence time and accumulates the number of repetitions within the time window; behavior baseline deviation determination uses records that have been filtered and deduplicated by clustering, and ensures that the high-risk threat events formed continue to carry evidence location keys and rule identifiers.
[0012] Furthermore, a historical behavior baseline is maintained for each IP or business system. The historical behavior baseline includes the number of accesses, the distribution of access time, and the amount of traffic. When the historical standard deviation is greater than zero, a standardized deviation is calculated for the behavior volume of the current observation window, and a high-risk threat event is determined based on the deviation of the number of accesses, the distribution of access time, and the amount of traffic.
[0013] Furthermore, before generating session logs, port identification results, DPI payload feature matching results, DFI statistical feature results, TLS SNI results, DNS query results, IP and domain name or certificate tracking results, protocol state machine results, and active probing results are obtained for the same session. The obtained results are merged to form an application tag and written into the session log, so that the application tag and evidence location key are jointly transmitted to high-risk threat events.
[0014] Furthermore, multiple acquisition probes periodically report CPU usage, memory usage, online status, packet capture throughput, and packet loss status. The platform performs NTP time synchronization on multiple acquisition probes and generates offline alarms based on the reported status. Evidence location keys and their bidirectional evidence from different network locations are centrally retrieved and sorted by attack timeline according to the synchronized timestamp and acquisition probe identifier.
[0015] Furthermore, the platform records rule identifiers, rule messages, and rule categories for each rule, and supports setting whitelists or disabling individual rules. After the rule base is updated, a reload command is sent to the rule engine to update the rules without stopping the packet capture process. At the same time, Berkeley Packet Filter parameters and DPI processing parameters are sent to the collection probe. Threat records generated before and after rule updates all save the rule identifier used when the record was generated. Replay records and audit records are transmitted along the rule identifier and evidence location key.
[0016] Furthermore, the work order status includes unread, read, and processed in sequence. After the processing action is executed, the action return result is recorded. When any of the ipset interface, firewall application interface, honeypot interface, Webhook interface, or SMTP interface times out or fails to execute, the reason for the failure is recorded and the operation and maintenance personnel are notified. The association between high-risk threat events, two-way evidence, rule identifiers, and evidence location keys is retained so that the original message can be viewed and processed again based on the unchanged evidence location keys.
[0017] Secondly, this invention also provides a network threat monitoring and tracing system based on full traffic acquisition, including an acquisition probe module, a data storage module, a detection and noise reduction module, a two-way playback module, a correlation and tracing module, a response module, and a probe management module. The acquisition probe module receives traffic from a switch mirror port, performs Berkeley PacketFilter filtering, generates session logs, URL logs, and key object sorting and statistics logs, forms application tags, and archives original packets. The data storage module writes structured logs into a wide database table partitioned by time and indexed by a composite index, archives the original packets into PCAP files, and saves original packet references, forming evidence locating keys for locating PCAP files and their original packet location information. The detection and noise reduction module performs rule detection, rule filtering, time window clustering deduplication, and behavior baseline deviation determination, outputting high-risk threat events carrying evidence locating keys and rule identifiers. The two-way playback module locates PCAP files based on the composite index and evidence locating keys, reconstructs request and response packets within a defined range, forming request-direction playback records and response-direction playback records, and stores the request-direction playback records... The request and response replay records are jointly written into the same high-risk threat event, forming bidirectional evidence composed of the request and response replay records. The association and tracing module associates the bidirectional evidence with session logs, URL logs, HTTP behavior logs, threat intelligence, malicious file information, and sandbox reports to form an attack timeline, target asset topology, and attack chain. After the bidirectional evidence is written into the high-risk threat event, the handling and response module generates a work order, and after confirmation, calls the ipset interface, firewall application interface, or honeypot interface to perform the handling, sends a notification through the Webhook interface or SMTP interface, and associates the handling audit record with the evidence location key and rule identifier. The probe management module receives the health status of the collection probes, performs NTP time synchronization, and sends rules and collection parameters to the collection probes.
[0018] The beneficial effects of this invention include: Compared with the prior art, the composite index of structured logs first returns the time range and session records, and then the original message reference selects the PCAP file and location information. Therefore, a transmission order of filtering followed by playback is formed between the filtered data and the original data. When maintenance personnel enter the original message for viewing from a high-risk threat event, they return to the corresponding session along the same evidence location key. Rule filtering, time window clustering, and behavioral baselines are processed in the order of the previous stage's output as the input of the next stage. The denoised high-risk threat events still carry the evidence location key and the rule identifier when the record was generated, and the source context of the detection results is not affected by the rules. The update or record merging is replaced by the current state; before the work order is generated, the request message and response message are replayed and written to the same high-risk threat event, so that the attack chain is established on the bidirectional original evidence of the same high-risk threat event, rather than based on a single digest or one-way message; when the external handling interface fails, only the status and reason are appended, and the existing replay records, evidence location keys and rule identifiers are retained, so that the operation and maintenance personnel can review and handle it again along the original evidence relationship; when multiple probes are deployed, the timestamp and probe identifier are synchronized to participate in the sorting of high-risk threat events, so that the attack timeline across network locations retains the time source and collection source. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating the network threat monitoring and tracing method based on full traffic collection according to the present invention. Detailed Implementation
[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings. It should be understood that the described embodiments are for illustrating the present invention, and not for limiting the scope of protection of the present invention.
[0021] In one implementation scenario, a company's traffic analysis platform is used by security operations personnel to monitor traffic at the office network egress, data center egress, or branch office egress. The operations personnel configure mirror ports on switches, bypassing one or more data collection probes to the mirror link. The platform receives session logs, URL logs, key object sorting statistics logs, threat records, original packet references, and probe health statuses generated by the data collection probes. It also sends filtering conditions, identification parameters, rule update instructions, and collection parameters to the data collection probes through the management channel. The initial state in this scenario is that the data collection probes can receive replicated traffic from the mirror link, the platform can receive structured records and status reports, the PCAP archive space has been established according to time ranges, and the rule engine is in a running state capable of generating threat records with rule identifiers.
[0022] Combination Figure 1The network interface card (NIC) of the acquisition probe receives replicated traffic in both inbound and outbound directions on the mirrored link side. The full traffic refers to the set of packets sent into the current monitoring range via the mirrored port. The Berkeley Packet Filter filters out packets not included in the analysis within this set according to the filtering conditions issued by the platform, retaining packets that enter session processing. The acquisition probe forms session objects for the retained packets based on source IP, source port, destination IP, destination port, and protocol, and writes a session timestamp to each session object. Simultaneously, it generates session logs, URL logs, and key object sorting statistics logs. The key object sorting statistics logs are used to store the traffic object sorting results formed according to predetermined statistical dimensions, serving as structured statistical input, but do not replace session logs and PCAP files. Original packets are written to PCAP files with time ranges according to the archiving strategy. The structured logs record original packet references associated with the original packets in the PCAP files during generation. At this point, the session logs serve as the structured input required for subsequent filtering and association, while the PCAP files serve as the original input required for subsequent playback. The two maintain a transmission relationship through the timestamp, 5-tuple, and original packet references of the same session.
[0023] After receiving structured logs, the platform writes session logs, URL logs, and threat records generated by the rule engine into a time-partitioned wide table in the database. The wide table stores the following fields: time, source IP, source port, destination IP, destination port, protocol, threat level, rule signature, action status, rule identifier, and original packet reference. A composite index is created on the time, source IP, and destination IP fields. Time partitioning first limits the query to time partitions that might contain the target session, and then the address field narrows down the range of session records. The results returned by the composite index do not directly replace the original PCAP evidence, but rather serve as the first round of filtering results for selecting the original packet reference and the PCAP file.
[0024] The original message reference written to the structured log is used to point to the PCAP file and information related to the location of the target session message. The evidence location key is a group of associated fields consisting of a session timestamp, a 5-tuple, and an original message reference, used to locate the PCAP file and its original message location information.
[0025] ;
[0026] In the formula, The evidence location key is a group of associated fields consisting of a session timestamp, a 5-tuple, and a reference to the original message. It is used to locate the PCAP file and the location information of the original message within it. It serves as the associated object for transmission between session logs, threat records, high-risk threat events, replay records, attack chain data, ticket data, and audit logs. The session timestamp is derived from the time field written when the acquisition probe receives a message or forms a session record. For the source IP, For the source port, For destination IP, For destination port, For the protocol, the above five items are derived from the quintuple fields of the session log; This is a reference to the original message, derived from fields in the structured log related to the PCAP file and message location. Each field is stored as a structured value in the log; the time field uses the timestamp unit from the acquisition probe, and the address, port, and protocol fields use the session parsing results. The platform transmits the same set of fields or references to that set of fields in subsequent stages, without rewriting the handling status, rule identifier, or current page status to new evidence location keys; therefore, playback, correlation, and auditing can utilize the already generated original session location relationships.
[0027] In this specification, threat records generated by the rule engine are the original detection records after a rule is hit; high-risk threat events are record objects formed after the threat records are sequentially filtered by rules, deduplicated by time window clustering, and determined by behavioral baseline deviation; request-direction replay records and response-direction replay records are the original message records of two directions restored from the PCAP file, and together they constitute bidirectional evidence. The rule engine saves the rule identifier, rule message, and rule category for the hit threat records. The rule identifier is used to indicate the rule record used when the threat record was generated, and the evidence location key is used to indicate the original session that the record can return to; the two play different transmission roles. The platform writes threat records with rule identifiers into a structured wide table. When performing rule filtering, deduplication by clustering, and behavioral baseline determination, the rule identifier and evidence location key of the threat record are used together as the context of the high-risk threat event. In this way, when the rule base is updated later, historical threat records can still retain the detection source at that time, and the original message can still be replayed according to the evidence location key.
[0028] In one embodiment, the platform first performs noise reduction processing on the detected threat records, and then replays the original evidence. The noise reduction output is a high-risk threat event with evidence location keys and rule identifiers. The replay output is the request-direction replay record and the response-direction replay record corresponding to the high-risk threat event. The request-direction replay record and the response-direction replay record together constitute bidirectional evidence. The association output is an attack timeline, target asset topology, and attack chain organized by the bidirectional evidence with logs, intelligence, malicious file information, and sandbox reports. The handling output is a work order and audit record with action return results or failure status. The output of the previous stage serves as the input for the next stage. Unless the replay record and high-risk threat event association processing have been completed, the platform does not enter the work order generation stage.
[0029] The platform queries a combined index based on the timestamp, source IP, and destination IP of a high-risk threat event to obtain the time range and session records. It then retrieves the evidence location key from the session records and selects the PCAP file and message location information based on the original message reference. The selected PCAP read object is only used for the playback of this high-risk threat event; the structured index results are still retained in the high-risk threat event record for easy post-event review and filtering. Within the selected range, the platform retrieves request and response messages respectively, forming request-direction playback records and response-direction playback records. These two records are then jointly written into the same high-risk threat event, forming bidirectional evidence composed of both. The request-direction and response-direction records respectively store the direction identifier, session timestamp, 5-tuple, and original message reference. Retaining records for both directions within the same high-risk threat event allows subsequent correlation processing to read the request and response content separately, without one direction record overwriting the other.
[0030] After writing bidirectional evidence into high-risk threat events, the platform then integrates session logs, URL logs, HTTP behavior logs, threat intelligence, malicious file information, and sandbox reports. When using IP and domain names as entry points, the platform determines the initiating party and target domain name from the request direction, and then associates subsequent sessions and malicious file activities with port and file hashes. Threat intelligence and sandbox reports are attached as external evidence to the same high-risk threat event. The platform arranges the behavior timeline according to the timestamps and collection probe identifiers in the high-risk threat event, and forms the target asset topology and attack chain based on the correlation between source address, destination address, domain name, port, and file hash. Each original packet in the attack chain continues to point to the PCAP file along the evidence location key, so operations personnel do not need to reselect another evidence identifier when viewing packets from attack chain nodes.
[0031] After the two-way evidence and rule identifiers have been written into a high-risk threat event, the platform generates a work order. Security operations personnel review the request message, response message, rule message, attack timeline, and target asset topology to confirm the handling action. After confirmation, the platform calls the ipset interface to add the attack source IP to the blacklist set, or calls the firewall application programming interface to add a blocking policy, or sends a trapping policy to the honeypot interface, and sends a notification via the Webhook interface or SMTP interface. The platform writes the handler, handling time, handling action, interface return result, rule identifier, two-way evidence reference, and evidence location key into the audit log. If any external interface times out or fails to execute, the platform only appends the failure status, failure reason, and current handling status to the work order and audit log, without deleting or rewriting the already formed two-way evidence, rule identifier, and evidence location key; operations personnel can review the original message again and re-execute the handling based on the unchanged evidence location key.
[0032] Furthermore, in one embodiment, the structured log fields are jointly maintained by the acquisition probe and the platform. When forming session logs, the acquisition probe writes time, source IP, source port, destination IP, destination port, and protocol fields, and supplements them with threat level, rule signature, rule identifier, and handling status fields after rule detection is completed. Original packet references are written to the structured logs after the original packet archiving action is completed or usable archive location information is obtained. The platform establishes partitions according to time ranges, and then performs a composite index query on the time, source IP, and destination IP fields. The composite index first returns the time range and session records. The platform reads the original packet references from the session records to form a candidate set of PCAP files, and finally selects the target PCAP file and location information based on the original packet references. This process passes the structured filtering results to the playback module, rather than directly using the summary field in the structured logs as original evidence. After the query is completed, the time range, session records, and selected PCAP information are still stored in the retrieval context of the high-risk threat event for playback and audit review.
[0033] Furthermore, in one embodiment, the bidirectional playback module first reads the evidence location key and retrieval context from the high-risk threat event, and then performs a limited-scope message reading on the target PCAP file. For sessions determined by the 5-tuple and timestamp, the module organizes messages sent to the target direction into a request-direction playback record and messages sent back into a response-direction playback record; the two directions are respectively written with a direction identifier, session timestamp, 5-tuple, and original message reference. The playback module does not merge the request record and response record into an indistinguishable message digest, but saves the two records as two evidentiary components of the same high-risk threat event. After the request record and response record are formed, the association and tracing module reads the two types of records with the same high-risk threat event as the entry point. The attack timeline is sorted using the time fields of the two types of records, and the attack chain is expanded using the source, destination, and entity clues of the two types of records. The handling and response module generates a work order using the bidirectional evidence and evidence location key from the same high-risk threat event. In this way, the directional state formed in the playback stage can directly become the input for the association and handling stages, avoiding subsequent modules from replacing the bidirectional original messages with a one-way digest.
[0034] Furthermore, in one embodiment, the noise reduction module first performs rule filtering on the threat records generated by the rule engine. Whitelists and filtering conditions are used to exclude internal monitoring, backups, and known scanning noise. The filtering conditions are configured by the platform and distributed to the rule processing stage; excluded records do not enter subsequent high-risk threat events. For records not excluded, the platform clusters them within a time window according to the attack source, attack purpose, and hit rules, grouping duplicate records under the same conditions into the same processing batch, retaining the first occurrence time and accumulating the number of repetitions; the first occurrence time is used as the starting point of the attack timeline, and the number of repetitions indicates the number of times the batch appears within the time window. The platform then uses the clustered records as input for behavior baseline deviation judgment, determining whether the current behavior deviates from historical behavior based on the number of accesses to the IP or business system, access time distribution, and traffic volume. The three stages are strictly executed in the order of the output of the previous stage as the input of the next stage. After a high-risk threat event is formed, the high-risk threat event is still written with the original message reference, evidence location key, and rule identifier at the time the record was generated, so as to enter bidirectional playback.
[0035] Furthermore, in one embodiment, the behavioral baseline is maintained by the platform separately for each IP address or business system. The platform obtains the number of accesses, access time distribution, and traffic volume for the current observation window from records that have already been filtered and deduplicated through rule filtering and clustering, and obtains the historical mean and historical standard deviation of the same type of behavior from the historical records of the same IP address or business system. When the historical standard deviation is greater than zero, the first... Standardized deviation of item behavior quantity calculation:
[0036] ;
[0037] In the formula, For the first The standardized deviation of the behavior quantity is an intermediate result of the platform comparing the current observation window with historical behavior; The first observation window The number of actions is derived from records that have already undergone rule filtering and clustering for deduplication; This is the historical average of the behavior volume of the same IP or business system, derived from historical baseline records; The historical standard deviation corresponds to the historical behavior data, derived from the same baseline records. When the deviation is greater than zero, the platform performs the calculation and submits the deviation results of the number of accesses, access time distribution, and traffic volume to the configured judgment conditions. After a high-risk threat event is identified, the platform also retains the evidence location key and rule identifier of the high-risk threat event. The subsequent playback module does not directly use the standardized deviation to replace the original message. The historical window length, baseline update method, and judgment threshold are provided by the platform configuration.
[0038] Furthermore, in one embodiment, application tags are obtained before the session log is generated. The acquisition probe performs port identification, DPI payload feature matching, and DFI statistical feature analysis on the same session, and obtains supplementary results from TLS SNI, DNS queries, IP and domain name or certificate tracking, protocol state machine, and active probing. DPI results come from packets with obtainable payload features; DFI results come from statistical features such as packet length, arrival interval, and connection mode; TLS SNI and DNS query results provide clues about access targets in encrypted interactions; address and domain name or certificate tracking results maintain the correspondence between entities; protocol state machine results indicate the session interaction phase; and active probing results supplement application clues that existing passive observations could not obtain. The platform collects the obtained results within the same session to form application tags and writes the application tags to the session log; the application tags, along with the session timestamp, 5-tuple, and original message reference, are then included in high-risk threat events associated with the evidence location key. Application tags are used for naming objects in attack timelines, entity associations, and behavior auditing; the original message reference is still responsible for returning to the PCAP file, and the two are not interchangeable.
[0039] Furthermore, in one embodiment, the platform centrally manages the time and health status of multiple acquisition probes. Each acquisition probe periodically reports CPU usage, memory usage, online status, packet capture throughput, and packet loss status, carrying the acquisition probe identifier in the status report. The platform performs NTP time synchronization with multiple acquisition probes, receiving the synchronized timestamps and status reports; when a probe goes offline or its packet capture status is abnormal, the platform generates an operation and maintenance alarm and writes the corresponding probe identifier and status into the operation and maintenance record. High-risk threat events from the office network egress, data center egress, and branch office egress are sorted on the platform side according to the synchronized timestamps, and the acquisition probe identifier is used to distinguish the evidence location keys generated by the same address in different network locations. After bidirectional evidence is formed, the timeline module uses request records, response records, probe identifiers, and synchronized timestamps as sorting inputs, and outputs a cross-location attack timeline; this timeline still retains the evidence location key for each record, so as to return to the respective PCAP files from the cross-location sequence. In this embodiment, NTP provides the time base required for cross-probe sorting.
[0040] Furthermore, in one embodiment, the platform maintains continuous packet capture when the rule base changes. Each rule in the rule base stores a rule identifier, rule message, and rule category. Security operations personnel can set whitelists or disable individual rules. After the rule base is updated, the platform sends a reload command to the rule engine, which completes the rule update without stopping the packet capture process. Simultaneously, the platform sends Berkeley PacketFilter filtering parameters and DPI processing parameters to the collection probes through a configuration interface, enabling the collection scope and identification processing to operate according to the new configuration. Before and after the rule update, each threat record generated by the rule engine stores the rule identifier used when the record was generated. The rule identifier, along with the evidence location key, is entered into high-risk threat events, replay records, and audit records. When the platform replays historical high-risk threat events, it reads the rule identifier stored in the high-risk threat event to identify the rule source when the high-risk threat event occurred. When the platform performs auditing, it still obtains the original packets along the evidence location key, without overwriting the historical records with the currently reloaded rule identifier. If rule reloading or parameter delivery fails, the platform records the update failure status and notifies the operations and maintenance personnel. The obtained session logs and PCAP files will not be deleted.
[0041] Furthermore, in one embodiment, the handling and response module manages external actions according to the work order status. After two-way evidence is written to a high-risk threat event, the platform establishes a work order ledger, and the work order is initially in an unread state. After security operations personnel open the high-risk threat event and view the request message, response message, rule source, attack timeline, and attack chain, the work order enters the read state. After security operations personnel confirm the attack source IP, blocking policy, or trapping policy, the platform calls the ipset interface, firewall application programming interface, or honeypot interface to execute the handling. The execution result is written to the action return field, and the work order enters the handled state after the action is completed. The Webhook interface and SMTP interface are used to send high-risk threat events or handling results to external notification objects. The return status and message of the notification interface are also written to the audit log. When any handling or notification interface times out, refuses, or fails to execute, the platform writes the failure reason and current status without deleting the original work order and two-way evidence, and associates the failure status with the rule identifier, two-way evidence reference, and evidence location key. Operations personnel use the unchanged evidence location key to re-view the original message and execute the handling again based on the new confirmation result. Since the execution status of external interfaces is only written as a new audit status, the existing request and response replay records will not be overwritten by the failure results.
[0042] In one embodiment, internal system modules and the platform transmit structured data to the acquisition probes via HTTP or HTTPS. The data includes log fields, raw message references, rule identifiers, probe identifiers, health status, configuration parameters, and interface return status. The platform frontend reads data through a unified log and statistics interface. Real-time situational statistics can be written to a cache for frontend reading; the cache does not replace the database wide table and PCAP file. The acquisition probe module processes mirrored traffic into session logs, URL logs, key object sorting statistics logs, and PCAP files. The data storage module writes these results to a partitioned wide table and saves the evidence location keys. The detection and noise reduction module processes threat records with rule identifiers as high-risk threat events. The bidirectional playback module converts the evidence location keys in high-risk threat events into request and response playback records. The correlation and tracing module converts bidirectional evidence into attack timelines, target asset topologies, and attack chains. The handling and response module converts high-risk threat events into work orders and audit statuses. The probe management module transmits health status, rules, and acquisition parameters to the acquisition probes. The modules transmit pre-defined log fields, statuses, and reference relationships. If any external interface malfunctions, the already formed evidence location keys can still be used for subsequent queries.
[0043] In another implementation scenario, a single data collection probe bypasses the office network egress, and the platform receives data via a local server or cloud service. In yet another scenario, multiple branch offices deploy data collection probes, with the headquarters platform centrally receiving logs and health status, issuing rules and collection parameters, and forming a cross-location attack timeline based on the probe identifiers. In single-point deployment, request and response replay records for high-risk threat events originate from the same probe's PCAP file. In multi-point deployment, each replay record retains the identifier of the probe that generated it, and the platform sorts them under a unified time base without deleting the original source. The number of data collection probes, archiving cycle, database implementation, caching implementation, rule source, notification interface, and deployment location can be adjusted, as long as the evidence location relationship between structured logs and PCAP, the rule identifier transmission relationship, the order from noise reduction to bidirectional replay, the order from bidirectional evidence to work orders, and the relationship that failure status does not overwrite the original evidence are preserved, the technical effects of this invention can be achieved.
[0044] In a common implementation scenario, the joint index location, rule context preservation, and bidirectional playback are executed consecutively. The platform first obtains the time range and session records using a joint index of the time field, source IP, and destination IP. Then, it selects the PCAP file and message location information based on the original message references in the session records. During the noise reduction phase, the evidence location key is not changed, and the rule identifier at the time of record generation is passed to the playback record along with the high-risk threat event. After bidirectional playback is completed, the correlation and tracing module reads the two types of records in the request and response directions to form an attack chain, and the handling and response module generates a work order based on the bidirectional evidence in the high-risk threat event. In this way, the index results, rule sources, and directional evidence jointly constrain subsequent processing in the same data transmission chain, and the state formed in the previous stage can become the input of the next stage.
[0045] In another common implementation scenario, rule base updates, multi-probe time management, and external handling audits run simultaneously. During rule base updates, the packet capture process continues running, and threat records generated before and after the update retain their respective rule identifiers from the time of generation. Synchronization timestamps and probe identifiers from multiple probes are used for attack timeline sorting. After a work order is confirmed, the external interface's return status or failure reason is written as a new audit status, while existing request directions, response directions, and original message references are retained. This combination ensures that rule changes and cross-location sorting do not replace historical evidence, and that handling interface anomalies does not sever the path from audit records back to the PCAP file.
[0046] In the above implementation, the first round of filtering for the composite index can be represented by a candidate session set as follows:
[0047] ;
[0048] In the formula, This is the set of candidate session records obtained under the conditions of time range and address. The time range is determined by the search criteria for high-risk threat events. and These are the source IP and destination IP to be retrieved, respectively. For the first 1 conversation record For the time field of this session record, and These are the source IP and destination IP fields for the session record, respectively. The platform uses... The results are returned as a structured composite index and saved to the retrieval context; this set represents session records that may need to be read further, does not indicate that the original PCAP message has been obtained, and does not require that the records that meet the conditions be unique.
[0049] When reading raw message references from candidate session records, the set of references entering the replay module can be represented as:
[0050] ;
[0051] In the formula, The set of original message references obtained from candidate session records. In order to be with the first The original message reference associated with the session record. For candidate session records, This refers to the aforementioned set of candidate session records. The platform, based on... Each original message reference in the process obtains information related to the PCAP file and message location, and then passes the corresponding PCAP file and location information to the playback module. Therefore, the structured index first narrows down the candidate range of sessions, and the original message reference then connects the candidate records to the actual playback object. The outputs of the two stages are connected to each other in the data stream without substituting for each other.
[0052] Within the defined PCAP read targets and scope, request-direction playback records and response-direction playback records are generated separately, and both are stored as bidirectional evidence in the same high-risk threat event, which can be represented as:
[0053] ;
[0054] In the formula, Two-way evidence in the same high-risk threat event. To request a directional playback record, This is in response to the replay records in both directions. The records for each direction carry a direction identifier, session timestamp, 5-tuple, and a reference to the original message; the tracing module retrieves these records from... The system reads content from both directions separately. The attack timeline uses time fields from both directions, the attack chain uses source, destination, and entity clues from both directions, and the response module uses evidence from both directions to form a work order. Therefore, the response message is not an overwrite of the request message's fields, but rather is written alongside the request message as subsequent input to the same high-risk threat event.
[0055] When performing time-window clustering on threat records that have already passed rule filtering, the clustering conditions can be expressed as:
[0056] ;
[0057] In the formula, For the first The clustering keys used in each processing batch and These are the attack source and the attack target, respectively. These are the rule identifiers or rule conditions that match the rule. The platform only reads these three fields within the record range of the rule filtering output, grouping duplicate records with the same clustering key that fall within the same time window into the record set. Records excluded by the rule filtering will not be included. The clustering process does not involve modifying the input records filtered by the rules.
[0058] The time of the first occurrence of the same processing batch can be expressed as:
[0059] ;
[0060] In the formula, For the first The first occurrence time of each processing batch For the first batch The time field of the record Belonging to The platform will record. Write the clustering output and use it as one of the starting times of the attack timeline; when subsequent records still meet the same clustering key and time window conditions, do not create another equivalent batch starting record, but continue to accumulate occurrences within the record range of the original batch.
[0061] The number of times the same processing batch is repeated can be expressed as the number of batch records:
[0062] ;
[0063] In the formula, For the first The number of records per processing batch This refers to the set of records contained in this batch. This indicates the number of records in the set. The repetition count, as part of the clustering output, is passed to behavioral baseline determination and high-risk threat event display. The platform still retains the evidence location keys and rule identifiers recorded in the batch, and does not use [the following text is incomplete and likely refers to a separate context:] the repetition count in the clustering output is passed to behavioral baseline determination and high-risk threat event display. Instead of referencing the original message, noise reduction alters the organization of duplicate records without severing the path from batch results back to the original session.
[0064] For the current observation window that has been filtered and deduplicated by clustering, the three types of behavioral variables—number of visits, distribution of visit time, and traffic volume—can be combined into a behavioral variable vector:
[0065] ;
[0066] In the formula, This is the behavior vector for the current observation window. The number of visits is obtained from the clustering output statistics. This refers to the access time distribution statistics obtained from the time field of the clustering output. This represents the traffic volume obtained from the packets or sessions output by clustering. Each of the three components is compared to a historical baseline component of the same IP or service system. If a component lacks a corresponding historical standard deviation condition, the platform does not perform standardization calculations for that component, but continues to process other available results according to the configured baseline judgment conditions.
[0067] When the historical standard deviation is greater than zero and the platform has obtained the corresponding judgment threshold, the deviation judgment of a certain behavior quantity can be expressed as:
[0068] ;
[0069] In the formula, For the first The comparison results of the number of behaviors This is the standardized deviation already obtained from the current observation window, historical mean, and historical standard deviation. Configure the judgment threshold corresponding to this behavior quantity. It only indicates whether the current comparison meets the configuration conditions, without limiting it to a fixed threshold or combining the three components into a single score; the platform determines whether a high-risk threat event is formed based on the judgment results of the number of accesses, the distribution of access time, and the traffic volume, and continues to pass the evidence location key and rule identifier on which the high-risk threat event is based to the replay stage.
[0070] When multiple source identification results obtained within the same session are used to form application tags, the tag formation process can be represented as follows:
[0071] ;
[0072] In the formula, For the first Application tags for each session, This is a collection of port identifications, DPI payload characteristics, DFI statistical characteristics, TLS SNI, DNS queries, IP and domain name or certificate traces, protocol state machines, and active probe results already obtained within the scope of this session. This is the process by which the platform merges the obtained results and writes them into the tag field. This indicates a result merging relationship rather than a newly added specific model; when a certain type of signal is not obtained in the current session, that signal is not used in this session. The input. The resulting... Write it to the session log and include it along with the session timestamp, 5-tuple, and original message reference in subsequent high-risk threat event association processing, so that the application naming clues can return to the original message of the same session.
[0073] When sorting high-risk threat events in a multi-probe scenario, the sorting input must include at least the synchronized timestamp and the probe identifier, which can be represented as:
[0074] ;
[0075] In the formula, For the first The sorting key for cross-location evidence records. This is the timestamp written to the record after NTP synchronization. The platform first identifies the acquisition probe that generated the record. Arrange request and response records from different network locations, and only use them when the time field is the same or when it is necessary to distinguish the source. The acquisition location is preserved; the sorted output still carries the original evidence location key for each record, and the sort key is not used as a new original message reference. Therefore, the cross-location timeline can return from the sequential node to the PCAP file of its respective probe.
[0076] When a rule base update fails, or an external interface times out, rejects, or fails to execute, the platform's handling of the record status can be represented as follows:
[0077] ;
[0078] In the formula, This field represents a work order or audit record that was created before the action was executed. Add new fields for failure status, failure reason, or API return status. This is the set of record fields appended to the status. The set union operation in the formula represents appending fields to a record, not deleting or rewriting existing bidirectional playback records, evidence location keys, and rule identifiers; therefore, when the processing or notification interface times out, rejects, or fails to execute, operations personnel can still access the record from... Read the reason for the failure and then review the original message again using the unchanged evidence location key.
[0079] In the same actual deployment, the above computational relationships apply to actual data objects at different stages: and Constrain the order in which structured logs are read from PCAP. , , and The way threat records are organized during the noise reduction phase. , and Constrain the comparison between the current behavior volume and the historical baseline. How to store the direction of requests and responses in high-risk threat events. and Supplement the session attributes required for application naming and cross-probe ordering, respectively. This constrains the way the state is written after an abnormal external action. Each intermediate result enters the next processing stage along the established fields, references, or state relationships, ultimately enabling structured filtering, original message playback, threat denoising, attack chain association, and handling auditing to be continuously linked on the same network session evidence.
[0080] During the aforementioned field lifecycle, the acquisition probe first writes a time field, a five-tuple field, a session log, a URL log, and a key object sorting statistics log for recording traffic object sorting results for the same session. After the original message is archived or usable archive location information is obtained, a reference to the original message is added. The platform then writes the structured fields into a time-partitioned wide table in the database and writes the rule identifiers and threat records generated by the rule engine into the same retrieval context. During the noise reduction and detection phase, when the aforementioned records are read to form a high-risk threat event, the session field, original message reference, or rule identifier is not deleted. During the bidirectional playback phase, a direction identifier and playback record are added to this context. During the correlation and tracing phase, bidirectional records and related logs, intelligence, and reports are read to form an attack timeline, target asset topology, and attack chain. During the work order and audit phase, the handler, handling time, action return status, or failure reason are added. Each stage's added fields have a clearly defined writing subject and subsequent reading subject. Fields already formed continue to be used as input in subsequent stages rather than being replaced by the current page status or interface return results. Therefore, those skilled in the art can implement a complete data flow from acquisition to auditing according to the field generation order, reading order, and status appending position.
[0081] Those skilled in the art will recognize that the Berkeley Packet Filter filtering, session parsing, PCAP archiving, federated indexing, rule engine, NTP synchronization, ipset interface, firewall application programming interface, honeypot interface, webhook interface, and SMTP interface in the implementation can be handled by existing data collection probes, servers, databases, and communication interfaces; component models, deployment locations, and communication protocols do not constitute necessary limitations of this invention. The essential relationship of this invention lies in the following: the structured log filtering results are used to form evidence location keys through reference to the original messages; the threat record, after noise reduction, still carries this key and rule identifier; before the work order is generated, request and response replay records are formed and written to the same high-risk threat event; the handling status only appends records, without rewriting the already formed bidirectional evidence and evidence location keys.
Claims
1. A method for network threat monitoring and tracing based on full traffic collection, characterized in that, include: Traffic is received by bypassing the mirror port of a switch at at least one network location. The traffic is filtered by Berkeley Packet Filter, and session logs, URL logs, and key object sorting statistics logs for recording traffic object sorting results are generated by session. The filtered raw packets are archived into PCAP files according to time range. The session logs and threat records with rule identifiers generated by the rule engine are written into a time-partitioned wide table of the database. A composite index is created on the session timestamp, source IP, and destination IP fields. The original message reference pointing to the corresponding original message in the PCAP file is saved in the structured log. The session timestamp, the 5-tuple, and the original message reference are combined to form an evidence locating key for locating the PCAP file and the original message locating information therein. The threat records are sequentially subjected to rule filtering, clustering and deduplication within a time window based on attack source, attack purpose and hit rule, and deviation judgment based on historical behavior baseline. This ensures that clustering and deduplication only processes threat records output by rule filtering, deviation judgment only processes threat records output by clustering and deduplication, and outputs high-risk threat events carrying the evidence location key and rule identifier. Before generating a work order, the time range and session records are determined using the composite index, and the corresponding PCAP file and message location information are determined using the evidence location key. Within the determined range, the request message and response message are restored to form a request-direction replay record and a response-direction replay record. The request-direction replay record and the response-direction replay record are written to the same high-risk threat event along with the rule identifier. The request-direction replay record and the response-direction replay record together constitute bidirectional evidence of the high-risk threat event. The high-risk threat event is then associated with session logs, URL logs, HTTP behavior logs, threat intelligence, malicious file information, and sandbox reports to form an attack timeline, target asset topology, and attack chain. After the bidirectional evidence is written to the high-risk threat event, a work order is generated. After the work order is confirmed, the ipset interface, firewall application interface, or honeypot interface is called to perform the disposal and record the disposal person, disposal time, disposal action, and disposal result. The disposal record is associated with the evidence location key and the rule identifier and written to the audit log. When the disposal interface times out or fails, the failure status and failure reason are appended without changing the playback record, the evidence location key, and the rule identifier.
2. The network threat monitoring and tracing method based on full traffic collection according to claim 1, characterized in that, The structured log includes at least the following fields: time field, source IP field, source port field, destination IP field, destination port field, protocol field, threat level field, rule signature field, handling status field, and original message reference field. The combined index sequentially returns the time range, session records, and a candidate set of PCAP files. The original message reference selects the target PCAP file and message location information from the candidate set and uses the selected file and message location information as input to the playback record.
3. The network threat monitoring and tracing method based on full traffic collection according to claim 1, characterized in that, When reconstructing the request message and the response message respectively, a replay record with a direction identifier, session timestamp, 5-tuple and original message reference is formed respectively. The two replay records are written together into the same high-risk threat event, and the original message references of the two directions are saved in the high-risk threat event respectively.
4. The network threat monitoring and tracing method based on full traffic collection according to claim 1, characterized in that, The rules filter out known normal traffic and known scanning noise; The clustering deduplication uses the same attack source, the same attack purpose, and the same hit rule as the clustering conditions, and retains the first occurrence time and accumulates the number of repetitions within the time window; The behavior baseline deviation determination uses records filtered by the rules and deduplicated by the clustering, and ensures that the resulting high-risk threat events continue to carry the evidence location key and the rule identifier.
5. The network threat monitoring and tracing method based on full traffic collection according to claim 4, characterized in that, Maintain historical behavior baselines for each IP or business system, including access counts, access time distribution, and traffic volume. When the historical standard deviation is greater than zero, the standardized deviation of the behavior in the current observation window is calculated, and the deviation of the number of visits, the distribution of visit time, and the amount of traffic is used to determine whether the high-risk threat event has been formed.
6. The network threat monitoring and tracing method based on full traffic collection according to claim 1, characterized in that, Before generating the session log, port identification results, DPI payload feature matching results, DFI statistical feature results, TLS SNI results, DNS query results, IP and domain name or certificate tracking results, protocol state machine results, and active probing results are obtained for the same session. The obtained results are merged to form an application tag and written into the session log, so that the application tag and the evidence location key are jointly transmitted to the high-risk threat event.
7. The network threat monitoring and tracing method based on full traffic collection according to claim 1, characterized in that, Multiple acquisition probes periodically report CPU usage, memory usage, online status, packet capture throughput, and packet loss status. The platform performs NTP time synchronization on multiple acquisition probes and generates offline alarms based on the reported status. The platform performs centralized retrieval and attack timeline sorting of the evidence location keys and their bidirectional evidence from different network locations according to the synchronized timestamp and acquisition probe identifier.
8. The network threat monitoring and tracing method based on full traffic collection according to claim 1, characterized in that, The platform records rule identifiers, rule messages, and rule categories for each rule, and supports setting whitelists or disabling individual rules. After the rule base is updated, a reload command is sent to the rule engine to update the rules without stopping the packet capture process, and Berkeley Packet Filter parameters and DPI processing parameters are sent to the collection probe. Threat records generated before and after rule updates all retain the rule identifier used when the record was generated. The replay records and audit records are transmitted along the rule identifier and the evidence location key.
9. The network threat monitoring and tracing method based on full traffic collection according to claim 8, characterized in that, The work order status includes unread, read, and processed in sequence. After the processing action is executed, the action return result is recorded. When any of the ipset interface, firewall application interface, honeypot interface, webhook interface, or SMTP interface times out or fails to execute, record the reason for the failure and notify the operation and maintenance personnel. Retain the association between the high-risk threat event, the two-way evidence, the rule identifier, and the evidence location key so that the original message can be viewed and the action taken can be taken again based on the unchanged evidence location key.
10. A network threat monitoring and tracing system based on full traffic acquisition, used to execute the network threat monitoring and tracing method based on full traffic acquisition as described in any one of claims 1 to 9, characterized in that, include: The acquisition probe module is used to receive mirrored traffic, perform Berkeley Packet Filter filtering, generate session logs, URL logs, key object sorting and statistics logs, and application tags, and archive raw packets; The data storage module is used to write session logs and threat records with rule identifiers into a partitioned wide table to create a composite index, and to save PCAP and original message references to form evidence locating keys for locating PCAP files and the original message locating information within them. The detection and noise reduction module is used to perform rule detection, filtering, cluster deduplication and baseline deviation judgment, and output high-risk threat events with evidence location keys and rule labels; The bidirectional playback module is used to locate the PCAP based on the joint index and evidence location key, restore the request message and response message respectively, form a request-direction playback record and a response-direction playback record, and write the request-direction playback record and the response-direction playback record together into the same high-risk threat event to form bidirectional evidence composed of the request-direction playback record and the response-direction playback record. The correlation and tracing module is used to correlate bidirectional evidence with session logs, URL logs, HTTP behavior logs, threat intelligence, malicious file information, and sandbox reports to form an attack timeline, target asset topology, and attack chain; The handling and response module is used to generate a work order after writing bidirectional evidence to a high-risk threat event. After confirmation, it calls the ipset interface, firewall application interface or honeypot interface to handle the event, and notifies and associates auditing via Webhook interface or SMTP interface. The probe management module is used to receive health status, perform NTP synchronization, and distribute rules and collection parameters.
